Skip to content

Add tool to display emergency log message full-sceen on boot failure. - #28077

Merged
bluca merged 1 commit into
systemd:mainfrom
1awesomeJ:bsod
Aug 3, 2023
Merged

bluca merged 1 commit into
systemd:mainfrom
1awesomeJ:bsod

Conversation

@1awesomeJ

@1awesomeJ 1awesomeJ commented Jun 19, 2023 •

Copy link
Copy Markdown
Contributor

No description provided.

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

I want to start with a function that fetches the oldest message in the journal and displays it to the console.
Currently I set PRIORITY to 5(notice) to see if I'll get any messages. I'm currently not getting any.

@1awesomeJ
1awesomeJ force-pushed the bsod branch 2 times, most recently from 2022bbe to 7bd44d9 Compare June 19, 2023 13:53
@1awesomeJ

Copy link
Copy Markdown
Contributor Author

@bluca
Following Poettering's guidance, I created this bsod tool.
Currently it just tries to fetch the oldest message from the current boot with a log level of "emergency", and prints it to a console.

Subsequently, I have to modify it to take over the entire screen, turn it blue, and display the QR code.

Howevever, I am currently not getting an output at all, even when i change the PRIORITY to numbers other than 0.
What am I doing wrong sir?

Thank you.

Comment thread man/rules/meson.build
Comment thread man/systemd-bsod.xml Outdated
Comment thread man/systemd-bsod.xml Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
_cleanup_close_ int fd = -EBADF;
char * message = first_emerg_boot_message();

fd = open_terminal("/dev/console", O_WRONLY|O_NOCTTY|O_CLOEXEC);

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

so for this I think we should not go to /dev/console (which is a magic device that basically points to wherever boot time logs and stuff should go to which can be serial or similar) but the VT subsystem (which is the virtual terminal, i.e. a textual interface to a local physical display device, i.e. no serial), and I think we should not interfere with the usual log/progress output that /dev/console gets. Hence I'd suggest we allocate a full new VT for this, and then switch to that.

(in case you wonder what a VT is, it's this archaic textual display logic that the linux kernel uses to do early boot logging before wayland/x11 take over, and that you can log into via Alt-F2, Alt-F3, …)

i.e. open /dev/tty1 temporarily, then issue the VT_GETSTATE ioctl call on it which tells you which VTs are currently allocated via a bitmask. Look for the first free VT (i.e. determine lowest unset bit, then format /dev/tty%i with it plus one. Then open that, and use that. switch to it via the VT_ACTIVATE ioctl, then clear it by output ANSI_HOME_CLEAR on it, then display the message there.

If this sounds like a bit much, grep our sources for VT_GETSTAT, we already call that ioctl elsewhere for other reasons.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VT_OPENQRY might be easier than VT_GETSTATE and __builtin_ctz(~vt_stat.v_state) or whatever.

Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
@poettering poettering added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks and removed please-review PR is ready for (re-)review by a maintainer labels Jun 19, 2023
@github-actions github-actions Bot added please-review PR is ready for (re-)review by a maintainer and removed reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels Jun 19, 2023
Comment thread man/systemd-bsod.xml Outdated
@poettering poettering added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks and removed please-review PR is ready for (re-)review by a maintainer labels Jun 19, 2023
@1awesomeJ

1awesomeJ commented Jun 20, 2023 •

Copy link
Copy Markdown
Contributor Author

I read that the APIs return errno-style errors on failure. Kindly review my choice of error types sir.

I made some of the requested changes.

I still have the SD_ID128_FORMAT_STR() and SD_ID128_FORMAT_VAL() outstanding. Still need to study them more.
And of course VT.

But I don't want to miss the chance of getting a review in case you view the PR, so I decided to push some first.

@github-actions github-actions Bot added please-review PR is ready for (re-)review by a maintainer and removed reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels Jun 20, 2023
@1awesomeJ

Copy link
Copy Markdown
Contributor Author

I looked up the format string and format value macros sir, and I've now used them in the patch.

So it's VT I need to look up now.

perhaps I have too many log messages?

@1awesomeJ 1awesomeJ changed the title PID1: Implement systemd blue screen of death for boot failures. PID1: Display emergency log message full-sceen on boot failure. Jun 21, 2023
@1awesomeJ

Copy link
Copy Markdown
Contributor Author

Now, we need to register this service as a varlink API client that subscribes to journald to get notified when messages with log_level "emerg" are processed, so it can fetch the message and display it.

I've been stuck on implementing this subscribe feature in journald. Please could I get some high level guidance - not so high level, or some resources or docs recommendations? Thank you sirs.

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

@bluca, Have you been in touch with Poettering though? I looked the the PRs -open and closed ones, It looks like he hasn't been with us for over a week. Did he go on vacation without inviting us or at least share photos?

I'm sort of bothered I've been on just one of the 4 points from his last guidance for over 2 weeks.
If I'm unbale to complete them by Aug 25, would you guys still spare some more time to guide beyond Aug 25?

Thank you.

@bluca

bluca commented Aug 1, 2023

Copy link
Copy Markdown
Member

there's a merge conflict, needs a rebase

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

okay sir.
I'll be home in 30minutes

Comment thread src/journal/meson.build Outdated
Comment thread src/journal/bsod.c Outdated

xsprintf(tty, "/dev/tty%d", free_vt + 1);

fd = open_terminal(tty, O_RDWR|O_NOCTTY|O_CLOEXEC);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You leak tty1's fd here

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You leak tty1's fd here

😂😂

You remember me from that initrd guide you gave in May?
Thanks again for that.
Thanks for showing up here too.

Comment thread src/journal/bsod.c
@1awesomeJ

Copy link
Copy Markdown
Contributor Author

Looks like I'm hearing you say "tty1's fd should be closed before we assign fd to hold the descriptor of another tty" doesn't _cleanup_close_ handle such scenarios?

Comment thread src/journal/bsod.c Outdated
Comment on lines +131 to +142
fd_tty1 = open_terminal("/dev/tty1", O_RDWR|O_NOCTTY|O_CLOEXEC);
if (fd_tty1 < 0)
return log_error_errno(fd, "Failed to open tty1: %m");

r = find_next_free_vt(fd_tty1, &free_vt, &original_vt);
if (r < 0)
return log_error_errno(r, "Failed to find a free VT: %m");

xsprintf(tty, "/dev/tty%d", free_vt + 1);

fd = open_terminal(tty, O_RDWR|O_NOCTTY|O_CLOEXEC);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Like so?

@AdrianVovk

Copy link
Copy Markdown
Contributor

doesn't _cleanup_close_ handle such scenarios

The _cleanup_* macros (aka __attribute__((__cleanup__(*)))) only activate when the variable goes out of scope. In other words, when the block that the variable is declared in ends.

{
    _cleanup_close_ void* int fd = -EBADF;
    fd = open(...); /* fd = 4 */
    fd = open(...); /* fd = 5 */
    /* we're about to leave the block... compiler adds: */ close(fd)
}

Notice that in the above code, close(fd) is called when fd=5, but fd 4 never gets closed. Thus, a leak.

What you want to do is something like this:

{
    _cleanup_close_ void* int fd = -EBADF;
    fd = open(...); /* fd = 4 */
    close(fd) /* we close fd 4, since we're about to replace it with fd 5, which will get auto-closed when we leave the block */
    fd = open(...); /* fd = 5 */
    /* we're about to leave the block... compiler adds: */ close(fd)
}

That is what close_and_replace does

Comment thread src/journal/bsod.c Outdated
if (tty1_fd < 0)
return log_error_errno(fd, "Failed to open tty1: %m");

close_and_replace(fd, tty1_fd);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrong place!

Basically where you put it now, you close fd (which is already -EBADF, so closing it does nothing). Then you replace fd with tty1_fd (and at the same time you make tty1_fd = -EBADF). Then later you overwrite fd with the real value you want, and so you're still leaking.

What you really want to do is something like this:

_cleanup_close int fd = -EBADF;
int r;

fd = open_terminal("/dev/tty1", ...);
if (fd < 0) ...

...

r = open_terminal(tty, ...)
if (r < 0) ...
close_and_replace(fd, r) /* this closes fd (which is /dev/tty1) and replaces it with r (which is the tty you want) */

PS: also, both fd and tty1_fd have _cleanup_close_ attached to them. Not sure if that's intended or not.

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

Wow... awesome.
Thanks @AdrianVovk,
You explain like you're a college professor.
Are you?

Thanks again.

Let me fix.

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

hey @AdrianVovk, I found you do actually have some business in college, although not actually as a professor.
I'd like to stay in touch if your schedule permits. I found a twitter profile which you don't seem to have used in a while. I followed you anyways.

Please let me know if your schedule has room for some sort of extra friend.
Thank you.

Comment thread src/journal/meson.build Outdated
Comment thread src/basic/terminal-util.h Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
@1awesomeJ

Copy link
Copy Markdown
Contributor Author

Thank you @yuwata for the detailed review.
I'm very grateful!

Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c
@1awesomeJ

Copy link
Copy Markdown
Contributor Author

@yuwata, I have made all the requested changes.

Comment thread src/basic/terminal-util.c Outdated
Comment thread src/journal/bsod.c
Comment thread src/journal/bsod.c
Comment thread src/journal/bsod.c Outdated
Comment thread src/journal/bsod.c Outdated
Comment thread src/shared/qrcode-util.c Outdated
Comment thread src/shared/qrcode-util.c Outdated
Comment thread src/shared/qrcode-util.c Outdated
Comment thread src/shared/qrcode-util.c Outdated
Comment thread src/shared/qrcode-util.c Outdated
@1awesomeJ

Copy link
Copy Markdown
Contributor Author

Thank you so much @yuwata for the very detailed review.
Let me fix.

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

@yuwata, I made all the requested changes.
Sorry I took long.

@bluca

bluca commented Aug 3, 2023

Copy link
Copy Markdown
Member

needs a rebase on main due to conflicts

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

needs a rebase on main due to conflicts

Okay sir.
I'll fix in about an hour.

@yuwata yuwata left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Let's go in this form at least now.
Future tasks are

  • merge two qrcode logic more,
  • add tests for bsod.service.

@bluca

bluca commented Aug 3, 2023

Copy link
Copy Markdown
Member

When you rebase and push, please also amend the commit message, as this is not about PID1, it's a different tool

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

When you rebase and push, please also amend the commit message, as this is not about PID1, it's a different tool

Certainly sir.

@1awesomeJ

Copy link
Copy Markdown
Contributor Author

LGTM. Let's go in this form at least now. Future tasks are

  • merge two qrcode logic more,
  • add tests for bsod.service.

Alright sir.
I hope "future" turns out to be next week or next month.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

7 participants