Skip to content

timer: Adjust calendar timers based on monotonic timer instead of realtime - #16428

Merged
keszybz merged 1 commit into
systemd:masterfrom
filbranden:issue16347
Jul 15, 2020
Merged

keszybz merged 1 commit into
systemd:masterfrom
filbranden:issue16347

Conversation

@filbranden

@filbranden filbranden commented Jul 11, 2020 •

Copy link
Copy Markdown
Contributor

When the RTC time at boot is off in the future by a few days, OnCalendar= timers will be scheduled based on the time at boot. But if the time has been adjusted since boot, the timers will end up scheduled way in the future, which may cause them not to fire as shortly or often as expected.

Update the logic so that the time will be adjusted based on monotonic time. We do that by calculating the adjusted manager startup realtime from the monotonic time stored at that time, by comparing that time with the realtime and monotonic time of the current time.

Added a test case to validate this works as expected. The test case creates a QEMU virtual machine with the clock 3 days in the future. Then we adjust the clock back 3 days, and test creating a timer with an OnCalendar= for every 15
minutes.

Test output without the corresponding code changes that fix the issue:

  Timer elapse outside of the expected 20 minute window.
    next_elapsed=1594686119
    now=1594426921
    time_delta=259198

With the code changes in, the test passes as expected.

Fixes #16347. cc @anitazha.

@keszybz
keszybz requested a review from anitazha July 11, 2020 11:42

@anitazha anitazha left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! Love the new test and I also like that the manager timestamps will reflect realtime relative to the current clock value now.

Comment thread src/core/timer.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe remove this comment about t->persistent since this also applies to non-persistent timers?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

@anitazha

Copy link
Copy Markdown
Member

bionic "boot-and-services" are not doing so hot. Any clue if it's this PR or more widespread?

@yuwata yuwata added ci-fails/needs-rework 🔥 Please rework this, the CI noticed an issue with the PR pid1 labels Jul 14, 2020
@filbranden

Copy link
Copy Markdown
Contributor Author

So I decided to make this PR smaller, by only fixing the issue with the OnCalendar= timers, but not touching the manager startup timestamp.

My logic here is that if the realtime clock for some reason fluctuates and starts becoming unreliable, then the manager startup time might start to drift, which might just look odd... So I think we might need more discussion on that one before merging.

This one to me is a clear fix, since scheduling timers based on monotonic time just makes sense.

Yeah not sure why the bionic tests failed, I looked at the logs and didn't look related to my changes at all... Let's see how they do on this second run.

Done removing the comment line you mentioned. Thanks for the review!

…ltime

When the RTC time at boot is off in the future by a few days, OnCalendar=
timers will be scheduled based on the time at boot. But if the time has been
adjusted since boot, the timers will end up scheduled way in the future, which
may cause them not to fire as shortly or often as expected.

Update the logic so that the time will be adjusted based on monotonic time.
We do that by calculating the adjusted manager startup realtime from the
monotonic time stored at that time, by comparing that time with the realtime
and monotonic time of the current time.

Added a test case to validate this works as expected. The test case creates a
QEMU virtual machine with the clock 3 days in the future. Then we adjust the
clock back 3 days, and test creating a timer with an OnCalendar= for every 15
minutes. We also check the manager startup timestamp from both `systemd-analyze
dump` and from D-Bus.

Test output without the corresponding code changes that fix the issue:

  Timer elapse outside of the expected 20 minute window.
    next_elapsed=1594686119
    now=1594426921
    time_delta=259198

With the code changes in, the test passes as expected.
@anitazha anitazha added good-to-merge/waiting-for-ci 👍 PR is good to merge, but CI hasn't passed at time of review. Please merge if you see CI has passed and removed ci-fails/needs-rework 🔥 Please rework this, the CI noticed an issue with the PR labels Jul 14, 2020
@keszybz keszybz added ci-failure-appears-unrelated and removed good-to-merge/waiting-for-ci 👍 PR is good to merge, but CI hasn't passed at time of review. Please merge if you see CI has passed labels Jul 15, 2020
@keszybz
keszybz merged commit 2669833 into systemd:master Jul 15, 2020
@filbranden
filbranden deleted the issue16347 branch July 15, 2020 23:08
@poettering

Copy link
Copy Markdown
Member

I dont get this. Why is this necessary? we watch wallclock time changes, and should reschedule everything in that case. Using the monotonic clock looks like a hack to me, we should schedule wallclock events by wallclock time, and watch wallclock time changes to deal with time jumps.

There might be a bug with rescheduling things on time jumps, but I am pretty sure that should not be fixed like this...

@poettering

Copy link
Copy Markdown
Member

hmm, look at this again, i think I grok why you want this, i.e. it's not actually about rescheduling things when time changes but it's only about figuring out the timestamps for the initial elapsing, i.e. when cross connecting the boot time timestamps to the timer scheduler. I think this is more acceptable then. So ignore my comment above.

I must say I am not too fond of the idea that we fill in a dual_timestamp here though, even though we just care about the realtime value...

@poettering

Copy link
Copy Markdown
Member

I prepped a fix for the dual_timestamp mapping stuff I complained about, please have a look at #16536

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

Timer using OnCalendar= has next elapse stuck on MANAGER_TIMESTAMP_USERSPACE

5 participants