Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#997943: systemd: Persistent attribute of timer is ignored
163 views
Skip to first unread message
Michael Biebl
Oct 27, 2021, 9:10:04 AM10/27/21
to
Control: tags -1 + unreproducible moreinfo
Hello,
unfortunately, I can not confirm the issue (thus marking the bug report
accordingly).
Persistent timers are correctly "remembered" when shutdown/rebooted.
Have you seen that apt-daily-upgrade.service has
ConditionACPower=true
Maybe your system is not updated because it ran on battery when the
timer fired?
Can you send me the "journalctl -u apt-daily-upgrade.service"
log after the dist upgrade until you locally modified the timer.
Regards,
Michael
Hendrik Buchner
Oct 27, 2021, 10:10:03 AM10/27/21
to
Hello Michael,
my system isn't running on battery because it's a Desktop-PC and not a
laptop. I've observed this behaviour now on 3 of my computers (2
Desktop-PCs and 1 laptop).
Attached you can find the output of your "journalctl", but on this
system I haven't already changed the timer. If you need the output from
a system which has already a modified timer please contact me.
Regards
Hendrik
my system isn't running on battery because it's a Desktop-PC and not a
laptop. I've observed this behaviour now on 3 of my computers (2
Desktop-PCs and 1 laptop).
Attached you can find the output of your "journalctl", but on this
system I haven't already changed the timer. If you need the output from
a system which has already a modified timer please contact me.
Regards
Hendrik
Michael Biebl
Oct 27, 2021, 2:30:03 PM10/27/21
to
you shutdown/started the system and where exactly you expected the timer
to fire
Looking at the log, I currenly only see that it is executed rather
regularly on a daily basis, i.e. the log looks rather normal.
Michael Biebl
Oct 27, 2021, 2:30:03 PM10/27/21
to
On 27.10.21 16:05, Hendrik Buchner wrote:
> Hello Michael,
>
> my system isn't running on battery because it's a Desktop-PC and not a
> laptop. I've observed this behaviour now on 3 of my computers (2
> Desktop-PCs and 1 laptop).
>
Just to recap here:
> Hello Michael,
>
> my system isn't running on battery because it's a Desktop-PC and not a
> laptop. I've observed this behaviour now on 3 of my computers (2
> Desktop-PCs and 1 laptop).
>
- Is this issue only reproducible with apt-daily.timer and
apt-daily-upgrade.timer works correctly?
- Does this issue happen reproducibly?
Say "systemctl status apt-daily.timer" shows that the timer is about to
elapse in 5 hours. If you shut down the system before that and start it
after that, apt-daily.service is not run with 100% certainty?
Hendrik Buchner
Oct 27, 2021, 2:50:03 PM10/27/21
to
>Is this issue only reproducible with apt-daily.timer and
>apt-daily-upgrade.timer works correctly?
I think so, but I will observe it for the next days.
>apt-daily-upgrade.timer works correctly?
>Does this issue happen reproducibly? Say "systemctl status
>apt-daily.timer" shows that the timer is about to elapse in 5 hours. If
>you shut down the system before that and start it after that,
>apt-daily.service is not run with 100% certainty?
>Can you mark in the log, when you upgraded your system
>at what times you shutdown/started the system and where exactly you
>expected the timer to fire
you can see the last execution of "apt-daily" was "Okt 24 19:34:45".
In the log of "apt-daily-upgrade" you see that the system was running
at "Okt 25 16:08:30" and "Okt 26 21:10:30" and "Okt 27 15:45:00".
So with the options of "apt-daily.timer" (OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h) it should have been executed since then because
there were missed timers.
For normal the computer is not running for only 1 minute so that
the timespan is too short for execution (apt-daily-upgrade had also
enough time for execution).
Michael Biebl
Oct 27, 2021, 3:00:03 PM10/27/21
to
Hello once again
So, I reread the documentation:
> Persistent=
> Takes a boolean argument. If true, the time when the service unit was last triggered is stored on disk. When the timer is activated, the service
> unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive. Such triggering is
> nonetheless subject to the delay imposed by RandomizedDelaySec=. This is useful to catch up on missed runs of the service when the system was
> powered down.
Looking at /var/lib/systemd/timers/ you should be able to see a stamp
file which indicates, when the corresponding service was last activated.
This is true in my case.
Important is this sentence:
"Such triggering is nonetheless subject to the delay imposed by
RandomizedDelaySec="
Keep in mind that RandomizedDelaySec= is not stored on disk, only the
timestamp of the last execution.
So, let's assume the service was last triggered 2 days ago and you shut
down the system yesterday. You boot the system this morning at 9:00, so
the timer has elapsed. But a RandomizedDelaySec= is applied, which is
computed dynamically. Let's say the delay is 7 hours. If you shut down
the system before that, your service won't be executed.
The next day when you start your system again, systemd will notice
again, that the timer has elapsed and again will apply a
RandomizedDelaySec=12h, in this case maybe 9 hours.
If you only ever use your system for a very short period of time, this
would mean that you never actually see the service being executed.
A RandomizedDelaySec=12h is rather large, if you run your system for
only 6 hours a day, you have a chance of hitting the timer by 50% I'd say.
> Persistent=
> Takes a boolean argument. If true, the time when the service unit was last triggered is stored on disk. When the timer is activated, the service
> unit is triggered immediately if it would have been triggered at least once during the time when the timer was inactive. Such triggering is
> nonetheless subject to the delay imposed by RandomizedDelaySec=. This is useful to catch up on missed runs of the service when the system was
> powered down.
Looking at /var/lib/systemd/timers/ you should be able to see a stamp
file which indicates, when the corresponding service was last activated.
This is true in my case.
Important is this sentence:
"Such triggering is nonetheless subject to the delay imposed by
RandomizedDelaySec="
Keep in mind that RandomizedDelaySec= is not stored on disk, only the
timestamp of the last execution.
So, let's assume the service was last triggered 2 days ago and you shut
down the system yesterday. You boot the system this morning at 9:00, so
the timer has elapsed. But a RandomizedDelaySec= is applied, which is
computed dynamically. Let's say the delay is 7 hours. If you shut down
the system before that, your service won't be executed.
The next day when you start your system again, systemd will notice
again, that the timer has elapsed and again will apply a
RandomizedDelaySec=12h, in this case maybe 9 hours.
If you only ever use your system for a very short period of time, this
would mean that you never actually see the service being executed.
A RandomizedDelaySec=12h is rather large, if you run your system for
only 6 hours a day, you have a chance of hitting the timer by 50% I'd say.
Michael Biebl
Oct 27, 2021, 3:10:04 PM10/27/21
to
On 27.10.21 20:47, Michael Biebl wrote:
> Important is this sentence:
> "Such triggering is nonetheless subject to the delay imposed by
> RandomizedDelaySec="
>
> Keep in mind that RandomizedDelaySec= is not stored on disk, only the
> timestamp of the last execution.
>
> So, let's assume the service was last triggered 2 days ago and you shut
> down the system yesterday. You boot the system this morning at 9:00, so
> the timer has elapsed. But a RandomizedDelaySec= is applied, which is
> computed dynamically. Let's say the delay is 7 hours. If you shut down
> the system before that, your service won't be executed.
> The next day when you start your system again, systemd will notice
> again, that the timer has elapsed and again will apply a
> RandomizedDelaySec=12h, in this case maybe 9 hours.
> If you only ever use your system for a very short period of time, this
> would mean that you never actually see the service being executed.
> A RandomizedDelaySec=12h is rather large, if you run your system for
> only 6 hours a day, you have a chance of hitting the timer by 50% I'd say.
This would be consistent with your observation that by reducing
> Important is this sentence:
> "Such triggering is nonetheless subject to the delay imposed by
> RandomizedDelaySec="
>
> Keep in mind that RandomizedDelaySec= is not stored on disk, only the
> timestamp of the last execution.
>
> So, let's assume the service was last triggered 2 days ago and you shut
> down the system yesterday. You boot the system this morning at 9:00, so
> the timer has elapsed. But a RandomizedDelaySec= is applied, which is
> computed dynamically. Let's say the delay is 7 hours. If you shut down
> the system before that, your service won't be executed.
> The next day when you start your system again, systemd will notice
> again, that the timer has elapsed and again will apply a
> RandomizedDelaySec=12h, in this case maybe 9 hours.
> If you only ever use your system for a very short period of time, this
> would mean that you never actually see the service being executed.
> A RandomizedDelaySec=12h is rather large, if you run your system for
> only 6 hours a day, you have a chance of hitting the timer by 50% I'd say.
RandomizedDelaySec= to 0h you are reliably triggering the timer.
So, what you see is actually consistent with the documented behaviour
I'd argue. Nonetheless, if I understand you correctly there is a
difference of behaviour between buster and bullseye if I understand you
correctly.
I went through the changelog/NEWS file of systemd. This seems to be related:
v247:
> * Timer units gained a new FixedRandomDelay= boolean setting. If
> enabled, the random delay configured with RandomizedDelaySec= is
> selected in a way that is stable on a given system (though still
> different for different units).
and from the man page
> RandomizedDelaySec=
> Delay the timer by a randomly selected, evenly distributed amount of time between 0 and the specified time value.
> Defaults to 0, indicating that no randomized delay shall be applied. Each timer unit will determine this delay
> randomly before each iteration, and the delay will simply be added on top of the next determined elapsing time, unless
> modified with FixedRandomDelay=, see below.
>
> This setting is useful to stretch dispatching of similarly configured timer events over a certain time interval, to
> prevent them from firing all at the same time, possibly resulting in resource congestion.
>
> Note the relation to AccuracySec= above: the latter allows the service manager to coalesce timer events within a
> specified time range in order to minimize wakeups, while this setting does the opposite: it stretches timer events
> over an interval, to make it unlikely that they fire simultaneously. If RandomizedDelaySec= and AccuracySec= are used
> in conjunction, first the randomized delay is added, and then the result is possibly further shifted to coalesce it
> with other timer events happening on the system. As mentioned above AccuracySec= defaults to 1 minute and
> RandomizedDelaySec= to 0, thus encouraging coalescing of timer events. In order to optimally stretch timer events over
> a certain range of time, set AccuracySec=1us and RandomizedDelaySec= to some higher value.
>
> FixedRandomDelay=
> Takes a boolean argument. When enabled, the randomized offset specified by RandomizedDelaySec= is reused for all
> firings of the same timer. For a given timer unit, the offset depends on the machine ID, user identifier and timer
> name, which means that it is stable between restarts of the manager. This effectively creates a fixed offset for an
> individual timer, reducing the jitter in firings of this timer, while still avoiding firing at the same time as other
> similarly configured timers.
>
> This setting has no effect if RandomizedDelaySec= is set to 0. Defaults to false.
Michael Biebl
Oct 27, 2021, 3:30:04 PM10/27/21
to
into the loop here.
Julian, please have a look at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=997943#42 and
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=997943#47
What were your expectations regarding the timer and the currently
documented behavior?
Afaics, Persistent=true is functional, but the behaviour of
RandomizedDelaySec= changed slighly which might affect timers which
chose a long delay.
Michael Biebl
Oct 27, 2021, 3:50:03 PM10/27/21
to
Michael Biebl
Oct 27, 2021, 4:20:04 PM10/27/21
to
On 27.10.21 21:42, Michael Biebl wrote:
> Possibly related
> https://github.com/systemd/systemd/pull/11608
The commit message explains it rather well:
https://github.com/systemd/systemd/pull/11608/commits/a87c1d3a979f8c2641471bed93577558a1027a24
> core: delay persistent timers by "RandomizedDelaySec=" at boot.
> Fixes #5659.
> Currently, if Persistent=true and the machine is off at the scheduled time of the timer unit, the timer
> will be triggered immediately at the next boot even if RandomizedDelaySec= is specified.
>
> As a result, if multiple timers meet that condition, they will be triggered at the same time and too
> much CPU/IO work makes boot slow down.
>
> With this commit, if the scheduled time of the persistent timer has already elapsed at boot,
> set the time when systemd first started as the scheduled time and RandomizedDelaySec= is applied to it.
> Possibly related
> https://github.com/systemd/systemd/pull/11608
The commit message explains it rather well:
https://github.com/systemd/systemd/pull/11608/commits/a87c1d3a979f8c2641471bed93577558a1027a24
> core: delay persistent timers by "RandomizedDelaySec=" at boot.
> Fixes #5659.
> Currently, if Persistent=true and the machine is off at the scheduled time of the timer unit, the timer
> will be triggered immediately at the next boot even if RandomizedDelaySec= is specified.
>
> As a result, if multiple timers meet that condition, they will be triggered at the same time and too
> much CPU/IO work makes boot slow down.
>
> With this commit, if the scheduled time of the persistent timer has already elapsed at boot,
> set the time when systemd first started as the scheduled time and RandomizedDelaySec= is applied to it.
Julian Andres Klode
Oct 27, 2021, 5:30:03 PM10/27/21
to
as the old situation meant that all your containers would run
unattended-upgrades at boot which was a terrible experience. Or was it
at resume where I always had problems with it?
But users not getting their systems updated once a day is a major
concern as well. Unfortunately the answer is hard: You might only
resume your laptop for 5 minutes a day, so even a RandomizedBootDelaySec
option would not help if we'd set it to 30 min (I assume resume acts the
same way as boot? I have not used either in a long time with timers).
Need to think more, but certainly not running directly at boot is
something we actively want.
--
debian developer - deb.li/jak | jak-linux.org - free software dev
ubuntu core developer i speak de, en
Hendrik Buchner
Oct 28, 2021, 3:10:04 AM10/28/21
to
Hello Michael and Julian,
that explains why the behaviour of "RandomizedDelaySec" together with
"Persistent" is different between buster and bullseye. If that's the
intended behaviour, my systems seems to work like they should and it's
no bug.
So for me I will change the delay from 12h to maybe 5min. I think that
will fit better to my use case.
Thanks a lot for your research and explanation of the timers.
Regards
Hendrik
that explains why the behaviour of "RandomizedDelaySec" together with
"Persistent" is different between buster and bullseye. If that's the
intended behaviour, my systems seems to work like they should and it's
no bug.
So for me I will change the delay from 12h to maybe 5min. I think that
will fit better to my use case.
Thanks a lot for your research and explanation of the timers.
Regards
Hendrik
Michael Biebl
Oct 28, 2021, 4:20:04 AM10/28/21
to
On 28.10.21 09:05, Hendrik Buchner wrote:
> Hello Michael and Julian,
>
> that explains why the behaviour of "RandomizedDelaySec" together with
> "Persistent" is different between buster and bullseye. If that's the
> intended behaviour, my systems seems to work like they should and it's
> no bug.
While it fixes the thundering herd issue during boot, it is definitely
problematic that for systems that aren't up for a longer period of time,
a timer with a large RandomizedDelaySec= will basically never fire.
I do think that we have a similar thundering herd issue with timers that
elapse while the system is suspended and ttbomk, systemd will *not*
reapply a RandomizedDelaySec= on resume.
This appears to be inconsistent imho.
Somehow I think systemd could be more clever here and scale down
RandomizedDelaySec= on boot depending on how far in the past the
timestamp of the stampfile is.
Say the last activation of the service was 12 hours ago, then scale down
RandomizedDelaySec=12h by dividing it by 12, so it would become
RandomizedDelaySec=1h. The further in the past, the shorter the delay
would become and the more likely that the timer fires eventually.
And for consistencies sake, a (scaled) RandomizedDelaySec=1 should also
be reapplied on resume.
Regards,
Michael
Michael Biebl
Oct 28, 2021, 4:30:03 AM10/28/21
to
codesearch.debian.net shows the following timers with a
RandomizedDelaySec of 12h, which will likely run into the same problem.
fwupd_1.5.7-5/data/motd/fwupd-refresh.timer
[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
apt_2.3.11/debian/apt-daily.timer
[Timer]
OnCalendar=*-*-* 6,18:00
RandomizedDelaySec=12h
Persistent=true
plocate_1.1.12-1/plocate-updatedb.timer
[Timer]
OnCalendar=daily
RandomizedDelaySec=12h
AccuracySec=20min
Persistent=true
python-certbot_1.18.0-1/debian/certbot.timer
[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=43200
Persistent=true
josm-installer_0.0.1+svn18171/josm-installer.timer
[Timer]
OnCalendar=*-*-* 00,12:00:00
RandomizedDelaySec=12h
Persistent=true
Hendrik Buchner
Oct 28, 2021, 4:50:03 AM10/28/21
to
> This appears to be inconsistent imho.
>
> Somehow I think systemd could be more clever here and scale down
> RandomizedDelaySec= on boot depending on how far in the past the
> timestamp of the stampfile is.
> Say the last activation of the service was 12 hours ago, then scale
> down RandomizedDelaySec=12h by dividing it by 12, so it would become
> RandomizedDelaySec=1h. The further in the past, the shorter the delay
> would become and the more likely that the timer fires eventually.
>
> And for consistencies sake, a (scaled) RandomizedDelaySec=1 should
> also be reapplied on resume.
>
That's right, if systemd would be capable of this behaviour, it would be
>
> Somehow I think systemd could be more clever here and scale down
> RandomizedDelaySec= on boot depending on how far in the past the
> timestamp of the stampfile is.
> Say the last activation of the service was 12 hours ago, then scale
> down RandomizedDelaySec=12h by dividing it by 12, so it would become
> RandomizedDelaySec=1h. The further in the past, the shorter the delay
> would become and the more likely that the timer fires eventually.
>
> And for consistencies sake, a (scaled) RandomizedDelaySec=1 should
> also be reapplied on resume.
>
very helpfull to not end up in a never updated system for example (like
it might happen to my systems).
Martin Lottermoser
Sep 5, 2023, 9:50:04 AM9/5/23
to
Hello,
I first came across this problem on a computer I maintain for an elderly
relative who uses it only occasionally, typically two or three times a
week for about an hour each time. As that includes online banking, the
system was configured to use the "unattended-upgrades" package to keep up
with security updates.
After the upgrade from Debian 10 to Debian 11 (switch from anacron to
systemd as the entity triggering automatic upgrades), I noticed that
while unattended-upgrades was still being executed, it didn't find any
packages to upgrade any more although such packages had been released.
At the time I fixed that by brute force with a daily anacron job for
"apt-get update".
Recently I observed similar behaviour on my own PC (three days without
installing security updates although such packages existed already for
at least two days). I therefore looked closer into this behaviour and
traced it to the problem described in this report, i.e., the
"RandomizedDelaySec=12h" line in the default configuration for
apt-daily.timer (which is part of the "apt" package and not of the
"systemd" package as implied by the current assignation of this report).
In the scenario as described above (typical run time 1 hour), this leads
to the system having only a chance of 1/12 (8 %) per boot session to
update its package lists, irrespective of how often it boots. (Note that
in contrast apt-daily-upgrade.timer is configured with
"RandomizedDelaySec=60m".)
Systems running continuously should obviously have no problems with the
default configuration, while for the use case above I would consider
this behaviour totally unacceptable for security reasons. In my opinion
the "apt" package should either provide a default configuration which
achieves the (presently misleading) "daily" claim implied by the name of
the units or at least prominently document the limitations of the
configuration so users will know when they have to take action.
For those who prefer to do the latter by restoring the old behaviour I
append a shell script I'm presently using with anacron. Because
apt-compat refuses to act if systemd is running at all, irrespective of
the status of the apt timers, we can't use it unmodified in its present
form; I suggest that it should be changed to behave similar to my
script which would make the latter obsolete. The initial comments in the
script describe how it is intended to be used.
Regards,
Martin Lottermoser
--
Martin Lottermoser martin.lo...@htp-tel.de
Greifswaldstrasse 28
38124 Braunschweig http://home.htp-tel.de/lottermose2
Germany Telephone: +49 531 6802747
I first came across this problem on a computer I maintain for an elderly
relative who uses it only occasionally, typically two or three times a
week for about an hour each time. As that includes online banking, the
system was configured to use the "unattended-upgrades" package to keep up
with security updates.
After the upgrade from Debian 10 to Debian 11 (switch from anacron to
systemd as the entity triggering automatic upgrades), I noticed that
while unattended-upgrades was still being executed, it didn't find any
packages to upgrade any more although such packages had been released.
At the time I fixed that by brute force with a daily anacron job for
"apt-get update".
Recently I observed similar behaviour on my own PC (three days without
installing security updates although such packages existed already for
at least two days). I therefore looked closer into this behaviour and
traced it to the problem described in this report, i.e., the
"RandomizedDelaySec=12h" line in the default configuration for
apt-daily.timer (which is part of the "apt" package and not of the
"systemd" package as implied by the current assignation of this report).
In the scenario as described above (typical run time 1 hour), this leads
to the system having only a chance of 1/12 (8 %) per boot session to
update its package lists, irrespective of how often it boots. (Note that
in contrast apt-daily-upgrade.timer is configured with
"RandomizedDelaySec=60m".)
Systems running continuously should obviously have no problems with the
default configuration, while for the use case above I would consider
this behaviour totally unacceptable for security reasons. In my opinion
the "apt" package should either provide a default configuration which
achieves the (presently misleading) "daily" claim implied by the name of
the units or at least prominently document the limitations of the
configuration so users will know when they have to take action.
For those who prefer to do the latter by restoring the old behaviour I
append a shell script I'm presently using with anacron. Because
apt-compat refuses to act if systemd is running at all, irrespective of
the status of the apt timers, we can't use it unmodified in its present
form; I suggest that it should be changed to behave similar to my
script which would make the latter obsolete. The initial comments in the
script describe how it is intended to be used.
Regards,
Martin Lottermoser
--
Martin Lottermoser martin.lo...@htp-tel.de
Greifswaldstrasse 28
38124 Braunschweig http://home.htp-tel.de/lottermose2
Germany Telephone: +49 531 6802747
0 new messages