Bug#635267: cron: @reboot jobs are not run
Helge Kreutzmann
Version: 3.0pl1-118
Severity: normal
In a freshly installed wheezy systems, @reboot jobs are not run. I see the
following in /var/log/syslog (grepped for cron):
Jul 23 12:53:53 sneo anacron[2161]: Job `cron.daily' terminated
Jul 23 12:53:53 sneo anacron[2161]: Normal exit (1 job run)
Jul 24 18:04:50 sneo anacron[2178]: Anacron 2.3 started on 2011-07-24
Jul 24 18:04:50 sneo anacron[2178]: Will run job `cron.daily' in 5 min.
Jul 24 18:04:50 sneo anacron[2178]: Will run job `cron.weekly' in 10 min.
Jul 24 18:04:50 sneo anacron[2178]: Jobs will be executed sequentially
Jul 24 18:04:51 sneo /usr/sbin/cron[2252]: (CRON) INFO (pidfile fd = 3)
Jul 24 18:04:51 sneo /usr/sbin/cron[2253]: (CRON) STARTUP (fork ok)
Jul 24 18:04:51 sneo /usr/sbin/cron[2253]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
Jul 24 18:09:50 sneo anacron[2178]: Job `cron.daily' started
Jul 24 18:09:50 sneo anacron[2933]: Updated timestamp for job `cron.daily' to 2011-07-24
(Please note this is done at every reboot, and yes, it *is* system startup).
Adding $all to required_start does not change this behaviour (thus I
think this bug is different from #634215)
-- Package-specific info:
--- EDITOR:
not set
--- usr/bin/editor:
/usr/bin/vim.basic
--- /usr/bin/crontab:
-rwxr-sr-x 1 root crontab 33984 Jun 5 11:27 /usr/bin/crontab
--- /var/spool/cron
drwxr-xr-x 5 root root 4096 Jul 9 13:12 /var/spool/cron
--- /var/spool/cron/crontabs
drwx-wx--T 2 root crontab 4096 Jun 5 11:27 /var/spool/cron/crontabs
-- System Information:
Debian Release: wheezy/sid
APT prefers testing
APT policy: (500, 'testing')
Architecture: amd64 (x86_64)
Kernel: Linux 2.6.39.3sneo.05
Locale: LANG=de_DE.UTF-8, LC_CTYPE=de_DE.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages cron depends on:
ii adduser 3.113 add and remove users and groups
ii debianutils 4.0.2 Miscellaneous utilities specific t
ii dpkg 1.16.0.3 Debian package management system
ii libc6 2.13-10 Embedded GNU C Library: Shared lib
ii libpam-runtime 1.1.3-2 Runtime support for the PAM librar
ii libpam0g 1.1.3-2 Pluggable Authentication Modules l
ii libselinux1 2.0.98-1.1 SELinux runtime shared libraries
ii lsb-base 3.2-27 Linux Standard Base 3.2 init scrip
Versions of packages cron recommends:
ii exim4 4.76-2 metapackage to ease Exim MTA (v4)
ii exim4-daemon-light [mail-tran 4.76-2 lightweight Exim MTA (v4) daemon
Versions of packages cron suggests:
ii anacron 2.3-14 cron-like program that doesn't go
pn checksecurity <none> (no description available)
ii logrotate 3.7.8-6 Log rotation utility
Versions of packages cron is related to:
pn libnss-ldap <none> (no description available)
pn libnss-ldapd <none> (no description available)
pn libpam-ldap <none> (no description available)
pn libpam-mount <none> (no description available)
pn nis <none> (no description available)
pn nscd <none> (no description available)
-- no debconf information
--
Dr. Helge Kreutzmann deb...@helgefjell.de
Dipl.-Phys. http://www.helgefjell.de/debian.php
64bit GNU powered gpg signed mail preferred
Help keep free software "libre": http://www.ffii.de/
Christian Kastner
thanks
On 07/24/2011 04:46 PM, Helge Kreutzmann wrote:
> In a freshly installed wheezy systems, @reboot jobs are not run.
>
> Jul 24 18:04:51 sneo /usr/sbin/cron[2252]: (CRON) INFO (pidfile fd = 3)
> Jul 24 18:04:51 sneo /usr/sbin/cron[2253]: (CRON) STARTUP (fork ok)
> Jul 24 18:04:51 sneo /usr/sbin/cron[2253]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
> Jul 24 18:09:50 sneo anacron[2178]: Job `cron.daily' started
> Jul 24 18:09:50 sneo anacron[2933]: Updated timestamp for job `cron.daily' to 2011-07-24
>
> (Please note this is done at every reboot, and yes, it *is* system startup).
cron decides on whether to run @reboot jobs by checking if the file
/var/run/crond.reboot exists. If it exists, cron skips @reboot jobs; if
it doesn't, it runs @reboot jobs, then creates it (so they are skipped
next time). cron does this to ensure @reboot jobs are really run only
once, instead of eg: when cron is restarted for some reason.
In your case, the /var/run/crond.reboot seems to be preserved across
reboots, which I cannot reproduce. Could you post the output of
`mount | grep run` here?
Christian
Helge Kreutzmann
thanks for your very fast reply.
On Sun, Jul 24, 2011 at 08:42:04PM +0200, Christian Kastner wrote:
> On 07/24/2011 04:46 PM, Helge Kreutzmann wrote:
> > In a freshly installed wheezy systems, @reboot jobs are not run.
> >
> > Jul 24 18:04:51 sneo /usr/sbin/cron[2252]: (CRON) INFO (pidfile fd = 3)
> > Jul 24 18:04:51 sneo /usr/sbin/cron[2253]: (CRON) STARTUP (fork ok)
> > Jul 24 18:04:51 sneo /usr/sbin/cron[2253]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
> > Jul 24 18:09:50 sneo anacron[2178]: Job `cron.daily' started
> > Jul 24 18:09:50 sneo anacron[2933]: Updated timestamp for job `cron.daily' to 2011-07-24
> >
> > (Please note this is done at every reboot, and yes, it *is* system startup).
>
> cron decides on whether to run @reboot jobs by checking if the file
> /var/run/crond.reboot exists. If it exists, cron skips @reboot jobs; if
root@sneo:~# ls -l /var/run/crond.reboot
---------- 1 root root 0 9. Jul 13:36 /var/run/crond.reboot
Clearly looks like it has not been removed.
> it doesn't, it runs @reboot jobs, then creates it (so they are skipped
> next time). cron does this to ensure @reboot jobs are really run only
> once, instead of eg: when cron is restarted for some reason.
Thanks for the explanation.
> In your case, the /var/run/crond.reboot seems to be preserved across
> reboots, which I cannot reproduce. Could you post the output of
> `mount | grep run` here?
tmpfs on /var/run/lock type tmpfs
(rw,noexec,nosuid,nodev,size=5242880,mode=1777,size=5242880,mode=1777)
tmpfs on /var/run/shm type tmpfs
(rw,nosuid,nodev,size=20%,mode=1777,size=20%,mode=1777)
tmpfs on /var/run type tmpfs
(rw,noexec,nosuid,relatime,size=818528k,mode=755)
Shouldn't a tmpfs be empty after a reboot?
Thanks!
Greetings
Helge
Christian Kastner
> On Sun, Jul 24, 2011 at 08:42:04PM +0200, Christian Kastner wrote:
>> cron decides on whether to run @reboot jobs by checking if the file
>> /var/run/crond.reboot exists. If it exists, cron skips @reboot jobs; if
>
> root@sneo:~# ls -l /var/run/crond.reboot
> ---------- 1 root root 0 9. Jul 13:36 /var/run/crond.reboot
>
> Clearly looks like it has not been removed.
>
>> reboots, which I cannot reproduce. Could you post the output of
>> `mount | grep run` here?
>
> tmpfs on /var/run/lock type tmpfs
> (rw,noexec,nosuid,nodev,size=5242880,mode=1777,size=5242880,mode=1777)
> tmpfs on /var/run/shm type tmpfs
> (rw,nosuid,nodev,size=20%,mode=1777,size=20%,mode=1777)
> tmpfs on /var/run type tmpfs
> (rw,noexec,nosuid,relatime,size=818528k,mode=755)
>
> Shouldn't a tmpfs be empty after a reboot?
Definitely yes.
/var/run/crond.reboot still being there would indicate that there's
another FS in play, but I wouldn't really know why this is happening.
One possibility would be that something went wrong during the recent
/var/run -> /run transition (I see the above mounts under /run)
Could you look into /etc/fstab (and maybe /etc/default/tmpfs) and see if
you can spot something fishy?
Christian
Helge Kreutzmann
/run is a symlink to /var/run on that machine.
> Could you look into /etc/fstab (and maybe /etc/default/tmpfs) and see if
> you can spot something fishy?
/etc/fstab:
/dev/mapper/system_vg-var_vg /var ext3 nodev,nosuid,data=ordered 0 2
/dev/mapper/system_vg-aptcache_lv /var/cache/apt-cacher-ng
ext3 nodev,nosuid,noexec,data=ordered 0 2
Those tmpfs file systems are not mentioned.
I did not touch /etc/default/tmpfs, the non comment options are:
TMPFS_SIZE=20%
RUN_SIZE=10%
LOCK_SIZE=5242880 # 5MiB
SHM_SIZE=
TMP_SIZE=
RW_SIZE=5242880 # 5 MiB
However, I see the following error message:
syslog.1:Jul 24 18:31:11 sneo kernel: tmpfs: Bad mount option data
So somehow the tmpfs-entries inherit the mount options from /var?
I removed the "data=ordered" from /var and together with the following
changes, the "Bad mount option" error message is gone.
Next I tried to get tmpfs mounted on /var/run
I disabled the other two tmpfs entries in /etc/default/rcS, i.e.
/run/lock and /runshm.
I added /var/run explicitly into /etc/fstab
tmpfs /var/run tmpfs nodev,nosuid,size=5242880,mode=755
0 0
Now, after a reboot, tmpfs is supposedly mounted (i.e. mount|grep
tmpfs). However, the files in /var/run look like before. When I umount
/var/run, I get an error message that the file system is not mounted.
Now, in mount|grep tmpfs /var/run is no longer listed. When I mount
/var/run explicitly afterwards, /var/run is empty as expected (and
tmpfs mounted).
I'm clearly running out of ideas how to get /var/run mounted as tmpfs.
Actually, I probably don't understand which script mounts what and
when; ideally, /var/run would simply be zeroed at reboot...
If you have any ideas, they are gladly appreciated.
Christian Kastner
> On Sun, Jul 24, 2011 at 11:12:41PM +0200, Christian Kastner wrote:
>> One possibility would be that something went wrong during the recent
>> /var/run -> /run transition (I see the above mounts under /run)
>
> /run is a symlink to /var/run on that machine.
Well, that's the first weird thing -- AFAIK it should be the other way
around. Looking at sysvinit's changelog, the above could be a result of
the transition process, but I'm not sure.
>> Could you look into /etc/fstab (and maybe /etc/default/tmpfs) and see if
>> you can spot something fishy?
>
> /etc/fstab:
> /dev/mapper/system_vg-var_vg /var ext3 nodev,nosuid,data=ordered 0 2
> /dev/mapper/system_vg-aptcache_lv /var/cache/apt-cacher-ng
> ext3 nodev,nosuid,noexec,data=ordered 0 2
>
> Those tmpfs file systems are not mentioned.
From a brief look, it appears as though /etc/init.d/mountkernfs.sh
automatically takes care of them.
> syslog.1:Jul 24 18:31:11 sneo kernel: tmpfs: Bad mount option data
>
> So somehow the tmpfs-entries inherit the mount options from /var?
Nope
> I removed the "data=ordered" from /var and together with the following
> changes, the "Bad mount option" error message is gone.
I'd put that back in. It's irrelevant to this case, but there was some
controversy regarding data safety around it (although I really don't
recall what it was)
> Next I tried to get tmpfs mounted on /var/run
>
> I disabled the other two tmpfs entries in /etc/default/rcS, i.e.
> /run/lock and /runshm.
>
> I added /var/run explicitly into /etc/fstab
> tmpfs /var/run tmpfs nodev,nosuid,size=5242880,mode=755
> 0 0
>
> Now, after a reboot, tmpfs is supposedly mounted (i.e. mount|grep
> tmpfs). However, the files in /var/run look like before. When I umount
> /var/run, I get an error message that the file system is not mounted.
> Now, in mount|grep tmpfs /var/run is no longer listed. When I mount
> /var/run explicitly afterwards, /var/run is empty as expected (and
> tmpfs mounted).
>
> I'm clearly running out of ideas how to get /var/run mounted as tmpfs.
> Actually, I probably don't understand which script mounts what and
> when; ideally, /var/run would simply be zeroed at reboot...
Again, to me it looks like something went wrong during the /run
transition. For example, this is an excerpt from my /etc/fstab:
# /var
UUID=<foo> /var jfs defaults 0 2
# This mount for /run replaces the default configured in
# /etc/default/tmpfs
tmpfs /run tmpfs nosuid,noexec,size=1m,mode=755 0 0
# This mount for /run/lock replaces the default configured in
# /etc/default/tmpfs
tmpfs /run/lock tmpfs nodev,noexec,nosuid,size=1m,mode=1777 0 0
I'm not sure I can help any further because I followed the transition
only in passing, but perhaps you can find some hints in the changelog
entries[0] for that transition?
Christian
[0] see 2.88dsf-13.3, 2.88dsf-13.5
http://packages.debian.org/changelogs/pool/main/s/sysvinit/current/changelog
Helge Kreutzmann
reassign -1 sysvinit
severity -1 normal
found -1 2.88dsf-13.10
thanks
Hello Christian,
On Mon, Jul 25, 2011 at 06:26:07PM +0200, Christian Kastner wrote:
> On 07/25/2011 01:40 PM, Helge Kreutzmann wrote:
> > On Sun, Jul 24, 2011 at 11:12:41PM +0200, Christian Kastner wrote:
> >> One possibility would be that something went wrong during the recent
> >> /var/run -> /run transition (I see the above mounts under /run)
> >
> > /run is a symlink to /var/run on that machine.
>
> Well, that's the first weird thing -- AFAIK it should be the other way
> around. Looking at sysvinit's changelog, the above could be a result of
> the transition process, but I'm not sure.
Well looking at the changelog, the transition happend before my
install. (Which happend on July 9th, cf. #633306 for the details). So
somehow the install created the wrong symlinks and sysvinit did not
change them because it assumed the transition to be over, I assume?
Cloning to sysvinit to get the best course of action. Simply turn the
symlinks around?
> >> you can spot something fishy?
> >
> > /etc/fstab:
> > /dev/mapper/system_vg-var_vg /var ext3 nodev,nosuid,data=ordered 0 2
> > /dev/mapper/system_vg-aptcache_lv /var/cache/apt-cacher-ng
> > ext3 nodev,nosuid,noexec,data=ordered 0 2
> >
> > Those tmpfs file systems are not mentioned.
>
> From a brief look, it appears as though /etc/init.d/mountkernfs.sh
> automatically takes care of them.
That's where I grepped their existance, too.
> > syslog.1:Jul 24 18:31:11 sneo kernel: tmpfs: Bad mount option data
> >
> > So somehow the tmpfs-entries inherit the mount options from /var?
>
> Nope
I don't think so neither, but I'm totally lost where they come from. I
only specified them explicitly for the file systems in /etc/fstab.
> > I removed the "data=ordered" from /var and together with the following
> > changes, the "Bad mount option" error message is gone.
>
> I'd put that back in. It's irrelevant to this case, but there was some
> controversy regarding data safety around it (although I really don't
> recall what it was)
I turned it on for syslog and the error remains absent. Probably
related to my other changes.
(Btw. I picked it up on lwn some time ago and added it as a safety
measure).
I don't have those comments and prior to adding it myself, /var/run
was not mentioned in /etc/fstab.
> I'm not sure I can help any further because I followed the transition
> only in passing, but perhaps you can find some hints in the changelog
> entries[0] for that transition?
>
> Christian
>
>
> [0] see 2.88dsf-13.3, 2.88dsf-13.5
> http://packages.debian.org/changelogs/pool/main/s/sysvinit/current/changelog
I'll clone the bug, maybe some d-i/sysvinit interaction is broken.
Unless I hear otherwise, I'll try to manually switch the symlinks
around and see how it goes.
Thanks again *very much* for your help!
Christian Kastner
On 07/25/2011 07:18 PM, Helge Kreutzmann wrote:
> clone 635267 -1
> reassign -1 sysvinit
> severity -1 normal
> found -1 2.88dsf-13.10
> thanks
> On Mon, Jul 25, 2011 at 06:26:07PM +0200, Christian Kastner wrote:
>> I'm not sure I can help any further because I followed the transition
>> only in passing, but perhaps you can find some hints in the changelog
>> entries[0] for that transition?
>>
>> Christian
>>
>> [0] see 2.88dsf-13.3, 2.88dsf-13.5
>> http://packages.debian.org/changelogs/pool/main/s/sysvinit/current/changelog
I only now noticed a bookmark I created earlier and seem to have
forgotten about. Perhaps this is helpful, too.
http://lists.debian.org/debian-devel/2011/05/msg00598.html
> I'll clone the bug, maybe some d-i/sysvinit interaction is broken.
> Unless I hear otherwise, I'll try to manually switch the symlinks
> around and see how it goes.
You forgot to CC: con...@b.d.o, could you try again?
Thanks,
Christian
Helge Kreutzmann
reassign -1 sysvinit
severity -1 normal
found -1 2.88dsf-13.10
thanks
Hello Christian,
On Tue, Jul 26, 2011 at 12:42:09AM +0200, Christian Kastner wrote:
> On 07/25/2011 07:18 PM, Helge Kreutzmann wrote:
> > clone 635267 -1
> > reassign -1 sysvinit
> > severity -1 normal
> > found -1 2.88dsf-13.10
> > thanks
>
> > On Mon, Jul 25, 2011 at 06:26:07PM +0200, Christian Kastner wrote:
> >> I'm not sure I can help any further because I followed the transition
> >> only in passing, but perhaps you can find some hints in the changelog
> >> entries[0] for that transition?
> >>
> >> Christian
> >>
> >> [0] see 2.88dsf-13.3, 2.88dsf-13.5
> >> http://packages.debian.org/changelogs/pool/main/s/sysvinit/current/changelog
>
> I only now noticed a bookmark I created earlier and seem to have
> forgotten about. Perhaps this is helpful, too.
>
> http://lists.debian.org/debian-devel/2011/05/msg00598.html
Thanks!
> > I'll clone the bug, maybe some d-i/sysvinit interaction is broken.
> > Unless I hear otherwise, I'll try to manually switch the symlinks
> > around and see how it goes.
>
> You forgot to CC: con...@b.d.o, could you try again?
Done now.
Helge Kreutzmann
I manually transitioned (the initscript maintainer did not respond for
more than 1 month to the cloned bug report, so he had ample time to
figure out the root cause (to avoid this issue for other users).
Result looks good:
Sep 18 19:30:58 sneo /usr/sbin/cron[2945]: (CRON) INFO (pidfile fd = 3)
Sep 18 19:30:58 sneo /usr/sbin/cron[2946]: (CRON) STARTUP (fork ok)
Sep 18 19:30:58 sneo /usr/sbin/cron[2946]: (CRON) INFO (Running @reboot jobs)
Sep 18 19:30:59 sneo /USR/SBIN/CRON[2955]: (logcheck) CMD ([2955]
if [ -x /usr/sbin/logcheck ]; then nice -n10 /usr/sbin/logcheck -R; fi)
(Don't mind the time, my clock is still off 1 hour)
Feel free to close this bug, the initscript maintainer can handle his
cloned one at his convenience (I'll add more info there in a moment).
Greetings & Thanks again for your quick and good help