Scripts in init.d getting run twice...

[stainl...@gmail.com]

It appears that /etc/init.d scripts are getting run twice during system boot up.  Most scripts handle this gracefully, but it's inefficient, and scripts which don't handle being run twice gracefully will cause problems.

I found the following article: http://forum.xda-developers.com/showthread.php?t=1777263, but it is written for a different platform, so I don't know how applicable it is to Alt-F.  I tried the suggestion to remove the symlink between busybox and run-parts and it didn't fix the problem, so I'm looking for a different fix or some suggestions.

Thanks.

[João Cardoso]

On Friday, February 21, 2014 1:10:50 AM UTC, stainl...@gmail.com wrote:

It appears that /etc/init.d scripts are getting run twice during system boot up.

I don't think so, why do you say that? based on what facts? diagnostics or logs are needed.

What should happens:

-/etc/init.d/rcS starts, near its end, all active scripts,

-when the user initscript runs, the *status* of all services is logged,

-when the /Alt-F folder is detected, any active initscript found under /Alt-F/etc/init.d will be executed

[stainl...@gmail.com]

I don't think so, why do you say that? based on what facts? diagnostics or logs are needed.

I have an init script which writes to a log file.  It writes twice about 10 seconds apart.

I can also clarify the only script that I'm dealing with is one that is in /Alt-F/etc/init.d (I had to add it there or it would not survive reboots).  It appears from your description below, that it will indeed be executed twice...

What should happens:

-/etc/init.d/rcS starts, near its end, all active scripts,

This is one execution of the scripts.
 

-when the user initscript runs, the *status* of all services is logged,

-when the /Alt-F folder is detected, any active initscript found under /Alt-F/etc/init.d will be executed

Because every file in /Alt-F/etc/init.d also appears on /etc/init.d, doesn't this mean that all the scripts in /Alt-F/etc/init.d are executed twice?  Or is there something about the way the shadow filesystem is handled that I'm misunderstanding?

Thanks.

[João Cardoso]

On Friday, February 21, 2014 8:59:28 PM UTC, stainl...@gmail.com wrote:

I don't think so, why do you say that? based on what facts? diagnostics or logs are needed.

I have an init script which writes to a log file.  It writes twice about 10 seconds apart.

I can also clarify the only script that I'm dealing with is one that is in /Alt-F/etc/init.d (I had to add it there or it would not survive reboots).  It appears from your description below, that it will indeed be executed twice...

What should happens:

-/etc/init.d/rcS starts, near its end, all active scripts,

This is one execution of the scripts.
 

-when the user initscript runs, the *status* of all services is logged,

-when the /Alt-F folder is detected, any active initscript found under /Alt-F/etc/init.d will be executed

Because every file in /Alt-F/etc/init.d also appears on /etc/init.d,

Only after hotplugging detects a new filesystem and an Alt-F folder is found there

 

doesn't this mean that all the scripts in /Alt-F/etc/init.d are executed twice?

No, but...

 

  Or is there something about the way the shadow filesystem is handled that I'm misunderstanding?

As disk spinup and filesystem detection is a "slow" process, *usually*  filesystem/Alt-F detection happens after the initscripts have already been run.

But if the disk is small, spins-up fast, or the Alt-F folder is on a small filesystem, it might happens that /Alt-F exists when the initscripts are run, and then yes, they will be run twice.

In short, the sequence:

1-/etc/init.d/rcS starts and, near its end, all active scripts are executed

2-the /Alt-F folder is detected, and any active initscript found under /Alt-F/etc/init.d will be executed

is what I observe in my system. But if

1-the /Alt-F folder is detected early, and active initscript found under /Alt-F/etc/init.d will be executed

2-/etc/init.d/rcS runs all active scripts, which now also contains the /Alt-F ones, which are executed twice.

Is this what you observe in your system? You could add some diagnostics to your initscripts, such as 'logger -t myinit "here we go"' and compare the dates with the ones in the system log and the hot*.logs, which shows the hotplugging events.

I can easily block hotplugging until rcS ends, which will solve the issue, but I want to be sure that the diagnostic is correct.

Thanks

Thanks.

[stainl...@gmail.com]

From your description and my observed system behavior, it appears that there is a race condition between the scripts and the filesystem mounting/hotplugging.  I would strongly recommend you consider a solution where you do not run any init.d script until all of the filesystems are fully mounted, hotplugged, etc.  It can easily be the case that the scripts rely on files on non-Alt-F filesystems.

As far as confirming your diagnosis:

• The Alt-F folder is on a small filesystem--only 4.5 GB.  The other filesystems are much larger (10x and 100x).
• I detected the double runs when switching between Debian and Alt-F using the "debian" and "alt-f" scripts.  In these cases, there is no disk spin-up. 
  
• I believe I also saw the behavior on a cold-start but I would have to play with the system again to be sure, so for now, we should consider it not confirmed on cold-start (when we do wait for disk spin-up).
  
• I have confirmed that the other filesystems are not mounted during the first run of the script.  In fact, this is how I detected the multiple runs in the first place--a file required by the script is on a different file system than Alt-F and was not being found.
• Further, I'm taking advantage of the fact the other filesystems are not mounted to work around the current behavior.  If no other filesystems are mounted, the script just exits. This makes the first run a no-op.  I do this by calling mountpoint on a required filesystem.  This works great, but it is a hack...
  

Is this enough confirmation, or do you still need me to set up and run the syslog stuff?


Thanks.

[João Cardoso]

On Monday, February 24, 2014 5:39:00 PM UTC, stainl...@gmail.com wrote:

I would strongly recommend you consider a solution where you do not run any init.d script until all of the filesystems are fully mounted.

I can't do that because some initscripts need to run even when no disks are present.

Alt-F runs from flash-memory, no disks are needed, and disks can easily be removed/added/swapped (this is the difference from a normal PC).

And some services might need different/several filesystems to be mounted, some of which might be available while others aren't, either because the fs is being checked or it has simply disappear and will never get back online.

Samba, NFS, ftp, ssh, etc, seems to handle this situation fine, while others, e.g., minidlna, might not.

I can't devise any approach (there is none!) to automatically deal with this issue, as it depends on how the user has setup the service or filesystems.

 

  It can easily be the case that the scripts rely on files on non-Alt-F filesystems, and they will fail simply due to the race condition.

As far as confirming your diagnosis:

• The Alt-F folder is on a small filesystem--only 4.5 GB.  The other filesystems are much larger.
• I have confirmed that the other filesystems are not mounted during the first run of the script.  In fact, this is how I detected the multiple runs in the first place--a file required by the script is on a different file system and was not being found.

• Further, I'm taking advantage of the fact the other filesystems are not mounted to work around the current behavior.  If no other filesystems are mounted, the script just exits. This makes the first run a no-op.  I do this by calling mountpoint on a required filesystem.

Is this enough confirmation, or do you still need me to set up and run the syslog stuff?

That's OK, thanks.

What I have done is to make disk-installed initscripts to wait until all the base initscripts have run.

This avoids them to run twice when installed on a small filesystem (which is a good thing to do), but does not guaranty that all fs will be available, which can't be done, as explained above.

Of course I could rely on a pure static fstab approach, and not run initscripts until all fs are mounted, but then disk plugging/removal would rise other issues.

Thanks

[stainl...@gmail.com]

OK.  I understand what you are saying.

One option would be to do a pure static fstab approach for only some of the init.d scripts.  Say all of the ones greater than S59...  So, if the script is named anything in S00* thru S59* it runs whenever, and if named S60* thru S99* it waits for the initial mount process to be complete.  I don't think disk plugging is an issue because either Alt-F has completed its initial filesystem mount or it hasn't--after that it's just regular operation of the system and disk plugging happens.

However, I totally understand if you don't want to bother with coding something like this up...

Strictly speaking, init scripts should be OK with being run twice (that's why there's all of the pidfile stuff), and a carefully coded script should fail gracefully when a needed file system is not present.  I brought the running twice issue up here because most *nix implementations don't run the scripts twice.  You have now resolved this.  Thank you.

What I have learned here is that a script writer for Alt-F needs to treat a missing filesystem as a "soft" error--it may be due to a startup race condition, not that the disk is truly missing...  Is there a simple way to determine that Alt-F is still in the process of initial file system mounts?  If there is, then my script can tell the difference between the race condition and a missing filesystem.  (In the first case, it will wait, and in the second case, it will write an error to the log and exit.)


Thanks!

[João Cardoso]

On Wednesday, February 26, 2014 6:38:36 AM UTC, stainl...@gmail.com wrote:

OK.  I understand what you are saying.

One option would be to do a pure static fstab approach for only some of the init.d scripts.  Say all of the ones greater than S59...  So, if the script is named anything in S00* thru S59* it runs whenever, and if named S60* thru S99* it waits for the initial mount process to be complete.  I don't think disk plugging is an issue because either Alt-F has completed its initial filesystem mount or it hasn't--after that it's just regular operation of the system and disk plugging happens.

However, I totally understand if you don't want to bother with coding something like this up...

Strictly speaking, init scripts should be OK with being run twice (that's why there's all of the pidfile stuff), and a carefully coded script should fail gracefully when a needed file system is not present.  I brought the running twice issue up here because most *nix implementations don't run the scripts twice.  You have now resolved this.  Thank you.

What I have learned here is that a script writer for Alt-F needs to treat a missing filesystem as a "soft" error--it may be due to a startup race condition, not that the disk is truly missing...  Is there a simple way to determine that Alt-F is still in the process of initial file system mounts?

I don't think it is possible to that that safely. One can make reasonable assumptions, however.

-The kernel generates hotplug events when new disks or partitions are found; this depends on the disk itself, bigger disks take several seconds to spinup.

-/usr/sbin/hot.sh is called for each event

-for a recognized filesystem /usr/sbin/hot_aux.sh is called in the background and fsck the fs; this step can take a second or an hour; then the fs is mounted, fstab modified and and special directories are handled

hot*.sh are Alt-F simple-minded equivalent to udev.

hot_aux.sh fscks fs in parallel when they are on different disks, and sequences fsck operations on the same disk. A /tmp/check-<disk-part> file is created.

The problem is that we don't know when/if an event will appear.

A static fstab (which it should be, by definition) would solve all this issues, but it had to be maintained by users, and Alt-F was "designed" with sloopy users in mind.

NAME

       fstab - static information about the filesystems

 

DESCRIPTION

       The  file  fstab contains descriptive information about the various file systems.  fstab is

       only read by programs, and not written; it is the duty of the system administrator to prop-

       erly  create  and  maintain  this  file.

And of course that is not what Alt-F does.

I guess that I was marveled with the box hot-plugging capabilities, and always hot-swapping disks, so I wanted most of that to be automatic. Now, however, my setup is fixed :-(

Thinking...

A possible approach would be to tag required fstab entries, and those would not be changed by Alt-F; they could be used to verify that the system has fully booted. But a long fsck on one fs would block the whole boot, or at least some services -- sda2 and sdb2 are used by samba; should samba wait until both are available? Of course not!

Initially, the Alt-F boot sequence was blocked when a fs was being checked, but with most users using (dirty) ext2 fs inherited from the vendor's fw, that was a pain. So I changed that and created the Filesystem Operation section on the Status page, to warn users. And the power led blinks with a heart-beat rhythm. For RC4 an explicit warning even appears in the login page!

OK, I decided, I will not change this. Perhaps only for Alt-G :-)