Boot Dependancies - a weird wacky wonderful new idea
Robert Woodcock
Hello, I have an idea (ok, maybe not an original idea, but bear with me :)
It involves the sysvinit (/etc/init.d/rcS and a lotta other stuff) and
dpkg packages. (update-rc.d)
Parallelized booting. What this means is we run multiple bootup scripts
simultaneously. It's a *lot* faster on mid-to-higher-end machines, even
with just one CPU - it'd be wickedly fast with SMP.
"Hey wait a second that won't work!!! What if the netstd_nfs script runs
before the netbase script is done processing?"
Well, you're right, but I have another (slightly more original) idea that
will make it work. Boot-time dependancies.
* Instead of relying on those icky priority numbers we just had a flame
war or two about, the packages tell us how they *really* feel and state
precisely in a file (I'm thinking /etc/init.d/deps/[KS]packagename) what
needs to be successfully running (or not running) before or after they
can start. Do we need to make this information separate for each runlevel?
(i.e. make it /etc/rc?.d/deps/[KS]packagename instead?)
* /etc/init.d/rc is modified to call a program that determines the order
the scripts should be run in, on the fly. I figure this won't be much
of a speed hit. Slrn can thread thousands of messages per second across
a localhost news connection on my machine, sorting 30 scripts shouldn't
take long either. Actually, we don't even need to thread, we can probably
get away with just checking on script exit (any script exit) whether
there's something else available to run or not - if there's nothing else
available but there's still stuff that wants to be run, we can just run
it anyway.
* Because a fatal error on bootup is simply not an acceptable option,
dependancies are considered 'soft' - if a dependancy can not be met
for whatever reason, it is simply ignored. (This is what our existing
setup with priority numbers does anyway.) We'd want to keep a debugging
flag in the works to reveal such a situation though.
* update-rc.d is modified to ignore priority numbers and keep things in
the new format. (hmmm, perhaps we can be trickier than this.)
Problems looming:
* Installing new packages using the new dependancy format on older machines
without thread-scripts needs to be handled - perhaps we can keep the NN
switch to update-rc.d calls in new packages (at least for a while)
* Going back and forth from the old setup to the new one won't be a walk in
the park. I'd need to make a conversion utility.
* My coding skills ;)
I still have yet to create proof-of-concept code for this but the general
idea has created such a buzz in #debian that we might just see something :)
As for the file formats of /etc/init.d/deps/[KS]packagename, I was
thinking something along the lines of:
After: <packagename> <anotherpackagename> <etc>
Before: <morestuff> <goeshere>
I feel this information absolutely positively must be admin-editable, which
dictates some type of text file format.
Perhaps we could give it the idea of psuedo-packagenames such as 'everything'
so we can have 'After: everything' for Srmnologin and Sxdm.
i.e. /etc/init.d/deps/Sbind would have:
-CUT-
After: sysklogd kerneld netstd_init
-CUT-
If we are ever going to do this it would require at least the agreement of
Miquel van Smoorenburg, Klee Dienes, and Ian Jackson (current sysvinit and
dpkg maintainers) and a good concensus among the Debian developers,
preceeded by a LOT of discussion, so I figured I'd throw it out onto
debian-devel and await the replies: "so... what *is* that you're smoking?" :)
Please let me know what you think of this.
--
Robert Woodcock - r...@oz.net
"Unix and C are the ultimate computer viruses" -- Richard Gabriel
--
To UNSUBSCRIBE, email to debian-dev...@lists.debian.org
with a subject of "unsubscribe". Trouble? Contact listm...@lists.debian.org
Avery Pennarun
On Wed, Jun 10, 1998 at 07:57:18PM -0700, Robert Woodcock wrote:
> Parallelized booting. What this means is we run multiple bootup scripts
> simultaneously. It's a *lot* faster on mid-to-higher-end machines, even
> with just one CPU - it'd be wickedly fast with SMP.
>
> "Hey wait a second that won't work!!! What if the netstd_nfs script runs
> before the netbase script is done processing?"
>
> Well, you're right, but I have another (slightly more original) idea that
> will make it work. Boot-time dependancies.
Parallelized booting is a good idea (I'm almost certain that _most_ of my
bootup time, and almost ALL of my shutdown time, is spent waiting for tasks
to decide they're ready) but boot-time dependencies are way too much work.
What's wrong with priority levels? Programs start up in alphabetical order.
I wish they would die in reverse-alphabetical order (then we could have
S99xdm and K99kdm, which would make the file-rc package look nicer) but
priority levels do exactly what you want -- they express boot-time
dependencies.
We can achieve a huge degree of parallelism simply by running all boot
scripts of each priority level in parallel. That is, run all scripts at
level 01 (and wait to finish), then run all scripts at level 02, and so on.
To see why this should be "parallel enough," look at the large number of
scripts running at the default level 20.
Also, the only guarantee we've ever made is that scripts are executed in
priority order, so running all scripts at level 20 simultaneously shouldn't
cause a problem.
The hardest thing would be keeping the log messages from getting tangled up.
That shouldn't really be too hard for a C/C++ program to do. I've done
something almost appropriate in my wvdial package (see class WvLog and
WvLogRcv).
If enough people dare me, I bet I could whip something like this up (using
the existing priority scheme) in an hour or two.
Have fun,
Avery
P.S. why isn't file-rc the standard? It's _so_ much better! And why can't
we kill things in reverse alphabetical order?
Philip Hands
> Parallelized booting. What this means is we run multiple bootup scripts
> simultaneously. It's a *lot* faster on mid-to-higher-end machines, even
> with just one CPU - it'd be wickedly fast with SMP.
I like it.
This sounds like a job for make (which can run things in parallel)
It shouldn't be too much trouble to come up with a few make rules that (on an
unmodified system) run all the rc scripts, one at a time, in the same order,
as they would be run now (for backward compatibility).
Then, as you make scripts parallel-isable, you do something[1] so that make can
pick up on the real dependencies (rather that the over-cautious default it
would have used), and your done.
This way, new scripts on an old system work as normal.
Old scripts with a new paraliasble make based /etc/init.d/rc script work fine
too.
You can use make's load/job limiting options to tune the best number of things
to run at once.
The only problem I see with this is making it survive things like file
corruption in the Makefiles, but I guess corrupted init scripts are fairly bad
news as it is, so that's not making thinks much worse.
Cheers, Phil.
[1] like make /etc/init.d/deps/[KS]packagename be makefile fragments that
get included into the master makefile.
Yann Dirson
Avery Pennarun writes:
> What's wrong with priority levels? Programs start up in alphabetical order.
> I wish they would die in reverse-alphabetical order (then we could have
> S99xdm and K99kdm, which would make the file-rc package look nicer) but
> priority levels do exactly what you want -- they express boot-time
> dependencies.
No. I'd say priorities are a higher-level abstraction than real
deps. You loose a lot of information when translating real deps into
priorities, which can make it difficult in case of deps changes: one
dep disapearing may well stay unnoticed, and thus not paralelized; one
appearing may eventually cause trouble...
> We can achieve a huge degree of parallelism simply by running all boot
> scripts of each priority level in parallel. That is, run all scripts at
> level 01 (and wait to finish), then run all scripts at level 02, and so on.
> To see why this should be "parallel enough," look at the large number of
> scripts running at the default level 20.
...which would mean that level 20 is nearly all that can be
parallelized ? The only other duplicate priorities on my system are
2 S10 and 2 S50... I guess there may be some unnecessary distinct
levels down there.
> The hardest thing would be keeping the log messages from getting tangled up.
> That shouldn't really be too hard for a C/C++ program to do. I've done
> something almost appropriate in my wvdial package (see class WvLog and
> WvLogRcv).
Miquel told me some time ago of a boot-logging daemon he plans, that
would save these boot messages. Maybe we should just modify boot
sctipts to redirect their outputs individually through a bootlog
facility, not unlike syslog ? That would just have to be done in "rc"
itself !
Philip Hands writes:
> This sounds like a job for make (which can run things in parallel)
/bin/make ? That would just be the 3rd largest binary there, after
bash and tar :(
> [1] like make /etc/init.d/deps/[KS]packagename be makefile fragments that
> get included into the master makefile.
sure "include /etc/init.d/deps/*" is an easy thing.
> You can use make's load/job limiting options to tune the best number of things
> to run at once.
Yes.
Maybe we can just add a series of #ifdef in make sources to make a
/bin/boot-make that would have small footprint both in RAM and in / ?
But maybe it's not worth the time to do it ?
--
Yann Dirson <ydi...@mygale.org> | Stop making M$-Bill richer & richer,
isp-email: <ydi...@a2points.com> | support Debian GNU/Linux:
debian-email: <dir...@debian.org> | more powerful, more stable !
http://www.mygale.org/~ydirson/ | Check <http://www.debian.org/>