To Whom It May Amuse (systemd-journald)

[Rainer Weikusat]

Ever since Lennart the Poettering start 'improving' syslog support for
Linux, I've been wondering how his 'improved' code would fare when
coming somewhat under actual load, as his 'improvement ideas' sounded
like the exact kind of lemon all his other ideas sounded like.

Now I know: In order to test some output queueing code, I'm writing 8K
blocks of base64 data to a socketpair. Another process on the other end
of the pair receives them and writes them to disk. I'm sending four of
these in a loop and then reschedule the send function through the
top-level I/O loop so that the (single-threaded) sending program also
gets some other stuff done. This causes a couple of lines of text to be
logged for each data block.

Outcome: The doubtlessly 'optimized' journald-code at 100% CPU usage, my
absolutely unoptimized Perl code sending the data at about 30% and the
writing process which also needs to do base64 decoding (written in C)
doesn't even show up.

Colour me unsurprised :-)))

[John McCue]

Rainer Weikusat <rwei...@talktalk.net> wrote:
<snip>

>
> Outcome: The doubtlessly 'optimized' journald-code at 100% CPU usage, my
> absolutely unoptimized Perl code sending the data at about 30% and the
> writing process which also needs to do base64 decoding (written in C)
> doesn't even show up.
>
> Colour me unsurprised :-)))

Curious, did you try that on a syslog system ?
Also how long did each process run (systend vs your perl) ?

Since everything goes through syslogd(8) I wonder if the
same would happen. If you can describe how you did this
or post the script I would like to try it uder OpenBSD.

John

--
csh(1) - "An elegant shell, for a more... civilized age."
- Paraphrasing Star Wars

[William Ahern]

John McCue <jmc...@fuzzball.mhome.org> wrote:
> Rainer Weikusat <rwei...@talktalk.net> wrote:
> <snip>
>>
>> Outcome: The doubtlessly 'optimized' journald-code at 100% CPU usage, my
>> absolutely unoptimized Perl code sending the data at about 30% and the
>> writing process which also needs to do base64 decoding (written in C)
>> doesn't even show up.
>>
>> Colour me unsurprised :-)))
>
> Curious, did you try that on a syslog system ?
> Also how long did each process run (systend vs your perl) ?
>
> Since everything goes through syslogd(8) I wonder if the
> same would happen. If you can describe how you did this
> or post the script I would like to try it uder OpenBSD.

For several years now there is no separate syslogd daemon for inbound logs.
systemd handles that directly:

$ ls -l /dev/log
lrwxrwxrwx 1 root root 28 Jan 26 13:52 /dev/log -> /run/systemd/journal/dev-log
$ sudo netstat -anlp | grep dev-log | tr -s ' '
unix 9 [ ] DGRAM 17134 1/init /run/systemd/journal/dev-log

On these systems you may still find a familiar daemon like rsyslogd managing
traditional text-based log files under /var/log. But it receives log events
from systemd, rather than directly from application processes.

It can all be rather confusing. On a default Ubuntu 20.04 server system you
can see rsyslog with an open descriptor to /run/systemd/journal/syslog, but
it's actually systemd which originally created that file and passed it to
rsyslog (see syslog.service and syslog.socket units) via systemd socket
activation. This isn't obvious from /etc/rsyslog.conf. This behavior is
baked into (Ubuntu's?) rsyslog/plugins/imuxsock/imuxsock.c. Similarly, it's
not obvious why or how systemd sends events to /run/systemd/journal/syslog.
On this Ubuntu system it's the default behavior--the default value for
ForwardToSyslog is "yes" (see /etc/systemd/journald.conf and man 5
journald.conf), and the implicit path when "yes" is to open and send log
events to /run/systemd/journal/syslog. (I don't even think this path is
configurable.)

Another alternative is for a service like rsyslog to use journald's event
listening facility. I had thought this was how rsyslogd was configured on
my Ubuntu systems until I just now confirmed.

And people wonder why systemd rubs some people the wrong way....

[Mut...@dastardlyhq.com]

Unfortunately the herd (or if you want to be cruel - sheep) mentality is
alive and well in the Linux distro community and where Red Hat goes the rest of
them follow. At least Devuan is trying to remain with old style init and
Slackware (if he ever bothers to release 15.0 it might be worth a look).

[Scott Lurndal]

William Ahern <wil...@25thandClement.com> writes:
>John McCue <jmc...@fuzzball.mhome.org> wrote:
>> Rainer Weikusat <rwei...@talktalk.net> wrote:
>> <snip>
>>>
>>> Outcome: The doubtlessly 'optimized' journald-code at 100% CPU usage, my
>>> absolutely unoptimized Perl code sending the data at about 30% and the
>>> writing process which also needs to do base64 decoding (written in C)
>>> doesn't even show up.
>>>
>>> Colour me unsurprised :-)))
>>
>> Curious, did you try that on a syslog system ?
>> Also how long did each process run (systend vs your perl) ?
>>
>> Since everything goes through syslogd(8) I wonder if the
>> same would happen. If you can describe how you did this
>> or post the script I would like to try it uder OpenBSD.
>
>For several years now there is no separate syslogd daemon for inbound logs.
>systemd handles that directly:

That depends entirely on which distribution you are using. There
are many distributions (including most BSDs) that eschew the steaming
pile of dog turds called systemd.

[Rainer Weikusat]

Mut...@dastardlyhq.com writes:
> On Tue, 01 Feb 2022 20:00:40 +0000
> Rainer Weikusat <rwei...@talktalk.net> wrote:
>>Ever since Lennart the Poettering start 'improving' syslog support for
>>Linux, I've been wondering how his 'improved' code would fare when
>>coming somewhat under actual load, as his 'improvement ideas' sounded
>>like the exact kind of lemon all his other ideas sounded like.

[...]

>>Outcome: The doubtlessly 'optimized' journald-code at 100% CPU usage,

[...]

>>Colour me unsurprised :-)))

> Unfortunately the herd (or if you want to be cruel - sheep) mentality
> is alive and well in the Linux distro community and where Red Hat goes
> the rest of them follow.

I think this is more a case of "what RedHat pays for, the others will
use".

> At least Devuan is trying to remain with old style init and
> Slackware (if he ever bothers to release 15.0 it might be worth a look).

I'm running Devuan at home but support systemd professionally (system
integration only). If it actually improved anything, instead of just
changing everything, I've yet to find it. Even service start scripts are
still with us because the systemd-provided facilities are too primitive
to replace them completely.

[Rainer Weikusat]

John McCue <jmc...@fuzzball.mhome.org> writes:
> Rainer Weikusat <rwei...@talktalk.net> wrote:
> <snip>
>>
>> Outcome: The doubtlessly 'optimized' journald-code at 100% CPU usage, my
>> absolutely unoptimized Perl code sending the data at about 30% and the
>> writing process which also needs to do base64 decoding (written in C)
>> doesn't even show up.
>>
>> Colour me unsurprised :-)))
>
> Curious, did you try that on a syslog system ?

I think I did worse things on syslog based systems, like running racoon
at log level debug for remote access VPNs for thousands of mobile users.

[...]

> Since everything goes through syslogd(8) I wonder if the
> same would happen.

Probably not: A traditional syslog daemon basically just writes log lines it
received from some set of sources to a set of text files, it's basically
just a data relay with no integrated data processing.

> If you can describe how you did this
> or post the script I would like to try it uder OpenBSD.

That's part of a >7000 LOC set of programs for enabling cloud-based
management of network threat detection appliances (and obviously, all
proprietary stuff).

[Mut...@dastardlyhq.com]

On Wed, 02 Feb 2022 15:45:13 +0000
Rainer Weikusat <rwei...@talktalk.net> wrote:

>Mut...@dastardlyhq.com writes:
>I'm running Devuan at home but support systemd professionally (system
>integration only). If it actually improved anything, instead of just
>changing everything, I've yet to find it. Even service start scripts are
>still with us because the systemd-provided facilities are too primitive
>to replace them completely.

Not surprising. Shell script is extremely powerful and trying to replace it
with some config file Lennart pulled out of his arse is unlikely to improve
things.

Still, he's not the only guilty party - the launchd system on MacOS has a
complicated config with its xml based plist files yet its functionality is
quite primitive at the same time.

[William Ahern]

I do indeed use OpenBSD for the stuff that matters most =) But I figured the
context here was mainline Linux distributions using systemd (e.g. Debian,
Fedora), not alternative Linux distributions (e.g. Alpine) or different Unix
operating systems altogether.

[John McCue]

Rainer Weikusat <rwei...@talktalk.net> wrote:
> John McCue <jmc...@fuzzball.mhome.org> writes:
>> Rainer Weikusat <rwei...@talktalk.net> wrote:
<snip>
>

> That's part of a >7000 LOC set of programs for enabling cloud-based
> management of network threat detection appliances (and obviously, all
> proprietary stuff).

OK, thanks

Maybe I can come up with such a test on my Work RHEL Test
System and my home Laptop to see if big difference.
Hardware is similar.

[Andrea Biscuola]

On Wed, 02 Feb 2022 15:45:13 +0000

And then, to try and replicate what a shell script does, you look at the
Unit file man pages to see that there are _hundreads_ of options and knobs
available.


Andrea

[Mut...@dastardlyhq.com]

A few times in Linux's history a large subsystem has been thrown out because
it became too unwieldy. dbus comes to mind as the most recent. One can hope
that eventually enough people will get so sick of systemd that Red Hat
supported or not, it'll get removed from most distros.

[Spiros Bousbouras]

On Thu, 3 Feb 2022 16:46:27 -0000 (UTC)
Mut...@dastardlyhq.com wrote:
> A few times in Linux's history a large subsystem has been thrown out because
> it became too unwieldy. dbus comes to mind as the most recent. One can hope
> that eventually enough people will get so sick of systemd that Red Hat
> supported or not, it'll get removed from most distros.

In what way has dbus been thrown out ? en.wikipedia.org/wiki/D-Bus does
not say anything.

[John Tsiombikas]

On 2022-02-02, William Ahern <wil...@25thandClement.com> wrote:
>
> I do indeed use OpenBSD for the stuff that matters most =) But I figured the
> context here was mainline Linux distributions using systemd (e.g. Debian,
> Fedora), not alternative Linux distributions (e.g. Alpine) or different Unix
> operating systems altogether.

Debian, and I would guess most GNU/Linux distributions, also distribute
good-old SysV init and can be configured to use that. So it's really on
you, if you decide to stick with systemd. Removing systemd is the first
thing I do on any new GNU/Linux installation.

--
John Tsiombikas
http://nuclear.mutantstargoat.com/

[Mut...@dastardlyhq.com]

On 26 May 2022 21:36:35 GMT

Just install Slackware, it uses SysV by default.

[Kaz Kylheku]

Speaking of which, I recently installed Slackware 15 into a VM.

Very refreshing experience.

But, I must say, Slackware is quite harmed by the insane piece of
vandalism that some dastardly knaves perpetrated against the kernel
source sometime in the past couple of years. I'm talking about the
vicious removal of the Shift-PgUp scrollback history available in the
console.

When you are doing a text-only install using only the console,
in a distro such as Slackware, it's very annoying not to be able to
scroll back. I needed to see something that scrolled off the screen
while doing partitioning with fdisk. This was so vexing, I did the
unthinkable and finished the job with cfdisk.

Burn in hell, scrollback-removing cretins!

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

[Kenny McCormack]

In article <202205272...@kylheku.com>,
Kaz Kylheku <480-99...@kylheku.com> wrote:
...

>Speaking of which, I recently installed Slackware 15 into a VM.
>
>Very refreshing experience.
>
>But, I must say, Slackware is quite harmed by the insane piece of
>vandalism that some dastardly knaves perpetrated against the kernel
>source sometime in the past couple of years. I'm talking about the
>vicious removal of the Shift-PgUp scrollback history available in the
>console.
>
>When you are doing a text-only install using only the console,
>in a distro such as Slackware, it's very annoying not to be able to
>scroll back. I needed to see something that scrolled off the screen
>while doing partitioning with fdisk. This was so vexing, I did the
>unthinkable and finished the job with cfdisk.
>
>Burn in hell, scrollback-removing cretins!

+1 (!)

Two comments:

1) I heard about this on some Raspberry Pi forum. Apparently, what
happened was that somebody discovered a bug (presumably, a non-trivial one)
in the code for this, but there was no one around (i.e., no one left) who
could fix it. So, they made the executive decision to just remove it - on
the assumption that (almost) nobody would miss it. Essentially, the code
had become unmaintainable.

2) I don't run any recent enough kernels to be affected by this, so I have
no way to test it. But I'm wondering something. Do you know if they
removed the underlying buffer (i.e., the fact that there is more stored
than actually fits [i.e., is visible] on the screen) or just the ability to
access it via certain keystrokes? I ask, because I have some scripts that
access the console buffer via /dev/vcsN - and these scripts are able to get
the part of the buffer that is no longer visible onscreen. I'm wondering
if that still works...

--
The motto of the GOP "base": You can't *be* a billionaire, but at least you
can vote like one.

[Kaz Kylheku]

On 2022-05-28, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> In article <202205272...@kylheku.com>,
> Kaz Kylheku <480-99...@kylheku.com> wrote:
> ...
>>Speaking of which, I recently installed Slackware 15 into a VM.
>>
>>Very refreshing experience.
>>
>>But, I must say, Slackware is quite harmed by the insane piece of
>>vandalism that some dastardly knaves perpetrated against the kernel
>>source sometime in the past couple of years. I'm talking about the
>>vicious removal of the Shift-PgUp scrollback history available in the
>>console.
>>
>>When you are doing a text-only install using only the console,
>>in a distro such as Slackware, it's very annoying not to be able to
>>scroll back. I needed to see something that scrolled off the screen
>>while doing partitioning with fdisk. This was so vexing, I did the
>>unthinkable and finished the job with cfdisk.
>>
>>Burn in hell, scrollback-removing cretins!
>
> +1 (!)
>
> Two comments:
>
> 1) I heard about this on some Raspberry Pi forum. Apparently, what
> happened was that somebody discovered a bug (presumably, a non-trivial one)
> in the code for this, but there was no one around (i.e., no one left) who
> could fix it.

This is the part that is idiotic, because unless those bugs are so
severe that they make possible a *remote* exploit, they should
just stay.

I enjoyed "years of trouble-free operation" with the console, and now
some fuckfaces who can't deal with a few corner cases think it's fine to
deal a blow to the user-friendliness of Linux?

Make it a command line option, you know?

linux root=whatever ... console_scroll=1 ,,,

Then those who are paranoid about the bug can just leave out the option.

[Scott Lurndal]

The source is available for download. Feel free to fix it and
submit the patch to the kernel mailing list.

[Richard Kettlewell]

sc...@slp53.sl.home (Scott Lurndal) writes:
> The source is available for download. Feel free to fix it and
> submit the patch to the kernel mailing list.

The vulnerability was already fixed. The requirement for restoring
functionality seems to be for someone to volunteer to maintain the code
in the long term.

https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=973c096f6a85e5b5f2a295126ba6928d9a6afd45
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=ebfdfeeae8c01fcb2b3b74ffaf03876e20835d2d

--
https://www.greenend.org.uk/rjk/

[Mut...@dastardlyhq.com]

Don't worry , Poettering will probably come along soon and add the functionality
to systemd. It'll require an incomprehensible setup in some undocumented
file and won't work properly but its the thought that counts.

[Kaz Kylheku]

On 2022-05-29, Mut...@dastardlyhq.com <Mut...@dastardlyhq.com> wrote:
> Don't worry , Poettering will probably come along soon and add the functionality
> to systemd. It'll require an incomprehensible setup in some undocumented
> file and won't work properly but its the thought that counts.

True systemd story here.

Almost something like a decade ago, while working at Broadcom, I
developed a nice kernel feature in the TTY subsystem.

1. Firstly, I ensured that when you unplug a a USB Serial adapter from
a port and then re-insert it, it would get the same /dev/ttyUSB<N>
device entry.

2. Secondly, I hacked the USB Serial and TTY code such that not only
would you get the same device, but your TTY session would be intact.
As a test case, you could log in to the board over the USB Serial
console, start editing a file in vi, and then unplug the USB Serial
dongle. Plug it back in, hit Ctrl-L to refresh the screen and, voila,
keep editing. You're not logged out, your edit session didn't die,
nothing.

3. I also hacked it so that you could boot the board with the USB-Serial
in an unplugged state, and get a console upon plugging it in.

4. The hot-plugging-with-session intact worked even if you swapped
brands. E.g, unplug a serial adapter based on a FTDI chip, and plug
in one based on a PL2302: no problem. Though a different driver is
now handling the I/O at the low level, your session is intact.

Now when I was debugging this into working, I kept getting a mysterious
issue: for some unknown reason, the session was dying when I unplugged
the USB-Serial adapter. I was sure the code was correct. There was
no reason that should be happening. I spent a day trying to root cause
this problem. Through some kernel printk's, I traced the behavior to
some system calls issued by ... that fucking systemd!

systemd implemented a behavior whereby if it detects that the USB Serial
device is removed, it turfs the surrounding TTY session. I found this
behavior in the source code; it was not documented anywhere. I grepped
all available docs backward and forward, to no avail.

Moreover, no such behavior should be implemented. If a TTY session is
going to be turfed due to removed hardware, it should be the operating
system doing it. Systemd thinks it's the operating system. That's
what its name means.

So all my code was fine; the biggest issue in debugging the feature was
what undocumented blurb to stick into some systemd file, after wasting
time uncovering that cause.

Send that Le Nerd Puttering bastard to hell, and throw his third-rate C
coding skills after him.

[Mut...@dastardlyhq.com]

Indeed. I've read numerous stories about systemd fucking things up. The
mystery isn't why it does it - Poetterings lack of decent coding ability is
only matched by his arrogance - the true mystery is why any distro never mind
almost all of them is using this 2nd rate POS. The only reason I've ever seen
that made sense was it booted faster than SysV. Which it does for a given
amount of functionality. So fucking what? Who cares about getting a prompt
30 secs faster if every other admin task is a bloody nightmare especially in
a server system which might only be rebooted once a year anyway!

After the abortion that is pulseaudio why did anyone let this guy near a
keyboard again never mind allow his pet init project to take over linux?
Its just the insanity of the herd mentality squared IMO.

[Richard Kettlewell]

Mut...@dastardlyhq.com writes:
> Indeed. I've read numerous stories about systemd fucking things
> up.

People write angry posts about things breaking, they don’t write them
about things that work. If you only look at the negatives then you don’t
have any basis for thinking that that the positives don’t outweigh them.

> The mystery isn't why it does it - Poetterings lack of decent
> coding ability is only matched by his arrogance - the true mystery is
> why any distro never mind almost all of them is using this 2nd rate
> POS.

In general if a large group of people are making technical decisions
that you don’t understand then they are probably working from different
information to you; either they know or believe something you don’t, or
the other way around.

In this case, the extra information Linux distributions have that you
don’t is the collection of practical issues that they and their users
have run into with sysvinit: things that didn’t work reliably or things
that were impractical to achieve with it.

Because of those issues something had to change (and in many other Unix
platforms, it already had). systemd wasn’t the only horse in the race;
it just happens to be the one that won.

--
https://www.greenend.org.uk/rjk/

[Spiros Bousbouras]

On Sat, 28 May 2022 05:38:39 -0000 (UTC)
Kaz Kylheku <480-99...@kylheku.com> wrote:
> Speaking of which, I recently installed Slackware 15 into a VM.
>
> Very refreshing experience.
>
> But, I must say, Slackware is quite harmed by the insane piece of
> vandalism that some dastardly knaves perpetrated against the kernel
> source sometime in the past couple of years. I'm talking about the
> vicious removal of the Shift-PgUp scrollback history available in the
> console.
>
> When you are doing a text-only install using only the console,
> in a distro such as Slackware, it's very annoying not to be able to
> scroll back. I needed to see something that scrolled off the screen
> while doing partitioning with fdisk. This was so vexing, I did the
> unthinkable and finished the job with cfdisk.
>
> Burn in hell, scrollback-removing cretins!

I wasn't familiar with this functionality. Does it offer any advantages over
using GNU screen or script ? The latter is not interactive but will record
the information you want. With the kernel thing , is it per user configurable
how much scrollback to store ? If not , it seems to me it's best to use
something which is per user.

--
vlaho.ninja/prog

[mut...@dastardlyhq.com]

On Sun, 29 May 2022 10:06:27 +0100
Richard Kettlewell <inv...@invalid.invalid> wrote:
>Mut...@dastardlyhq.com writes:
>> Indeed. I've read numerous stories about systemd fucking things
>> up.
>
>People write angry posts about things breaking, they don’t write them
>about things that work. If you only look at the negatives then you don’t
>have any basis for thinking that that the positives don’t outweigh them.

I've seen enough negatives to know that systemd design is fundamentally
broken.

>In this case, the extra information Linux distributions have that you
>don’t is the collection of practical issues that they and their users
>have run into with sysvinit: things that didn’t work reliably or things
>that were impractical to achieve with it.

Feel free to list them. I haven't noticed the BSDs pounding at Poetterings
door to do a port for them.

[Kenny McCormack]

In article <t7034i$1eb0$1...@gioia.aioe.org>, <mut...@dastardlyhq.com> wrote:
...

>I've seen enough negatives to know that systemd design is fundamentally
>broken.

...

>Feel free to list them. I haven't noticed the BSDs pounding at Poetterings
>door to do a port for them.

I'm no fan of systemd, so don't assume that I am. I'd just as soon it had
not happened, but it has, and I have to live with it. It's the nature of
computers that when you run out of things that are broken, you start fixing
things that are not broken. MS (e.g.) does this frequently.

That said, I say to you: Give it time.

As time goes by, two things will happen with systemd:

1) The bugs will, slowly, be filtered out.

2) It will become a community project, rather than a one-man-band, as
it currently is.

The point is that, in, say, 30 years, it will be as accepted and normalized
as the old init system is/was. And people will be clamoring for something
new to replace it.

P.S. I do agree, though, that if it is an init system, it would stay an
init system. The grand design of its being an extension (and by
"extension to", I mean "an integral part of") to the kernel/OS is
fundamentally wrong. That it should be doing what Kaz describes above is
fundamentally wrong. I expect these parts also to be filtered out as time
goes by.

--
People who say they'll vote for someone else because Obama couldn't fix
*all* of Bush's messes are like people complaining that he couldn't cure
cancer, so they'll go and vote for (more) cancer.

[mut...@dastardlyhq.com]

On Mon, 30 May 2022 11:38:03 -0000 (UTC)
gaz...@shell.xmission.com (Kenny McCormack) wrote:
>In article <t7034i$1eb0$1...@gioia.aioe.org>, <mut...@dastardlyhq.com> wrote:

>....

>>I've seen enough negatives to know that systemd design is fundamentally
>>broken.

>....

>>Feel free to list them. I haven't noticed the BSDs pounding at Poetterings
>>door to do a port for them.
>
>I'm no fan of systemd, so don't assume that I am. I'd just as soon it had
>not happened, but it has, and I have to live with it. It's the nature of
>computers that when you run out of things that are broken, you start fixing
>things that are not broken. MS (e.g.) does this frequently.

True.

>That said, I say to you: Give it time.
>
>As time goes by, two things will happen with systemd:
>
> 1) The bugs will, slowly, be filtered out.

No one is complaining about bugs, thats the nature of software. Its the
designed in fuckups that are the issue.

> 2) It will become a community project, rather than a one-man-band, as
> it currently is.

Out of his cold dead hands IMO.

>fundamentally wrong. That it should be doing what Kaz describes above is
>fundamentally wrong. I expect these parts also to be filtered out as time
>goes by.

Pffft. When did a codebase ever get smaller with less functionality?

[Kaz Kylheku]

On 2022-05-30, Kenny McCormack <gaz...@shell.xmission.com> wrote:
> In article <t7034i$1eb0$1...@gioia.aioe.org>, <mut...@dastardlyhq.com> wrote:
> ...
>>I've seen enough negatives to know that systemd design is fundamentally
>>broken.
> ...
>>Feel free to list them. I haven't noticed the BSDs pounding at Poetterings
>>door to do a port for them.
>
> I'm no fan of systemd, so don't assume that I am. I'd just as soon it had
> not happened,

Let's make it clear; systemd (or something like it) is necessary and
that is why distros bundle it and use it.

The problem is, systemd is just yet another instance in the continuing
Unix history of settling on the first, maybe second thing that comes
along that solves a problem, just because some random goofball hit upon
the idea and hacked something up and promoted it in the right place at
the right time. And then you're stuck with it for decades to come.

The main problem with systemd is that it creates a big impetus against
replacing it. And that would be fine if systemd were something other
than it is.

(The answer isn't necessarily going back to some scattered shell
scripts to duct-tape everything together; that's the strawman
characterization of the systemd nay-sayers as a group, even though
a few do advocate exactly that.)

> The point is that, in, say, 30 years, it will be as accepted and normalized
> as the old init system is/was. And people will be clamoring for something
> new to replace it.

The GNU Guix project has already replaced it; it manages services with
something called GNU Shepherd. Shepherd runs as PID 1, and manages
service units and all that stuff.

Guix isn't currently very popular though.

[mut...@dastardlyhq.com]

On Mon, 30 May 2022 20:10:43 -0000 (UTC)
Kaz Kylheku <480-99...@kylheku.com> wrote:
>(The answer isn't necessarily going back to some scattered shell
>scripts to duct-tape everything together; that's the strawman
>characterization of the systemd nay-sayers as a group, even though
>a few do advocate exactly that.)

The thing people forget about shell scripts is that all shells are turing
complete languages and therefor FAR more flexible than some config scripts and
a load of flags which is the systemd and other init system methods. With
systemd you're effectively limited to the functionality that systemd provides
but with rc scripts you have complete flexibility over how you control the
system. old -ne bad.

[Kenny McCormack]

I think that the gist of the systemd mentality is that "flexibility" is not
necessarily a good thing. I.e., too much "flexibility" is a bad thing.

I.e., the problem with making people write shell scripts is that it:
1) Is more work than is needed in 97.1237% of the cases.
and
2) Allows people to screw up by making things more complicated and thus
hard-to-maintain than they should be doing.

And, when all is said and done, I do agree with point 1 above. That for
most cases, being able to do it as a "config file" rather than as a
full-blown executable script, is a benefit.

Now I know, of course, that nobody ever actually writes one of these things
from scratch - you just take an existing one and modify it - but, still,
you have to be concerned with the fact that you are writing a script, not
just a "config file".

[Nicolas George]

mut...@dastardlyhq.com, dans le message <t74hcm$1fiu$1...@gioia.aioe.org>,
a écrit :

> The thing people forget about shell scripts is that all shells are turing
> complete languages and therefor FAR more flexible

The things you don't understand about shell scripts is that they can do
anything and they are a very fragile language. That combination unavoidably
leads to scripts that behave wildly and are a nightmare to control as soon
as the running environment is slightly different than expected.

[Kenny McCormack]

In article <6295e928$0$8502$426a...@news.free.fr>,

Nicolas George <nicolas$geo...@salle-s.org> wrote:
>mut...@dastardlyhq.com, dans le message <t74hcm$1fiu$1...@gioia.aioe.org>,

> a crit:

>> The thing people forget about shell scripts is that all shells are turing

>> complete languages and therefor FAR more flexible.

>
>The things you don't understand about shell scripts is that they can do
>anything and they are a very fragile language. That combination unavoidably
>leads to scripts that behave wildly and are a nightmare to control as soon
>as the running environment is slightly different than expected.

Yes. Well put.

The same general point as I was making.

It is a lot easier to prove that a config file is correct than it is to
prove a shell script correct.

--
I've learned that people will forget what you said, people will forget
what you did, but people will never forget how you made them feel.

- Maya Angelou -

[mut...@dastardlyhq.com]

On 31 May 2022 10:08:40 GMT

Rubbish. I've never had an rc script "behave wildly" in my entire career.
You sound like an apologist for Poettering.

[Rainer Weikusat]

Richard Kettlewell <inv...@invalid.invalid> writes:
> Mut...@dastardlyhq.com writes:

[...]

>> The mystery isn't why it does it - Poetterings lack of decent
>> coding ability is only matched by his arrogance - the true mystery is
>> why any distro never mind almost all of them is using this 2nd rate
>> POS.
>
> In general if a large group of people are making technical decisions
> that you don’t understand then they are probably working from different
> information to you; either they know or believe something you don’t, or
> the other way around.
>
> In this case, the extra information Linux distributions have that you
> don’t is the collection of practical issues that they and their users
> have run into with sysvinit: things that didn’t work reliably or things
> that were impractical to achieve with it.

Since RedHat had switched to sysvinit, Linux distribution had
accumulated a real mess of horrors in seriously bloated and buggy init
scripts whose authors seem to have been wedded to doing everything in
the wrong way. These didn't work reliably and some things just couldn't
be achieved in this way. One would have needed to write dedicated C
programs for them instead of the usual sysadmin coding horror tools
which try to implement every conceivable piece of process management
$sysadmin could presently think of in a single program with some 8000
different options. This was arguably a real problem, however, it wasn't
related to the program which caused these scripts to be executed.

Claiming otherwise was one of many sleights of hand used to market
systemd.

The sysvinit program (well worth of replacement due to it being old and
crufty and having an ill-defined feature set) does almost nothing for
system/ process management itself. It just initiates runlevel changes by
invoking a configurable, other program with certain arguments.

Obviously, the script mess has long since started to regrow in top of
systemd as the functionality of that is way to limited to handle a
wealth of real world problems.

[mut...@dastardlyhq.com]

On Tue, 31 May 2022 14:37:33 -0000 (UTC)
gaz...@shell.xmission.com (Kenny McCormack) wrote:
>In article <6295e928$0$8502$426a...@news.free.fr>,
>Nicolas George <nicolas$geo...@salle-s.org> wrote:
>>mut...@dastardlyhq.com, dans le message <t74hcm$1fiu$1...@gioia.aioe.org>,
>> a crit:
>>> The thing people forget about shell scripts is that all shells are turing
>>> complete languages and therefor FAR more flexible.
>>
>>The things you don't understand about shell scripts is that they can do
>>anything and they are a very fragile language. That combination unavoidably
>>leads to scripts that behave wildly and are a nightmare to control as soon
>>as the running environment is slightly different than expected.
>
>Yes. Well put.
>
>The same general point as I was making.
>
>It is a lot easier to prove that a config file is correct than it is to
>prove a shell script correct.

Great. Now prove the system that runs the config file is correct too because
that is the thing producing the same functionality as the shell script.

[mut...@dastardlyhq.com]

On Tue, 31 May 2022 09:01:36 -0000 (UTC)
gaz...@shell.xmission.com (Kenny McCormack) wrote:
>I think that the gist of the systemd mentality is that "flexibility" is not
>necessarily a good thing. I.e., too much "flexibility" is a bad thing.

I'd better restrict myself to a subset of C++ then next time I write an
application, don't want too much flexibility!

>Now I know, of course, that nobody ever actually writes one of these things
>from scratch - you just take an existing one and modify it - but, still,
>you have to be concerned with the fact that you are writing a script, not
>just a "config file".

All unix admins can write shell, particularly if they're managing the boot
system. That may not be the case for a-n-other config file syntax.

[Kaz Kylheku]

On 2022-05-31, mut...@dastardlyhq.com <mut...@dastardlyhq.com> wrote:
> Great. Now prove the system that runs the config file is correct too
> because that is the thing producing the same functionality as the
> shell script.

Why that's a piece of cake; we just refer to the thorough documentation
and exhaustive regression test suite which accompanies the mass of shell
scripts and defines what they actually do!

[Rainer Weikusat]

Kaz Kylheku <480-99...@kylheku.com> writes:
> On 2022-05-31, mut...@dastardlyhq.com <mut...@dastardlyhq.com> wrote:
>> Great. Now prove the system that runs the config file is correct too
>> because that is the thing producing the same functionality as the
>> shell script.
>
> Why that's a piece of cake; we just refer to the thorough documentation
> and exhaustive regression test suite which accompanies the mass of shell
> scripts and defines what they actually do!

This is a ghost discussion. The shell-script arrangement has absolutely
nothing to do with what program is initially started by the system. One
could easily have put a universal process manager for cooperating
processes, ie, the UNIX way and not the Poetterwindows-way, where it's
assumed that everybody else programming the system must be an enemy, on
top if it (not that I'd consider that to be a good design, but that's a
different question).

[mut...@dastardlyhq.com]

On Tue, 31 May 2022 20:36:53 -0000 (UTC)
Kaz Kylheku <480-99...@kylheku.com> wrote:
>On 2022-05-31, mut...@dastardlyhq.com <mut...@dastardlyhq.com> wrote:
>> Great. Now prove the system that runs the config file is correct too
>> because that is the thing producing the same functionality as the
>> shell script.
>
>Why that's a piece of cake; we just refer to the thorough documentation
>and exhaustive regression test suite which accompanies the mass of shell
>scripts and defines what they actually do!

You mean like what comes with systemd? rc shell scripts are by their
nature fairly simple and linear and can be grokked in minutes. Good luck
doing that with the systemd source code.

[Rainer Weikusat]

Ehh ... real-world distro init.d scripts are every-growing tumour-like
entities. Eg, the system I'm using to write this has 83 of them.

avg length 125.49 lines
median 84
min 8
max 1347

Length distribution:

0.000-99.000 47 (57.3%)
100.000-199.000 23 (28%)
200.000-299.000 8 (9.7%)
300.000-399.000 2 (2.4%)
400.000-499.000 1 (1.2%)
1300.000-1399.000 1 (1.2%)

In ancient times, when I had little else to do, I used to rewrite all of
this c**p just so that I didn't have to look at it. There's doubtlessly
a real problem here. It's just that system-smurfd is not a particularly good
solution for it.

[Kenny McCormack]

In article <202205311...@kylheku.com>,

Kaz Kylheku <480-99...@kylheku.com> wrote:
>On 2022-05-31, mut...@dastardlyhq.com <mut...@dastardlyhq.com> wrote:
>> Great. Now prove the system that runs the config file is correct too
>> because that is the thing producing the same functionality as the
>> shell script.
>
>Why that's a piece of cake; we just refer to the thorough documentation
>and exhaustive regression test suite which accompanies the mass of shell
>scripts and defines what they actually do!

I knew someone would post to that effect - that using systemd instead of
shell scripts just transfers the burden of proof from your shell scripts to
systemd itself. But the point is that:

1) It probably is correct, and, unless you really good (and most people
currently employed in IT aren't), it is more likely to be correct
than are your scripts.

2) If there is a bug, it's not your problem. It's that guy at Red
Hat's problem. You might even get a thrill from
finding/documenting/reporting the bug.

--
Donald Drumpf claims to be "the least racist person you'll ever meet".

This would be true if the only other person you've ever met was David Duke.

[Rainer Weikusat]

gaz...@shell.xmission.com (Kenny McCormack) writes:
> Kaz Kylheku <480-99...@kylheku.com> wrote:
>>On 2022-05-31, mut...@dastardlyhq.com <mut...@dastardlyhq.com> wrote:
>>> Great. Now prove the system that runs the config file is correct too
>>> because that is the thing producing the same functionality as the
>>> shell script.
>>
>>Why that's a piece of cake; we just refer to the thorough documentation
>>and exhaustive regression test suite which accompanies the mass of shell
>>scripts and defines what they actually do!
>
> I knew someone would post to that effect - that using systemd instead of
> shell scripts just transfers the burden of proof from your shell scripts to
> systemd itself. But the point is that:
>
> 1) It probably is correct, and, unless you really good (and most people
> currently employed in IT aren't), it is more likely to be correct
> than are your scripts.

The systemd-242 that's part of Debian 10 has 444,451 lines of
code. Anybody who seriously claims that a C program of this size is
"probably correct" and "more likely to be correct" than a much smaller
shell scripts is seriously deluded, not the least because the two
statements contradict each other.

The Debian build process applies no less than 63 patches with a total
length of 5477 lines to it. And that's certainly not done because it was
"probably correct" out of the box.

> 2) If there is a bug, it's not your problem. It's that guy at Red
> Hat's problem. You might even get a thrill from
> finding/documenting/reporting the bug.

As everybody knows, systemd has no bugs, just undocumented features some
of its users like less than its developers, IOW, somebody who gets hit
by a systemd bug most certainly has a problem and not exactly a small
one.

Of course, the justifcation for "I cannot possibly fix that!" is a lot
better, but that's not always an option.

[mut...@dastardlyhq.com]

On Wed, 01 Jun 2022 19:39:12 +0100
Rainer Weikusat <rwei...@talktalk.net> wrote:
>mut...@dastardlyhq.com writes:
>> On Tue, 31 May 2022 20:36:53 -0000 (UTC)
>> Kaz Kylheku <480-99...@kylheku.com> wrote:
>>>On 2022-05-31, mut...@dastardlyhq.com <mut...@dastardlyhq.com> wrote:
>>>> Great. Now prove the system that runs the config file is correct too
>>>> because that is the thing producing the same functionality as the
>>>> shell script.
>>>
>>>Why that's a piece of cake; we just refer to the thorough documentation
>>>and exhaustive regression test suite which accompanies the mass of shell
>>>scripts and defines what they actually do!
>>
>> You mean like what comes with systemd? rc shell scripts are by their
>> nature fairly simple and linear and can be grokked in minutes. Good luck
>> doing that with the systemd source code.
>
>Ehh ... real-world distro init.d scripts are every-growing tumour-like
>entities. Eg, the system I'm using to write this has 83 of them.

Why's that a bad thing? Would you prefer one huge script that did everything?
My /bin directory alone has 209 utils in it. Should I exchange them all for
BusyBox?

>avg length 125.49 lines
>median 84

Seems perfectly reasonable to me.

[mut...@dastardlyhq.com]

On Wed, 1 Jun 2022 18:50:02 -0000 (UTC)
gaz...@shell.xmission.com (Kenny McCormack) wrote:
>In article <202205311...@kylheku.com>,
>Kaz Kylheku <480-99...@kylheku.com> wrote:
>>On 2022-05-31, mut...@dastardlyhq.com <mut...@dastardlyhq.com> wrote:
>>> Great. Now prove the system that runs the config file is correct too
>>> because that is the thing producing the same functionality as the
>>> shell script.
>>
>>Why that's a piece of cake; we just refer to the thorough documentation
>>and exhaustive regression test suite which accompanies the mass of shell
>>scripts and defines what they actually do!
>
>I knew someone would post to that effect - that using systemd instead of
>shell scripts just transfers the burden of proof from your shell scripts to
>systemd itself. But the point is that:
>
> 1) It probably is correct, and, unless you really good (and most people
> currently employed in IT aren't), it is more likely to be correct
> than are your scripts.
>
> 2) If there is a bug, it's not your problem. It's that guy at Red
> Hat's problem. You might even get a thrill from
> finding/documenting/reporting the bug.

It most definately is your problem if you're system is fucked because of
it. With a shell script you edit it yourself, problem solved. With systemd
you'd have to wait for pottyring to get his arse into gear and even
acknowledge there is a bug first. Good luck with that.

[Rainer Weikusat]

I didn't claim this (the number of scripts) was a bad thing, I just
provided it as necessary part of the other statistics.

>>avg length 125.49 lines
>>median 84
>
> Seems perfectly reasonable to me.

That's because you've deleted everything which openly contradicts your
"simple and linear and can be grokked in minutes" claim.

[Rainer Weikusat]

If The Poetter (or one of his accomplices) would understand why XXX is a
problem, it wouldn't exist. Fortunately, nobody needs his help to fix
bugs in systemd. But that's nevertheless very likely more involved and
more time-consuming than fixing a bug in a much smaller shell script.

[Nicolas George]

Rainer Weikusat , dans le message
<87zgiwu...@doppelsaurus.mobileactivedefense.com>, a écrit :

> The systemd-242 that's part of Debian 10 has 444,451 lines of
> code. Anybody who seriously claims that a C program of this size is
> "probably correct" and "more likely to be correct" than a much smaller
> shell scripts is seriously deluded, not the least because the two
> statements contradict each other.

How many lines of C code in the shell interpreter and core utilities
necessary to run the shell scripts?

[William Ahern]

$ find coreutils-8.30/lib coreutils-8.30/src -name '*.[ch]' -exec cat {} + | wc -l
217752
$ find dash-0.5.10.2/src -name '*.[ch]' -exec cat {} + | wc -l
18963

The need for coreutils and the shell doesn't go away with systemd, though.
So it's not either/or; it's and.

[Nicolas George]

William Ahern , dans le message
<mgdlmi-...@wilbur.25thandClement.com>, a écrit :

> The need for coreutils and the shell doesn't go away with systemd, though.
> So it's not either/or; it's and.

Indeed. And there are other considerations: any case only uses a subset of
any framework. But things built of shell scripts will probably use a larger
subset of the shell scripting framework than things that just run commands.

My point is that counting the lines of C in systemd is just a stupid
argument.

[Scott Lurndal]

William Ahern <wil...@25thandClement.com> writes:
>Nicolas George <nicolas$geo...@salle-s.org> wrote:
>> Rainer Weikusat , dans le message
>> <87zgiwu...@doppelsaurus.mobileactivedefense.com>, a écrit :
>>> The systemd-242 that's part of Debian 10 has 444,451 lines of
>>> code. Anybody who seriously claims that a C program of this size is
>>> "probably correct" and "more likely to be correct" than a much smaller
>>> shell scripts is seriously deluded, not the least because the two
>>> statements contradict each other.
>>
>> How many lines of C code in the shell interpreter and core utilities
>> necessary to run the shell scripts?
>
>$ find coreutils-8.30/lib coreutils-8.30/src -name '*.[ch]' -exec cat {} + | wc -l
>217752

That's not lines of code. Look for the 'sloccount' utility.

[Rainer Weikusat]

225870 coreutils-8.30 ansic=191706,sh=24897,perl=7361,yacc=1843,python=47,
sed=16
21361 dash-0.5.10.2 ansic=20047,sh=1276,perl=38

(as per sloccount)

This means 20,047 lines of C code for the shell and another 191,706 for
all of the coreutils. This is less than half of (47.94%) of what systemd
needs and - that's also a factor - these are multiple programs the
largest of which (presumably the shell) has ony 4.81% of the size of the
systemd codebase (possibly also more than one program, but didn't
check).

But this doesn't really matter: Both (combined) codebases are large
enough that they're pretty much guaranteed to have bugs.

There's also another case of the usual system-smurf-d marketing
dishonesty in here: It's assumed that systemd must be 'better quality
code' because it's being programmed by professionals paid by RedHat
while start scripts would be amateurishly hacked together by 'mere'
sysadmins. This is obviously nonsense: Both codebases are/ were
developed by roughly the same set of professional system programmers,
or, worded in a more polemic way, the people who fucked up the init
scripts are now fucking up system startup and operation on a much larger
style.

What could possibly go wrong?

TBH: I'm meanwhile forced to support systemd professionally and while it
has a quirky, ill-designed interface on top of an equally quirky,
incomplete and ill-designed feature set, the only real problem I had
with it so far is that the default settings for init.d script based
units are completely unsuitable for the use cases I need to support.

OTOH, I had to fix exactly one bug in the shell in more than 25 years of
using it.

[Mut...@dastardlyhq.com]

On Wed, 01 Jun 2022 20:26:27 +0100

Are you suggesting you'd have trouble understanding a 100 line shell script?

[Rainer Weikusat]

Mut...@dastardlyhq.com writes:
> Rainer Weikusat <rwei...@talktalk.net> wrote:

[...]

>>>>Ehh ... real-world distro init.d scripts are every-growing tumour-like
>>>>entities. Eg, the system I'm using to write this has 83 of them.
>>>
>>> Why's that a bad thing? Would you prefer one huge script that did everything?
>>
>>> My /bin directory alone has 209 utils in it. Should I exchange them all for
>>> BusyBox?
>>
>>I didn't claim this (the number of scripts) was a bad thing, I just
>>provided it as necessary part of the other statistics.
>>
>>>>avg length 125.49 lines
>>>>median 84
>>>
>>> Seems perfectly reasonable to me.
>>
>>That's because you've deleted everything which openly contradicts your
>>"simple and linear and can be grokked in minutes" claim.
>
> Are you suggesting you'd have trouble understanding a 100 line shell
> script?

Let me put it this way: I'm not the person who recently posted that the
only way to handle a bug in systemd would be to complain to its author.
But that's completely besides the point:

You wrote that init.d scripts would be "simple and linear and can be
grokked in minutes" and this claim is wrong. Some are. And some others
are not.

[John Tsiombikas]

On 2022-05-27, Mut...@dastardlyhq.com <Mut...@dastardlyhq.com> wrote:
> On 26 May 2022 21:36:35 GMT John Tsiombikas <nuc...@member.fsf.org>
> wrote:
>>
>>Removing systemd is the first thing I do on any new GNU/Linux
>>installation.
>
> Just install Slackware, it uses SysV by default.

Slackware is a perfectly good system, and I have used it in the past,
but Debian is more convenient day to day, and as I said, the option to
use SysV init is there. It not being the default doesn't make much
difference. Default installation options are meaningless when you
install a system once and keep using it for the next 12 years :)
I think my current Debian installation dates from 2010.

--
John Tsiombikas
http://nuclear.mutantstargoat.com/

[John Tsiombikas]

On 2022-05-30, Kaz Kylheku <480-99...@kylheku.com> wrote:
>
> Let's make it clear; systemd (or something like it) is necessary and
> that is why distros bundle it and use it.

[citation needed]

> (The answer isn't necessarily going back to some scattered shell
> scripts to duct-tape everything together; that's the strawman
> characterization of the systemd nay-sayers as a group, even though
> a few do advocate exactly that.)

I'll have my scattered shell scripts thank you very much.

Look. The requirements of distributions do not necessarilly align with
the requirements of users. I'm sure random init scripts are tough for
distributions to automatically manage, but I as a user, don't care about
that. I care about how easy it is for me to manually understand and
manage the boot process. And for me a bunch of simple shell scripts
running in pre-determined order due to runlevel changes are
*significantly* easier to understand and maintain, and as others
mentioned, much more flexible to boot.

It's the same story with the proliferation of breaking a simple
configuration file into a .d subdirectory with a bilion random config
snippets. It's all to make it easier for automated tools to manage, but
it's definitely much harder for users to manage manually (unless of
course it happens to be really humongous, which it usually isn't...).

[Bit Twister]

First off, I am no fan of systemd. BUT it is here to stay so learn to work
with it.

run the following:
systemd-analyze plot > /var/tmp/blame.svg
firefox /var/tmp/blame.svg &

Makes it dead easy to see what is dependent on who and how long it takes
for something to complete initialization. There are lots of shell init scripts.
Use "systemctl status whatever.unit_name_here" to get what is executed to
see what actually is executed to start it.


The drop-in directory is a very nice method for users/sys admins to
make their configuration changes verses an app configuration file.

I just cut the configuration section from the app configuration, paste
into my file, change value(s), and place file or link in the app's
drop-in directory, and restart service.

Picking a naming convention makes it easy to find them.
$ locate xx__ | wc -l
57
Two example names:
/etc/dovecot/conf.d/xx__local_dovecot.conf
/etc/sysctl.d/xx__sysctl.conf
easy to find/see what app configuration I have changed.


System Change Management is pretty easy for me, I automate all my changes,
one per file, because I always do clean installs instead of upgrades.
$ ls /local/bin/*change* | wc -l
198

Same for having to do special app installation procedures.
$ ls /local/bin/*install* | wc -l
61

Also comes in handy if app upgrade creates a new config file and wipes
out your changes. Example sshd :(

[William Ahern]

John Tsiombikas <nuc...@member.fsf.org> wrote:
> On 2022-05-30, Kaz Kylheku <480-99...@kylheku.com> wrote:
>>
>> Let's make it clear; systemd (or something like it) is necessary and
>> that is why distros bundle it and use it.
>
> [citation needed]
>
>> (The answer isn't necessarily going back to some scattered shell
>> scripts to duct-tape everything together; that's the strawman
>> characterization of the systemd nay-sayers as a group, even though
>> a few do advocate exactly that.)
>
> I'll have my scattered shell scripts thank you very much.

Shell scripts cannot fix bad process management in the daemons themselves.
So either process management in the daemons needs to be comprehensively
fixed and maintained (see OpenBSD), or it needs to be removed, in which case
you still need infrastructure such as daemontools or systemd to juggle
processes.

Process management can't always be removed, though, as some daemons need to
manage subprocesses for various legitimate reasons. And that's where we end
up with irreducible complexity. Process management isn't *that* difficult,
but manifestly people on average suck at it. Herculean efforts at mitigating
and papering over those issues is where most of the complexity of systemd's
init system arises.

For my tastes, I prefer fixing the actual software. But in the context of
the broader Linux ecosystem, that approach alone isn't reasonable. (It's
more reasonable on OpenBSD, where the most commonly used daemons are
maintained as part of the system, and the general user and developer base is
a self-selective bunch.) I'm not a systemd fan, but in some ways systemd
seems almost inevitable.

If I had more time and motivation I might try to implement a different
compromise: introducing more sophisticated process management primitives
directly into the shell. Job management is, afterall, a core competency of
the shell, and despite it's innumerable problems it still excels relative to
other languages, at least for the simple stuff. I suspect it might make
sense to implement a POSIX-compliant shell from scratch, probably in a
higher-level language like Lua or Go, making it easier and faster to
implement and integrate more complex builtins (commands, operators,
semantics) for process management--e.g. process descriptors, asynchronous
in-process events beyond simple signals, etc. But, realistically, extending
bash or dash might be the better approach to maximize chances of widespread
adoption.

Success might still be contingent on fixing and complementing existing
kernel facilities, especially if you want to achieve correctness without
centralizing everything under, e.g., PID 1 init. (For example, process
descriptor semantics and interfaces might not be improved.) Even systemd
still falls short of semantic correctness as it's still technically
susceptible to some of the same races as traditional PID files. When I've
brought this up on LWN.net, people have pointed out that correctness might
be achievable by leveraging more recent Linux kernel facilities, but systemd
doesn't yet use them and might not ever be able to use them as-is.
Ultimately, I doubt a more holistic approach can be avoided. Confining a
solution to ad hoc shell scripts preserves the same fundamental error as
systemd preserved, which also tries to solve everything without fixing code
elsewhere. (I know systemd developers do sometimes propose and submit kernel
patches, but approaches and outcomes still seem lacking.)

[William Ahern]

Dead easy may be a stretch. It's extremely useful, but there are caveats.

1) Service restarts aren't shown in those plots; only the most recent run.
(And possibly failures, if they're not persistent.) On several occassions
this has hidden or obscured problems.

2) You cannot reliably detect dependency conflicts. When working on the
Packer build for a baseline OS image at a previous job I tried to
comprehensively write ordering assertions, Depends checks, etc, in the
provisioning scripts. But they proved unreliable, and issues still often
made it through the CI/CD pipeline. systemd can sometimes detect and report
on these issues, but not always, and it lacks some necessary functionality
for enforcing strict correctness or comprehensively analyzing correctness.

> The drop-in directory is a very nice method for users/sys admins to
> make their configuration changes verses an app configuration file.

Yes, it's nice. But some directives are additive (e.g. Depends) while others
override prior directives, and there's no way to change the behavior. If you
want to replace dependency sets, you need to remove the original unit
file(s). Such limitations are especially annoying (and confusing) for
SysV-compat generated units. (I've accidentally yet silently--see
above--introduced DAG conflicts this way.)

Unit files are intended to be declarative, but they're based on ini-style
key=value pairs, which is too simplistic, not very descriptive, and
difficult to manually analyze. That makes the fragments system *deceptively*
simple. It works well for simple stuff, but handling or just debugging
complex issues is maddening. OpenBSD daemons also emphasize declarative
configs, but they use proper grammars (lex+yacc) which make it easier to
express both simple and complex configurations as well as tend to make
limitations and pitfalls more obvious.

Many problems would be easier to address if there were a generic facility
for conditional directives. But then it'd need expressions, a library of
functions (or shell substitutions), etc. At which point you quickly come
dangerously close to a full blown language. In theory it could end up
looking like Prolog or a Makefile, but it'd probably look more like a
generic procedural language (C or shell) as I don't think the underlying
systemd code is amenable to a strict yet generic declarative system. In any
event, they seem wedded to the horrendous key=value approach.

[Bit Twister]

Yep.

> (And possibly failures, if they're not persistent.) On several occassions
> this has hidden or obscured problems.

That is what journalctl is for. I created a systemd-journal and added to
my user login. That allows me to read the system journal without having
to be root or require root privileges.

I then have a cron hourly job to tell me about system problems.
My script uses xmessage to pop up list of problems found in the journal
since the last time it looked.

> 2) You cannot reliably detect dependency conflicts. When working on the
> Packer build for a baseline OS image at a previous job I tried to
> comprehensively write ordering assertions, Depends checks, etc, in the
> provisioning scripts. But they proved unreliable, and issues still often
> made it through the CI/CD pipeline. systemd can sometimes detect and report
> on these issues, but not always, and it lacks some necessary functionality
> for enforcing strict correctness or comprehensively analyzing correctness.

Cannot comment on the above. I have not had to work at that level.

>> The drop-in directory is a very nice method for users/sys admins to
>> make their configuration changes verses an app configuration file.
>
> Yes, it's nice. But some directives are additive (e.g. Depends) while others
> override prior directives, and there's no way to change the behavior. If you
> want to replace dependency sets, you need to remove the original unit
> file(s).

Hmmmm, I thought all I had to do was put new dependency in my drop-in file.
Have not looked into having to remove a factory unit dependency since I want
to be able to just delete my drop-in file and verify the problem does
not exist in the install setup.

[Mut...@dastardlyhq.com]

On Thu, 02 Jun 2022 18:51:32 +0100
Rainer Weikusat <rwei...@talktalk.net> wrote:
>Mut...@dastardlyhq.com writes:
>> Rainer Weikusat <rwei...@talktalk.net> wrote:
>
>[...]
>
>>>>>Ehh ... real-world distro init.d scripts are every-growing tumour-like
>>>>>entities. Eg, the system I'm using to write this has 83 of them.
>>>>
>>>> Why's that a bad thing? Would you prefer one huge script that did
>everything?
>>>
>>>> My /bin directory alone has 209 utils in it. Should I exchange them all for
>
>>>> BusyBox?
>>>
>>>I didn't claim this (the number of scripts) was a bad thing, I just
>>>provided it as necessary part of the other statistics.
>>>
>>>>>avg length 125.49 lines
>>>>>median 84
>>>>
>>>> Seems perfectly reasonable to me.
>>>
>>>That's because you've deleted everything which openly contradicts your
>>>"simple and linear and can be grokked in minutes" claim.
>>
>> Are you suggesting you'd have trouble understanding a 100 line shell
>> script?
>
>Let me put it this way: I'm not the person who recently posted that the
>only way to handle a bug in systemd would be to complain to its author.

Turn of phrase. Systemd bugs get fixed slowly if at all on the main tree.

>But that's completely besides the point:
>
>You wrote that init.d scripts would be "simple and linear and can be
>grokked in minutes" and this claim is wrong. Some are. And some others
>are not.

The vast majority are and anyone with even a passing knowledge of shell script
can understand them. Most admins still struggle with systemd and only know
a few sections of it well.

[Mut...@dastardlyhq.com]

On 02 Jun 2022 20:21:25 GMT

John Tsiombikas <nuc...@member.fsf.org> wrote:
>On 2022-05-27, Mut...@dastardlyhq.com <Mut...@dastardlyhq.com> wrote:
>> On 26 May 2022 21:36:35 GMT John Tsiombikas <nuc...@member.fsf.org>
>> wrote:
>>>
>>>Removing systemd is the first thing I do on any new GNU/Linux
>>>installation.
>>
>> Just install Slackware, it uses SysV by default.
>
>Slackware is a perfectly good system, and I have used it in the past,
>but Debian is more convenient day to day, and as I said, the option to
>use SysV init is there. It not being the default doesn't make much
>difference. Default installation options are meaningless when you
>install a system once and keep using it for the next 12 years :)
>I think my current Debian installation dates from 2010.

Depends what you want to do on install. I add, disable and change a number of
things in the boot system when I install Slackware. Usually takes about 5-10
mins. God knows how long it would take in systemd or even if some of it would
be possible as I add my own functionality to a couple of the scripts.

[Rainer Weikusat]

Mut...@dastardlyhq.com writes:
> On Thu, 02 Jun 2022 18:51:32 +0100
> Rainer Weikusat <rwei...@talktalk.net> wrote:

[...]

>>Let me put it this way: I'm not the person who recently posted that the
>>only way to handle a bug in systemd would be to complain to its author.
>
> Turn of phrase. Systemd bugs get fixed slowly if at all on the main
> tree.

Well, so what? If it works for me, I'm good. Interactions with the
so-called OSS community are cumbersome and time-consuming at
best. Often, they're also fruitless and seriously unpleasant. I remember
one OSS conslutant who had come up with a written design re: how to fix
a certain problem which also affected me. Hence, I implemented his
design after consulting him about that. This convinced him that his idea for
fixing the issue at hand was really completely misguided and that, for
now, the project would be better off with keeping the bug until he had
come up with a new and even more uselessly complicated scheme for
eventually fixing it.

>>You wrote that init.d scripts would be "simple and linear and can be
>>grokked in minutes" and this claim is wrong. Some are. And some others
>>are not.
>
> The vast majority are and anyone with even a passing knowledge of shell script
> can understand them. Most admins still struggle with systemd and only know
> a few sections of it well.

About 60% are. Whether or not that's a vast majority is open to anyone's
opinion. Someone who has only a passing knowledge of some programming
language will be hard pressed to understand anything but the most
trivial code written in it. In a Suse init.d script, I once encountered a
comment complaining about the need to work around a bug in another Suse
init.d script. Fixing the bug instead apparently never crossed the mind
of the author of the comment.

It's perfectly possible to keep init.d scripts simple and thus, easily
modifiable. But that's something Linux distributors were never good at.

[Mut...@dastardlyhq.com]

On Fri, 03 Jun 2022 15:26:44 +0100
Rainer Weikusat <rwei...@talktalk.net> wrote:

>Mut...@dastardlyhq.com writes:
>> The vast majority are and anyone with even a passing knowledge of shell
>script
>> can understand them. Most admins still struggle with systemd and only know
>> a few sections of it well.
>
>About 60% are. Whether or not that's a vast majority is open to anyone's
>opinion. Someone who has only a passing knowledge of some programming
>language will be hard pressed to understand anything but the most

Someone who doesn't understand or program in shell script has no business
administering a unix box. They should go and be a click monkey admin in the
windows world.

>trivial code written in it. In a Suse init.d script, I once encountered a
>comment complaining about the need to work around a bug in another Suse
>init.d script. Fixing the bug instead apparently never crossed the mind
>of the author of the comment.

I'd have fixed it myself. Couldn't do that with a systemd bug.

>It's perfectly possible to keep init.d scripts simple and thus, easily
>modifiable. But that's something Linux distributors were never good at.

I'll give you that. But they've made even more of a hash of systemd.

[Rainer Weikusat]

I'm not using it privately and professionally, this is more a case of
"learn how to work around it". At the moment, this looks stereotypically
like this:

[Service]
KillMode=none
NotifyAccess=all
RemainAfterExit=no

which is autogenerated (for packages needing it) by a script I've chosen
to name poetter-em-all plus a 60 line C program which transparently
invokes the systemd start command when the start action of an init
script is executed manually. As the systemd feature set isn't capable of
supporting all of the features I need and I don't care about all of the
misfeatures I don't need, I just need it to stay out of may way.

We can read documentation, BTW. And even code.

[Rainer Weikusat]

William Ahern <wil...@25thandClement.com> writes:
> John Tsiombikas <nuc...@member.fsf.org> wrote:
>> On 2022-05-30, Kaz Kylheku <480-99...@kylheku.com> wrote:
>>>
>>> Let's make it clear; systemd (or something like it) is necessary and
>>> that is why distros bundle it and use it.
>>
>> [citation needed]
>>
>>> (The answer isn't necessarily going back to some scattered shell
>>> scripts to duct-tape everything together; that's the strawman
>>> characterization of the systemd nay-sayers as a group, even though
>>> a few do advocate exactly that.)
>>
>> I'll have my scattered shell scripts thank you very much.
>
> Shell scripts cannot fix bad process management in the daemons themselves.
> So either process management in the daemons needs to be comprehensively
> fixed and maintained (see OpenBSD), or it needs to be removed, in which case
> you still need infrastructure such as daemontools or systemd to juggle
> processes.

[...]

> If I had more time and motivation I might try to implement a different
> compromise: introducing more sophisticated process management primitives
> directly into the shell. Job management is, afterall, a core competency of
> the shell, and despite it's innumerable problems it still excels relative to
> other languages, at least for the simple stuff.

Around the time when systemd was still being debated (ie, before its
proponents forced it into all major distributions via technopolitical
coups d'etat) an old project I had been working on was finally
discontinued and I started working on a new one. The old had a dedicated
init I wrote myself. For the new project, I said to myself "Ok, I'm not
going to do that again. sysvinit is old and crufty but I'm perfectly
happy with making it do whatever I need it to do. As an experiment,
whenever I run into another process management problem, I'll write a
small C program specifically for solving it and see how far this will
carry me."

This was 13 years ago. Meanwhile, there are 37 of these tools with a
combined size of 5355 lines of code and I haven't run into something
which couldn't be solved in this way so far.

Morale: Their are two kinds of programmers, those who create immensly
complicated programs to solve simple problems and those who strive to
create simple programs to solve complicated problems. The members of the
first group are much more admired than the members of the second.

[...]

> Success might still be contingent on fixing and complementing existing
> kernel facilities, especially if you want to achieve correctness without
> centralizing everything under, e.g., PID 1 init. (For example, process
> descriptor semantics and interfaces might not be improved.) Even systemd
> still falls short of semantic correctness as it's still technically
> susceptible to some of the same races as traditional PID files.

When I learned that systemd still uses PID files, as was only halfway
surpised. Something like "Well, why have I ever believed they to would
discontinue this habit?" :-)

[John Tsiombikas]

On 2022-06-02, Bit Twister <BitTw...@mouse-potato.com> wrote:
>
> First off, I am no fan of systemd. BUT it is here to stay so learn to work
> with it.

The beauty of UNIX, and even more so of free software, is that I don't
have to. I can choose to never run systemd on my computers and that's
what I intend to do. No need to learn to work with shitty programs.

[Kenny McCormack]

In article <slrnt9kl4c....@goat.mutantstargoat.com>,

John Tsiombikas <nuc...@member.fsf.org> wrote:
>On 2022-06-02, Bit Twister <BitTw...@mouse-potato.com> wrote:
>>
>> First off, I am no fan of systemd. BUT it is here to stay so learn to work
>> with it.
>
>The beauty of UNIX, and even more so of free software, is that I don't
>have to. I can choose to never run systemd on my computers and that's
>what I intend to do. No need to learn to work with shitty programs.

Of course you can. And you do, as do I.

But that's not really what this thread is about.

--
"There's no chance that the iPhone is going to get any significant market share. No chance." - Steve Ballmer

[Mut...@dastardlyhq.com]

On Fri, 03 Jun 2022 17:34:33 +0100
Rainer Weikusat <rwei...@talktalk.net> wrote:
>complicated programs to solve simple problems and those who strive to
>create simple programs to solve complicated problems. The members of the
>first group are much more admired than the members of the second.

So very true sadly.

[Marc Haber]

Spiros Bousbouras <spi...@gmail.com> wrote:
>I wasn't familiar with this functionality. Does it offer any advantages over
>using GNU screen or script ?

It works even in early stages of boot when it's not yet possible to
log in. Used to be very helpful if the cause of the error was followed
by three pages of kenel oops and stack traces.

--
-------------------------------------- !! No courtesy copies, please !! -----
Marc Haber | " Questions are the | Mailadresse im Header
Mannheim, Germany | Beginning of Wisdom " |
Nordisch by Nature | Lt. Worf, TNG "Rightful Heir" | Fon: *49 621 72739834