Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
sshd package systemd misconfiguration?
201 views
Skip to first unread message
Michael
Sep 15, 2022, 6:30:05 AM9/15/22
to
hey,
i recently had problems to reach some of my host with ssh. as it turned
out, it was b/c sshd refused the connection due to a missing /run/sshd
directory.
the logfile entry:
Aug 28 00:10:08 mail sshd[151893]: fatal: Missing privilege separation
directory: /run/sshd
so i started digging, and to my surprise i found out that on the affected
servers sshd was configured to be invoked by ssh.socket (via ssh@.service
with the -i option), wheras on all other hosts sshd was running as a daemon
(via ssh.service whith the -D option).
so, my first question is: why?
all servers run debian 11 (bullseye), updated from debian 10 (buster), and
i cannot remember changing this, i.e. enabling ssh.socket. why would i?
now that i know the problem, i have essentially three choices (assuming not
to change the invocation via ssh.socket):
1: create /run/sshd whenever it disappears
2: prevent /run/sshd from being deleted
3: make each ssh@.service session use its own directory
1: that's what i started with to monitor what was wrong, and to be able to
use ssh but this is not a solution rather than a mitigation.
2: i added a drop-in at /etc/systemd/system/ssh@.service.d/ with
'RuntimeDirectoryPreserve=yes', and it works,
3: first, i also added a drop in at /etc/systemd/system/ssh@.servide.d/
with 'RuntimeDirectory=sshd.%i', but it just added the new directory to the
already defined 'sshd', resulting again in the deletion of /run/sshd. so i
copied /usr/lib/systemd/system/ssh@.service to /etc/systemd/system/ and
changed 'RuntimeDirectory=sshd' to 'RuntimeDirectory=sshd.%i', and it
works.
is it safe to say, that this issue is a misconfiguration? should the
package maintainer be notified? or did i miss something?
greetings...
i recently had problems to reach some of my host with ssh. as it turned
out, it was b/c sshd refused the connection due to a missing /run/sshd
directory.
the logfile entry:
Aug 28 00:10:08 mail sshd[151893]: fatal: Missing privilege separation
directory: /run/sshd
so i started digging, and to my surprise i found out that on the affected
servers sshd was configured to be invoked by ssh.socket (via ssh@.service
with the -i option), wheras on all other hosts sshd was running as a daemon
(via ssh.service whith the -D option).
so, my first question is: why?
all servers run debian 11 (bullseye), updated from debian 10 (buster), and
i cannot remember changing this, i.e. enabling ssh.socket. why would i?
now that i know the problem, i have essentially three choices (assuming not
to change the invocation via ssh.socket):
1: create /run/sshd whenever it disappears
2: prevent /run/sshd from being deleted
3: make each ssh@.service session use its own directory
1: that's what i started with to monitor what was wrong, and to be able to
use ssh but this is not a solution rather than a mitigation.
2: i added a drop-in at /etc/systemd/system/ssh@.service.d/ with
'RuntimeDirectoryPreserve=yes', and it works,
3: first, i also added a drop in at /etc/systemd/system/ssh@.servide.d/
with 'RuntimeDirectory=sshd.%i', but it just added the new directory to the
already defined 'sshd', resulting again in the deletion of /run/sshd. so i
copied /usr/lib/systemd/system/ssh@.service to /etc/systemd/system/ and
changed 'RuntimeDirectory=sshd' to 'RuntimeDirectory=sshd.%i', and it
works.
is it safe to say, that this issue is a misconfiguration? should the
package maintainer be notified? or did i miss something?
greetings...
Greg Wooledge
Sep 15, 2022, 7:20:06 AM9/15/22
to
On Thu, Sep 15, 2022 at 12:02:21PM +0200, Michael wrote:
> i recently had problems to reach some of my host with ssh. as it turned out,
> it was b/c sshd refused the connection due to a missing /run/sshd directory.
>
> the logfile entry:
> Aug 28 00:10:08 mail sshd[151893]: fatal: Missing privilege separation
> directory: /run/sshd
>
> so i started digging, and to my surprise i found out that on the affected
> servers sshd was configured to be invoked by ssh.socket (via ssh@.service
> with the -i option), wheras on all other hosts sshd was running as a daemon
> (via ssh.service whith the -D option).
>
> so, my first question is: why?
I'm afraid only you can answer that one.
> i recently had problems to reach some of my host with ssh. as it turned out,
> it was b/c sshd refused the connection due to a missing /run/sshd directory.
>
> the logfile entry:
> Aug 28 00:10:08 mail sshd[151893]: fatal: Missing privilege separation
> directory: /run/sshd
>
> so i started digging, and to my surprise i found out that on the affected
> servers sshd was configured to be invoked by ssh.socket (via ssh@.service
> with the -i option), wheras on all other hosts sshd was running as a daemon
> (via ssh.service whith the -D option).
>
> so, my first question is: why?
> now that i know the problem, i have essentially three choices (assuming not
> to change the invocation via ssh.socket):
> 1: create /run/sshd whenever it disappears
> 2: prevent /run/sshd from being deleted
> 3: make each ssh@.service session use its own directory
the unit file:
unicorn:/lib/systemd/system$ grep RuntimeDirectory ssh@.service
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755
unicorn:/lib/systemd/system$ grep RuntimeDirectory ssh.service
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755
That should cause the /run/sshd directory to be created when the service
is started, and removed when it's stopped.
It's possible that someone has created a local override file on your
system which suppresses this, or that someone has actually edited the
unit file in /lib. The latter should never happen, as a package upgrade
will replace the file and destroy the edits.
If the former has happened, you should see ssh.* files or directories
under /etc/systemd/system/. If you have anything there which isn't
a symlink named sshd.service pointing to /lib/systemd/system/ssh.service
please show it to us -- "ls -l" output on the file, plus the file's
contents.
If your /lib/systemd/system/ssh@.service doesn't contain the two lines
I showed, then you might start by reinstalling the package, or perhaps
by noting the timestamp on the file, which should tell you when it was
modified. Perhaps that will jog your memory about someone who might
have changed it. Or maybe the file is simply corrupted by file system
damage, which should be obvious in a text editor.
Michael
Sep 16, 2022, 3:40:06 AM9/16/22
to
On Thursday, 15 September 2022 13:01:45 CEST, Greg Wooledge wrote:
of course the first thing i did was to check if all the files from the
package were as they should be, and everything was fine!
> It's supposed to be created as needed. There should be two lines in
> the unit file:
>
> unicorn:/lib/systemd/system$ grep RuntimeDirectory ssh@.service
> RuntimeDirectory=sshd
> RuntimeDirectoryMode=0755
> unicorn:/lib/systemd/system$ grep RuntimeDirectory ssh.service
> RuntimeDirectory=sshd
> RuntimeDirectoryMode=0755
>
i never questioned that! my problem wasn't based on these lines are missing
or in some way altered. my problem resulted in these lines being there as
they are! ;)
> That should cause the /run/sshd directory to be created when the service
> is started, and removed when it's stopped.
you got it! :)
having 'RuntimeDirectory=sshd' in ssh.service is totally fine! that is, b/c
sshd runs as a daemon, and each new connection is handled by that exact
daemon. so the /run/sshd directory stays there no matter how many
connections to the host exist or being terminated, etc.
with ssh@.service it is completely different. for each connection there is
a dedicated sshd process being started, and each one of them has the same
/run/sshd directory assigned. and that's the problem if you have more than
one connection to a given host. as soon as the first connection is
terminated, the /run/sshd directory disappears, and the other sshd's might
run into problems.
but i still don't understand, why this scenario prevents me from contacting
a host with the formerly mentioned error message in the log file...
greetings...
of course the first thing i did was to check if all the files from the
package were as they should be, and everything was fine!
> It's supposed to be created as needed. There should be two lines in
> the unit file:
>
> unicorn:/lib/systemd/system$ grep RuntimeDirectory ssh@.service
> RuntimeDirectory=sshd
> RuntimeDirectoryMode=0755
> unicorn:/lib/systemd/system$ grep RuntimeDirectory ssh.service
> RuntimeDirectory=sshd
> RuntimeDirectoryMode=0755
>
or in some way altered. my problem resulted in these lines being there as
they are! ;)
> That should cause the /run/sshd directory to be created when the service
> is started, and removed when it's stopped.
having 'RuntimeDirectory=sshd' in ssh.service is totally fine! that is, b/c
sshd runs as a daemon, and each new connection is handled by that exact
daemon. so the /run/sshd directory stays there no matter how many
connections to the host exist or being terminated, etc.
with ssh@.service it is completely different. for each connection there is
a dedicated sshd process being started, and each one of them has the same
/run/sshd directory assigned. and that's the problem if you have more than
one connection to a given host. as soon as the first connection is
terminated, the /run/sshd directory disappears, and the other sshd's might
run into problems.
but i still don't understand, why this scenario prevents me from contacting
a host with the formerly mentioned error message in the log file...
greetings...
Jonathan Dowland
Sep 16, 2022, 4:50:05 AM9/16/22
to
I've been hit by this too. Likewise I haven't deliberately
configured sshd for socket activation nor tampered with
unit files. In my case the host was a newly imaged raspberry
pi using the images linked from the Debian Wiki. I haven't
done any further investigation.
--
Jonathan Dowland
https://jmtd.net
configured sshd for socket activation nor tampered with
unit files. In my case the host was a newly imaged raspberry
pi using the images linked from the Debian Wiki. I haven't
done any further investigation.
--
Jonathan Dowland
https://jmtd.net
Greg Wooledge
Sep 16, 2022, 7:50:06 AM9/16/22
to
On Fri, Sep 16, 2022 at 09:17:10AM +0200, Michael wrote:
> On Thursday, 15 September 2022 13:01:45 CEST, Greg Wooledge wrote:
> > unicorn:/lib/systemd/system$ grep RuntimeDirectory ssh@.service
> > RuntimeDirectory=sshd
> > RuntimeDirectoryMode=0755
> On Thursday, 15 September 2022 13:01:45 CEST, Greg Wooledge wrote:
> > unicorn:/lib/systemd/system$ grep RuntimeDirectory ssh@.service
> > RuntimeDirectory=sshd
> > RuntimeDirectoryMode=0755
> with ssh@.service it is completely different. for each connection there is a
> dedicated sshd process being started, and each one of them has the same
> /run/sshd directory assigned. and that's the problem if you have more than
> one connection to a given host. as soon as the first connection is
> terminated, the /run/sshd directory disappears, and the other sshd's might
> run into problems.
OK then. This is a bit beyond my experience level, and should probably
> dedicated sshd process being started, and each one of them has the same
> /run/sshd directory assigned. and that's the problem if you have more than
> one connection to a given host. as soon as the first connection is
> terminated, the /run/sshd directory disappears, and the other sshd's might
> run into problems.
be filed as a bug against Debian.
I did find this paragraph in systemd.exec(5):
Use RuntimeDirectory= to manage one or more runtime directories for
the unit and bind their lifetime to the daemon runtime. This is
particularly useful for unprivileged daemons that cannot create
runtime directories in /run/ due to lack of privileges, and to make
sure the runtime directory is cleaned up automatically after use.
For runtime directories that require more complex or different
configuration or lifetime guarantees, please consider using
tmpfiles.d(5).
Maybe you can find a workaround there, and/or contribute your workaround
when you file your bug report.
Personally, I've never configured sshd to use socket activation, nor do I
see any advantage in doing so. Back in the old days, we were always
warned *not* to do that, because the startup time could be extremely long
due to entropy exhaustion. Yeah, I know, entropy is handled differently
by the kernel now. Still, I don't see any reason to move away from the
tried-and-tested single daemon model.
Frank
Sep 16, 2022, 8:30:05 AM9/16/22
to
Op 16-09-2022 om 09:17 schreef Michael:
service section of the ssh@.service file currently includes
RuntimeDirectoryPreserve=yes.
Maybe file a bug report to have this added for bullseye?
Regards,
Frank
> with ssh@.service it is completely different. for each connection there
> is a dedicated sshd process being started, and each one of them has the
> same /run/sshd directory assigned. and that's the problem if you have
> more than one connection to a given host. as soon as the first
> connection is terminated, the /run/sshd directory disappears, and the
> other sshd's might run into problems.
Apparently this has already been 'fixed' for bookworm. That is, the
> is a dedicated sshd process being started, and each one of them has the
> same /run/sshd directory assigned. and that's the problem if you have
> more than one connection to a given host. as soon as the first
> connection is terminated, the /run/sshd directory disappears, and the
> other sshd's might run into problems.
service section of the ssh@.service file currently includes
RuntimeDirectoryPreserve=yes.
Maybe file a bug report to have this added for bullseye?
Regards,
Frank
David Wright
Sep 16, 2022, 12:10:05 PM9/16/22
to
https://bugzilla.redhat.com/show_bug.cgi?id=963268
and
http://0pointer.de/blog/projects/inetd.html
(not that I appreciate some of the points being made there).
> but i still don't understand, why this scenario prevents me from
> contacting a host with the formerly mentioned error message in the log
> file...
> > On Thu 15 Sep 2022 at 12:02:21 (+0200), Michael wrote:
> > > so i started digging, and to my surprise i found out that on the
> > > affected servers sshd was configured to be invoked by ssh.socket (via
> > > ssh@.service with the -i option), wheras on all other hosts sshd was
> > > running as a daemon (via ssh.service whith the -D option).
> > >
> > > so, my first question is: why?
I don't think you've yet confirmed what's in your /etc/systemd tree
> > > so i started digging, and to my surprise i found out that on the
> > > affected servers sshd was configured to be invoked by ssh.socket (via
> > > ssh@.service with the -i option), wheras on all other hosts sshd was
> > > running as a daemon (via ssh.service whith the -D option).
> > >
> > > so, my first question is: why?
on the different hosts. Mine all show:
$ find /etc/systemd/ -name \*ssh\* -ls | squeeze-whitespace
1700765 0 lrwxrwxrwx 1 root root 31 Apr 16 20:31 /etc/systemd/system/sshd.service -> /lib/systemd/system/ssh.service
1700799 0 lrwxrwxrwx 1 root root 31 Apr 16 20:31 /etc/systemd/system/multi-user.target.wants/ssh.service -> /lib/systemd/system/ssh.service
1702117 0 lrwxrwxrwx 1 root root 42 Apr 16 20:56 /etc/systemd/user/sockets.target.wants/gpg-agent-ssh.socket -> /usr/lib/systemd/user/gpg-agent-ssh.socket
$
with no trace of the @ version, below.
$ diff -u /lib/systemd/system/ssh*service
--- /lib/systemd/system/ssh.service 2022-07-01 17:37:41.000000000 -0500
+++ /lib/systemd/system/ssh@.service 2022-07-01 17:37:41.000000000 -0500
@@ -1,22 +1,11 @@
[Unit]
-Description=OpenBSD Secure Shell server
+Description=OpenBSD Secure Shell server per-connection daemon
Documentation=man:sshd(8) man:sshd_config(5)
-After=network.target auditd.service
-ConditionPathExists=!/etc/ssh/sshd_not_to_be_run
+After=auditd.service
[Service]
EnvironmentFile=-/etc/default/ssh
-ExecStartPre=/usr/sbin/sshd -t
-ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
-ExecReload=/usr/sbin/sshd -t
-ExecReload=/bin/kill -HUP $MAINPID
-KillMode=process
-Restart=on-failure
-RestartPreventExitStatus=255
-Type=notify
+ExecStart=-/usr/sbin/sshd -i $SSHD_OPTS ←
+StandardInput=socket ←
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755
-
-[Install]
-WantedBy=multi-user.target
-Alias=sshd.service
$
Cheers,
David.
Michael
Sep 17, 2022, 4:20:05 AM9/17/22
to
On Friday, 16 September 2022 14:10:01 CEST, Frank wrote:
> Apparently this has already been 'fixed' for bookworm. [...]
so, this issue is known and 'they' did something about it.
> Maybe file a bug report to have this added for bullseye?
since this issue is known, 'they' should be aware of it, too...
greetings...
> Apparently this has already been 'fixed' for bookworm. [...]
so, this issue is known and 'they' did something about it.
> Maybe file a bug report to have this added for bullseye?
greetings...
Michael
Sep 17, 2022, 4:30:04 AM9/17/22
to
On Friday, 16 September 2022 13:25:06 CEST, Greg Wooledge wrote:
> I did find this paragraph in systemd.exec(5):
me, too. if i run into a problem, the first thing i do is to read. and,
> I did find this paragraph in systemd.exec(5):
yes: i do read even man pages! ;)
> Maybe you can find a workaround there, and/or contribute your workaround
> when you file your bug report.
fixed in bookworm, i don't see any reason to file a bug.
> Personally, I've never configured sshd to use socket activation, nor do I
> see any advantage in doing so.
and went back to the good old daemon. yes, we all have our daemons! ;)
i just had a problem with ssh(d), and i wanted to know what caused it. now
that i know, i can sleep a lot better. :)
> Back in the old days, [...]
yeah, i miss them, too... sometimes...
greetings...
0 new messages