Bug#1002591: misdetects socket activated ssh
Marc Haber
Version: 3.5-5
Severity: normal
Hi,
when using ssh as a socket activated service (systemctl stop/disable
ssh.service, systemctl enable/start ssh.socket), after a library update
needrestart will offer to restart ssh.service. This fails since port 22
is occupied by the instance services and causes the machine to be
without listening process after logging out.
A possible workaround is masking ssh.service, see #1001320.
Restarting services...
systemctl restart console-log.service cron.service exim4.service haveged.service ippl.service ntp.service rsyslog.service serial...@ttyS0.service ssh.service systemd-journald.service systemd-networkd.service systemd-resolved.service systemd-udevd.service
Job for ssh.service failed because the control process exited with error code.
See "systemctl status ssh.service" and "journalctl -xeu ssh.service" for details.
Service restarts being deferred:
/etc/needrestart/restart.d/dbus.service
systemctl restart ge...@tty1.service
systemctl restart systemd-logind.service
systemctl restart us...@1001.service
and the following log entries:
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping LSB: Puts a logfile pager on virtual consoles...
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping Regular background program processing daemon...
Dec 8 12:58:26 emptybookworm82 systemd[1]: cron.service: Deactivated successfully.
Dec 8 12:58:26 emptybookworm82 cron[429258]: (CRON) INFO (pidfile fd = 3)
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopped Regular background program processing daemon.
Dec 8 12:58:26 emptybookworm82 systemd[1]: cron.service: Consumed 15min 4.856s CPU time.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Started Regular background program processing daemon.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping LSB: exim Mail Transport Agent...
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping Entropy Daemon based on the HAVEGE algorithm...
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping LSB: IP protocols logger...
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping Network Time Service...
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping System Logging Service...
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping Serial Getty on ttyS0...
Dec 8 12:58:26 emptybookworm82 systemd[1]: serial...@ttyS0.service: Deactivated successfully.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopped Serial Getty on ttyS0.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Started Serial Getty on ttyS0.
Dec 8 12:58:26 emptybookworm82 systemd[1]: ssh.socket: Deactivated successfully.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Closed OpenBSD Secure Shell server socket.
Dec 8 12:58:26 emptybookworm82 systemd[1]: ssh.socket: Consumed 10.571s CPU time.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Starting OpenBSD Secure Shell server...
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping Flush Journal to Persistent Storage...
Dec 8 12:58:26 emptybookworm82 systemd[1]: systemd-networkd-wait-online.service: Deactivated successfully.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopped Wait for Network to be Configured.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping Wait for Network to be Configured...
Dec 8 12:58:26 emptybookworm82 systemd[1]: Stopping Network Name Resolution...
Dec 8 12:58:26 emptybookworm82 systemd[1]: ssh.service: Main process exited, code=exited, status=255/EXCEPTION
Dec 8 12:58:26 emptybookworm82 systemd[1]: ssh.service: Failed with result 'exit-code'.
Dec 8 12:58:26 emptybookworm82 systemd[1]: Failed to start OpenBSD Secure Shell server.
Dec 8 12:58:26 emptybookworm82 ntpd[298]: ntpd exiting on signal 15 (Terminated)
Dec 8 12:58:26 emptybookworm82 ntpd[298]: 2a01:4f8:140:246a::2 local addr 2a01:4f8:140:246a::52:100 -> <null>
Dec 8 12:58:26 emptybookworm82 haveged[220]: haveged: Stopping due to signal 15
Dec 8 12:58:27 emptybookworm82 cron[429258]: (CRON) INFO (Skipping @reboot jobs -- not system startup)
Dec 8 12:58:27 emptybookworm82 systemd[1]: systemd-journal-flush.service: Deactivated successfully.
Dec 8 12:58:27 emptybookworm82 systemd[1]: Stopped Flush Journal to Persistent Storage.
Dec 8 12:58:27 emptybookworm82 exim4[429259]: exim4_listener.
Here is what Timo Weingärtner found out in relation to my bug report against sshd:
| To me it looks like a problem in needrestart. The (forked off) sshd process
| handling your client connection belongs to cgroup session-NN.scope, no matter
| if it was started by systemd socket activation or regular sshd.
|
| needrestart (invoked with "-vlp" here) detects a process with outdated libs:
|
| [main] #2111961 uses deleted /lib/x86_64-linux-gnu/libnss_files-2.32.so
| [main] #2111961 is a child of #2111904
|
| Then it figures out the binary and the cgroup:
|
| [main] #2111961 exe => /usr/sbin/sshd
| [main] trying systemctl status
|
| cgroup detection didn't work, so:
|
| [main] #2111961 running /etc/needrestart/hook.d/10-dpkg
| [main] #2111961 package: openssh-server
| [main] #2111961 running /etc/needrestart/hook.d/20-rpm
| [main] #2111961 running /etc/needrestart/hook.d/90-none
|
| /etc/needrestart/hook.d/10-dpkg also finds /etc/init.d/ssh and we end up with:
|
| Services:
| […]
| - spamassassin.service
| - ssh
| - systemd-journald.service
| […]
|
| Note the missing ".service". Then you have it invoke "systemctl restart
| ssh.service" and voilà.
|
| A workaround might be masking ssh.service.
A per-client process should not get restarted by needrestart. This might
happen even if ssh.service is running, but it is usually harmless.
You might find more information in #1001320 regarding ssh.
Greetings
Marc, thanking Timo for his insights and help
-- Package-specific info:
needrestart output:
Your outdated processes:
kaccess[6984], kalarm[14960], kdeconnectd[7001], kded5[6924], kdeinit5[6907], kglobalaccel5[6937], klauncher[6908], konsole[7011, 41112], krunner[837284], ksmserver[6965], kwalletd5[6727], kwin_x11[6928], obexd[7066], plasmashell[6989], pulseaudio[6717], telegram-deskto[7262], yakuake[7063]
checkrestart output:
-- System Information:
Debian Release: bookworm/sid
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'unstable')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 5.15.10-zgws1 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_DIE, TAINT_OOT_MODULE
Locale: LANG=de_DE.utf8, LC_CTYPE=de_DE.utf8 (charmap=UTF-8), LANGUAGE=en
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
Versions of packages needrestart depends on:
ii binutils 2.37-10
ii dpkg 1.21.1
ii gettext-base 0.21-4
ii libintl-perl 1.26-3
ii libmodule-find-perl 0.15-1
ii libmodule-scandeps-perl 1.31-1
ii libproc-processtable-perl 0.634-1
ii libsort-naturally-perl 1.03-2
ii libterm-readkey-perl 2.38-1+b2
ii perl 5.32.1-6
ii xz-utils 5.2.5-2
Versions of packages needrestart recommends:
ii libpam-systemd 249.7-1
Versions of packages needrestart suggests:
ii iucode-tool 2.3.1-1
ii libnotify-bin 0.7.9-3
-- no debconf information
Thomas Liske
could you please provide the content of /proc/$PID/cgroup for an socket
activated sshd instance?
As a workaround you might blacklist sshd in needrestart but I think a
generic approach handling socket activation services in needrestart
would be better. Therefore needrestart need a way to detect if the
process belongs to a socket activated service.
TIA & HTH,
Thomas
Marc Haber
On Sat, Jan 01, 2022 at 02:31:04PM +0100, Thomas Liske wrote:
> could you please provide the content of /proc/$PID/cgroup for an socket
> activated sshd instance?
1 [1/4996]mh@torres:~ $ pgrep ssh
315675
315738
[2/4997]mh@torres:~ $ sudo cat /proc/315675/cgroup
[sudo] password for mh on torres:
0::/user.slice/user-1001.slice/session-296.scope
[3/4998]mh@torres:~ $ sudo cat /proc/315738/cgroup
0::/user.slice/user-1001.slice/session-296.scope
[4/4999]mh@torres:~ $
> As a workaround you might blacklist sshd in needrestart but I think a
> generic approach handling socket activation services in needrestart
> would be better. Therefore needrestart need a way to detect if the
> process belongs to a socket activated service.
couse having the heuristic fixed would be better.
Greetings
Marc
--
-----------------------------------------------------------------------------
Marc Haber | "I don't trust Computers. They | Mailadresse im Header
Leimen, Germany | lose things." Winona Ryder | Fon: *49 6224 1600402
Nordisch by Nature | How to make an American Quilt | Fax: *49 6224 1600421
Thomas Liske
On Sat, 2022-01-01 at 20:55 +0100, Marc Haber wrote:
> Sure:
> 1 [1/4996]mh@torres:~ $ pgrep ssh
> 315675
> 315738
> [2/4997]mh@torres:~ $ sudo cat /proc/315675/cgroup
> [sudo] password for mh on torres:
> 0::/user.slice/user-1001.slice/session-296.scope
> [3/4998]mh@torres:~ $ sudo cat /proc/315738/cgroup
> 0::/user.slice/user-1001.slice/session-296.scope
> [4/4999]mh@torres:~ $
>
user slice cgroup. It does not work due to this check[1] in
needrestart.
[1] https://github.com/liske/needrestart/blob/v3.5/needrestart#L637
Looks like a systemd/cgroup related change in bullseye, buster seems
not to be affected.
Regards,
Thomas
Marc Haber
> Looks like a systemd/cgroup related change in bullseye, buster seems
> not to be affected.
in the past.