Google Groups no longer supports new Usenet posts or subscriptions. Historical content remains viewable.
Dismiss
Bug#990069: openssh-server: Not accepting new connections during Debian 10 -> 11 upgrade
18 views
Skip to first unread message
Olaf van der Spek
Jul 4, 2021, 5:40:03 AM7/4/21
to
Op zo 4 jul. 2021 om 00:42 schreef Colin Watson <cjwa...@debian.org>:
> Sorry for my delay - it took me a while to spot the problem. libc6's
> postinst does arrange to restart services where needed, but in this case
> it doesn't realize that you have the ssh service installed because you
> only have the openssh-server package installed, not the ssh metapackage
> (i.e. the package with the same name as the service).
>
> I've proposed
> https://salsa.debian.org/glibc-team/glibc/-/merge_requests/3 to fix
> this. glibc maintainers, if there's any way to get this into bullseye,
> I'm sure it would help avoid some people upgrading remote systems ending
> up being unable to fix them if something goes wrong between configuring
> libc6 and configuring openssh-server. Also CCing debian-release for
> their information, as I know it's pretty late for glibc changes.
Thanks Colin!
I assume openssh-server would then be restarted after the "There are
services installed on your system which need to be
restarted when certain libraries, such as libpam, libc, and libssl,
are upgraded." question. But ssh is already 'down' when that question
is being asked, so wouldn't there still be a time window, with
required user interaction, where ssh would be 'down'?
--
Olaf
> Sorry for my delay - it took me a while to spot the problem. libc6's
> postinst does arrange to restart services where needed, but in this case
> it doesn't realize that you have the ssh service installed because you
> only have the openssh-server package installed, not the ssh metapackage
> (i.e. the package with the same name as the service).
>
> I've proposed
> https://salsa.debian.org/glibc-team/glibc/-/merge_requests/3 to fix
> this. glibc maintainers, if there's any way to get this into bullseye,
> I'm sure it would help avoid some people upgrading remote systems ending
> up being unable to fix them if something goes wrong between configuring
> libc6 and configuring openssh-server. Also CCing debian-release for
> their information, as I know it's pretty late for glibc changes.
Thanks Colin!
I assume openssh-server would then be restarted after the "There are
services installed on your system which need to be
restarted when certain libraries, such as libpam, libc, and libssl,
are upgraded." question. But ssh is already 'down' when that question
is being asked, so wouldn't there still be a time window, with
required user interaction, where ssh would be 'down'?
--
Olaf
Paul Gevers
Jul 4, 2021, 2:00:03 PM7/4/21
to
Hi all,
On 04-07-2021 00:42, Colin Watson wrote:
> Sorry for my delay - it took me a while to spot the problem. libc6's
> postinst does arrange to restart services where needed, but in this case
> it doesn't realize that you have the ssh service installed because you
> only have the openssh-server package installed, not the ssh metapackage
> (i.e. the package with the same name as the service).
>
> I've proposed
> https://salsa.debian.org/glibc-team/glibc/-/merge_requests/3 to fix
> this. glibc maintainers, if there's any way to get this into bullseye,
> I'm sure it would help avoid some people upgrading remote systems ending
> up being unable to fix them if something goes wrong between configuring
> libc6 and configuring openssh-server. Also CCing debian-release for
> their information, as I know it's pretty late for glibc changes.
I think we really want this. I *think* I ran into exactly this issue two
days ago when I upgraded my NAS. It's really scary to notice that you
can't log into your system and your only connection is the current one
running the upgrade. In my case, it was asking questions along the way.
I had considered running the upgrade in screen. In the end it look
longer than expected and I left my laptop on. If I would have run in
screen and disconnected, I would have had no idea what hit me.
Paul
On 04-07-2021 00:42, Colin Watson wrote:
> Sorry for my delay - it took me a while to spot the problem. libc6's
> postinst does arrange to restart services where needed, but in this case
> it doesn't realize that you have the ssh service installed because you
> only have the openssh-server package installed, not the ssh metapackage
> (i.e. the package with the same name as the service).
>
> I've proposed
> https://salsa.debian.org/glibc-team/glibc/-/merge_requests/3 to fix
> this. glibc maintainers, if there's any way to get this into bullseye,
> I'm sure it would help avoid some people upgrading remote systems ending
> up being unable to fix them if something goes wrong between configuring
> libc6 and configuring openssh-server. Also CCing debian-release for
> their information, as I know it's pretty late for glibc changes.
days ago when I upgraded my NAS. It's really scary to notice that you
can't log into your system and your only connection is the current one
running the upgrade. In my case, it was asking questions along the way.
I had considered running the upgrade in screen. In the end it look
longer than expected and I left my laptop on. If I would have run in
screen and disconnected, I would have had no idea what hit me.
Paul
Olaf van der Spek
Jul 31, 2021, 3:10:03 AM7/31/21
to
Package: libc6
Version: 2.31-13
Followup-For: Bug #990069
X-Debbugs-Cc: olafv...@gmail.com
Dear Maintainer,
The original issue of not being able to open a new SSH connection during the upgrade still seems to be present.
Greetings,
Olaf
-- System Information:
Debian Release: 11.0
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'testing'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-17-amd64 (SMP w/1 CPU thread)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages libc6 depends on:
ii libcrypt1 1:4.4.18-4
ii libgcc-s1 10.2.1-6
Versions of packages libc6 recommends:
ii libidn2-0 2.3.0-5
ii libnss-nis 3.1-4
ii libnss-nisplus 1.3-4
Versions of packages libc6 suggests:
ii debconf [debconf-2.0] 1.5.77
pn glibc-doc <none>
ii libc-l10n 2.31-13
ii locales 2.31-13
-- debconf information:
glibc/disable-screensaver:
glibc/kernel-not-supported:
* libraries/restart-without-asking: true
glibc/upgrade: true
glibc/kernel-too-old:
glibc/restart-services:
glibc/restart-failed:
Version: 2.31-13
Followup-For: Bug #990069
X-Debbugs-Cc: olafv...@gmail.com
Dear Maintainer,
The original issue of not being able to open a new SSH connection during the upgrade still seems to be present.
Greetings,
Olaf
-- System Information:
Debian Release: 11.0
APT prefers stable-updates
APT policy: (500, 'stable-updates'), (500, 'testing'), (500, 'stable')
Architecture: amd64 (x86_64)
Kernel: Linux 4.19.0-17-amd64 (SMP w/1 CPU thread)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled
Versions of packages libc6 depends on:
ii libcrypt1 1:4.4.18-4
ii libgcc-s1 10.2.1-6
Versions of packages libc6 recommends:
ii libidn2-0 2.3.0-5
ii libnss-nis 3.1-4
ii libnss-nisplus 1.3-4
Versions of packages libc6 suggests:
ii debconf [debconf-2.0] 1.5.77
pn glibc-doc <none>
ii libc-l10n 2.31-13
ii locales 2.31-13
-- debconf information:
glibc/disable-screensaver:
glibc/kernel-not-supported:
* libraries/restart-without-asking: true
glibc/upgrade: true
glibc/kernel-too-old:
glibc/restart-services:
glibc/restart-failed:
Ron
Aug 4, 2021, 1:10:03 AM8/4/21
to
On Tue, Aug 03, 2021 at 10:08:42PM +0200, Aurelien Jarno wrote:
> On 2021-08-01 00:05, Ron wrote:
> >
> > Unpacking openssh-sftp-server (1:8.4p1-5) over (1:7.9p1-10+deb10u2) ...
> > Preparing to unpack .../openssh-client_1%3a8.4p1-5_amd64.deb ...
> > Unpacking openssh-client (1:8.4p1-5) over (1:7.9p1-10+deb10u2) ...
> > Preparing to unpack .../runit-helper_2.10.3_all.deb ...
> > Unpacking runit-helper (2.10.3) ...
> > Preparing to unpack .../openssh-server_1%3a8.4p1-5_amd64.deb ...
> > Unpacking openssh-server (1:8.4p1-5) over (1:7.9p1-10+deb10u2) ...
> > Preparing to unpack .../libc6_2.31-13_amd64.deb ...
> > Checking for services that may need to be restarted...
> > Checking init scripts...
> > Unpacking libc6:amd64 (2.31-13) over (2.28-10) ...
> > Setting up libc6:amd64 (2.31-13) ...
> > Checking for services that may need to be restarted...
> > Checking init scripts...
> > Services to restart for GNU libc library upgrade:
> > cron atd
> > Restarting services possibly affected by the upgrade:
> > cron: restarting...done.
> > atd: restarting...done.
> > Services restarted successfully.
> > Preparing to unpack .../libc-bin_2.31-13_amd64.deb ...
> > Unpacking libc-bin (2.31-13) over (2.28-10) ...
> > Setting up libc-bin (2.31-13) ...
> > ...
> > <much later>
> > ...
> > Setting up openssh-server (1:8.4p1-5) ...
>
> Thanks for this upgrade log, with it I have been able to understand and
> reproduce the issue. The libc restart logic looks for installed
> packaged, however due to all the breaks and depends between
> openssh-server and libc6 in the buster -> bullseye upgrade path, it
> could happen that at the moment the libc6 postinst script is ran, the
> openssh-server has been degraded from installed to unpacked.
>
> I have tested the following fix, which works well when used in the same
> conditions as the above log:
>
> diff --git a/debian/script.in/nsscheck.sh b/debian/script.in/nsscheck.sh
> index 8406a543..ee99ac16 100644
> --- a/debian/script.in/nsscheck.sh
> +++ b/debian/script.in/nsscheck.sh
> @@ -1,8 +1,8 @@
> echo -n "Checking for services that may need to be restarted..."
> # Only get the ones that are installed, of the same architecture
> # as libc (or arch all) and configured
> - check=$(dpkg-query -W -f='${binary:Package} ${Status} ${Architecture}\n' $check 2> /dev/null | \
> - grep -E "installed (all|${DPKG_MAINTSCRIPT_ARCH})$" | sed 's/[: ].*//')
> + check=$(dpkg-query -W -f='${binary:Package} ${db:Status-Want} ${Architecture}\n' $check 2> /dev/null | \
> + grep -E "(install|hold) (all|${DPKG_MAINTSCRIPT_ARCH})$" | sed 's/[: ].*//')
> # some init scripts don't match the package names
> check=$(echo $check | \
> sed -e's/\bapache2.2-common\b/apache2/g' \
>
> However before uploading such a fix so close to the release, I think it
> requires a review by many more pair of eyes.
Again flying blind to a lot of important details -- but what happens if
libc postinst tries to restart a service that is unpacked but not yet
configured?
I'm guessing that depends a lot on what the service post* do, but how
horrible could that get? Does this really need to be a trigger that
(also?) restarts each of the half-installed services after they are
fully (re)configured again?
I was able to restart ssh manually during the upgrade, but by the time I
figured out that was what was needed, the install was probably far enough
through that it had likely been configured and was fully installed again ...
Cheers,
Ron
> On 2021-08-01 00:05, Ron wrote:
> >
> > Unpacking openssh-sftp-server (1:8.4p1-5) over (1:7.9p1-10+deb10u2) ...
> > Preparing to unpack .../openssh-client_1%3a8.4p1-5_amd64.deb ...
> > Unpacking openssh-client (1:8.4p1-5) over (1:7.9p1-10+deb10u2) ...
> > Preparing to unpack .../runit-helper_2.10.3_all.deb ...
> > Unpacking runit-helper (2.10.3) ...
> > Preparing to unpack .../openssh-server_1%3a8.4p1-5_amd64.deb ...
> > Unpacking openssh-server (1:8.4p1-5) over (1:7.9p1-10+deb10u2) ...
> > Preparing to unpack .../libc6_2.31-13_amd64.deb ...
> > Checking for services that may need to be restarted...
> > Checking init scripts...
> > Unpacking libc6:amd64 (2.31-13) over (2.28-10) ...
> > Setting up libc6:amd64 (2.31-13) ...
> > Checking for services that may need to be restarted...
> > Checking init scripts...
> > Services to restart for GNU libc library upgrade:
> > cron atd
> > Restarting services possibly affected by the upgrade:
> > cron: restarting...done.
> > atd: restarting...done.
> > Services restarted successfully.
> > Preparing to unpack .../libc-bin_2.31-13_amd64.deb ...
> > Unpacking libc-bin (2.31-13) over (2.28-10) ...
> > Setting up libc-bin (2.31-13) ...
> > ...
> > <much later>
> > ...
> > Setting up openssh-server (1:8.4p1-5) ...
>
> Thanks for this upgrade log, with it I have been able to understand and
> reproduce the issue. The libc restart logic looks for installed
> packaged, however due to all the breaks and depends between
> openssh-server and libc6 in the buster -> bullseye upgrade path, it
> could happen that at the moment the libc6 postinst script is ran, the
> openssh-server has been degraded from installed to unpacked.
>
> I have tested the following fix, which works well when used in the same
> conditions as the above log:
>
> diff --git a/debian/script.in/nsscheck.sh b/debian/script.in/nsscheck.sh
> index 8406a543..ee99ac16 100644
> --- a/debian/script.in/nsscheck.sh
> +++ b/debian/script.in/nsscheck.sh
> @@ -1,8 +1,8 @@
> echo -n "Checking for services that may need to be restarted..."
> # Only get the ones that are installed, of the same architecture
> # as libc (or arch all) and configured
> - check=$(dpkg-query -W -f='${binary:Package} ${Status} ${Architecture}\n' $check 2> /dev/null | \
> - grep -E "installed (all|${DPKG_MAINTSCRIPT_ARCH})$" | sed 's/[: ].*//')
> + check=$(dpkg-query -W -f='${binary:Package} ${db:Status-Want} ${Architecture}\n' $check 2> /dev/null | \
> + grep -E "(install|hold) (all|${DPKG_MAINTSCRIPT_ARCH})$" | sed 's/[: ].*//')
> # some init scripts don't match the package names
> check=$(echo $check | \
> sed -e's/\bapache2.2-common\b/apache2/g' \
>
> However before uploading such a fix so close to the release, I think it
> requires a review by many more pair of eyes.
Again flying blind to a lot of important details -- but what happens if
libc postinst tries to restart a service that is unpacked but not yet
configured?
I'm guessing that depends a lot on what the service post* do, but how
horrible could that get? Does this really need to be a trigger that
(also?) restarts each of the half-installed services after they are
fully (re)configured again?
I was able to restart ssh manually during the upgrade, but by the time I
figured out that was what was needed, the install was probably far enough
through that it had likely been configured and was fully installed again ...
Cheers,
Ron
Aurelien Jarno
Aug 4, 2021, 4:50:03 AM8/4/21
to
Hi,
Well this bug report complains that SSH is unavailable during an upgrade
until the package is fully configured again. For making it available
again as soon as possible, it has to be restarted after the new glibc
has been installed, even if it has only been unpacked at this stage.
That said you are fully right that it might not work for some other
services.
> I was able to restart ssh manually during the upgrade, but by the time I
> figured out that was what was needed, the install was probably far enough
> through that it had likely been configured and was fully installed again ...
Given I have been able to reproduce the issue, I have tested the above
patch, and I can confirm that openssh can be restarted when it has only
been unpacked.
Regards,
Aurelien
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aure...@aurel32.net http://www.aurel32.net
until the package is fully configured again. For making it available
again as soon as possible, it has to be restarted after the new glibc
has been installed, even if it has only been unpacked at this stage.
That said you are fully right that it might not work for some other
services.
> I was able to restart ssh manually during the upgrade, but by the time I
> figured out that was what was needed, the install was probably far enough
> through that it had likely been configured and was fully installed again ...
patch, and I can confirm that openssh can be restarted when it has only
been unpacked.
Regards,
Aurelien
--
Aurelien Jarno GPG: 4096R/1DDD8C9B
aure...@aurel32.net http://www.aurel32.net
Aurelien Jarno
Aug 4, 2021, 6:40:02 AM8/4/21
to
systems where the ssh meta-package was installed. I am afraid that the
change actually made things worse for those systems, although it was
only working by chance.
Andres Salomon
Aug 12, 2021, 10:10:04 PM8/12/21
to
Hi,
So this only affects users who do or do not have the ssh metapackage
installed? This bug report is a bit confusing.
Thanks,
Andres
So this only affects users who do or do not have the ssh metapackage
installed? This bug report is a bit confusing.
Thanks,
Andres
Paul Gevers
Aug 13, 2021, 10:40:03 AM8/13/21
to
On Thu, 12 Aug 2021 21:51:09 -0400 Andres Salomon <dili...@queued.net>
wrote:
Paul
wrote:
> So this only affects users who do or do not have the ssh metapackage
> installed?
I'm pretty sure it effects both.
> installed?
Paul
Aurelien Jarno
Aug 16, 2021, 9:00:04 AM8/16/21
to
Hi,
On 2021-08-12 21:51, Andres Salomon wrote:
> Hi,
>
> So this only affects users who do or do not have the ssh metapackage
> installed? This bug report is a bit confusing.
>
It used to affects all users who do or do not have the ssh metapackage
installed, however following the fix that went into 2.31-13, all users
are now affected.
On 2021-08-12 21:51, Andres Salomon wrote:
> Hi,
>
> So this only affects users who do or do not have the ssh metapackage
> installed? This bug report is a bit confusing.
>
installed, however following the fix that went into 2.31-13, all users
are now affected.
0 new messages