Bug#993755: libcrypt.so.1: cannot open shared object file when upgrading from Stretch to Sid

[Otto Kekäläinen]

Package: libcrypt1

The file /lib/x86_64-linux-gnu/libcrypt.so.1 used to be in package
libc6 on Stretch and Buster until it was split off into a separate
libcrypt in Bullseye (and in Sid).

Something in the upgrade logic to adopt the new pacakge is lacking
when upgrading libc6 as the system ends in a situation where
libcrypt.so.1 is removed before the replacement from a new package is
installed.

I noticed this on Salsa-CI for my own package. It happens in a
reproducible way on a clean Docker image. See full log at:
https://salsa.debian.org/mariadb-team/mariadb-server/-/jobs/1911234

Here is the relevant part:

# Replace any old repos with just Sid
echo 'deb http://deb.debian.org/debian sid main' > /etc/apt/sources.list
# Upgrade minimal stack first
apt-get update
apt-get install -y apt
apt-get install -y pkg-config libmariadb2 libmariadb-dev libmariadb-dev-compat
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
bzip2 libdpkg-perl libffi6 libfile-fcntllock-perl libgdbm3 libglib2.0-0
libglib2.0-data libicu57 liblocale-gettext-perl libperl5.24 libssl1.1
libxml2 netbase perl perl-modules-5.24 rename sgml-base shared-mime-info
xdg-user-dirs xml-core xz-utils
Suggested packages:
bzip2-doc debian-keyring gnupg | gnupg2 gcc | c-compiler binutils patch
perl-doc libterm-readline-gnu-perl | libterm-readline-perl-perl make
sgml-base-doc debhelper
The following NEW packages will be installed:
bzip2 libdpkg-perl libffi6 libfile-fcntllock-perl libgdbm3 libglib2.0-0
libglib2.0-data libicu57 liblocale-gettext-perl libmariadb-dev
libmariadb-dev-compat libmariadb2 libperl5.24 libssl1.1 libxml2 netbase perl
perl-modules-5.24 pkg-config rename sgml-base shared-mime-info xdg-user-dirs
xml-core xz-utils
0 upgraded, 25 newly installed, 0 to remove and 0 not upgraded.
...
Preparing to unpack .../libc6_2.31-17_amd64.deb ...
Checking for services that may need to be restarted...
Checking init scripts...
Unpacking libc6:amd64 (2.31-17) over (2.24-11+deb9u4) ...
Setting up libc6:amd64 (2.31-17) ...
Installing new version of config file
/etc/ld.so.conf.d/x86_64-linux-gnu.conf ...
/usr/bin/perl: error while loading shared libraries: libcrypt.so.1:
cannot open shared object file: No such file or directory
dpkg: error processing package libc6:amd64 (--configure):
subprocess installed post-installation script returned error exit status 127
Errors were encountered while processing:
libc6:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

[Marco d'Itri]

On Sep 06, Otto Kekäläinen <ot...@debian.org> wrote:

> Something in the upgrade logic to adopt the new pacakge is lacking
> when upgrading libc6 as the system ends in a situation where
> libcrypt.so.1 is removed before the replacement from a new package is
> installed.

Yes, I think that this happens because perl-base Pre-Depends on
libcrypt1, but since perl-base is not being upgraded in your example
then there is nothing which will pull libcrypt1 in.
The interesting questions here is why this does not happen when
upgrading from Debian 10 to 11, or at least not frequently enough to be
noticed?
If the fix (Breaks perl-base in libc6?) does not have too many
collateral effects then it may be worth implementing it anyway even if
9 -> 12 upgrades are not officially supported.

--
ciao,
Marco

[Marco d'Itri]

Control: tag -1 wontfix

On Sep 06, Marco d'Itri <m...@Linux.IT> wrote:

> If the fix (Breaks perl-base in libc6?) does not have too many
> collateral effects then it may be worth implementing it anyway even if
> 9 -> 12 upgrades are not officially supported.

Upgrades to Debian 11 work because libc6 used to depend on libcrypt1,
but since circular dependencies are suboptimal it was removed in
unstable: see #975077.
This /may/ be fixed with adding "Breaks: perl-base (<< 5.32.1-4)" to
libc6, but the libc6 maintainer has requested that this would be tested
extensively to show that it does not cause any troubles.
But since skipping releases when upgrading is unsupported this is not
a priority and somebody will need to volunteer for this work.

--
ciao,
Marco

[Otto Kekäläinen]

Thanks for a quick reply. I understand why libc6 maintainers prefer
not to touch the dependencies they have unless the issue is severe.

I guess the workaround for this is to manually start the upgrade by
installing perl-base before upgrading the rest of the system.

[Otto Kekäläinen]

This issue is a bit nasty though, as if a user encounters it,
recovering with apt is impossible as dpkg depends on perl and will not
run if libcrypt1 got broken:

$ apt -y install libcrypt1
Reading package lists... Done
Building dependency tree
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
libc-bin : Depends: libc6 (< 2.25) but 2.32-2 is to be installed
Recommends: manpages but it is not going to be installed
libcrypt1 : Conflicts: libpam0g (< 1.4.0-10) but 1.1.8-3.6 is to be installed
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages
(or specify a solution).

$ apt -y --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done

The following additional packages will be installed:

libc-bin
Recommended packages:
manpages
The following packages will be upgraded:
libc-bin
1 upgraded, 0 newly installed, 0 to remove and 95 not upgraded.
1 not fully installed or removed.
Need to get 829 kB of archives.
After this operation, 341 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian sid/main amd64 libc-bin amd64 2.32-2 [829 kB]
Fetched 829 kB in 0s (5968 kB/s)

/usr/bin/perl: error while loading shared libraries: libcrypt.so.1:
cannot open shared object file: No such file or directory

Setting up libc6:amd64 (2.32-2) ...

/usr/bin/perl: error while loading shared libraries: libcrypt.so.1:
cannot open shared object file: No such file or directory
dpkg: error processing package libc6:amd64 (--configure):
subprocess installed post-installation script returned error exit status 127
Errors were encountered while processing:
libc6:amd64
E: Sub-process /usr/bin/dpkg returned an error code (1)

Here are manual steps on how to recover:

$ cd /tmp
$ apt -y download libcrypt1
$ dpkg-deb -x libcrypt1_1%3a4.4.25-2_amd64.deb .
$ cp -av lib/x86_64-linux-gnu/* /lib/x86_64-linux-gnu/
$ apt -y --fix-broken install
Reading package lists... Done
Building dependency tree
Reading state information... Done
Correcting dependencies... Done

The following additional packages will be installed:

libc-bin
Recommended packages:
manpages
The following packages will be upgraded:
libc-bin
1 upgraded, 0 newly installed, 0 to remove and 95 not upgraded.
1 not fully installed or removed.
Need to get 829 kB of archives.
After this operation, 341 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian sid/main amd64 libc-bin amd64 2.32-2 [829 kB]
Fetched 829 kB in 0s (5292 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Setting up libc6:amd64 (2.32-2) ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog
based frontend cannot be used. at
/usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline

Checking for services that may need to be restarted...
Checking init scripts...

Nothing to restart.
(Reading database ... 9176 files and directories currently installed.)
Preparing to unpack .../libc-bin_2.32-2_amd64.deb ...
Unpacking libc-bin (2.32-2) over (2.24-11+deb9u4) ...
Setting up libc-bin (2.32-2) ...
Updating /etc/nsswitch.conf to current default.
W: APT had planned for dpkg to do more than it reported back (5 vs 9).
Affected packages: libc6:amd64


Also note that there is no workaround to *not* hit this bug, as any
attempt to install libcrypt1 will also pull in libc and thus break
Perl and dpkg before libcrypt1 is unpacked.

$ apt-get install -y libcrypt1
Reading package lists... Done
Building dependency tree
Reading state information... Done

The following additional packages will be installed:

gcc-11-base libc-bin libc6 libgcc-s1 libidn2-0 libpam0g libunistring2
Suggested packages:
glibc-doc libc-l10n locales libnss-nis libnss-nisplus libpam-doc
Recommended packages:
manpages

The following NEW packages will be installed:

gcc-11-base libcrypt1 libgcc-s1 libidn2-0 libunistring2
The following packages will be upgraded:
libc-bin libc6 libpam0g
3 upgraded, 5 newly installed, 0 to remove and 94 not upgraded.
Need to get 4590 kB of archives.
After this operation, 5108 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian sid/main amd64 gcc-11-base amd64
11.2.0-4 [205 kB]
Get:2 http://deb.debian.org/debian sid/main amd64 libgcc-s1 amd64
11.2.0-4 [41.7 kB]
Get:3 http://deb.debian.org/debian sid/main amd64 libc6 amd64 2.32-2 [2817 kB]
Get:4 http://deb.debian.org/debian sid/main amd64 libc-bin amd64 2.32-2 [829 kB]
Get:5 http://deb.debian.org/debian sid/main amd64 libpam0g amd64
1.4.0-10 [130 kB]
Get:6 http://deb.debian.org/debian sid/main amd64 libcrypt1 amd64
1:4.4.25-2 [89.2 kB]
Get:7 http://deb.debian.org/debian sid/main amd64 libunistring2 amd64
0.9.10-6 [384 kB]
Get:8 http://deb.debian.org/debian sid/main amd64 libidn2-0 amd64
2.3.2-2 [95.3 kB]
Fetched 4590 kB in 1s (3459 kB/s)
debconf: delaying package configuration, since apt-utils is not installed
Selecting previously unselected package gcc-11-base:amd64.
(Reading database ... 9179 files and directories currently installed.)
Preparing to unpack .../gcc-11-base_11.2.0-4_amd64.deb ...
Unpacking gcc-11-base:amd64 (11.2.0-4) ...
Setting up gcc-11-base:amd64 (11.2.0-4) ...
Selecting previously unselected package libgcc-s1:amd64.
(Reading database ... 9184 files and directories currently installed.)
Preparing to unpack .../libgcc-s1_11.2.0-4_amd64.deb ...
Unpacking libgcc-s1:amd64 (11.2.0-4) ...
Replacing files in old package libgcc1:amd64 (1:6.3.0-18+deb9u1) ...
Setting up libgcc-s1:amd64 (11.2.0-4) ...
(Reading database ... 9186 files and directories currently installed.)
Preparing to unpack .../libc6_2.32-2_amd64.deb ...
debconf: unable to initialize frontend: Dialog
debconf: (No usable dialog-like program is installed, so the dialog
based frontend cannot be used. at
/usr/share/perl5/Debconf/FrontEnd/Dialog.pm line 76.)
debconf: falling back to frontend: Readline

Checking for services that may need to be restarted...
Checking init scripts...

Unpacking libc6:amd64 (2.32-2) over (2.24-11+deb9u4) ...
Setting up libc6:amd64 (2.32-2) ...

[Otto Kekäläinen]

Hello!

Seems this libcrypt1 change not only affects upgrades from Stretch,
but in some cases even upgrades from Buster:
- https://salsa.debian.org/mariadb-team/galera-4/-/jobs/2008098
- https://salsa.debian.org/mariadb-team/galera-4/-/jobs/2008103

I wonder if there is some workaround? Could we somehow force the new
librcypt1 to install before libc upgrades?

[Marco d'Itri]

On Sep 27, Otto Kekäläinen <ot...@debian.org> wrote:

> Seems this libcrypt1 change not only affects upgrades from Stretch,
> but in some cases even upgrades from Buster:

Buster to Bookworm still means skipping a release.

--
ciao,
Marco

[Otto Kekäläinen]

Yeah, but one should be able to skip at least one release, right? What
is the point with e.g. Debian LTS if you can't upgrade from the Debian
LTS version to the newest stable?

Could you please consider if the libcrypt1 change was done in a way so
that at least Buster to Bookworm upgrades would work?

[Marco d'Itri]

On Oct 10, Otto Kekäläinen <ot...@debian.org> wrote:

> Yeah, but one should be able to skip at least one release, right? What

No.

> Could you please consider if the libcrypt1 change was done in a way so
> that at least Buster to Bookworm upgrades would work?

This has already been discussed: feel free to do the work required by
the glibc maintainer.

--
ciao,
Marco

[Thomas Braun]

I just upgraded from buster to bullseye (so no jumping releases) and encountered this bug as well.

Thanks for Otto for the workaround. This worked without issues.

[David Wagner]

Otto's workaround also resolved the issue for me, thanks Otto!

I do not know what caused the issue as I did not upgrade, but some details below just in case it is helpful for anybody:

I'm on Debian 10. Did not upgrade. Last I remember before issue was adding some cryptographic libraries with pip/conda and following instructions here [0] to add to apt sources to upgrade system Python. Found out because could not SSH into GCP instance next day after restart. sshd was failing on start finding libcrypto. Tried adding apt commands to startup script for VM and using chpasswd to add temporary password for access through the serial console, but both of those commands similarly failed without libcrypto. Adding Otto's fixes to the VM's startup script got SSH working again. 

[0] - https://www.linode.com/docs/guides/how-to-install-python-on-debian-10/#how-to-upgrade-from-python-37-to-39

[Otto Kekäläinen]

There seems to be a lot of people hitting this:

* https://forums.debian.net/viewtopic.php?f=17&t=152419
* https://www.reddit.com/r/debian/comments/qcwpfr/sudo_is_completely_broken_after_i_accidentally/

Thus I am posting this to clarify things:

Release notes of Debian 11 (bullseye)
https://www.debian.org/releases/stable/amd64/release-notes/ch-upgrading.en.html#upgrade-to-debian-oldrelease

> 4.2.1. Upgrade to Debian 10 (buster)
> Direct upgrades from Debian releases older than 10 (buster) are not supported. Display your Debian version with:
> $ cat /etc/debian_version
> Please follow the instructions in the Release Notes for Debian 10 to upgrade to Debian 10 first.

This means that people running Debian 9 (stretch) can't upgrade to
Debian 11 (bullseye). If they attempt to do so, they will run into
this libcrypt.1 issue.

Also people running Debian 10 (buster) can't upgrade to upcoming
Debian 12 (bookworm). . If they attempt to do so, they will also run
into this libcrypt.1 issue.


These are the steps I use in my CI scripts to automate upgrade from
Stretch to Sid or Buster to Sid:

.test-enable-sid-repos: &test-enable-sid-repos

# Replace any old repos with just Sid

- echo 'deb http://deb.debian.org/debian sid main' > /etc/apt/sources.list

# Upgrade minimal stack first

- apt-get update
# Next step will fail on https://bugs.debian.org/993755
# /usr/bin/perl: error while loading shared libraries: libcrypt.so.1: cannot
# open shared object file: No such file or directory
# dpkg: error processing package libc6:amd64 (--configure):
- apt-get install -y apt || true
# Apply workaround
- cd $(mktemp -d)
- apt -y download libcrypt1
- dpkg-deb -x libcrypt1_*.deb .
- cp -ra lib/* /lib/
- find /lib/*/libcrypt.* -ls
- apt -y --fix-broken install
# Complete upgrade of minimal stack
- apt-get install -y apt


Live example at
https://salsa.debian.org/mariadb-team/mariadb-server/-/merge_requests/22