Bug#993821: After upgrading libc, some services are unable to restart (including systemd-resolved)

[Vincent Bernat]

Package: systemd
Version: 247.9-1
Severity: normal

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hey!

After upgrading to libc6 2.32-1, some services are unable to restart.
In my case, systemd-resolved, systemd-timesyncd and colord. Using
"systemctl daemon-reexec" fixes the issue. Unsure if there is really
something to be fixed but as I didn't find anything about that, a bug
report may help others. I suppose the problem is related to NSS.

Sep 06 23:06:43 chocobo systemd[1]: Starting Network Time Synchronization...
Sep 06 23:06:43 chocobo systemd[236983]: systemd-timesyncd.service: Failed to determine user credentials: No such process
Sep 06 23:06:43 chocobo systemd[236983]: systemd-timesyncd.service: Failed at step USER spawning /lib/systemd/systemd-timesyncd: No such process



- -- Package-specific info:

- -- System Information:
Debian Release: bookworm/sid
APT prefers unstable-debug
APT policy: (500, 'unstable-debug'), (500, 'unstable'), (500, 'testing'), (101, 'experimental-debug'), (101, 'experimental')
Architecture: amd64 (x86_64)

Kernel: Linux 5.13.0-trunk-amd64 (SMP w/4 CPU threads)
Kernel taint flags: TAINT_USER
Locale: LANG=C.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages systemd depends on:
ii adduser 3.118
ii libacl1 2.3.1-1
ii libapparmor1 3.0.3-2
ii libaudit1 1:3.0.5-1
ii libblkid1 2.37.2-2
ii libc6 2.32-1
ii libcap2 1:2.44-1
ii libcrypt1 1:4.4.25-2
ii libcryptsetup12 2:2.4.0-1
ii libgcrypt20 1.9.4-2
ii libgnutls30 3.7.2-2
ii libgpg-error0 1.42-3
ii libip4tc2 1.8.7-1
ii libkmod2 29-1
ii liblz4-1 1.9.3-2
ii liblzma5 5.2.5-2
ii libmount1 2.37.2-2
ii libpam0g 1.4.0-10
ii libseccomp2 2.5.1-1
ii libselinux1 3.1-3
ii libsystemd0 247.9-1
ii libzstd1 1.4.8+dfsg-2.1
ii mount 2.37.2-2
ii systemd-timesyncd [time-daemon] 247.9-1
ii util-linux 2.37.2-2

Versions of packages systemd recommends:
ii dbus 1.12.20-2

Versions of packages systemd suggests:
ii policykit-1 0.105-31
ii systemd-container 247.9-1

Versions of packages systemd is related to:
pn dracut <none>
ii initramfs-tools 0.140
ii libnss-systemd 247.9-1
ii libpam-systemd 247.9-1
ii udev 247.9-1

- -- Configuration Files:
/etc/systemd/logind.conf changed:
[Login]
HandlePowerKey=ignore

/etc/systemd/resolved.conf changed:
[Resolve]
DNSSEC=allow-downgrade


- -- no debconf information

-----BEGIN PGP SIGNATURE-----

iQJGBAEBCAAwFiEErvI0h2bzccaJpzYAlaQv6DU1JfkFAmE2jA8SHGJlcm5hdEBk
ZWJpYW4ub3JnAAoJEJWkL+g1NSX5HXcQAKG5+toLrAqmXBjrFCHUauoUJ1MKEXYp
gMk11uNJd61yY0gW7eQHNJ1bl1aeXQ5FBOw830xMVFn9qXcCAghrDcB8yT8Vlsw9
XQQakme/+qUNU6xg4RW9IRbrqH32AhV1hp4rkrchjVYpoLng26JOV7zKSs7PrhL/
THtVzuxRnqgyQ0j742yDmw5X6y/jqBIyOgdVWm176kYIaIvkob8i8YzF8eSjQCgq
0PEDVwtbkDUu8M79lA2QPTya+3y9xD/vp01/hbyMA7+lOfQKC5ylX5rklsMhsWez
mLwRBj3UYGGYm6jRWwYRbciSCpYLxsz0RVz6+T8FStyGqh3vNBAWFQHYn47QtypU
2t4JGgJV+Z6yDeY2eh/ltk2kk+yhsMJtDzEoY7unsG4fWa8MoxgdKeIwbQ3Hujir
uyQMD170q5/AkPWsmKeJm2OdBF9nRs8dGU+VGi80XZzld0Ociu0tcUQu9F9LbrNj
5HdvUlkY6rkW0OsTpDBjKqVyB/DMHT9ciS5o9a/C6ZwRMspyItxKrxtN+9M2VIN3
5BHv/2vuZfh5OqDCtlVVwEhj2YZUVlEmtIJ+UfA7VfupXHwNiCvI8QOWkKcOvkMJ
1FurFmmJ3WED/2qLfr25GhZgyUGDb/3NluuHHnwRIkpCBYyV4qOSnAd/z4SUa+wv
p2k968FNOmUz
=jj8P
-----END PGP SIGNATURE-----

[Michael Biebl]

Am 06.09.21 um 23:45 schrieb Vincent Bernat:

> Package: systemd
> Version: 247.9-1
> Severity: normal
>

> Hey!
>
> After upgrading to libc6 2.32-1, some services are unable to restart.
> In my case, systemd-resolved, systemd-timesyncd and colord. Using
> "systemctl daemon-reexec" fixes the issue. Unsure if there is really
> something to be fixed but as I didn't find anything about that, a bug
> report may help others. I suppose the problem is related to NSS.
>
> Sep 06 23:06:43 chocobo systemd[1]: Starting Network Time Synchronization...
> Sep 06 23:06:43 chocobo systemd[236983]: systemd-timesyncd.service: Failed to determine user credentials: No such process
> Sep 06 23:06:43 chocobo systemd[236983]: systemd-timesyncd.service: Failed at step USER spawning /lib/systemd/systemd-timesyncd: No such process
>
>

@libc maintainers: any ideas what could be causing this? If this is
triggered by a libc6 update, should this be reassigned to glibc?

[Michael Hudson-Doyle]

We went through this in Ubuntu recently and decided that restarting systemd in glibc's postinst was the safest option: https://bugs.launchpad.net/ubuntu/+source/glibc/+bug/1942276

What's happening is that systemd is running with the old glibc, forks and then does NSS things that cause the new glibc's NSS modules to load and they don't necessarily work, leading to failures in any unit that specifies User=. At least for Ubuntu's builds the NSS modules seem to be ABI compatible between 2.32 and 2.33 (I didn't try 2.31 vs 2.32) but they are definitely not between 2.33 and 2.34.

Cheers,

mwh 

[Michael Biebl]

Control: reassign -1 libc6
Control: found -1 2.32-1
Control: severity -1 serious
Control: affects -1 + systemd

Hi Michael

Am 07.09.21 um 00:39 schrieb Michael Hudson-Doyle:

> On Tue, 7 Sept 2021 at 10:21, Michael Biebl <bi...@debian.org

Thanks for this information. This is indeed an icky issue and I feel
like we are between a rock and a hard place.
I'm not a huge fan of going back to re-exec systemd again directly in
libc6.postinst, but your proposed patch to at least check that the
systemd binary can be sucessfully executed should at least deal with the
situation sufficiently, where a library is (temporarily) missing.
I do wonder though, if this this will mean that on dist-upgrades the
daemon-reexec will be skipped.

Anyway, I think it's best to reassign this libc6 for now and mark it as
RC so the package doesn't migrate to testing for now.

Regards,
Michael

[Michael Hudson-Doyle]

Yeah. I guess one could say that having a long running process that forks and then does NSS stuff is skating on thin ice a bit. At least the changes in glibc 2.34 to move nss_files functionality into glibc itself will reduce the fallout of this considerably.

 

I'm not a huge fan of going back to re-exec systemd again directly in
libc6.postinst, but your proposed patch to at least check that the
systemd binary can be sucessfully executed should at least deal with the
situation sufficiently, where a library is (temporarily) missing.
I do wonder though, if this this will mean that on dist-upgrades the
daemon-reexec will be skipped.

FWIW I had a long chat with Julian (the apt maintainer) about this and he thought there were three potential situations that could be a problem:

1) a new systemd is unpacked before its Depends
2) one of systemd dependencies has a Breaks: systemd (<< new)

3) in some cases a cycle has to be broken by removing a package with --force-deps

It think 1) is by some margin the most likely to actually happen, and at least in that situation systemd will be restarted shortly by its own postinst.

Cheers,

Michael

[Michael Biebl]

Hi Aurelien

Am 07.09.21 um 12:41 schrieb Aurelien Jarno:
> Hi,

>
> On 2021-09-07 10:39, Michael Hudson-Doyle wrote:

>> What's happening is that systemd is running with the old glibc, forks and
>> then does NSS things that cause the new glibc's NSS modules to load and
>> they don't necessarily work, leading to failures in any unit that specifies
>> User=. At least for Ubuntu's builds the NSS modules seem to be ABI
>> compatible between 2.32 and 2.33 (I didn't try 2.31 vs 2.32) but they are
>> definitely not between 2.33 and 2.34.
>

> Thanks for this feedback and the pointer to the patch used in Ubuntu. It
> seems to be a good solution, and matches what is done for other init
> systems.
>
> On the other hand, the problem is supposed to only happen for major
> glibc version upgrade where the NSS modules might have a different ABI.
> In that regard, I would be tempted to restart it only for major versions
> upgrade like it's done for other daemons. Now if the systemd maintainers
> consider it's fine restarting it for each glibc upgrade, we should
> probably go that way.

I guess you are in a better position to make a judgement call here. If I
read the glibc bug report correctly, there aren't strictly any
guarantees regarding NSS modules. What that means for glibc minor
updates, I'm not really in a position to tell.

Fwiw, I don't have a better proposal then Michael's patch he added to
Ubuntu. We could run with that and if it causes problems, reiterate on it.

Regards,
Michael