Bug#993947: Time lost, /etc/systemd/timesyncd.conf gets replaced with a default one

[Santiago Garcia Mantinan]

Package: systemd-timesyncd
Version: 247.3-6
Severity: important

Hi!

On upgrade from Buster to Bullseye I have seen my machines, without Internet
ntp access, loose their time sync because of the upgrade.

On the upgrade to the new packages, the old /etc/systemd/timesyncd.conf file
which specifies the right ntp servers to use, will get replaced with a default
one which will not work for a setup where Internet ntp access is not available.

Luckily the old file is left as /etc/systemd/timesyncd.conf.dpkg-bak, so a
mv /etc/systemd/timesyncd.conf.dpkg-bak /etc/systemd/timesyncd.conf
and proper reloading will get things back to normal.

I don't know why this file is being replaced, this happened on all the
machines that had a modified timesyncd.conf file and the system never asked
if I wanted to override the setup or leave the old one, no choice was given,
it was just replaced and that's it.

If you feel that this is the right behaviour, I believe that we should at
least add a note on the Release Notes or similar.

Regards.


-- System Information:
Debian Release: 11.0
APT prefers stable-security
APT policy: (990, 'stable-security'), (990, 'stable'), (500, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 5.10.0-8-amd64 (SMP w/2 CPU threads)
Locale: LANG=gl_ES.UTF-8, LC_CTYPE=gl_ES.UTF-8 (charmap=UTF-8), LANGUAGE not set
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages systemd-timesyncd depends on:
ii adduser 3.118
ii libc6 2.31-13
ii systemd 247.3-6

systemd-timesyncd recommends no packages.

systemd-timesyncd suggests no packages.

-- no debconf information

[Michael Biebl]

Am 08.09.21 um 14:04 schrieb Santiago Garcia Mantinan:

> Package: systemd-timesyncd
> Version: 247.3-6
> Severity: important
>
> Hi!
>
> On upgrade from Buster to Bullseye I have seen my machines, without Internet
> ntp access, loose their time sync because of the upgrade.
>
> On the upgrade to the new packages, the old /etc/systemd/timesyncd.conf file
> which specifies the right ntp servers to use, will get replaced with a default
> one which will not work for a setup where Internet ntp access is not available.
>
> Luckily the old file is left as /etc/systemd/timesyncd.conf.dpkg-bak, so a
> mv /etc/systemd/timesyncd.conf.dpkg-bak /etc/systemd/timesyncd.conf
> and proper reloading will get things back to normal.
>
> I don't know why this file is being replaced, this happened on all the
> machines that had a modified timesyncd.conf file and the system never asked
> if I wanted to override the setup or leave the old one, no choice was given,
> it was just replaced and that's it.

systemd-timesyncd was split into a separate binary package in bullseye.
Transferring the ownership of the conffile from systemd to
systemd-timesyncd is a tricky business as dpkg does not have native
support for that and so we need to go behind dpkg's back for that.

We do have some custom maintainer scripts code where we try to preserve
local modifications:
https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/systemd-timesyncd.postinst

Apparently this does not work properly.

Regards,
Michael

[Michael Biebl]

Am 08.09.21 um 19:25 schrieb Michael Biebl:

> We do have some custom maintainer scripts code where we try to preserve
> local modifications:
> https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/systemd-timesyncd.postinst
>
>
> Apparently this does not work properly.

Needless to say, this is a bug and needs investigation.
I don't have an idea yet, why it fails.

[Michael Biebl]

Hi Felipe

Am 08.09.21 um 19:25 schrieb Michael Biebl:

> systemd-timesyncd was split into a separate binary package in bullseye.
> Transferring the ownership of the conffile from systemd to
> systemd-timesyncd is a tricky business as dpkg does not have native
> support for that and so we need to go behind dpkg's back for that.
>
> We do have some custom maintainer scripts code where we try to preserve
> local modifications:
> https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/systemd-timesyncd.postinst

This code is based on
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=797048
i.e.
https://salsa.debian.org/systemd-team/systemd/commit/f5f5028055f25fa733a45265e7c008e06960e0a7

A typescript log from an upgrade is attached.
One can see, that systemd-timesyncd.postinst is run before
systemd.postinst and as a result timesyncd.conf.dpkg-bak does not exist yet.

Afaics, this could only be fixed with a versioned Pre-Depends on systemd
(>= 245.4-2). I don't think a Depends gives us any guarantees regarding
the order postinst is run?

[Santiago Garcia Mantinan]

Hi again!

I have just migrated another machine and I found out that it worked out
well.

The thing is exactly what you have descrived, here the order was (luckily)
the right one, so when systemd-timesyncd is configured, systemd had already
created the dpkg-bak file like we can see here:

Configurando systemd (247.3-6) ...M
Instalando una nueva versión del fichero de configuración /etc/systemd/journald.conf ...M
Instalando una nueva versión del fichero de configuración /etc/systemd/logind.conf ...M
Instalando una nueva versión del fichero de configuración /etc/systemd/networkd.conf ...M
Instalando una nueva versión del fichero de configuración /etc/systemd/resolved.conf ...M
Instalando una nueva versión del fichero de configuración /etc/systemd/system.conf ...M
Instalando una nueva versión del fichero de configuración /etc/systemd/user.conf ...M
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-pstore.service → /lib/systemd/system/systemd-pstore.service.M
Removing obsolete conffile /etc/dhcp/dhclient-exit-hooks.d/timesyncd ...M
Obsolete conffile /etc/systemd/timesyncd.conf has been modified by you.M
Saving as /etc/systemd/timesyncd.conf.dpkg-bak ...M
Removing obsolete conffile /etc/pam.d/systemd-user ...M
Configurando libmono-system-core4.0-cil (6.8.0.105+dfsg-3.2) ...M
Configurando systemd-timesyncd (247.3-6) ...M
Created symlink /etc/systemd/system/dbus-org.freedesktop.timesync1.service → /lib/systemd/system/systemd-timesyncd.service.M
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-timesyncd.service → /lib/systemd/system/systemd-timesyncd.service.M
Configurando libmono-system-security4.0-cil (6.8.0.105+dfsg-3.2) ...M

Regards...
--
Manty/BestiaTester -> http://manty.net

[Felipe Sateler]

Hi,

It should give us the guarantees[1]:

> The postinst script may be called in the following ways:
> postinst configure most-recently-configured-version
>   The files contained in the package will be unpacked. 

>   All package dependencies will at least be “Unpacked”. 

>   If there are no circular dependencies involved, 

>   all package dependencies will be configured

AFAICS we don't have circular dependencies, but maybe the versioned breaks/replaces + versioned depends makes dpkg think there is one?

[1]https://www.debian.org/doc/debian-policy/ch-maintainerscripts.html#summary-of-ways-maintainer-scripts-are-called 

--

Saludos,
Felipe Sateler

[Michael Biebl]

Am 09.09.21 um 15:15 schrieb Felipe Sateler:

> It should give us the guarantees[1]:
>
> > The postinst script may be called in the following ways:
> > postinst configure most-recently-configured-version
> >   The files contained in the package will be unpacked.
> >   All package dependencies will at least be “Unpacked”.
> >   If there are no circular dependencies involved,
> >   all package dependencies will be configured
>
> AFAICS we don't have circular dependencies, but maybe the versioned
> breaks/replaces + versioned depends makes dpkg think there is one?

Hm, we do have systemd -> systemd-timesyncd | time-daemon and
systemd-timesyncd -> systemd

This is a circular dep afaiu.

That said, keep in mind that we don't have a versioned systemd dep
(yet), only an unversioned one [1].
So maybe turning this into a versioned dep is "good enough" and should
work in most cases.
If possible, I think we should get this into the first stable release,
as we can't really fix this for good, only for users who haven't
upgraded yet.

Regards,
Michael

[1]
https://salsa.debian.org/systemd-team/systemd/-/blob/debian/master/debian/control#L192

[Michael Biebl]

Am 09.09.21 um 16:15 schrieb Michael Biebl:

> That said, keep in mind that we don't have a versioned systemd dep
> (yet), only an unversioned one [1].

Actually, what I said is not true. We do get a strict, versioned Depends
via shlibs.local as systemd-timesyncd links against libsystemd-shared


$ cat debian/shlibs.local.in
udeb: libudev 1 libudev1-udeb
libsystemd 0 libsystemd0 (= ${binary:Version})
libsystemd-shared SHARED_LIB_VERSION systemd (= ${binary:Version})

[Michael Biebl]

Am 09.09.21 um 16:15 schrieb Michael Biebl:

> Am 09.09.21 um 15:15 schrieb Felipe Sateler:
>> It should give us the guarantees[1]:
>>
>>  > The postinst script may be called in the following ways:
>>  > postinst configure most-recently-configured-version
>>  >   The files contained in the package will be unpacked.
>>  >   All package dependencies will at least be “Unpacked”.
>>  >   If there are no circular dependencies involved,
>>  >   all package dependencies will be configured
>>
>> AFAICS we don't have circular dependencies, but maybe the versioned
>> breaks/replaces + versioned depends makes dpkg think there is one?
>
> Hm, we do have systemd -> systemd-timesyncd | time-daemon and
>                systemd-timesyncd -> systemd
>
> This is a circular dep afaiu.
>

I guess the only way to break this dep cycle is to drop (or rather
demote) the Depends: systemd-timesyncd | time-daemon to a Recommends.

To ensure that systemd-timesyncd is installed during the initial
bootstrap, we'd have to bump it's prio, similar to what we did for
libpam-systemd. Either important or standard.

Related here:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=986651

If we can get such a change into bullseye remains to be seen.

Does anyone else have a different idea how to approach this?

Michael

[Felipe Sateler]

No, I don't have any other idea. Perhaps the installer or apt teams have any ideas?

--

Saludos,
Felipe Sateler