Bug#1036821: NTP does not keep accurate time on bookworm

[Rob Janssen]

Package: ntpsec
Version: 1.2.2+dfsg1-1

I upgraded a system running on VMware from bullseye to bookworm.
It had a simple NTP setup with two local servers (no pool servers).
Before the upgrade, all was OK.

After the upgrade, the system does not keep accurate time.  It is synced,
but it hovers at a ~1300 msec offset from the servers.
I have rebooted the system and restarted ntpd, left it running for
several days, but problem remains.

On another system that runs on a physical machine, all is OK.
No timesync is configured on VMware.

I removed ntp+ntpsec and installed chrony.  Timesync is OK now.

[Rob Janssen]

status before I removed ntp:

root@**-video:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset   jitter
===============================================================================
+router.**.**    216.239.35.4     2 u   50   64  377   0.3705 -1604.11   1.2697
+**-linux.**.**  35.73.197.144    2 u   44   64  377   0.4070 -1604.26   1.2953

root@**-video:~# ntpq -p
     remote           refid      st t when poll reach   delay   offset   jitter
===============================================================================
+router.**.**    216.239.35.4     2 u   51   64  377   0.3606 -1623.06   1.2701
+**-linux.**.**  44.137.41.102    2 u   34   64  377   0.4630 -1623.29   1.2684

[Richard Laager]

Since you've moved to chrony, this is probably moot for you. But in case
it affects anyone else, I forwarded this upstream:
https://gitlab.com/NTPsec/ntpsec/-/issues/790

Can you confirm you were running the "ntp" package on bullseye, not
"ntpsec"?

--
Richard

[Richard Laager]

The answer is so obvious as soon as someone said it!

The default "minsane" is "3" (see "tos minclock 4 minsane 3" in
/etc/ntpsec/ntp.conf). Try commenting out that line, or if that doesn't
work, set both to "1".

--
Richard

[Rob Janssen]

Yes I was using the "ntp" package before.
I have upgraded and it installed "ntpsec".  I tried to remove it as I have no need
for the "security" part but it removed "ntp" as well.  It looks like "ntp" is only a dummy package.

I am in the process of upgrading a couple of systems so I will likely encounter this again and
then will try to remove the "tos minclock 4 minsane 3".

Please don't fall in the common trap of trying to make everything "top secure" and then making it
unusable or causing problems for people that do not require that.  Probably you should put that
config line commented in the default config so people who like it can easily enable it.

Rob

[Richard Laager]

On 2023-06-07 02:37, Rob Janssen wrote:
> Yes I was using the "ntp" package before.
> I have upgraded and it installed "ntpsec".  I tried to remove it as I have no need
> for the "security" part but it removed "ntp" as well.

And then you presumably reinstalled it. Did this result in you starting
over with a default ntp.conf, where you then manually removed (or
commented out) the pool lines and added your server lines?

> Please don't fall in the common trap of trying to make everything "top secure" and then making it
> unusable or causing problems for people that do not require that.

NTPsec is a fork of NTP. Most of the security benefit of NTPsec comes
from NTPsec simply removing and cleaning up decades of code cruft in
NTP. NTPsec is a drop-in replacement for NTP.

> It looks like "ntp" is only a dummy package.

In Debian, NTPsec was packaged alongside NTP for some time. This release
cycle, the Debian ntp maintainer suggested it was time to retire ntp,
and the consensus was to do so and migrate existing ntp installs to ntpsec:
https://lists.debian.org/debian-devel/2022/01/msg00172.html

> Probably you should put that
> config line commented in the default config so people who like it can
> easily enable it.

This configuration exists for correctness. If a given system has two
time sources and they disagree, which one is correct? There is no way to
be sure. If you have three sources, then you take whichever two agree.

"A man with a watch knows what time it is. A man with two watches is
never sure."
https://en.wikipedia.org/wiki/Segal%27s_law

If you're only running your own servers, then the best practice is to
run 3 (or more) servers. (Some sources say 4, so if one server is down,
you can still detect a falseticker.) And I say that as someone who runs
two. But my clients use my two servers plus the pool.

https://access.redhat.com/solutions/58025
https://www.tenable.com/audits/items/CIS_Cisco_NX-OS-v1.0.0_Level_2.audit:6a5be86b59dc9342bd22dfc2b7c70cb4
https://insights.sei.cmu.edu/blog/best-practices-for-ntp-services/
https://labs.ripe.net/author/christer-weinigel/best-practices-for-connecting-to-ntp-servers/

--
Richard

[Rob Janssen]

On 6/7/23 10:13, Richard Laager wrote:
> On 2023-06-07 02:37, Rob Janssen wrote:
>> Yes I was using the "ntp" package before.
>> I have upgraded and it installed "ntpsec".  I tried to remove it as I have no need
>> for the "security" part but it removed "ntp" as well.
>
> And then you presumably reinstalled it. Did this result in you starting over with a default ntp.conf, where you then manually removed (or commented out) the pool lines and added your server lines?

No, then I removed everything and installed chrony.  That resolved the problem so then I made a bugreport.

>
>> Please don't fall in the common trap of trying to make everything "top secure" and then making it
>> unusable or causing problems for people that do not require that.
> NTPsec is a fork of NTP. Most of the security benefit of NTPsec comes from NTPsec simply removing and cleaning up decades of code cruft in NTP. NTPsec is a drop-in replacement for NTP.

Except that it isn't.  Or at least the default configuration isn't.

>
> > Probably you should put that
> > config line commented in the default config so people who like it can
> > easily enable it.
>
> This configuration exists for correctness. If a given system has two time sources and they disagree, which one is correct? There is no way to be sure. If you have three sources, then you take whichever two agree.

In my opinion it is not good to enforce such policy on the users of the package.
I know very well how NTP works and what issues there may be, but indeed the NTP servers are local and I deem them
sufficiently reliable FOR MY PURPOSE.
It worked fine on bullseye, it failed on upgrade to bookworm.
And the config line that is responsible for the problem has a comment that does not indicate at all that you want to
remove it when you have fewer than 3 servers.  Maybe change that, I would have noticed it when I reviewed the config diffs.

I originally commented that it works ok on another machine and believed it may be due to the VMware/Physical
difference, but that wasn't the cause: that other machine was on another network and happend to have 3 servers configured.
But I commented that line now (I do not want time sync to fail because one of the servers is unavailable!)

[Richard Laager]

On 2023-06-07 03:22, Rob Janssen wrote:
> On 6/7/23 10:13, Richard Laager wrote:
>> On 2023-06-07 02:37, Rob Janssen wrote:
>>> Yes I was using the "ntp" package before.
>>> I have upgraded and it installed "ntpsec".  I tried to remove it as I have no need
>>> for the "security" part but it removed "ntp" as well.
>>
>> And then you presumably reinstalled it. Did this result in you starting over with a default ntp.conf, where you then manually removed (or commented out) the pool lines and added your server lines?
>
> No, then I removed everything and installed chrony.

I thought the sequences of events was this:

0. You are running ntp on bullseye.
1. You upgrade to bookworm. This results in ntpsec being installed.
2. You removed ntpsec.
3. [The part I was asking about.] You reinstalled ntpsec.
4. You found that ntpd was not syncing the clock.
5. You switched to chrony.

Was it this instead?

0. You are running ntp on bullseye.
1. You upgrade to bookworm. This results in ntpsec being installed.
2. You found that ntpd was not syncing the clock.
3. You removed ntpsec.
4. You switched to chrony.

I'm trying to understand what happened with your ntp.conf. Upgrading
from ntp to ntpsec should result in your existing /etc/ntp.conf being
copied to /etc/ntpsec/ntp.conf by ntpsec.preinst.

--
Richard

[Rob Janssen]

On 6/7/23 10:54, Richard Laager wrote:
>
> I thought the sequences of events was this:
>
> 0. You are running ntp on bullseye.
> 1. You upgrade to bookworm. This results in ntpsec being installed.
> 2. You removed ntpsec.
> 3. [The part I was asking about.] You reinstalled ntpsec.
> 4. You found that ntpd was not syncing the clock.
> 5. You switched to chrony.
>
> Was it this instead?
>
> 0. You are running ntp on bullseye.
> 1. You upgrade to bookworm. This results in ntpsec being installed.
> 2. You found that ntpd was not syncing the clock.
> 3. You removed ntpsec.
> 4. You switched to chrony.
>
> I'm trying to understand what happened with your ntp.conf. Upgrading from ntp to ntpsec should result in your existing /etc/ntp.conf being copied to /etc/ntpsec/ntp.conf by ntpsec.preinst.
>

It was the second scenario.
As usual during upgrade I got those messages "you changed this config file, do you want
to keep it or use the maintainers version", then I normally select "keep" every time (the default)
and afterwards I do a "find / -name '*.dpkg-*'" to find all the new config files, review the diffs, and modify
the new version with the same changes I made to the previous one.
(in this case to remove the pool servers and add my own)

So it was basically running with default config, except that I added two of my own servers
and removed the pool servers.
I do it this way to avoid situations where new default files are better and/or required with
new packages, and I do not want to keep my old config all the time.

[Richard Laager]

I've made a suggestion upstream of how we could do better here by making
this either a fatal error or at least a warning. If it was a fatal error
with a good error message, you would have figured it out immediately.
Let's see what people think of that.

--
Richard