NTP community feels broken

[Phillip Hellewell]

I've found it impossible to contribute to the NTP reference implementation (with patches, bug reports, or anything).

I've also found it impossible to communicate this problem to the NTP authors.

It feels like they've closed their doors to any community involvement in this area. Maybe it's on purpose?

The github repo (https://github.com/ntp-project/ntp/) is outdated; it is not being synced with bitkeeper, even though the support.ntp.org website says they are "fully and automatically synchronized".

The issue created on github about it being outdated has been ignored since 2019: https://github.com/ntp-project/ntp/issues/1

The issue about it on bugs.ntp.org has been ignored since 2017: https://bugs.ntp.org/show_bug.cgi?id=3413

Creating an account on bugs.ntp.org results in "User account creation has been restricted." That happens with my gmail. Then I tried a different email and it "worked". Except it didn't really work because I am still unable to log in. I go through the motions, but I'm still not logged in.

Emails to webm...@ntp.org bounce with "Address not found" / "User unknown".

There are only 10 people on #ntp on freenode and no questions get answered there.

I see links to mailing lists (https://lists.ntp.org/), but I can't find instructions on how to join them.

Emails to specific individuals at ntp.org get ignored.

With this inability to communicate, I can't even get an answer to a simple question like, how/why is the ntp-dev branch older than ntp-stable in bitkeeper? Bug fixes make it into stable but not dev?

Speaking of bitkeeper, why are the NTP devs still using bitkeeper when everyone else in the world is using git?

There is tons of info on the various websites (www.ntp.org, bugs.ntp.org, support.ntp.org), but they are all completely different and archaic in their own unique ways, making navigation difficult.

Why is there no ability to contribute or even communicate with the NTP devs / authors? Why do README.hackers, README.patches, and README.pullrequests even exist if you can't actually follow those workflows?

I'm trying a post to this group as a last resort. But no one uses usenet these days either, so I fully expect this to be ignored.

Phillip

[chris]

This sort of thing has been going on for years. The system software is
written in go fwir, which is fine if you want a gig at google, but
it's obscurity means that maintenance is difficult, especially as the
original engineer no longer seems to be interested in the project. Also,
lack of full disclosure, documented, is a serious bar to anyone wishing
to contribute to the project. Perhaps that's intentional, but if so,
then say so. Candour being the best disinfectant etc.

I joined in the forum with discussions and suggestions, as have others,
but the lack of leadership and direction means that any progress has
stalled. I don't know what the answer is and perhaps some politics
is in the background, but various issues need to be dealt with.
Where so many depend on the service worldwide, there is a duty to
pass the baton on with full disclosure, if the originators need to
move on to other things, but I don't see any evidence of that...

Chris

[David Woolley]

On 17/06/2022 00:30, chris wrote:
> lack of full disclosure, documented

I'm having trouble understanding what this means. If you mean that the
documentation is poor, that is a common problem with open source
software, as it relies on volunteer effort, and programmers don't like
writing documentation.

Actually, in terms of end user documentation, I find most technology
poor. Businesses tend to document the small number of marketing claims,
in marketing language, and don't provide detailed functional descriptions.

For software, the only really good documentation is often the test plan,
but that is considered highly confidential for commercial software.
Open source coders are less likely to write formal test suites.

The original author of ntpd saw it as a maths problem, and was
frustrated by the inability of the RFC system to cope with mathematical
notation. The primary documentation, the version 4 RFC, is written as a
maths paper, but with the Greek letters spelled out.

ntpd will have been documented to the same sort of detail as an
experimental rig in an academic paper. The main detail would have gone
into the particualr property of the core algorithm that the paper was
trying to investigate.

It may also be significant that its primary developer is now 84.

Although I say lack of documentation is a problem for all technology,
what I sometimes find out with hardware is that it is all based on a
small number of special purpose ICs, and if you can establish what is
being used you can get a long way towards a real specification, rather
than the half page marketing hype on Amazon, by looking at the 100 page
data sheet for the ICs. Semiconductors seem to be an area where
detailed end user documentation is still available in the public domain.

It is common for the consumer products to be more or less direct
implementations of the typical application circuits, from the IC data
sheet. This doesn't work so well with software, as every user can end
up customising its use to a level that only the final manufacturer would
do for hardware, and they are more prepared to do so than the people who
sell products on Amazon.

[chris]

No problem with ntp client / server softfware which seems to work very
well. Getting close to a year's continuous uptime for the server here,
though it is on a ups, as are the internet facing switches and routers
in the path.

The problem is the monitoring software, which uses a fairly crude
algorithm to determine if a server is up. Had long discussions in
the forum about a year ago about this. Because of the obscure
implementation language, lack of comments in the code, lack of
documentation in terms of overall system design and finally, the
original implementer has moved on to other things and seems to have
no further interest in the project. That makes it almost impossible to
add improvements to the system. In summary, it's a hacked effort, not
a software engineered solution.

Probably the only long solution is to rewrite the lot in C or C++,
but it would take leadership from the center to make that happen
and organise it...

Chris

[Paul G]

On Friday, June 17, 2022 at 9:37:15 AM UTC-4, chris wrote:
> The problem is the monitoring software

What software product/program do you mean?

[chris]

It's the code that polls ntp servers to verify that they are up.
The website itself seems to work fine. Was quite interested in
contributing to the project at the time, as have been many
others. Did some work in collaboration using tcpdump and logfiles,
but nothing more after that...

Regards,

Chris

[Garrett Wollman]

In article <t8i6bd$13fh$1...@gioia.aioe.org>,

chris <chris-...@tridac.net> wrote:
>On 06/17/22 15:48, Paul G wrote:
>> On Friday, June 17, 2022 at 9:37:15 AM UTC-4, chris wrote:
>>> The problem is the monitoring software
>>
>> What software product/program do you mean?
>
>It's the code that polls ntp servers to verify that they are up.

That's not narrowing it down at all.

-GAWollman

--
Garrett A. Wollman | "Act to avoid constraining the future; if you can,
wol...@bimajority.org| act to remove constraint from the future. This is
Opinions not shared by| a thing you can do, are able to do, to do together."
my employers. | - Graydon Saunders, _A Succession of Bad Days_ (2015)

[chris]

On 06/17/22 17:58, Garrett Wollman wrote:
> In article<t8i6bd$13fh$1...@gioia.aioe.org>,
> chris<chris-...@tridac.net> wrote:
>> On 06/17/22 15:48, Paul G wrote:
>>> On Friday, June 17, 2022 at 9:37:15 AM UTC-4, chris wrote:
>>>> The problem is the monitoring software
>>>
>>> What software product/program do you mean?
>>
>> It's the code that polls ntp servers to verify that they are up.
>
> That's not narrowing it down at all.
>
> -GAWollman
>

How much more specific can I be ?. Somewhere there is a module that
does the monitoring and sends emails when a server can't be reached.

There are no docs on the internals, so how is it possible to say any
more ?. Perhaps you know better ?...

Chris

[Garrett Wollman]

In article <t8ie8u$hjr$1...@gioia.aioe.org>,

chris <chris-...@tridac.net> wrote:
>On 06/17/22 17:58, Garrett Wollman wrote:
>> In article<t8i6bd$13fh$1...@gioia.aioe.org>,
>> chris<chris-...@tridac.net> wrote:
>>> On 06/17/22 15:48, Paul G wrote:
>>>> On Friday, June 17, 2022 at 9:37:15 AM UTC-4, chris wrote:
>>>>> The problem is the monitoring software
>>>>
>>>> What software product/program do you mean?
>>>
>>> It's the code that polls ntp servers to verify that they are up.
>>
>> That's not narrowing it down at all.
>>
>> -GAWollman
>>
>
>How much more specific can I be ?. Somewhere there is a module that
>does the monitoring and sends emails when a server can't be reached.

Um, every site monitors its own NTP servers. We use Nagios for ours
but there are many other such products.

[chris]

On 06/17/22 18:46, Garrett Wollman wrote:
> In article<t8ie8u$hjr$1...@gioia.aioe.org>,

> chris<chris-...@tridac.net> wrote:
>> On 06/17/22 17:58, Garrett Wollman wrote:
>>> In article<t8i6bd$13fh$1...@gioia.aioe.org>,
>>> chris<chris-...@tridac.net> wrote:
>>>> On 06/17/22 15:48, Paul G wrote:
>>>>> On Friday, June 17, 2022 at 9:37:15 AM UTC-4, chris wrote:
>>>>>> The problem is the monitoring software
>>>>>
>>>>> What software product/program do you mean?
>>>>
>>>> It's the code that polls ntp servers to verify that they are up.
>>>
>>> That's not narrowing it down at all.
>>>
>>> -GAWollman
>>>
>>
>> How much more specific can I be ?. Somewhere there is a module that
>> does the monitoring and sends emails when a server can't be reached.
>
> Um, every site monitors its own NTP servers. We use Nagios for ours
> but there are many other such products.
>
> -GAWollman
>

Nothing to do with products. ntp.org has a monitoring system that polls
every server in its database to verify that it's reachable. It marks a
server bad if it can't be reached after a given number of retries. The
issue has been for years that monitoring false flags a server down, due
to ineffective monitoring. Much of the time it does work, but it could
be improved.

ntp is a volunteer effort via thousands of independent servers
worldwide, with coordination and management from ntp.org. RTFM ?...

Chris

[Paul G]

On Friday, June 17, 2022 at 11:24:31 AM UTC-4, chris wrote:
> It's the code that polls ntp servers to verify that they are up.

Where is it in this tarball: http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-4.2/ntp-4.2.8p15.tar.gz

If it's not there then you're probably in the wrong list/group.

[Paul G]

On Friday, June 17, 2022 at 2:12:52 PM UTC-4, chris wrote:

> Nothing to do with products. ntp.org has a monitoring system that polls
> every server in its database to verify that it's reachable.

Perhaps you mean pool.ntp.org. It's in the ntp.org namespace but it's a separate project run by Ask Bjørn Hansen.

[chris]

The ntp package works fine, that's the client server package,
nothing to do with the monitoring functions at ntp.org.

ntp.org runs a pool of remote ntp servers and dishes out requests
to them based on current assessment of available servers. Monitoring
continually polls the server list to make sure they are available...

Chris

[chris]

That's correct, but the various issues with the system have been
discussed for years, yet nothing ever gets done about it. That's the
point that Philip above was making...

Chris

[David Woolley]

On 17/06/2022 19:14, Paul G wrote:
> Where is it in this tarball:http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-4.2/ntp-4.2.8p15.tar.gz
>
> If it's not there then you're probably in the wrong list/group.

This group is about the NTP protocol, not just the version 4 reference
implementation. chrony, the SNTP client that Debian use, etc., are also
on topic. I would say that the management of the pool servers was
definitely on topic.

[Paul G]

On Friday, June 17, 2022 at 2:33:35 PM UTC-4, David Woolley wrote:
> On 17/06/2022 19:14, Paul G wrote:
> > Where is it in this tarball:http://www.eecis.udel.edu/~ntp/ntp_spool/ntp4/ntp-4.2/ntp-4.2.8p15.tar.gz
> >
> > If it's not there then you're probably in the wrong list/group.
> This group is about the NTP protocol

I said probably because other major projects have (or should have) their own discussion channels.

[Paul G]

On Friday, June 17, 2022 at 2:24:05 PM UTC-4, chris wrote:
> That's correct, but the various issues with the system have been
> discussed for years, yet nothing ever gets done about it. That's the
> point that Philip above was making...

Of course working with Harlan is difficult. Coming here to advise us of that won't result in any changes. Fortunately there are alternatives so there's no need to fret about the "reference" implementation.

The issue with your posts is that they were confusing. Or wrong.

[Harlan Stenn]

If you're talking about questions@ and maybe even c.p.t.n, I think "this
group" is about things related to the NTP Project, originally considered
to be at UDel (from the 80s until 2011), and (since 2011) now at Network
Time Foundation.

--
Harlan Stenn <st...@nwtime.org>
http://networktimefoundation.org - be a member!
--
This is ques...@lists.ntp.org
Subscribe: questions...@lists.ntp.org
Unsubscribe: questions+...@lists.ntp.org

[chris]

On 06/17/22 20:13, Paul G wrote:
> On Friday, June 17, 2022 at 2:24:05 PM UTC-4, chris wrote:
>> That's correct, but the various issues with the system have been
>> discussed for years, yet nothing ever gets done about it. That's the
>> point that Philip above was making...
>
> Of course working with Harlan is difficult. Coming here to advise us of that won't result in any changes. Fortunately there are alternatives so there's no need to fret about the "reference" implementation.
>

Well, we have been saying all along that the reference implementation
works well, but perhaps you missed that ?.

> The issue with your posts is that they were confusing. Or wrong.

If you say so, then there must be a reason, but are we on the same
conversation ?...

Chris

[Phillip Hellewell]

Does anyone have answers to the specific issues I found, like why the github repo isn't being synced, or why I can't log into bugs.ntp.org, or why emails to webm...@ntp.org bounce?

[Harlan Stenn]

Phillip,

I have send you several emails. Have you received any of them?

On 6/17/2022 8:45 PM, Phillip Hellewell wrote:
> Does anyone have answers to the specific issues I found, like why the github repo isn't Philbeing synced, or why I can't log into bugs.ntp.org, or why emails to webm...@ntp.org bounce?

[Phillip Hellewell]

On Saturday, June 18, 2022 at 1:33:02 AM UTC-6, Harlan Stenn wrote:
> Phillip,
>
> I have send you several emails. Have you received any of them?

Unfortunately, no. Maybe they went to my Spam folder? If you want to try again, I'll keep a closer eye on that.

Or feel free to just answer here since this seems to be working. I do believe most of my questions are ones that a wider audience could benefit from.

I am happy to know that my concerns are not getting ignored. Thank you for that.

Phillip

[chris]

Sorry, no idea at all, but does seem to be a common thread running
through, the lack of community in terms of future direction and
progress.

The build here is earlier than I remembered, from the end of 2019, but
one of the reasons I sort of gave up, was the difficulty in sorting out
the pps input options. PPS handling at hardware interface level is a
critical function if the most accurate results are to be obtained.
At present there is just a single hardware option for that, the
rs232 data carrier detect (dcd) line, which as David has said, is
less than ideal because of the need for level translation, which
introduces a delay. In practice, that will be small, since the
data sheet figures for a typical max232 assume a 2.5nf capacitive
load on the output, whereas a few inches of wire into a rs232 line
receiver setup might be much faster. Having no other options for pps
is serious limitation though. The key thing is that whatever interface
is used, it must be capable of generating an interrupt to system,
to minimise offset. Both the dcd and the parallel port ack line
satisfy that requirement, though modern pc hardware often has neither
interface.

Do hw and sw here, but the limitations of the present system only
become evident when looking in depth at all aspects. Just wish there
could be some sort of plan to fix the various issues.

Chris

[Jim Pennino]

A delay in a hardware level translator will be a fixed amount easily
fudged out and with jitter far less than that of the computer reading
and processing the interrupt.

An industrial strength serial/parallel card costs about $40 these days.

You need more that just an interrupt to handle PPS, you need a high
priority interrupt.

[David Woolley]

On 19/06/2022 01:06, chris wrote:
> In practice, that will be small, since the
> data sheet figures for a typical max232 assume a 2.5nf capacitive
> load on the output, whereas a few inches of wire into a rs232 line
> receiver setup might be much faster.

As we are talking about compliant RS232, which is the only real world
reason for not just connecting TTL directly to the RS232 port, the the
2.5nF condition is the maximum capacitance that can appear across the
driver output as the result of what it is driving. That sets a minimum
possible slew rate.

However a compliant RS232 system also has a maximum permitted slew rate,
intended to minimise cross talk, and probably also to ensure that long
cables don't ring, as the initial transient reflects backwards and
forwards. 30V/µs is the maximum permitted slew rate for a compliant
system. If your system exceeds that, even if it is using RS232 drivers,
it is not a compliant system.

[chris]

Yes, it does have a spec for the slew rate, but i'm not sure that would
always be met for modern devices, as such uarts can often be run at data
rates of up to 230 Kbytes / second, or about a 4uS bit time. That
implies much faster rise and fall times, which relates to slew rate.

We can theorise about that, but iirc, I bought two ttl to rs232
converters at the time. If I can find the second one, i'll set it up
on the bench and measure the propagation delay on a scope from input to
output. Also, see what effect the 2.5nF cap has on the waveform and
timing...

Chris

[chris]

Agreed and that is the best option. Polling methods will always suffer
from more jitter. There will be timing uncertainty around the polling
rate, of which there is a minimum practical limit. Direct interrupt
methods are really the only way to go for best precision.

>
> An industrial strength serial/parallel card costs about $40 these days.
>
> You need more that just an interrupt to handle PPS, you need a high
> priority interrupt.
>

Using FreeBSD here and that needed a kernel rebuild to enable PPS
support. Also enabled kernel thread preemption, which should help
make the os more deterministic. Unless you write your own os
optimised for the task, there's not much else you can do afaics...

Chris

[Harlan Stenn]

Hi Phillip,

Please let me know if you see this.

I'll dig thru my Sent folder to find my previous messages. Some of what
I want to chat about would be better handled via direct communication.

If I see something where others might have an interest, I make public
comments.

Thanks!

[David Woolley]

On 19/06/2022 11:30, chris wrote:
>
> Yes, it does have a spec for the slew rate, but i'm not sure that would
> always be met for modern devices, as such uarts can often be run at data
> rates of up to 230 Kbytes / second, or about a 4uS bit time. That
> implies much faster rise and fall times, which relates to slew rate.

But those aren't RS232 compliant devices, although the slew rate limit
is still low enough to support those data rates, but not with the full
cable length.

>
> We can theorise about that, but iirc, I bought two ttl to rs232
> converters at the time. If I can find the second one, i'll set it up
> on the bench and measure the propagation delay on a scope from input to
> output. Also, see what effect the 2.5nF cap has on the waveform and
> timing...

The 2.5nF is mainly intended to represent the capacitance of the cable,
which is deliberately operated with both source and load termination
impedances well above the characteristic impedances, so behaves like
capacitor. With the full length cable, rather than the lumped
capacitance, you would expect the voltage to staircase upwards, each
time the leading edge reflection returns.

The 2.5nF basically allows for the 50 foot maximum cable at 50pF/ft.

[Jim Pennino]

There is still the matter of interrupt priority.

Serial ports normally show up on irq 3 and 4. If it is an add in card
they can show up anywhere, and if it happens to be something like
irq 19, then you will find your jitter in the PPS to be something in the
order of 10 to 20 microseconds even on a very fast machine.

[William Unruh]

Who cares if it is compliant esp as most real RS232 do better. RS232
standards are about 50 years old. They were still using vacuum tubes:-)

Test your system to see what it can do.

[William Unruh]

On 2022-06-19, David Woolley <da...@ex.djwhome.demon.invalid> wrote:

So teminate it to get rid of the refections.

[Jim Pennino]

Hand-wringing over the serial physical characteristics are pointless
unless you have a very long cable or are designing your own interface.

Commercial hardware is orders of magnitude better at both latency and
jitter than most all computer hardware/software combinations unless you
are using computer hardware specifically designed for real time systems
with a real time OS.

The limiting factor for PC's and things like the Pi will be the jitter
in processing the interrupt generated by the PPS signal.

This tends to be on the order of 1 microsecond at best on every PC and
SBC board I have used.

FYI the best I've seen so far is about 900 nanoseconds on a Pi4.

[David Woolley]

On 19/06/2022 16:10, William Unruh wrote:
> So teminate it to get rid of the refections.

It's no longer RS232 if you terminate it anything less than, ISTR, 4k,
or reverse terminate in anything other than 300 ohms.

I think 100 ohm is typical for the characteristic impedance of a wire
pair, so given the open source voltage of the driver is 12 volts, by
terminating it you have removed all your noise margin, if you accept the
transition region is the full +/-3V.

If you want to use an EIA standard that uses terminated lines, you
should use RS 422, which also uses differential pairs, which also
improves noise immunity. Or RS 423, which, although it uses a
differential line receiver, is unbalanced, with one of the pair tied to
local functional ground at the transmitter.

[chris]

Yes and that depends on a raft of factors, including processor type,
hardware organisation and software architecture. Early processors
often had just a couple of interrupt pins, irq and nmi, so that
several devices had to be polled to find out the source of the
interrupt. That imposed it's own priority structure.
Later devices, starting in the 68000 era, used an interrupt vector
table, where interrupt sources were vectored directly to the interrupt
handler, with other methods used to organise priority.
All a bit of a wash now, with nanosecond processor response times, but
was a serious issue in the past, often requiring counting machine
cycles in interrupt handlers to ensure timing accuracy.

Real time embedded here for over 3 decades and still love the work, but
need to get more into the FreeBSD kernel to understand more about
timing. The FreeBSD internals book is well worth a read and is one
of the best documented ever seen here...

Chris

[David Woolley]

On 20/06/2022 00:14, chris wrote:
> Early processors
> often had just a couple of interrupt pins, irq and nmi,

You mean microprocessors. NMI seems to be a concept that was introduced
with microprocessors, and multi-priority interrupts existed before even
the earliest microprocessors.

In terms of software, I don't think typical Linux code really fully
supports multi-level interrupts, and, if anything, may effectively
invert the priority order.

[Terje Mathisen]

Back before OoO processors, spread spectrum clocks and variable
frequency boosts, i.e. Pentium days around 1994-1997, it was trivially
easy to get far below your stated 1 us jitter.

It is indeed harder today!

A propr GPS clock these days needs to be inverted, i.e. the server asks
the clock what time it is _now_, and gets back a timestamp.

So, by taking the local system clock, sending off this querey, and then
checking the local clock again, you can both measure the latency of this
reading and get the official/gps-based time.

In order to work this way, the GPS needs a way to use its local osc
(typically running at 10 MHz) to count ticks since the last second
transition, there has been at least one reference clock that worked this
way.

Repeat this call maybe 5 times and pick the one with the lowest
turnaround time.

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

[chris]

That's correct, thinking of things like early Arm, 6502, 6800 and many
others. Suspect some of the early minis used similar ideas, perhaps
under different names. The pdp11 mini, for example, had a fully vectored
interrupt dispatch table from day one back in 1969. Not sure about the
Data General Nove from the same era...

As for Linux, dumped that for good in frustration about the bloat after
systemd. Little more than a windows replacement these days imho. FreeBSD
is much more easily configured for real time work and comes with far
less baggage. Of course, traditional unix, including Linux tasks run
to completion, where a fully preemptive kernel is needed for real time
work...

Chris

[William Unruh]

On 2022-06-19, David Woolley <da...@ex.djwhome.demon.invalid> wrote:

> On 19/06/2022 16:10, William Unruh wrote:
>> So teminate it to get rid of the refections.
>
> It's no longer RS232 if you terminate it anything less than, ISTR, 4k,
> or reverse terminate in anything other than 300 ohms.

Who cares if it is no longer RS232, if it works. As has been pointed out
virtually no rs232 hardware is RS232 according to the standards. What
one wants is a signal pin that triggers an interrupt. If your bog
standard RS232 does that, then use it. I do agree that if you terminate
it with too low an impedance, you will get a small signal on the pin,
and it could be problematic, or could not, again depending on the
hardware.

[David Woolley]

On 21/06/2022 10:48, William Unruh wrote:
> Who cares if it is no longer RS232, if it works. As has been pointed out
> virtually no rs232 hardware is RS232 according to the standards.

Oddly that is the point I was trying to make. Someone was insisting
that you mustn't cut corners, and must use an RS232 driver when
interfacing to a nominally RS232 input port. I was analyzing the
consequences of actually conforming to RS232.