At the request of a national time standards laboratory I have removed
their NTP servers from the public lists. The timekeepers cited gross
violations of their access policy and the expense of the network
service, especially for unintended international users. As you know from
my previous grouse to this list, this is a growing problem and may well
lead to the loss of public time service altogether.
You may not have noticed it, but provisions added to recent NTP versions
includes symmetric and public key cryptography, which is my recommended
method for source authentication. It is a trivial matter to require this
for access control as well and I am preparing to do exactly this for our
public time servers and recommending it for the national laboratories.
It is to work like this. With NTPv4 you will need OpenSSL and an
encrypted identity key, as well as public/private keys you generate
yourself. We can provide the identity key by electric mail and the
password by ordinary mail. Unless you have the proper key, our servers
will respond with a kiss-of-death packet, a crypto-NAK, contrived really
bad time or no time at all. There are several cryptographic methods
already implemented in NTPv4, including one in which we can revoke
client keys individually without affecting others.
You will be asked to hold the password secret. A client can't use
another client's key without the password and every client password is
different. NTP maintains cryptographic auditing logs (cryptostats) so we
can find and prosecute villians.
You can find information on the how the schemes work and the procedures
to create and use keys in the latest NTP documentation on the web and on
the NTP project page linked from www.ntp.org. The documetns include
briefings, executive summaries and various scenario descriptions. There
is an Internet Draft now in the RFC standards pipeline that reveals the
technical model and protocol. See the citations on the NTP project page
We will phase the crypto requirement in over a transition period of a
few months. Some of the UDel servers already support the cryptographic
means and we will be handing out trial keys on request. However, at some
future flag day we will suspend all service unless cryptographically
enabled. Our experience will no doubt be a model for the national
laboratories, who without a doubt will be forced along the same path at
some point in the future.
It will take a little while to set up administrative procedures. I will
announce when we are ready for experimental use.
University of Delaware
I have written an SNTP client for our Unisys mainframe.
My client currently supports MD5 authentication only.
It appears that, sometime in the future, I might have to
support this stuff in my client. Where is the best place
to look for guidance in coding this? Would it be this:
>. . . There
>is an Internet Draft now in the RFC standards pipeline that reveals the
>technical model and protocol. See the citations on the NTP project page
I speak only for myself; not my employer
Please reply in the newsgroup. Don't try
The latest Autokey draft is at
Almost all the cryptographic means is in ./ntpd/ntp_crypto.c in the
latest distribution. Unfortunately, this version is not yet on the web
due circumstances beyond my control. Until this rock is turned, find the
latest at ftp.udel.edu/pub/ntp/software/ntp-4.1.72c.tar.gz.
Is there a quick way for people to verify that they haven't made a
mistake of this sort? Off the top of my head, verifying that one's
organization's NTP servers do not report a peer of stratum 1 would do
a good job.
> We will phase the crypto requirement in over a transition period of a
> few months.
Who is the "we" you are speaking for here? You don't seem to have
introduced them, unless you mean the unnamed "national time standards
laboratory" above. But in that case, how are the people who need to
know that the laboratory is changing its procedures to know that you
are talking about that particular laboratory?
I'm not meaning to be contentious here, but looking in from the
outside, this message wasn't as clear as it probably needs to be.
If that happens, then we might as well scrap ntp!
'We' is probably all the people who take part in the ntp.hackers mail
list, and helps develop/maintain the ntp source code.
There's lots of people, I'd put Harlan Stenn, Danny Mayer, John Hay and
Ulrich Windl high on the list.
There's several others as well, with Prof. Mills on top as a sort of
"almost all programming can be viewed as an exercise in caching"
I am aware of the problem that David Mills is talking about, but we have no
plans to limit public access to the NIST time servers at this time. Our
network is currently handling about 9,000 requests per second with no
particular difficulties. The main problem is load balancing -- we have 14
servers, but they do not all carry an equal share of the total load. I am
implementing a load-balancing scheme to address this problem, which I hope
will be completely transparent to the user community. (If you are currently
using any of our servers in Boulder, Colorado or in Redmond, Washington, you
could help balance the load by moving to one of our other systems.)
Although we have a formal access policy, I enforce it rarely and only
against serious abusers, most of whom turn out to be ignorant rather than
malicious. Part of the reason for this laid-back attitude is that many of
our users are required to use NIST services for reasons of legal
traceability and part of the reason is that we have always been strongly
encouraged to provide as wide an access as is possible to our time and
I cannot guarantee that this service model will continue forever, of course.
All NIST programs are affected by budgetary issues and by political
pressures. However, there is nothing on the horizon at the moment that would
suggest a change to our current services.
Time and Frequency Division
NIST, Boulder, Colorado
Or perhaps people might have to start using their ISP's
NTP servers more, or set up their own reference clocks.
Or switch to the remaining public-access servers. I don't
think there will be a _total_ loss of public time service.
It might well be considered the end of an era. Which
might be (incorrectly) viewed as cataclysmic, or simply
as reflecting on the achievement in succumbing to one's
> David J Taylor wrote
>>"David L. Mills" <mi...@udel.edu> wrote
>>> this is a growing problem and may well
>>> lead to the loss of public time service altogether.
>>If that happens, then we might as well scrap ntp!
> Or perhaps people might have to start using their ISP's
> NTP servers more, or set up their own reference clocks.
Not all ISPs offer it. When I asked my home ISP (broadband) if they had an
NTP server I could use, they said no, and sent me off to NIST.
Always check if your next-hop router has an NTP server. And whether
ntp.<isp>.net is an NTP server.
A year ago, life was simple. We got about 10 packets/server/s
and this was growing linearly, or at least close to linear
over a time scale of two years. Then, something changed.
Traffic started to grow exponentially and is now at 200 packets/s.
Projecting current growth we will have another factor of 10 in about
200 packets/s is about 1.5 GB per day or roughly $40 per day or $15000
per year. Not so frightening now, but in 3 months it will be 10 times
Here are a few things I have discovered while searching for a solution
to this problem:
(1) Not responding to an NTP query can be bad. When we tried dropping packets
at the firewall, traffic went up by a factor of 50. Sending back a "port
unreachable" didn't help. We only tried this for 24 hours before giving up
and there was no sign of a decrease in the traffic. The kiss of death
gets a less violent response - traffic only goes up by 10 or 20% after
a transient period of a day or so where it initially doubles.
(2) Traffic from a large chunk of networks has continued to grow (slowly)
during the four months they have been blocked.
(3) There are at least two companies who have hard-coded the
addresses of our NTP servers into their routers/NAT boxes.
They cannot be reconfigured.
(4) 85% of our traffic comes from a mere 7000 users, polling more than
1000 times per day each. That's not many users and there's clearly
plenty of room for growth here.
Finally, a question.
Has anyone else ever closed down a busy server and watched for NTP
traffic to this over a period of time to see how it dies away ?
I would be very interested to know.
Had the same experience here. I called my ISP, a broadband service which
shall remain nameless <coughCoxInternetcough> and was told they had no NTP
server for their users to sync to. Since an ISP needs synchronization as
much as any other network, I figured the real answer was "we do, but damned
if WE'RE going to tell you." Sure enough, somebody here on c.p.t.n pointed
ntp.cox.net was alive and well and, as it turned out, serving up fresh
helpings of accurate time. So you might check ntp.your isp'sdomain.com and
see if there's a server there.
My ISP's public ntp server is 15msec off (asymmetrical routing, I
You expect me to use that?
For the typical user, that is likely more than satisfactory. Getting
everyone within a minute of each other would be pretty impressive;
15ms is hardly stunningly off-base.
If you truly need better than 15ms, that means your needs are "greater
than average," and you'll be one of those looking for better quality
service. That doesn't seem an outrageous outcome...
output = ("cbbrowne" "@acm.org")
Have you ever considered beating yourself with a cluestick?
I suspect I was referring to the obvious lack of care in the way the
server was set up, rather than the absolute error. I agree that 15ms is
acceptable, but not when the provider neither acknowledges nor cares about
the error. Their own routers use ntp, and my local one is only a ms or so
out, but their public ntp service is the one with the errors.
> server for their users to sync to. Since an ISP needs synchronization as
> much as any other network, I figured the real answer was "we do, but damned
> if WE'RE going to tell you." Sure enough, somebody here on c.p.t.n pointed
I think it's more often the case of 'ntp? Never heard, so we can't be
offering that'. I mean, your average hotline person is not the one you'd
trust to administrate your servers...
Did they really hard-code the address instead of the host name? If so,
you could change your address but keep the same name. People accessing
you by name would follow the change. To be nice you could respond to
both addresses for a month or so to allow most everyone to switch to
the new address.
>(4) 85% of our traffic comes from a mere 7000 users, polling more than
> 1000 times per day each. That's not many users and there's clearly
> plenty of room for growth here.
>Finally, a question.
>Has anyone else ever closed down a busy server and watched for NTP
>traffic to this over a period of time to see how it dies away ?
>I would be very interested to know.
How about something like the GPS Selective Availability? Dither
the bottom bits of the answer. Increase the amount of dithering
more for the more abusive users.
Turn it off for properly authenticated users. (assuming they
aren't being abusive :)
The suespammers.org mail server is located in California. So are all my
other mailboxes. Please do not send unsolicited bulk e-mail or unsolicited
commercial e-mail to my suespammers.org address or any of my other addresses.
These are my opinions, not necessarily my employer's. I hate spam.
In context, "we" includes the 50 or so folks listed on the copyright
page. I may use "we" to designate the 300 or so timekeepers on the
public server lists. In a fit of naughtiness I may even stoop to the
royal "we" to include me as chief sorcerer in my lab plus my sorcerer's
apprentices. I agree my use is sometimes confusing. We should be more
Who says I am benevolent?
As a matter of personal opinion, I submit that any ISP worth
their cookies offer it.
>Always check if your next-hop router has an NTP server. And whether
>ntp.<isp>.net is an NTP server.
Or perhaps ntp0, ntp1, ntp2, ...
The thing that worries me most about your experience and others reported
to this list is the growing traffic when service is denied. Rascal
rackety.udel.edu has been a loyal primary server for over 16 years. The
machine is an ancient SPARC IPC running SunOS 4.1.3, so doesn't have a
lot of horsepower. In an effort to reduce the load, about a year ago I
denied service for all but the current NTP version (4). Today, nine
packets out of ten received are dropped as violators of that restriction
and there has been no noticeable decline in that fraction. I conclude
denial of service, with and without death, goes largely unnoticed by the
It's curious you noted that the increase after denial was less with
death than with discard. This may or may not be due to the clients
actually responding to death. Death packets are purposely coded so a
client won't believe the time. But, there is an Achilles heel: the
packet is marked unsyncrhonized and with stratum zero; however, the time
values are in fact valid. While stock NTP software of any version will
disregard those packets, some other software might use them anyway.
I'm evaluating a few changes that may help contain the problem. First,
the "nopeer" restriction bit applies to all broadcast and symmetric
active packets, not just ones that might mobilize an association. Since
Bill mistakenly uses symmetric active mode, setting this bit amounts to
a Bill filter. Second, I changed the behavior of the "notrust" bit.
Previously, it denied all service; now it denies service only if the
packet is not correctly authenticated by either symmetric or public key
cryptography. This makes it possible to implement a closed client group
we plan here.
Once upon a time the "limited" restriction was effective in denying
service when too many clients from the same network ganged up on the
server. This worked well when the population was only a few hundred and
just about all the time software in the world was NTP. The measure of
success in the scheme is how old the oldest monlist entry is. If it is
at least as old as the minimum poll interval - 64s - we find joy. Now,
at least for those servers I can reach out and touch, thelist overflows
quickly and the average entry lifetime is only a few seconds.
There's another way to go about this. While the overhead to properly
maintain a large MRU list in real time may be excessive, the scheme
might work better with random replacement rather than MRU. Rather than
search and order the list for every packet, search for multiple IP
addresses say once per second and insert a timeout entry on the restrict
list, as a crypto deny restriction does now.
I'm about to implement something like the GPS Selective Availability
(SA) we knew and hated, but in a much meaner way. Perhaps the simplest
means is to set all death packet timestamp fields to zero.
This stuff will in time appear in the development version on the web.
Your comments are welcome.
I hope some of that was a joke. I see definite signs of megalonaia
creeping in there.
It's a bit like the "war against terror" or what ever Bush and his
cronies call their flavour of the month.
It is all right to treat the symptom using autokey etc, but you ( I
say you because You are the recognised guru on the block ) aught to
be treating the CAUSE, something that Bush hasn't got the brains to
address in the other context.
Problem: For whatever reason, we all need precise globally certified
time. (independent of .gov)
Less than a few dozen publicly available sources.
Where is the logic in that?
Whoever runs the time show need to address that. We should be
able to put certified sources on about every block, and what is
as important, guarauntee that THE protocol doesn't get its
knickers in a twist when trading on asymetric networks.
Once upon a time when the Internet was young and only a few dozen knew
it wasn't a volleyball club, we asked ourselves "Who runs the Internet?"
Nobody. You know that. "Whoever runs the time show...?" Nobody, You knew
When folks start dropping of the lists, most of us get really nervous
and try to find appropriate defenses. The only thing the victim NTP
server can do is craft something that persuades the client to go away.
Sometimes the best defense is to make like a black hole. My experience
is that doesn't work. The death packet does that quite well, but only if
the client software believes it. You might not like additional less
friendly measures, but hey, that will never happen to you, right?
One would expect the answer then is to force everybody to run NTPv4,
warts and all. Yes, I know the rumors will fly that NTP is completely
broken if it jiggles the client clock bigtime, and the operator doesn't
realize why. This is a no-win win. I say again, the only widely
acceptable countermeasure is per-client cryptographic access control.
Granted, some are zombies, but there are currently 121 servers listed as
stratum 1 and 163 listed as stratum 2, somewhat more than the few dozen
you cite. Is it a terrorist campaign? I don't think so, just sloth and
ignorance and the very likely result of employee turnover.
I like your "megalonaia", which may indeed by politically applicable,
but hopefully not to me. I have indeed been accused a benevolent
dictator, but with regards your particular inventive spelling, maybe you
accuse me of spending to much time by myself.
As for "certified sources on every block," USNO and NIST both run
extensive time services and, as Judah points our, if automatic load
leveling works, maybe it will work in the wider community as well.
Sometimes desperation drives me to enhance violence as you suggest. A
possible lose is that the generally naive user will conclude NTP itself
is broken and switch to something else which does not respond to death
More constructively, there have been several prior discussions on this
frequency about NTP server servers. Our local DNS responds to a well
known host name by randomizing over a number of local servers. Let me
float an idea based on that experience.
The manycast paradigm is a really useful way for a bunch of ignorant
clients to find loving servers; however, that requires IP multicast,
which is not a ubiquitous Internet feature. The manycast paradigm is
described on the Autonomous Configuration page in the documentation. We
can, nowever, emulate that with clever unicasting.
It would work something like this. A cabal of maybe a dozen servers
would be preconfigured with each other as peers. DNS time.cabal.com
returns (some of or all of) their addresses. The client lights up
theassociations so revealed and plays manycast to prize out the best of
those servers and abandons the rest.
The problem with this is there is no way a servers can themself modulate
the replies on the basis of load leveling, for example. So, roll a
random value and compare with some kind of load indicator. The result
should be the server responds with less and less probability as the load
15ms sounds pretty good; my ISP is about 60ms behind the CHU beat.
Yes, that's allowing for the ~ 9.4 ms of propagation delay :). Time
to get an Oncore.
(780) 975 3505 cell / (780) 433 6725 home
Could you post a list of the servers you've removed from the list? If
you do, semi-ignorant users like me could check our ntp.conf files and
remove references to those servers immediately. I would gladly do that.
The ISP I mentioned (ntp.blueyonder.co.uk) is syncing off other NTP
servers, not a reference clock, that's why I find 15ms unusual - many
reports here have show ten times better accuracy than that. 60ms is
certainly unacceptable for an NTP referenced server, but I can see how
other references could be that much out.
I recently uncovered one named "time" at my ISP.
Theodore Heise <th...@heise.nu> West Lafayette, IN, USA
There is another hack you might want to consider. How about a BGP4
hack where many ntp servers are sprinkled around the world all
answering to the exact same IP address. BGP itself would work out the
shortest "distance" to each server, and magically route the packets to
the closest server. Probably you'd have to burn a /24 (or perhaps
larger) so that folks didn't drop the BGP advertisement as being too
small, but that would be a small price to pay.
Wolfgang S. Rupprecht http://www.wsrcc.com/wolfgang/
Decoding genes for the sake of cloning is against the DMCA
(NOTE: The email address above is valid. Edit it at your own peril.)
Nifty idea, but most working netadmins and sysadmins I've run across
consider NTP to be an frivolous luxury and wont spend any time at all to
intall and maintain, especially to modify BGP. More important, NTP
service does not earn revenue or defray expenses, unless as in Canada
you charge for the keys.
Already done; see the end of the lists.