Fixing the NTP server abuse problem
Nelson Minar
I was sad to read David Mills' threat to start requiring crypto keys
to access public servers. One of the great things about NTP has been
how open it is. It sounds like this is becoming untenable.
What's ironic is that NTP is the most lightweight useful service I can
think of on the Internet. If the load were just spread a little
better, there'd be no problem.
I've got four half-baked ideas on fixing the problem:
Invent an NTP administration model like DNS has
Get vendors involved in funding NTP service
Look at integrating NTP closer with the routing infrastructure
Look at using P2P discovery mechanisms for NTP
NTP administration: DNS manages to spread load pretty well, but only
in a heavyweight way that requires a lot of administration. It'd be
great if NTP were deployed the same way as DNS, where ISPs are
expected to provide the service and every computer has to be
configured (by hand or via DHCP) or else it doesn't work.
Vendors: it seems one of the big problems is vendors shipping servers
with public NTP server addresses baked in. This is shameful - at the
least, the vendor should ask permission first. Maybe there's a way to
get the vendors to fund better NTP service? The NTP server admins have
a very poweful tool here; if one day a switch were thrown and all of
router vendor Xs routers stopped working well, someone would notice.
Routing: it's a shame that multicast doesn't work on the Internet - it
would solve this problem neatly. Is there a way to do a one-off hack
in the routing structures of the net itself to support NTP? Someone in
another thread suggested publishing a single address and using BGP
magic to do shortest path routing; that seems like an idea with merit.
P2P: the P2P file sharing world has solved a similar problem. On any
given day there are 3 million KaZaA peers running, all self-organized
into a P2P network. What's amazing is how well it all works even
though most of the peers are crummy Windows PCs behind NAT firewalls
and the like. The bootstrapping and organization methods KaZaA, etc
are well known; maybe they could be appropriated to design a new NTP
discovery mechanism?
nel...@monkey.org
. . . . . . . . http://www.media.mit.edu/~nelson/
David Malone
>Routing: it's a shame that multicast doesn't work on the Internet - it
>would solve this problem neatly. Is there a way to do a one-off hack
>in the routing structures of the net itself to support NTP? Someone in
>another thread suggested publishing a single address and using BGP
>magic to do shortest path routing; that seems like an idea with merit.
IPv4 anycast would probably work quite well for NTP, as long as
routes changed slowly enough not to upset NTP (I suspect this is
the case already). IPv4 anycast is already being used to provide
default routes into the IPv6 internet for one of the trancision
mechanisms, and seems to work relatively well.
Maybe we should see if we can get IPv4 anycast address space for
NTP experiment - if it worked reasonably well, then public NTP
providers could start to advertise the anycast address rather than
their NTP server's "real" address.
For real NTP servers which need several peers another DNS name could
be used.
David.
Adrian 'Dagurashibanipal' von Bidder
[...]
> Look at using P2P discovery mechanisms for NTP
> P2P: the P2P file sharing world has solved a similar problem. On any
> given day there are 3 million KaZaA peers running, all self-organized
> into a P2P network. What's amazing is how well it all works even
> though most of the peers are crummy Windows PCs behind NAT firewalls
> and the like. The bootstrapping and organization methods KaZaA, etc
> are well known; maybe they could be appropriated to design a new NTP
> discovery mechanism?
The current P2P protocols do not really scale. Yes, it works, sort of, but
it's absolutely inefficient, generating way too much traffic.
I think the things that *would* work (some things already proposed by you and
others)
- encouraging the ISPs to run and advertise ntp services (creating an A
record for one/a few of their routers and adding one line to their FAQ is
all they'd have to do in most cases).
- requiring sensible behaviour when servers are unavailable by having it
iin the RFC (that's a long term measure and won't have effect in the next
two years, though. Also, it doesn't guarantee 3rd party implementations
will honour this)
- establish multicast in the Internet (I guess this is dreaming)
- pushing awareness of the ntp service to open source communities? Debian
and others have quite a few public machines which might be available as
public low precision stratum2 servers.
- ship ntp with a 'discover ntp services on next upstream ntp server
script', using traceroute and ntpq to find some router with ntp.
- asking the vendors of the devices in question wouldn't hurt either, I
guess.
Just a few thoughts.
-- vbi
those who know me have no need of my name
>NTP administration: DNS manages to spread load pretty well, but only
>in a heavyweight way that requires a lot of administration. It'd be
>great if NTP were deployed the same way as DNS, where ISPs are
>expected to provide the service and every computer has to be
>configured (by hand or via DHCP) or else it doesn't work.
the is already how ntp works, in general. most isp's are unaware of ntp
and what they'd have to do to provide such service -- yes, we know it's
moderately trivial, but they don't. most are overwhelmed with the need to
keep their service growing that the time to investigate and implement ntp
is just not on the radar. their servers can use it, but why provide it to
customers: what does the isp get from it? are customers demanding it? will
it bring them additional customers/revenue?
>Vendors: it seems one of the big problems is vendors shipping servers
>with public NTP server addresses baked in. This is shameful - at the
>least, the vendor should ask permission first. Maybe there's a way to
>get the vendors to fund better NTP service? The NTP server admins have
>a very poweful tool here; if one day a switch were thrown and all of
>router vendor Xs routers stopped working well, someone would notice.
yes, it would have been much better if auto-configuration via manycasting
were implemented, perhaps in addition to using dhcp ntp when present. but
manycasting hasn't been around as long as ntp itself, and in the rush to
get product to market the implementation of `optional features' is all too
often glossed-over or vetoed. some vendors do the right thing -- when they
`need to' hard-code a server address they set up a server themselves. but
when the value is merely a default it's usually aimed at a `public' server,
and it's unlikely that there is documentation that it's something that
should be customized. microsoft's windows xp is a combination, they worked
with nist to setup an ntp cluster at their data center which is the target
of their default value but with little documentation that it exists or
might be better if customized.
>Routing: it's a shame that multicast doesn't work on the Internet -
it does, and it works well. the `problem' is, again, that most isp's
aren't aware of it, what it can do for them, or what is necessary to bring
the service to their customers or themselves. an education campaign could
solve this, but the effort of reaching 30_000 isp's with this information
would be significant -- even a talk at significant conventions isn't likely
to reach more than a few hundred successfully (a few thousand would hear of
it, but only a few hundred would actually do something).
>Is there a way to do a one-off hack in the routing structures of the net
>itself to support NTP? Someone in another thread suggested publishing a
>single address and using BGP magic to do shortest path routing; that seems
>like an idea with merit.
certainly this is possible. the only issues are administrative; mainly
locating enough providers willing to participate, donate routing/bandwidth,
co-location, or equipment, so that critical mass is achieved so that the
project would succeed.
>P2P: the P2P file sharing world has solved a similar problem. On any
>given day there are 3 million KaZaA peers running, all self-organized
>into a P2P network. What's amazing is how well it all works even
>though most of the peers are crummy Windows PCs behind NAT firewalls
>and the like. The bootstrapping and organization methods KaZaA, etc
>are well known; maybe they could be appropriated to design a new NTP
>discovery mechanism?
the difference here is that in p2p `sharing' there is incentive on the part
of the user to `do what is necessary' -- they are after some stuff. almost
no users know or care about the accuracy of their system clock, it's not
used for much. twice a year they are reminded that the clock is being
changed, and they look at it when they wonder if they are too late/early
for something (and that's the human timescale, i.e., measured in minutes)
or to get the date, but the need to be correct within even a few seconds is
just not a common concern. before this could possibly succeed there would
have to be some sort of education initiative, and sufficient rationale for
`why' they would care would need to be found -- i.e., we need wondrous new
`killer' application that makes such demands before the common user will
care. (ms' windows xp cares that the clock be approximately correct, which
is why it has an sntp-ish client and is defaulted to be active.)
simply put: we need to educate people, about ntp and internet multicast --
the former is fairly simple. initially it would be best to target isp's.
reaching them all, or even a usefully significant minority, is the main
hurdle. (well, getting started is actually the main hurdle.) what david
is considering might force them to wake-up and look into the issue.
--
bringing you boring signatures for 17 years
Nelson Minar
> The current P2P protocols do not really scale.Yes, it works, sort of, but
> it's absolutely inefficient, generating way too much traffic.
On the other hand, KaZaA isn't about to go private-access-only because
of scalability issues. The self-organizing protocols behind KaZaA are
awfully clever. There's even better stuff in the academic literature,
but it hasn't been battle tested.
Yes, none of this is super efficient, and there's a lot more traffic
spent on discovery than you have in current NTP. But if it kept the
network alive without a few private servers paying the heavy burden,
would it be worth it?
Nelson Minar
>most isp's are unaware of ntp and what they'd have to do to provide
>such service
What's maddening is that someone at the ISPs *does* know what NTP is,
and does implement it. My upstream router on PacBell DSL is a stratum
2 clock. It's just not provided on through to the users. It would be
such a simple thing to do - just throw the NTP server info into the
DHCP lease along with the client IP address. Is that enough to appease
WindowsXP's built in NTP client, or is more necessary?
You could imagine educating the ISPs - block all NTP requests from
home machines on ISPs that don't themselves provide NTP service. Cut
them off from the public network until they serve their users. Shades
of old school Usenet peering policies.
> >Routing: it's a shame that multicast doesn't work on the Internet -
> it does, and it works well.
I guess this depends on what you mean by "works". Multicast works for
that tiny fraction of the net that routes it. As a practical matter,
deploying NTP clients that required multicast to consumer machines
won't work. Getting multicast (or even anycast) routed on the public
Internet is a battle larger than any of us.
>>P2P: the P2P file sharing world has solved a similar problem.
>the difference here is that in p2p `sharing' there is incentive on the part
>of the user to `do what is necessary' -- they are after some stuff.
Right. What I had in mind was a network of peers of several hundred
stratum 2 NTP servers who are incented to build a functioning network.
They'd maintain a list of several thousand NTP hosts (themselves, plus
their downstream peers). When a new random client comes online it asks
one of the well known addresses to hand it off to an NTP host.
This is roughly how KaZaA's supernode stuff works. And it works, in
practice. The one step beyond would be to occasionaly turn the random
client into a NTP server/supernode itself. I don't think that'd work
too well, mostly because of assymmetric bandwidth problems throwing
off timing accuracy. (ADSL makes a lousy NTP server).
Thinking out loud,
Nelson
Christopher Browne
>>Is there a way to do a one-off hack in the routing structures of the
>>net itself to support NTP? Someone in another thread suggested
>>publishing a single address and using BGP magic to do shortest path
>>routing; that seems like an idea with merit.
> certainly this is possible. the only issues are administrative;
> mainly locating enough providers willing to participate, donate
> routing/bandwidth, co-location, or equipment, so that critical mass
> is achieved so that the project would succeed.
Let me suggest a different direction, namely that of Internet
Registrars. There are about 150 of them, distributed around the
world, who then sell domain services on to the millions that register
domains.
Find a way for time services to be something /they/ can make a few
bucks a year on, and that can get some "publishing" going.
The merit of this would be that it could lead to an NTP server address
getting added into WHOIS entries, which provides a reasonable approach
to "easy discovery."
--
(reverse (concatenate 'string "gro.gultn@" "enworbbc"))
http://www3.sympatico.ca/cbbrowne/ntp.html
Nagging is the repetition of unpalatable truths. --Baroness Edith
Summerskill
Maarten Wiltink
[...]
> - requiring sensible behaviour when servers are unavailable by having it
>iin the RFC (that's a long term measure and won't have effect in the next
>two years, though. Also, it doesn't guarantee 3rd party implementations
>will honour this)
<whispers very softly> NTPv4 RFC? <tiptoes away>
Groetjes,
Maarten Wiltink
Adrian 'Dagurashibanipal' von Bidder
Good. I didn't read the rfcs, I just got the impression in this discussion
that the rfcs don't cover this. But all the better if they do.
cheers
-- vbi
Maarten Wiltink
<DNS tree>
>the is already how ntp works, in general. most isp's are unaware of ntp
>and what they'd have to do to provide such service -- yes, we know it's
>moderately trivial, but they don't. most are overwhelmed with the need to
>keep their service growing that the time to investigate and implement ntp
>is just not on the radar. their servers can use it, but why provide it to
>customers: what does the isp get from it? are customers demanding it? will
>it bring them additional customers/revenue?
Perhaps I'm from another era, but they should provide
(stratum 2 or 3) NTP because it's The Right Thing To Do.
What I'm missing in all these discussions is that NTP
is hierarchical. It's something of a Prisoner's Dilemma,
and everybody seems to be defecting. It would work *so*
much better overall if the tree were deeper on average.
I know, I know.
> [...] microsoft's windows xp is a combination, they
worked
>with nist to setup an ntp cluster at their data center ...
No foolin'. NTP? Real NTP? Well, let's not complain.
They seem to have got it right at one end.
Groetjes,
Maarten Wiltink
David L. Mills
Thanks for the good suggestions. At the very bedrock of the difference
between the DNS and NTP models is that DNS service with domain names
makes money and NTP service with network expense loses it.
At one time in the mid-eighties when the NSF network routers were
Fuzzballs running the Hello routing protocol, the routing and timing
information was aggregated in the same packet, so all routers were
synchronized NTP servers by design. That could, but I doubt it will,
happen again.
Also once upon a time when the MBONE flourished we could and did
synchronize European clocks from American servers and vice versa using
multicast without special configuration. You had to be a little careful
about coming up on 224.0.1.1, because everybody from here to Upper Volta
might come on frequency.
Dave
David L. Mills
You saw a lemur with huge round black eyes peeping around the corner and
then he was gone not with a tiptoe but a terrified leap.
I've got a few things cooking here; that lemur may land sometime before
sabbatical ends and classes begin in the Fall.
Dave
Marc A. Donges
> Nelson Minar <nel...@monkey.org> writes:
>
>>Routing: it's a shame that multicast doesn't work on the Internet - it
>>would solve this problem neatly. Is there a way to do a one-off hack
>>in the routing structures of the net itself to support NTP? Someone in
>>another thread suggested publishing a single address and using BGP
>>magic to do shortest path routing; that seems like an idea with merit.
>
> IPv4 anycast would probably work quite well for NTP, as long as
> routes changed slowly enough not to upset NTP (I suspect this is
> the case already). IPv4 anycast is already being used to provide
> default routes into the IPv6 internet for one of the trancision
> mechanisms, and seems to work relatively well.
True.
> Maybe we should see if we can get IPv4 anycast address space for
> NTP experiment - if it worked reasonably well, then public NTP
> providers could start to advertise the anycast address rather than
> their NTP server's "real" address.
>
> For real NTP servers which need several peers another DNS name could
> be used.
I guess that making SNTP-Service available via IPv4-Anycast would solve
most of the load-problems.
Those in need of *real* NTP-service, where flapping routes would hurt
and where several servers are required, can still use manual
configuration.
Marc
--
_ _ Marc A. Donges +49 721 6904-2130
'v'
/ \ Klosterweg 28 / E110
W W 76131 Karlsruhe PGP-Key(RSA): 1024R/429D9719
David L. Mills
Just looking over this thread again I am reminded of an occasion circa
1986-88 when the NSFnet was just a baby and the only real time servers
were the Fuzzball routers. The NSFnet NOC at Cornell kept track of the
monthly packet count, easily won by electric mail. One month a new
leader blew the minders off their three-legged stools.
It was NTP! When the electrified me, I saw the Fuzzballs in redline with
packets from all over the place ganging up on all six machines. Looking
back on it, this was very likely the first case of genuine Internet
flooding attack if you don't count broadcast storms and Chernobyl
packets in BSD 4.2. The only thing that saved the world was that
Fuzzballs aren't exactly blinding fast.
Turns out the problem was an early NTP implementation for Unix; I will
not reveal the author, but it wasn't me. In that version there was no
operation code and the difference between client and server packets was
in the port number. Well, the implementor had a wee trisket which made
every client a server as well and the server packet returned by the
Fuzzballs came right back from the client. Ooops.
Funny thing was that nobody noticed until the end of the month when
Cornell rolled the stats. Who knows, maybe it's happening right now in
some NTP shareware sharewar sharewa sharew share shar sha sh s [].
Dave
Ulrich Windl
Possibly the problem with not announcing NTP service is with
liability: Maybe they are afraid if some customer complains that the
time presented by NTP was not correct. It's similar in our
department: We operate and use the NTP service, but there are not
guarantees about quality of service whatsever.
Regards,
Ulrich
