Problem syncing NTP behind NAT

[Ken Link]

Hello,

I'm trying to sync two NTP clients behind the same NAT to an Internet
NTP server. Both machines behind the NAT have the same NTP
configuration file and are running v4.2.6p4 on Windows XP. The NTP
server outside the NAT is running v4.2.6p3 on Ubuntu 10.04 LTS. The
problem I'm having is that one of the machines behind the NAT is able
to sync to the external server, while the other isn't.

This is what I'm seeing: I'll start NTP on one of the machines behind
the NAT (let's call it machine A). Via wireshark on machine A and
tcpdump on the external server I can see the NTP v4 client request
leave the NAT and arrive at the external server. The NTP debug log on
the external server shows it got the request ("receive: at 5 [local
IP]<-[machine A's IP] mode 3 len 48") and immediately sends a response
as expected ("transmit: at 5 [local IP]->[machine A's IP] mode 4 len
48"). Machine A sees the server response and thanks to iburst quickly
syncs to the machine, all good.

Now I stop NTP on machine A and start NTP on machine B. The client
request goes out the NAT, and I see the request coming into the
external server with tcpdump. But, NTP on the external server doesn't
respond. In fact, the debug from NTP doesn't even have a "receive"
line for the request. Machine B never sees a response and continues to
retry, but gets stuck and keeps the external server in the init state,
never syncing.

The order I start/stop NTP doesn't make a difference. With both
machines running NTP it doesn't make a difference. The external server
will always respond to machine A, and never respond to machine B.
Tcpdump captures from both scenarios reveal very few differences
between the NTP client requests. What could be the problem? It
shouldn't have anything to do with port forwarding, since these are
outgoing requests. I don't have access to the router but I can
guarantee neither machine A or machine B have any unique routing rules
in the network.

Thanks in advance!

Ken

[E-Mail Sent to this address will be added to the BlackLists]

On 4/5/2012 7:38 PM, Ken Link wrote:
> Machine A sees the server response and thanks to iburst quickly
> syncs to the machine, all good.
>
> Now I stop NTP on machine A and start NTP on machine B.
> The client request goes out the NAT, and I see the request
> coming into the external server with tcpdump.
> But, NTP on the external server doesn't respond.

No message at all, not RATE or KOD?

If the external has restrict limited / kod,
it may not respond, if KOD is enabled, and limited is not,
or it it the rate limit for KODs.


Is Auth required by the external ntp?

> In fact, the debug from NTP doesn't even have a "receive"
> line for the request.

Does the external server still respond to A, if you restart A?

> The order I start/stop NTP doesn't make a difference. With both
> machines running NTP it doesn't make a difference. The external server
> will always respond to machine A, and never respond to machine B.

What client source ports through the NAT are seen by the external?

IIRC restrict ntpport at the external,
will make it only answer clients,
that it sees messages coming from port 123;
and if the NAT sends from port 123 for machine A,
and another port from machine B, ...
{You should be able to see this at the external's wireshark.}

--
E-Mail Sent to this address <Blac...@Anitech-Systems.com>
will be added to the BlackLists.

[Dave Hart]

On Fri, Apr 6, 2012 at 02:38, Ken Link <kl...@numberzero.org> wrote:
> Now I stop NTP on machine A and start NTP on machine B. The client
> request goes out the NAT, and I see the request coming into the
> external server with tcpdump. But, NTP on the external server doesn't
> respond. In fact, the debug from NTP doesn't even have a "receive"
> line for the request.

That's the key point. I don't know how you can expect ntpd to respond
to a packet it doesn't receive. There is some difference, probably to
be seen in comparing the tcpdump output, to explain why the request
never makes it to ntpd.

Good luck,
Dave Hart

[Ken Link]

I did some more testing with a total of four different machines behind
the NAT. Two of them synced in a few seconds, the other two were stuck
in INIT. For the machines that didn't sync, the external server did
not respond at all.

Here are the detailed packet captures of each session, as seen from
the external server. The same tcpdump filter string was used for each
capture, and NTP was running on only one NAT'd machine at a time. The
source IP is the same for each machine behind the NAT, while the
source ports are all different. Sorry for the image links, but this
would have been a LOT of text to paste.

Machine A (works): http://i.imgur.com/a5qL5.png
Machine B (doesn't work): http://i.imgur.com/F8ndL.png
Machine C (doesn't work): http://i.imgur.com/OxIpE.png
Machine D (works): http://i.imgur.com/Bcpfd.png

The restrict config on the external machine is this:

restrict -4 default limited kod notrap nomodify nopeer
restrict -6 default limited kod notrap nomodify nopeer
restrict 127.0.0.1
restrict ::1

The external machine has a pretty basic conf. A drift file, enabling
stats, a couple server/peer/pool machines defined, then those restrict
lines from above. That's it.

Maybe you can see a difference that I can't.

Ken

On Thu, Apr 5, 2012 at 10:37 PM, E-Mail Sent to this address will be
added to the BlackLists <Nu...@blacklist.anitech-systems.invalid>
wrote:

> _______________________________________________
> questions mailing list
> ques...@lists.ntp.org
> http://lists.ntp.org/listinfo/questions

[Dave Hart]

On Sat, Apr 7, 2012 at 16:53, Ken Link <kl...@numberzero.org> wrote:
> I did some more testing with a total of four different machines behind
> the NAT. Two of them synced in a few seconds, the other two were stuck
> in INIT. For the machines that didn't sync, the external server did
> not respond at all.

I notice the successful clients were querying using ports > 123, and
the failing ones < 123. I dimly recall seeing an inappropriate
less-than-123 source port comparison in ntpd long ago, in fact I'd
have guessed it had been removed before 4.2.6p3.

I suggest trying a newer ntpd outside the NAT, or if possible
reconfigure the NAT to avoid low source ports.

Cheers,
Dave Hart

[Dave Hart]

On Sat, Apr 7, 2012 at 21:36, Dave Hart <ha...@ntp.org> wrote:
> I notice the successful clients were querying using ports > 123, and
> the failing ones < 123.  I dimly recall seeing an inappropriate
> less-than-123 source port comparison in ntpd long ago, in fact I'd
> have guessed it had been removed before 4.2.6p3.

4.2.6p3 does suffer from the low-port bug:

/*
* Monitor the packet and get restrictions. Note that the packet
* length for control and private mode packets must be checked
* by the service routines. Some restrictions have to be handled
* later in order to generate a kiss-o'-death packet.
*/
/*
* Bogus port check is before anything, since it probably
* reveals a clogging attack.
*/
sys_received++;
if (SRCPORT(&rbufp->recv_srcadr) < NTP_PORT) {
sys_badlength++;
return; /* bogus port */
}

In 4.2.7 that code rejects port 0 alone.

Cheers,
Dave Hart