Compensating for asymmetric delay on a per-peer/server basis?

[Rich Wales]

Is there a way to compensate for asymmetric delay to/from one specific
peer or server?

My home LAN is connected to my school's network via a cable modem.
There appears to be a consistent asymmetry of 2-3 msec between my home
and the school's network. I can see this by comparing the time on
my home server (which is synced to a local GPS) with the time on the
school servers (which are synced to on-campus GPSes).

I would like to be able to use the campus network as a backup time
source for my home LAN, but only if I can somehow compensate for the
asymmetry introduced by the cable modem infrastructure.

The asymmetry does not appear to be traffic-dependent; I see pretty
much the same offset (between 2 and 3 msec) at any time, day or night.

I'm running NTP version 4.2...@1.2349-o.

Rich Wales
ri...@richw.org

[Rob]

Yes, you can use:

fudge 1.2.3.4 time1 -0.002

or similar. see the manual.

[Rich Wales]

Replying to "Rob":

> Yes, you can use: fudge 1.2.3.4 time1 -0.002 or similar.
> see the manual.

This didn't work. And the following error message appeared in my syslog:

inappropriate address 10.0.229.163 for the fudge command,
line ignored

As best I can tell from the online manual, the "fudge" command is
defined only for local reference clocks and has no effect on peers
or servers.

I checked the manual before asking my question but couldn't find
anything. If there is something there that will help me with this
problem, I'd be grateful if someone could tell me exactly where to
look.

Rich Wales
ri...@richw.org

[Paul]

On Tue, Sep 9, 2014 at 5:11 PM, Rich Wales <ri...@richw.org> wrote:
> I checked the manual before asking my question

Good start, so many don't -- even years later.

I might point you at The Huff-n'-Puff Filter but perhaps you could
explain your concern. An error in the small millisecond range is
often considered acceptable or unavoidable.

[Harlan Stenn]

The issue is that the huff-n-puff filter will work in the case where a
symmetrical delay becomes asymmetric, and it will "tolerate" or
"accommodate" an asymmetric delay (caused by a large download, for
example) for some period of time.

This is a case where there is reasonably-understood asymmetry for a
given server, or across a given interface.

That one is really hard to figure out.

--
Harlan Stenn <st...@ntp.org>
http://networktimefoundation.org - be a member!

[Rob]

Rich Wales <ri...@richw.org> wrote:
> Replying to "Rob":
>
>> Yes, you can use: fudge 1.2.3.4 time1 -0.002 or similar.
>> see the manual.
>
> This didn't work. And the following error message appeared in my syslog:
>
> inappropriate address 10.0.229.163 for the fudge command,
> line ignored
>
> As best I can tell from the online manual, the "fudge" command is
> defined only for local reference clocks and has no effect on peers
> or servers.

Ok you are right. In fact I filed bug #2598 myself for a similar
situation... In my case I wanted to compensate for the delay asymmetry
for external users using my GPS reference via my ADSL line. So I
would like to apply such a fudge command to a network interface, not
to a peer server.

I forgot that it is not even possible to apply it to a server, what you
would like to do. I think the only thing you can do is apply an offset
to your GPS and live with the fact that you are always 2ms off. At least
then you don't have a time wander when you switch to your backup and
the external users of your system get the correct time. That is what
I did as a workaround until someone fixes this in ntpd.

[Charles Elliott]

You can do it in broadcast mode, but I am not sure it works,
or has even been tried on a WAN. Broadcast mode was solid
as a rock on my LAN for more than a year, especially after
I started using GB Ethernet.

I don't use broadcast mode now after switching to using FreeNAS
as a time server, it addition to its other duties.

Charles Elliott

> _______________________________________________
> questions mailing list
> ques...@lists.ntp.org
> http://lists.ntp.org/listinfo/questions

[William Unruh]

On 2014-09-09, Harlan Stenn <st...@ntp.org> wrote:
> The issue is that the huff-n-puff filter will work in the case where a
> symmetrical delay becomes asymmetric, and it will "tolerate" or
> "accommodate" an asymmetric delay (caused by a large download, for
> example) for some period of time.
>
> This is a case where there is reasonably-understood asymmetry for a
> given server, or across a given interface.
>
> That one is really hard to figure out.

Not if you have gps reference at both ends, though why you would not use
the gps as the timesource then I do not know. (I guess you could have it
only temporarily). May home connections are much slower one way than the
other, and an assymmetric delay is very possible. If it really is
stable, it would be nice to be able to remove it (like the fudge time1
for refclocks). But I also could not see the equivalent for network
clocks.
I agree that huff and puff is for temporary assymmetry. There is not way
a permanant assymmeter could be detected by ntpd.
But if there was a sudden onset, then one could use the free running
computer clock to notice that (delta t vs roundtrip would show it).


>

[Martin Burnicki]

William Unruh wrote:
> Not if you have gps reference at both ends, though why you would not use
> the gps as the timesource then I do not know.

The case mentioned by the original poster is just one possible reason.

If you have a GPS controlled NTP server at home, and a fast internet
connection, and you are willing to contribute to the pool, then usually
external clients querying your server will see a systematic offset/error
depending on the ratio of the upload/download speed for your home
connection.

If you had a chance to measure this from an external node using another
GPS controlled NTP server you could try to compensate this, and thus
provide better accuracy to the clients coming from the pool.

This is also what Rob has mentioned in another post of this thread, and
I agree with Rob that a one approach could be to specify (and configure
for ntpd) the systematic error due to asymmetry of your internet connection.

However, this can also be pretty tricky if you have several NTP nodes on
your home network, if all nodes and the inet router are connected to the
same switch.

For different nodes on you home net there is no asymmetry (thus no time
error), but for each of them who contacts also an external server there
is. And often a specific machine contacts the other internal devices as
well as the external ones via the same own LAN interface.

So for your internal operation this means:

- If you specify a fudge time for a specific interface this may be OK
for external servers but yield an error for internal servers, i.e.
exactly the other way round as without compensation.

- You had indeed to specify a fudge time for servers of which you know
they are outside on the internet, e.g. other pool servers


On the other hand, if your local NTP server shall be accessible both for
external pool clients, and local clients, how should you know where a
specific request comes from? Based on the IP address? Only if the local
network and the internet interface are connected via different interfaces?

So even though it would be good to be able to specify some compensation
values, there should be different ways to do it, and putting all
together in a way that there is no error is tricky.


Martin
--
Martin Burnicki

Meinberg Funkuhren
Bad Pyrmont
Germany

[Martin Burnicki]

Rob wrote:
> Ok you are right. In fact I filed bug #2598 myself for a similar
> situation... In my case I wanted to compensate for the delay asymmetry
> for external users using my GPS reference via my ADSL line. So I
> would like to apply such a fudge command to a network interface, not
> to a peer server.
>
> I forgot that it is not even possible to apply it to a server, what you
> would like to do. I think the only thing you can do is apply an offset
> to your GPS and live with the fact that you are always 2ms off. At least
> then you don't have a time wander when you switch to your backup and
> the external users of your system get the correct time. That is what
> I did as a workaround until someone fixes this in ntpd.

Please see my reply to Bill Unruh.

[William Unruh]

On 2014-09-11, Martin Burnicki <martin....@meinberg.de> wrote:
> William Unruh wrote:
>> Not if you have gps reference at both ends, though why you would not use
>> the gps as the timesource then I do not know.
>
> The case mentioned by the original poster is just one possible reason.
>
> If you have a GPS controlled NTP server at home, and a fast internet
> connection, and you are willing to contribute to the pool, then usually
> external clients querying your server will see a systematic offset/error
> depending on the ratio of the upload/download speed for your home
> connection.

Agreed. There are two (at least) issues. One is making sure that the
clock on your local machine is accurate-- ie as close to UTC as
possible. The other is that the time you deliver to some remote machine
is as accurate as poosible (ie that the average of the timeout-timein on
the remote machine is as close to utc as possible. That of course is a
confluence of factors partly in your control or determination (ie
assymmetric delay on your own particular connection) and way outside
your control (assymetric connection of the remote machine's connection,
or assymetry on the network between you and them.)

I think that ntpd should allow you to solve the first problem-- making
sure your local machine's time be as close to accurate as possible-- but
I agree that the second problem really is beyond ntpd's capability.

However not being able even to accomplish the first is a in my mind a
bug.

[Rob]

Martin Burnicki <martin....@meinberg.de> wrote:
> This is also what Rob has mentioned in another post of this thread, and
> I agree with Rob that a one approach could be to specify (and configure
> for ntpd) the systematic error due to asymmetry of your internet connection.
>
> However, this can also be pretty tricky if you have several NTP nodes on
> your home network, if all nodes and the inet router are connected to the
> same switch.
>
> For different nodes on you home net there is no asymmetry (thus no time
> error), but for each of them who contacts also an external server there
> is. And often a specific machine contacts the other internal devices as
> well as the external ones via the same own LAN interface.
>
> So for your internal operation this means:
>
> - If you specify a fudge time for a specific interface this may be OK
> for external servers but yield an error for internal servers, i.e.
> exactly the other way round as without compensation.
>
> - You had indeed to specify a fudge time for servers of which you know
> they are outside on the internet, e.g. other pool servers
>
>
> On the other hand, if your local NTP server shall be accessible both for
> external pool clients, and local clients, how should you know where a
> specific request comes from? Based on the IP address? Only if the local
> network and the internet interface are connected via different interfaces?
>
> So even though it would be good to be able to specify some compensation
> values, there should be different ways to do it, and putting all
> together in a way that there is no error is tricky.

Well, in my own system I have a different IP address for the internet
than I have for my local network. In the bug report I asked for a
fudge time1 that could be specified per local IP addres. This would
work OK in my case. When you use the same address on a LAN and on
internet it is more difficult. I guess this only happens in cases
where there is a NAT router that translates requests from internet to
a local address. Not a configuration I would recommend when being
in the pool anyway.

[Paul]

On Tue, Sep 9, 2014 at 6:58 PM, Harlan Stenn <st...@ntp.org> wrote:
> The issue is that the huff-n-puff filter will work in the case where a
> symmetrical delay becomes asymmetric, and it will "tolerate" or
> "accommodate" an asymmetric delay (caused by a large download, for
> example) for some period of time.

I was overly casual. If you follow the breadcrumbs you find that
there is no general solution to the problem.

On Thu, Sep 11, 2014 at 9:35 AM, Martin Burnicki
<martin....@meinberg.de> wrote:
> The case mentioned by the original poster is just one possible reason.

Not to suggest that someone is doing something unreasonable but again
why does time derived from the back-up clock need to be as accurate as
the local clock (say .5ms versus 2ms)? If there's a legitimate need
then trying to solve the problem with the wrong tool will only lead to
frustration and complaints that NTP is buggy.

If I *needed* highly available and accurate time at home I'd get a
constellation of stable oscillators to drive time rather than hoping a
remote clock or set of clocks was going to solve my problem.

As an aside has anyone tried shaping traffic to make the
upstream/downstream latencies similar? It would seem more efficient
to apply network solutions to network problems if possible.

[Rob]

Paul <tik...@bodosom.net> wrote:
> Not to suggest that someone is doing something unreasonable but again
> why does time derived from the back-up clock need to be as accurate as
> the local clock (say .5ms versus 2ms)? If there's a legitimate need
> then trying to solve the problem with the wrong tool will only lead to
> frustration and complaints that NTP is buggy.

One possible explanation is that people don't want their clock to suddenly
track the changed offset in such cases. If only because ntpd will think
that the frequency offset it has calculated over a long time period is
suddenly wrong, and will change it to reflect a sudden time offset in a
short time interval. It will then slew to the correct time, overshoot,
and when the correct reference comes back online this will repeat in
the other direction.

This problem can partly be mitigated by ensuring that the offset between
your local clock and the external references is as small as possible.
(with some tweaking I can get these well below .5ms, often below .1ms)

[Rob]

Paul <tik...@bodosom.net> wrote:
> As an aside has anyone tried shaping traffic to make the
> upstream/downstream latencies similar? It would seem more efficient
> to apply network solutions to network problems if possible.

That does not work. The asymmetry is not caused by traffic but by
modem parameters.

[William Unruh]

Nope. You could have a local network in which each computer has its own
public IP addess, but the connection to that subnetwork is assymetric.
I doubt that NAT would add much assymetry. An adsl connection might well
since they advertise very different rates up from down.

[mike cook]

Did I miss something? The OP did not offer any evidence that there was path asymmetry, just that there was an unexplained offset between two GPS sync'd servers. It is often not possible to identify the origin of such an offset, but it would help if the suggested feature was implemented to take care of these corner cases. I saw that Dr Mills was in agreement back in 2005 but that the implementation is complex. If anyone wants a subject for an MSc project, this could be it.

[Paul]

On Thu, Sep 11, 2014 at 12:48 PM, Rob <nom...@example.com> wrote:
> That does not work. The asymmetry is not caused by traffic but by
> modem parameters.

The asymmetry is caused by asymmetric latency which is caused (for our
purposes) by asymmetric line speeds. Traffic shaping can change
various things (depending on where you do it) including effective line
speed (packets/s not bits/s).

Perhaps shaping UDP is too hard.

[William Unruh]

On 2014-09-11, mike cook <michae...@sfr.fr> wrote:
>
> Le 11 sept. 2014 ? 18:48, Rob a ?crit :

>
>> Paul <tik...@bodosom.net> wrote:
>>> As an aside has anyone tried shaping traffic to make the
>>> upstream/downstream latencies similar? It would seem more efficient
>>> to apply network solutions to network problems if possible.
>>
>> That does not work. The asymmetry is not caused by traffic but by
>> modem parameters.
>
> Did I miss something? The OP did not offer any evidence that there was path asymmetry, just that there was an unexplained offset between two GPS sync'd servers. It is often not possible to identify the origin of such an offset, but it would help if the suggested feature was implemented to take care of these corner cases. I saw that Dr Mills was in agreement back in 2005 but that the implementation is complex. If anyone wants a subject for an MSc project, this could be it.

No idea why a fudge parameter would be complicated. If you wanted to use
ntpd itself to figure out the assymmetry, that could well be
complicated. But if it is a fixed offset, I cannot see how that would be
complicated and it ihas already been implimented in the refclock case.

[Paul]

On Thu, Sep 11, 2014 at 2:08 PM, mike cook <michae...@sfr.fr> wrote:
> Did I miss something?

On Tue, Sep 9, 2014 at 3:17 PM, Rich Wales <ri...@richw.org> wrote:
> My home LAN is connected to my school's network via a cable modem.

If we make the (safe) assumption of a common cable ISP/FiOS in the
Palo Alto area the path is asymmetric.

[Rich Wales]

> Did I miss something? The OP did not offer any evidence that there was
> path asymmetry, just that there was an unexplained offset between two
> GPS sync'd servers.

The asymmetry in this case is being caused by the characteristics of the
cable modem that connects my residence with the campus network and the
rest of the Internet.

I've observed this consistently for several years, at various times of
day. I'm convinced that it is not being caused by traffic congestion
-- or, at least, that any traffic congestion factor is small compared
to a pretty constant offset of 2 - 3 msec.

> The asymmetry is caused by asymmetric latency which is caused (for
> our purposes) by asymmetric line speeds. Traffic shaping can change
> various things (depending on where you do it) including effective line
> speed (packets/s not bits/s). Perhaps shaping UDP is too hard.

I would certainly be open to doing experimentation in this regard. I
have essentially full control over two Linux boxes (one on either side
of the cable modem). However, up till now at least, I don't have any
experience at all with traffic shaping; suggested steps are welcome.

In the absence of a solution from the traffic-shaping domain, I would
like to be able to use the "fudge" command in conjunction with a peer
or server (currently, "fudge" works only with a reference clock). It
seems to me that the "time1" parameter would do what I want -- if it
could be specified for a peer or server, which currently it cannot.

Again, I'm running version 4.2.6p5 right now.

Rich Wales
ri...@richw.org

[Paul]

On Thu, Sep 11, 2014 at 2:29 PM, William Unruh <un...@invalid.ca> wrote:
> I doubt that NAT would add much assymetry

NAT is symmetric. Otherwise it wouldn't work. But I don't see how
that's part of anything at hand.
And yes the A in ADSL stands for Asymmetric. If you see the word
"home" in reference to a link in the US it's almost certainly
asymmetric modulo some special cases (ISDN, T1, Google Fiber etc.).

[mike cook]

Yup, AsymmetricDSL does have different up/down bit rates. What I really meant was that the difference would not explain his issue.
ex: with a 12Mbps down rate and 1.3Mbps up rate, the ratio is around 40usec to 300usec transfer of a 48byte NTP packet.

[Rob]

Paul <tik...@bodosom.net> wrote:
> On Thu, Sep 11, 2014 at 2:29 PM, William Unruh <un...@invalid.ca> wrote:
>> I doubt that NAT would add much assymetry
>
> NAT is symmetric. Otherwise it wouldn't work. But I don't see how
> that's part of anything at hand.

I never claimed it is part of the asymmetry, I only mentioned it because
usually you use a private IP range on a LAN, so when you are both on
a LAN and on Internet you either have two different IP addresses (as I do)
or you have NAT between an external address and your LAN range.
The NAT is not causing asymmetry but it makes it more difficult to separate
the internal from the external traffic.

[Rob]

mike cook <michae...@sfr.fr> wrote:
>
> Le 11 sept. 2014 à 18:48, Rob a écrit :
>
>> Paul <tik...@bodosom.net> wrote:
>>> As an aside has anyone tried shaping traffic to make the
>>> upstream/downstream latencies similar? It would seem more efficient
>>> to apply network solutions to network problems if possible.
>>
>> That does not work. The asymmetry is not caused by traffic but by
>> modem parameters.
>
> Did I miss something? The OP did not offer any evidence that there was path asymmetry, just that there was an unexplained offset between two GPS sync'd servers.

An offset between two GPS synced servers by definition means path asymmetry.

[Rob]

mike cook <michae...@sfr.fr> wrote:
>
> Le 11 sept. 2014 à 21:08, Paul a écrit :
>
>> On Thu, Sep 11, 2014 at 2:08 PM, mike cook <michae...@sfr.fr> wrote:
>>> Did I miss something?
>>
>> On Tue, Sep 9, 2014 at 3:17 PM, Rich Wales <ri...@richw.org> wrote:
>>> My home LAN is connected to my school's network via a cable modem.
>>
>> If we make the (safe) assumption of a common cable ISP/FiOS in the
>> Palo Alto area the path is asymmetric.
>
> Yup, AsymmetricDSL does have different up/down bit rates. What I really meant was that the difference would not explain his issue.
> ex: with a 12Mbps down rate and 1.3Mbps up rate, the ratio is around 40usec to 300usec transfer of a 48byte NTP packet.

Bitrate is not the only modem parameter.

[Charles Elliott]

The offset may be a function of distance.
Try this experiment:

Set up your ntp.conf file to have three servers (all examples assume you
are located in Eastern USA):
1. A relatively unused stratum 1 or 2 server as close to you as possible.
2. A relatively unused stratum 1 or 2 server about 1,000 miles away (e.g.,
ntp.melancthon.net)
3. A relatively unused stratum 1 or 2 server more than 2,000 miles away
(e.g., ntp1.tp.pl, ntp2.tp.pl, time.coi.pw.edu.pl, ntp.certum.pl).

On my computer, the offset is proportional to distance:

Remote Refid Stratum Type When Poll
Reach Delay Offset Jitter
BR-350P GPS 0 Local clock 7 16
017 0.000 -17.653 2.345
FreeNAS time-c.nist.gov 2 Unicast server 16 16
017 0.238 0.008 0.037
nist1-pa.ustiming.org ACTS 1 Unicast server 15 16
017 28.844 0.135 3.158
2a01:1102:0:b::2 ATOM 1 Unicast server 16
16 017 120.732 -5.145 2.151
2a01:1100:1::2 ATOM 1 Unicast server 15 16
017 128.756 -3.931 4.635
213.222.200.99 PPS 1 Unicast server 13 16
017 110.727 -0.968 4.119
ntp.coi.pw.edu.pl OCX0 1 Unicast server 14
16 017 122.100 -4.253 0.584
serenity.melancthon.net india.colorado.edu 2 Unicast server 35 32
003 53.520 2.019 3.556

Charles Elliott

> -----Original Message-----
> From: questions-bounces+elliott.ch=comca...@lists.ntp.org
> [mailto:questions-bounces+elliott.ch=comca...@lists.ntp.org] On
> Behalf Of mike cook
> Sent: Thursday, September 11, 2014 2:08 PM
> To: Questions List
> Subject: Re: [ntp:questions] Compensating for asymmetric delay on a
> per-peer/server basis?
>
>

[David Woolley]

The baud rate is 4kHz (approx). That means that the absolute minimum
packet time is 125 microseconds. As packets probably don't align with
signalling units, there is a good chance that you will need to add
another 125 microseconds. There is also going to be a delay of, on
average, 63 microseconds waiting for the start of a signalling unit.

To this you need to allow for any expansion due to FEC coding, and the
likely use of interleaving, which, to be effective, would need to spread
adjacent bits over quite a lot of signalling units. I can't find a
figure at the moment, but my guess is that it pushes the minimum delay
into the milliseconds range.

(Gamers tend to turn off interleaving, if they can, as they want low
latency at all costs. Businesses are most likely to want it on.)

[Rob]

David Woolley <da...@ex.djwhome.demon.invalid> wrote:
> On 11/09/14 22:11, Rob wrote:
>> mike cook <michae...@sfr.fr> wrote:
>>>
>>> Le 11 sept. 2014 à 21:08, Paul a écrit :
>>>
>>>> On Thu, Sep 11, 2014 at 2:08 PM, mike cook <michae...@sfr.fr> wrote:
>>>>> Did I miss something?
>>>>
>>>> On Tue, Sep 9, 2014 at 3:17 PM, Rich Wales <ri...@richw.org> wrote:
>>>>> My home LAN is connected to my school's network via a cable modem.
>>>>
>>>> If we make the (safe) assumption of a common cable ISP/FiOS in the
>>>> Palo Alto area the path is asymmetric.
>>>
>>> Yup, AsymmetricDSL does have different up/down bit rates. What I really meant was that the difference would not explain his issue.
>>> ex: with a 12Mbps down rate and 1.3Mbps up rate, the ratio is around 40usec to 300usec transfer of a 48byte NTP packet.
>>
>> Bitrate is not the only modem parameter.
>>
>
> The baud rate is 4kHz (approx). That means that the absolute minimum
> packet time is 125 microseconds. As packets probably don't align with
> signalling units, there is a good chance that you will need to add
> another 125 microseconds. There is also going to be a delay of, on
> average, 63 microseconds waiting for the start of a signalling unit.

The poster had no DSL, he mentioned a cable modem.
In a cable modem there is another issue: the subscribers share the
same uplink channel in an alternating fashion, while the downlink channel
is used only by the cable headend. To avoid collisions, there will
usually be some user timeslot allocation by the headend.

[David Lord]

mike cook wrote:
> Le 11 sept. 2014 � 21:08, Paul a �crit :

>
>> On Thu, Sep 11, 2014 at 2:08 PM, mike cook <michae...@sfr.fr> wrote:
>>> Did I miss something?
>> On Tue, Sep 9, 2014 at 3:17 PM, Rich Wales <ri...@richw.org> wrote:
>>> My home LAN is connected to my school's network via a cable modem.
>> If we make the (safe) assumption of a common cable ISP/FiOS in the
>> Palo Alto area the path is asymmetric.
>
> Yup, AsymmetricDSL does have different up/down bit rates. What I really meant was that the difference would not explain his issue.
> ex: with a 12Mbps down rate and 1.3Mbps up rate, the ratio is around 40usec to 300usec transfer of a 48byte NTP packet.

Hi

My experience is different. Due to uplink pipe being very much
less capable than downlink I had 10-100ms latencies if pipe was
full. Solution for me was to limit my outgoing rates to 80-90%
which actually increased upload speed by almost 2x.

I've adjusted that filter since 2005 each time my adsl has been
upgraded.


David

[Harlan Stenn]

There are a bunch of issues here, and I don't think there is a simple
answer.

For starters, there is static asymmetry and dynamic asymmetry.

One of the core issues is that NTP is frequently multihop, and the
routing for at least some of these connections can spontaneously change.

Declaring an asymmetry correction for an interface will affect all
connections over that interface. Sometimes that's OK, sometimes not.

Declaring an asymmetry correction for a given remote server hardcodes
assumptions that almost certainly will change over time.

The trick is that we don't know how many hops there are between "here"
and "there", and the number and location of these hops can change.

Precision Time Protocol looks closer at these issues, and PTP is
designed to work on point-to-point links.

So if one can use a local good reference time and use those timestamps
to compare with remote good reference times, one can have a better
chance to identify some of the static asymmetry issues.

Dealing with the dynamic ones is harder...

The soution seems to depend on having multiple sources of good time, and
having access to these good time sources via different "paths".
--
Harlan Stenn <st...@ntp.org>
http://networktimefoundation.org - be a member!

[detha]

That is, if you run ntpd on a directly internet-facing machine. In most
scenarios involving NAT, the NAT happens in a router/firewall/modem
appliance, not on the ntpd server itself.

Also, if/when IPv6 becomes more prevalent, all machines on the LAN will
hopefully have a public address.

You could specify a different offset based on /source/ address of the
query, or on incoming interface, but I am afraid that will open yet
another can of worms.


-d

[Martin Burnicki]

William Unruh wrote:
> No idea why a fudge parameter would be complicated. If you wanted to use
> ntpd itself to figure out the assymmetry, that could well be
> complicated. But if it is a fixed offset, I cannot see how that would be
> complicated and it ihas already been implimented in the refclock case.

I tried to explain this in my earlier email.

If you have a local (GPS controlled) NTP server plus some NTP clients
connected to the inet via an asymmetric connection you

- need to apply a fudge time if your local server contacts external servers

- need to apply the same fudge time with reversed sign if *you* try to
compensate the time error caused by *your* inet connection when external
clients try to get the time from your NTP server

- need to apply no fudge time at all for connections on your local network

I don't know how this is in other countries, but at least here in
Germany a typical home setup is a NAT router connected to the inet via
ADSL, and the router providing an internal switch with several ports to
which you can connect your devices, e.g. your NTP server and some NTP
clients.

As has been stated in some other posts:

- NAT doesn't hurt at all, unless you are trying to use NTP's authentication

- the asymmetry of the ADSL connection causes a time error of a certain
range, the sign of which depends on whether you look from an external
node to your home NTP server, or from your home NTP server to some
remote NTP server

So an overall solution might be:

- if the source IP of incoming client requests is not on your local
subnet, apply a fudge time

- if the destination address of reply packets you send is not on your
local subnet, apply a fudge time with reversed sign

- otherwise (source and destination on your local subnet) apply no fudge
time at all

[Martin Burnicki]

I think an IP based rules could do the job. See my other reply.

[Rob]

Martin Burnicki <martin....@meinberg.de> wrote:
> - NAT doesn't hurt at all, unless you are trying to use NTP's authentication

NAT in itself does not hurt, but when you want to be a timeserver for
a large number of clients, it can be a problem.

Many home routers have no "static NAT" but only "portforwarding" which
creates dynamic NAT entries on demand. As UDP has no session concept,
such NAT entries have a lifetime of usually a couple of minutes.

When you serve thousands of clients, this tends to overflow the NAT
table or stress the lookup code so much that it overloads the CPU.

[Martin Burnicki]

Rob wrote:
> An offset between two GPS synced servers by definition means path asymmetry.

+1

However, path asymmetry includes

- systematic asymmetry (e.g. ADSL) on one ore more (!) parts of the path
between 2 nodes

- errors due to different link speeds, e.g 100 MBit from 1 switch port
to one node, and GBit from another switch port to the other node

- different processing speeds for incoming and outgoing packets at the
client ans server side

The time from when a packet comes in from the wire until it arrives at
the application depends on the protocol stack, CPU power, and system load

Similarly, the time from when a packet is sent by ntpd until it actually
goes out on depends on the protocol stack, CPU power, and system load.

However, the mean processing time (without noticeable system load) is
usually different for sending and receiving, so this also a kind of
asymmetry.

As long as the ratio between these processing times is similar on the
client and server the resulting time error is low. However, if there's
only an asymmetry at one end the resulting time error is higher.

We have made some tests with hardware time stamping of NTP packets:

- if hardware time stamping is used at both ends then all processing
times are eliminate, and thus the time error is obviously small

- if hardware time stamping is only used at one end (client *or* server)
then the resulting error is usually *larger* than the mean error you see
if no hardware time stamping is used at all, which is due to the
difference/asymmetry in processing times for incoming and outgoing packets.

(I know there are additional conditons like IRQ coalescense which make
things even worse)

[Martin Burnicki]

Haven't had such case, yet since my home NTP server doesn't serv 1000s
of clients, but sounds reasonable and should be kept in mind.

[Rob]

It is sometimes a problem when you become member of the NTP pool.

[Terje Mathisen]

I'm in that situation, but my ntp server is only announced on IPv6 where
I do have a static/personal network range, and the server is also my
gateway machine, i.e. it gets all port 123 packets forwarded without any
NAT type source port rewriting.

I also have a nicely symmetric 50/50 fiber connection so I've stopped
worrying about asymmetric delays. :-)

The best ipv6 link I've seen has been to a peer in South Africa, where
we had many (100?) milliseconds of path delay, but our (gps-based) local
times agreed within 100 us, i.e. a completely symmetric link all the
way. :-)

Terje

--
- <Terje.Mathisen at tmsw.no>
"almost all programming can be viewed as an exercise in caching"

[Rob]

Terje Mathisen <terje.m...@tmsw.no> wrote:
> Rob wrote:
>> Martin Burnicki <martin....@meinberg.de> wrote:
>>> Rob wrote:
>>>> Martin Burnicki <martin....@meinberg.de> wrote:
>>>> When you serve thousands of clients, this tends to overflow the NAT
>>>> table or stress the lookup code so much that it overloads the CPU.
>>>
>>> Haven't had such case, yet since my home NTP server doesn't serv 1000s
>>> of clients, but sounds reasonable and should be kept in mind.
>>
>> It is sometimes a problem when you become member of the NTP pool.
>>
> I'm in that situation, but my ntp server is only announced on IPv6 where
> I do have a static/personal network range, and the server is also my
> gateway machine, i.e. it gets all port 123 packets forwarded without any
> NAT type source port rewriting.

Same for me, I am in the IPv6 pool as well. But that is on a small
system where I did not want the extra load of being in the IPv4 pool.

A number of years ago I had my system at home in the IPv4 pool, but
then I installed a new modem/router and even when it was configured
in "no NAT" mode it still has tracking always enabled and was swamped
by the number of "sessions". So I left the pool.

[Paul]

On Fri, Sep 12, 2014 at 4:03 AM, Martin Burnicki
<martin....@meinberg.de> wrote:
> +1
>
> However, path asymmetry includes

I think you're abusing the conventional notion of asymmetric latency.
Uncorrected bandwidth asymmetry will result in offsets between
truechimers. Offsets between clocks that we hope are truechimers
doesn't mean link-speed asymmetry. I think it's confusing to be
considering CPU effects (or NAT) in this conversation because the OP
probably doesn't have these issues.

[Paul]

On Thu, Sep 11, 2014 at 3:50 PM, mike cook <michae...@sfr.fr> wrote:
> Yup, AsymmetricDSL does have different up/down bit rates. What I really meant was that the difference would not explain his issue.
> ex: with a 12Mbps down rate and 1.3Mbps up rate, the ratio is around 40usec to 300usec transfer of a 48byte NTP packet.

I was thinking closer to 1ms assuming a 10:1 asymmetry. In any case
it would helpful to know the link speeds, delay and the sign of the
offset i.e. ntpq output from each end might be illuminating. I also
see consistent 2ms offsets between my stratum 1 servers and external
stratum 1 servers. Looking on the latter two I see the same offset
with opposite sign. I have a 30/5 link.

remote refid st t when poll reach delay offset jitter
==============================================================================
o127.127.22.0 .GPPS. 0 l 1 8 377 0.000 -0.007 0.008
2610:20:6f15:15 .ACTS. 1 u 111 128 377 35.726 2.098 0.153
2620:cc:8000:80 .GPPS. 1 u 47 64 377 35.532 2.034 0.304
2620:cc:8000:40 .GPPS. 1 u 2 64 377 35.308 2.054 0.675

[Rob]

Paul <tik...@bodosom.net> wrote:
> On Fri, Sep 12, 2014 at 4:03 AM, Martin Burnicki
> <martin....@meinberg.de> wrote:
>> +1
>>
>> However, path asymmetry includes
>
>
> I think you're abusing the conventional notion of asymmetric latency.
> Uncorrected bandwidth asymmetry will result in offsets between
> truechimers. Offsets between clocks that we hope are truechimers
> doesn't mean link-speed asymmetry.

No, not link-speed asymmetry but propagation-time asymmetry that may
be the result of link-speed asymmetry but also could be caused by
other asymmetries like interleaving in one direction, medium access
protocols that do time-slotting in one direction, etc.

[Paul]

On Fri, Sep 12, 2014 at 11:48 AM, Rob <nom...@example.com> wrote:
> No, not link-speed asymmetry but propagation-time asymmetry

Sure, you can say that after the fact. Only one other person in this
conversation *particularly, not the OP* meant that. As I said "the
conventional notion of asymmetric latency". It's not helpful to
hijack the thread.

I'm also quite confident that asymmetric network effects other than
link speed are second order.

[Rich Wales]

Replying to Charles Elliott:

> The offset may be a function of distance. Try this experiment:

> Set up your ntp.conf file to have three servers . . . :

> 1. A relatively unused stratum 1 or 2 server as close to you as possible

> 2. A relatively unused stratum 1 or 2 server about 1,000 miles away

> 3. A relatively unused stratum 1 or 2 server more than 2,000 miles away

OK, here is information taken from two local servers under my control.

==============================================================================

This first machine (171.67.203.16) is on the Stanford University campus. The
first two peers listed below are located on the Stanford campus; the third
peer is also run by Stanford but is about 50 miles east of the campus.

+171.64.7.105 .PPS. 1 u 1012 1024 377 0.465 -0.045 0.076
+171.64.7.67 .PPS. 1 u 217 1024 377 0.584 -0.043 0.081
*204.63.224.70 .PPS. 1 u 737 1024 377 1.803 -0.031 0.252

This next peer is my home machine (the one I described earlier as being
connected to the Internet via a cable modem), with its own local GPS clock.

-68.65.164.12 .PPS. 1 u 13 16 372 8.168 -2.126 4.585

Finally, these two servers are in Utah (Xmission) and Poland.

-198.60.22.240 .GPS. 1 u 721 1024 377 18.931 0.239 0.045
-213.222.200.99 .PPS. 1 u 833 1024 377 172.099 -4.083 1.167

==============================================================================

Now, here is the info from my home machine (68.65.164.12). First, my local
GPS reference clock:

*127.127.28.1 .PPS. 0 l 14 16 377 0.000 -0.003 0.025
x127.127.28.0 .GPS. 8 l 7 16 377 0.000 -37.762 13.399

Next, my campus machine (see above for details):

-171.67.203.16 204.63.224.70 2 u 2 16 333 9.689 2.146 3.930

And finally, the same two remote servers (in Utah and Poland) that I used on
my campus machine:

+198.60.22.240 .GPS. 1 u 45 64 377 35.222 6.542 2.864
+213.222.200.99 .PPS. 1 u 18 64 377 191.192 4.890 8.968

==============================================================================

Any thoughts?

Rich Wales
ri...@richw.org

[mike cook]

Le 13 sept. 2014 à 07:46, Rich Wales a écrit :

> Replying to Charles Elliott:
>
>> The offset may be a function of distance. Try this experiment:
>> Set up your ntp.conf file to have three servers . . . :
>> 1. A relatively unused stratum 1 or 2 server as close to you as possible
>> 2. A relatively unused stratum 1 or 2 server about 1,000 miles away
>> 3. A relatively unused stratum 1 or 2 server more than 2,000 miles away
>
> OK, here is information taken from two local servers under my control.
>
> ==============================================================================
>
> This first machine (171.67.203.16) is on the Stanford University campus. The
> first two peers listed below are located on the Stanford campus; the third
> peer is also run by Stanford but is about 50 miles east of the campus.
>
> +171.64.7.105 .PPS. 1 u 1012 1024 377 0.465 -0.045 0.076
> +171.64.7.67 .PPS. 1 u 217 1024 377 0.584 -0.043 0.081

delay is low so you are nearby in a network sense, maybe on the same net (what is your netmask).

> *204.63.224.70 .PPS. 1 u 737 1024 377 1.803 -0.031 0.252
>
> This next peer is my home machine (the one I described earlier as being
> connected to the Internet via a cable modem), with its own local GPS clock.
>
> -68.65.164.12 .PPS. 1 u 13 16 372 8.168 -2.126 4.585

I thought that you were connecting over a VPN? The delay is high but not exceptionally and you can get good stability even so. However, the jitter is huge and your link is not stable as your reach is not 377. Here you missing polls , possibly due to a timeout or dropped packets. These are not symptoms of link asymmetry, but might be due to unbalanced packet priority or something. Do you have HD TV on at the same time.

>
> Finally, these two servers are in Utah (Xmission) and Poland.
>
> -198.60.22.240 .GPS. 1 u 721 1024 377 18.931 0.239 0.045
> -213.222.200.99 .PPS. 1 u 833 1024 377 172.099 -4.083 1.167
>
> ==============================================================================
>
> Now, here is the info from my home machine (68.65.164.12). First, my local
> GPS reference clock:
>
> *127.127.28.1 .PPS. 0 l 14 16 377 0.000 -0.003 0.025
> x127.127.28.0 .GPS. 8 l 7 16 377 0.000 -37.762 13.399
>
> Next, my campus machine (see above for details):
>
> -171.67.203.16 204.63.224.70 2 u 2 16 333 9.689 2.146 3.930

again, reach is showing a 'broken' link.

>
> And finally, the same two remote servers (in Utah and Poland) that I used on
> my campus machine:
>
> +198.60.22.240 .GPS. 1 u 45 64 377 35.222 6.542 2.864
> +213.222.200.99 .PPS. 1 u 18 64 377 191.192 4.890 8.968

These are not showing bad reach, at least not on these samples. Do they ever show a non 377 value?

>
> ==============================================================================
>
> Any thoughts?

You are up late.

>
> Rich Wales
> ri...@richw.org

[mike cook]

Le 13 sept. 2014 à 07:46, Rich Wales a écrit :
>
>

> -68.65.164.12 .PPS. 1 u 13 16 372 8.168 -2.126 4.585
>

> -171.67.203.16 204.63.224.70 2 u 2 16 333 9.689 2.146 3.930
>

> Any thoughts?
>
I should have added
netstat -i

are you dropping packets, getting errors

> Rich Wales
> ri...@richw.org

[William Unruh]

On 2014-09-12, Martin Burnicki <martin....@meinberg.de> wrote:
> William Unruh wrote:
>> No idea why a fudge parameter would be complicated. If you wanted to use
>> ntpd itself to figure out the assymmetry, that could well be
>> complicated. But if it is a fixed offset, I cannot see how that would be
>> complicated and it ihas already been implimented in the refclock case.
>
> I tried to explain this in my earlier email.
>
> If you have a local (GPS controlled) NTP server plus some NTP clients
> connected to the inet via an asymmetric connection you
>
> - need to apply a fudge time if your local server contacts external servers

Again, let us separate the two questions-- one is trying to make the
time on your computer equal UTC, the other is to deliver that UTC to
others. At present it is the first that is most important to me, and for
the solution to THAT problem that it seems to me to be simple-- exactly
the same as the fudge for refclock.
Also if you have not solved the first, then the second-- delivering
exact time to others, cannot be solved.

One could argue that the second is the problem for the client, not the
server to solve. Ie, if you connect to some source, it is up to you to
figure out if there is an assymetry on the path from them to you. Now
the server could help in this if it knows that there are some local
assymmeteries, but that is help, not duty.

>
> - need to apply the same fudge time with reversed sign if *you* try to
> compensate the time error caused by *your* inet connection when external
> clients try to get the time from your NTP server
>
> - need to apply no fudge time at all for connections on your local network
>
> I don't know how this is in other countries, but at least here in
> Germany a typical home setup is a NAT router connected to the inet via
> ADSL, and the router providing an internal switch with several ports to
> which you can connect your devices, e.g. your NTP server and some NTP
> clients.
>
> As has been stated in some other posts:
>
> - NAT doesn't hurt at all, unless you are trying to use NTP's authentication
>
> - the asymmetry of the ADSL connection causes a time error of a certain
> range, the sign of which depends on whether you look from an external
> node to your home NTP server, or from your home NTP server to some
> remote NTP server

Agreed.

>
> So an overall solution might be:
>
> - if the source IP of incoming client requests is not on your local
> subnet, apply a fudge time

No. That is their problem not yours. Otherwise you havee too many cooks.

[Paul]

On Sat, Sep 13, 2014 at 1:46 AM, Rich Wales <ri...@richw.org> wrote:
> Any thoughts?

Something is wrong with your home machine but there's nothing you can
do with stock NTP to fix your offset.

As posted earlier I see exactly the same ~2ms offset. However as
noted -- given that you're on the same network -- these two lines are
junk.

-68.65.164.12 .PPS. 1 u 13 16 372 8.168 -2.126 4.585

-171.67.203.16 204.63.224.70 2 u 2 16 333 9.689 2.146 3.930

and these two are pretty bad (and given the same polling interval
might also be junk)

+198.60.22.240 .GPS. 1 u 45 64 377 35.222 6.542 2.864
+213.222.200.99 .PPS. 1 u 18 64 377 191.192 4.890 8.968

My original question remains. Do you have a production need to remove
the offset or is it for aesthetic reasons?

Regarding shaping. An extremely cursory look suggests you can use a
Linux qdisc to insert a delay (shape to effective speed) in NTP
traffic so if you have an appropriate box to place between you modem
and your home network you could try that -- after you resolve your
current problem.

[Martin Burnicki]

Phil W Lee wrote:
> William Unruh <un...@invalid.ca> considered Sat, 13 Sep 2014 11:56:37

> +0000 (UTC) the perfect time to write:
>
>> On 2014-09-12, Martin Burnicki <martin....@meinberg.de> wrote:
>>> William Unruh wrote:
>>>> No idea why a fudge parameter would be complicated. If you wanted to use
>>>> ntpd itself to figure out the assymmetry, that could well be
>>>> complicated. But if it is a fixed offset, I cannot see how that would be
>>>> complicated and it ihas already been implimented in the refclock case.
>>>
>>> I tried to explain this in my earlier email.
>>>
>>> If you have a local (GPS controlled) NTP server plus some NTP clients
>>> connected to the inet via an asymmetric connection you
>>>
>>> - need to apply a fudge time if your local server contacts external servers
>>
>> Again, let us separate the two questions-- one is trying to make the
>> time on your computer equal UTC, the other is to deliver that UTC to
>> others. At present it is the first that is most important to me, and for
>> the solution to THAT problem that it seems to me to be simple-- exactly
>> the same as the fudge for refclock.
>> Also if you have not solved the first, then the second-- delivering
>> exact time to others, cannot be solved.
>

> Additionally, the same solution, if applied by the client, would work
> just as well for them as it does for you.

Only if the client has explicitly configured my server as time source.
If I'm a member of the pool and clients get my IP or the IP of other
pool servers dynamically, how should they know if they had too apply a
fudge time because *my* server is accessed via an ADSL line while others
are not, or with a different ratio of upload/download speeds?

If I know the time error caused by *my* ADSL connection I could try to
compensate it as good as possible so it doesn't make a difference for a
client (in terms of time error) if he polls my server or different one
from the pool which is eventually connected via a symmetric line.

If another client is also connected via ADSL he could apply another
fudge time to compensate the asymmetry in *his* connection.

>> One could argue that the second is the problem for the client, not the
>> server to solve. Ie, if you connect to some source, it is up to you to
>> figure out if there is an assymetry on the path from them to you. Now
>> the server could help in this if it knows that there are some local
>> assymmeteries, but that is help, not duty.
>>

> Heck, you can only know about the part of the path on the local loop
> anyway, because after that the route diversifies, and may even change
> dynamically, as well as by destination.

I don't think there are systematic asymmetries on internet backbones as
there are with ADSL lines, so if every end node took care to compensate
the time error caused by his *own* inet connection the overall accuracy
should increase, regardless which time source you chose.

> And those who are seriously concerned about a ms or two are likely to
> want to verify their time sources anyway, in which case a small web
> page on the same IP address as the ntp server could be used to advise
> prospective clients what, if any, asymmetry is known on the local loop
> of the server, which version of ntp is needed for corrective
> parameters to be set, and what line(s) to include in the ntp.conf to
> achieve that.

As said above this wouldn't help in case of pool servers where clients
might get your IP automatically.

> I suppose there might be some value in recommending a different port
> for ntp related web information, just in case a single machine (or NAT
> gateway) is used to reach both an ntp server and a web server with
> other content - I'd suggest 8123 would be memorable for such a purpose
> (the 8 coming from the normal http port 80 and the 123 from the ntp
> port assignment).
> Most users aren't likely to be that bothered about a small (sub
> 1/100th sec) difference between real UTC and the time received,
> provided the time served is consistent.

So they can simply don't care about time corrections. What I mean is
that you *could* take care if you wanted, but you don't have to if the
accuracy you achieve is good enough for your requirements.

[Martin Burnicki]

Rob schrieb:

I just meant that there can be different overall propagation delays for
a packet from the client ntpd and to a server ntpd, and from the server
back to the client, and a possible ADSL connection is only is only one
component of the overall asymmetry.

[Martin Burnicki]

William Unruh wrote:
> On 2014-09-12, Martin Burnicki <martin....@meinberg.de> wrote:
>> William Unruh wrote:
>>> No idea why a fudge parameter would be complicated. If you wanted to use
>>> ntpd itself to figure out the assymmetry, that could well be
>>> complicated. But if it is a fixed offset, I cannot see how that would be
>>> complicated and it ihas already been implimented in the refclock case.
>>
>> I tried to explain this in my earlier email.
>>
>> If you have a local (GPS controlled) NTP server plus some NTP clients
>> connected to the inet via an asymmetric connection you
>>
>> - need to apply a fudge time if your local server contacts external servers
>
> Again, let us separate the two questions-- one is trying to make the
> time on your computer equal UTC, the other is to deliver that UTC to
> others. At present it is the first that is most important to me, and for
> the solution to THAT problem that it seems to me to be simple-- exactly
> the same as the fudge for refclock.

Yes. And if you make the fudge depending on the IP range because you
know it's *your* ADSL line causing the time error then the same
correction is applied for *all* NTP servers used as time source and
accessed via the inet. ;-)

Of course the simplest approach would be to allow a fudge time on a per
"server" line, but who says that ntpd couldn't support both approaches,
one per server, and another on per IP range.

> Also if you have not solved the first, then the second-- delivering
> exact time to others, cannot be solved.

That's not quite true. If my local server connected via ADSL is not
synchronized by GPS but only by servers on the internet then the time of
my local server is off by a few milliseconds tue to the ADSL asymmetry.

However, for other nodes on the internet which poll *my* local server
the asymmetry due to my ADSL connection is once more applied with
reversed sign, so from my client's point of view there is no time error
due to *my* ADSL line.

> One could argue that the second is the problem for the client, not the
> server to solve. Ie, if you connect to some source, it is up to you to
> figure out if there is an assymetry on the path from them to you. Now
> the server could help in this if it knows that there are some local
> assymmeteries, but that is help, not duty.

I agree, but I don't think we are discussing duties here. I was just
thinking about ways what *can* be done to make the time on my server as
accurate as possible, and also server time to others as accurately as
possible.

>> - need to apply the same fudge time with reversed sign if *you* try to
>> compensate the time error caused by *your* inet connection when external
>> clients try to get the time from your NTP server
>>
>> - need to apply no fudge time at all for connections on your local network
>>
>> I don't know how this is in other countries, but at least here in
>> Germany a typical home setup is a NAT router connected to the inet via
>> ADSL, and the router providing an internal switch with several ports to
>> which you can connect your devices, e.g. your NTP server and some NTP
>> clients.
>>
>> As has been stated in some other posts:
>>
>> - NAT doesn't hurt at all, unless you are trying to use NTP's authentication
>>
>> - the asymmetry of the ADSL connection causes a time error of a certain
>> range, the sign of which depends on whether you look from an external
>> node to your home NTP server, or from your home NTP server to some
>> remote NTP server
>
> Agreed.
>>
>> So an overall solution might be:
>>
>> - if the source IP of incoming client requests is not on your local
>> subnet, apply a fudge time
>
> No. That is their problem not yours. Otherwise you havee too many cooks.

Please see also my other reply to Phil W Lee.

[William Unruh]

It they are willing to put up with the pool, they are willing to accept
what they get. The quality of the servers in the pool varies a lot. Some
will have microsecond accuracy, some millisecond. It says that they
really do not care about the ultimate accuracy. If your assymmetric
delay is such that it puts out your time by say 10ms or 1 sec, perhaps
you should refrain from participating in the pool.

But in any case this is a very different situation from making sure that
your own clock is accurate, and having ntpd provide solutions for that
case should not wait for solutions for this far more difficult case. The
solutions for your own acuracy seem simple, kand just hneed an extention
fo the fudge time directrive to servers as well as refclocks.
As someone pointed out, the solution for compensating for delays in the
time you server to others is far more complex.