How to determin hardware latency for PPS offset given simple tools.

[Paul G]

I have four GPS receivers into four different computers. I've assumed that I
should be able to get small offsets between them but to do so I need
to set time1. If that's the correct approach how do I pick the "right"
system to be the benchmark. So far I've just been assuming that
the purpose built M. Tharp server should have the lowest latency but I'm not sure -- in any case, at present, I can't adjust it.

Each machine in the truncated table below (except 210) has a time1
value derived in an ad hoc fashion.

l r refid st delay offset jitter
==================================================
1 o22.0 .PPS. 0 0.000 -0.001 0.001
2 +244 .PPS. 1 0.068 0.000 0.006
2 *210 .GPS. 1 0.514 -0.008 0.136
2 +192 .PPS. 1 0.455 0.008 0.032


I feel as if this topic should be clearly explained somewhere but I've been
unable to find it.

[unruh]

On 2013-05-24, Paul G <bod...@gmail.com> wrote:
> I have four GPS receivers into four different computers. I've assumed that I

GPS receivers are not particularly good time sources UNLESS you use PPS.
If you use PPS they should all agree to the 1micro second level.
You should NOT need time1. If youjust use the serial data, you will be
lucky to get millisecond agreement, and that only with a lot of work.
Ie, if you are going to set up gps, set up pps.

Since you seem to be using it, you should get us accuracy from them
without any offset. grabbing times via the net from the those computers
will be dominated by network issues.

> should be able to get small offsets between them but to do so I need
> to set time1. If that's the correct approach how do I pick the "right"
> system to be the benchmark. So far I've just been assuming that
> the purpose built M. Tharp server should have the lowest latency but I'm not sure -- in any case, at present, I can't adjust it.
>
> Each machine in the truncated table below (except 210) has a time1
> value derived in an ad hoc fashion.

Are they a secret?

[David Taylor]

Paul,

Those PCs with PPS feed should have zero offset, and time1 should be set
to zero unless you /know/ that the GPS PPS signal is offset (e.g. you
are feeding it via several microseconds delay of cable, which is
unlikely). What NTP is actually syncing to is an internal PPS (if you
like), i.e. the external PPS signal delayed by whatever processing is
taking place inside the PC, but any external things you measure will be
subject to the same delay. What happens with the fudge set to zero?

Please show the table for each machine.

For the PC with just GPS sync, you might get it a PPS feed somehow, use
PPS as the reference, set GPS/NMEA to noselect in the ntp.conf, and
record the peersstats. I wrote this up with help from others here:

http://www.satsignal.eu/ntp/Garmin-GSP18x-LVC-firmware-issue.htm#analysis

and although the results in that case are wrong, the method is, I
believe, correct.
--
Cheers,
David
Web: http://www.satsignal.eu

[Miguel Gonçalves]

Hi Paul!

My recent experience with 4 stratum 1 servers (one of them an
appliance from Meinberg - great box BTW) is that you will never have
every server in sync with each other due to the latency and delays of
the network (even LAN, same segment as is in this case). I don't depend
on external NTP servers.

My Meinberg NTP server:

$ ntpq -p 10.0.2.1
remote refid st t when poll reach delay offset jitter
==============================================================================
LOCAL(0) .LOCL. 10 l 8 16 377 0.000 0.000 0.001
+GENERIC(0) .GPS. 0 l 13 16 377 0.000 0.000 0.001
oPPS(0) .PPS. 0 l 7 16 377 0.000 -0.001 0.001
-10.0.2.2 .PPS. 1 u 37 64 377 0.199 0.007 0.031
-10.0.2.3 .PPS. 1 u 6 64 377 0.171 -0.004 0.008
+10.0.2.4 .PPS. 1 u 51 64 377 0.157 -0.002 0.009

My Garmin 18 LVC on the roof:

$ ntpq -p 10.0.2.2
remote refid st t when poll reach delay offset jitter
==============================================================================
oGPS_NMEA(0) .PPS. 0 l 5 16 377 0.000 0.000 0.004
+10.0.2.1 .PPS. 1 u 2 16 377 0.157 0.005 0.013
+10.0.2.3 .PPS. 1 u 1 16 377 0.144 -0.001 0.004
+10.0.2.4 .PPS. 1 u 16 16 377 0.140 -0.002 0.004

My Sure GPS board on the window sill:

$ ntpq -p 10.0.2.3
remote refid st t when poll reach delay offset jitter
==============================================================================
oGPS_NMEA(0) .PPS. 0 l 13 16 377 0.000 0.001 0.004
+10.0.2.1 .PPS. 1 u 10 16 377 0.166 0.003 0.008
+10.0.2.2 .PPS. 1 u 5 16 377 0.138 0.004 0.004
+10.0.2.4 .PPS. 1 u 8 16 377 0.140 0.001 0.004

My 2nd Sure GPS board on the same window sill:

$ ntpq -p 10.0.2.4
remote refid st t when poll reach delay offset jitter
==============================================================================
oGPS_NMEA(0) .PPS. 0 l 7 16 377 0.000 0.000 0.004
+10.0.2.1 .PPS. 1 u 4 16 377 0.201 0.000 0.011
+10.0.2.2 .PPS. 1 u 3 16 377 0.136 0.002 0.004
+10.0.2.3 .PPS. 1 u 2 16 377 0.140 0.001 0.004

For me < 10 us offsets is OK and acceptable between stratum 1 machines.

I believe this is what is expected.

Regards,
Miguel

[unruh]

On 2013-05-26, Miguel Gon?alves <miguel.barbo...@gmail.com> wrote:
> Hi Paul!
>
> On Fri, May 24, 2013 at 9:28 PM, Paul G <bod...@gmail.com> wrote:
>> I have four GPS receivers into four different computers. I've assumed that I
>> should be able to get small offsets between them but to do so I need
>> to set time1. If that's the correct approach how do I pick the "right"
>> system to be the benchmark. So far I've just been assuming that
>> the purpose built M. Tharp server should have the lowest latency but I'm not sure -- in any case, at present, I can't adjust it.
>>
>> Each machine in the truncated table below (except 210) has a time1
>> value derived in an ad hoc fashion.
>>
>> l r refid st delay offset jitter
>> ==================================================
>> 1 o22.0 .PPS. 0 0.000 -0.001 0.001
>> 2 +244 .PPS. 1 0.068 0.000 0.006
>> 2 *210 .GPS. 1 0.514 -0.008 0.136
>> 2 +192 .PPS. 1 0.455 0.008 0.032
>>
>>
>> I feel as if this topic should be clearly explained somewhere but I've been
>> unable to find it.
>
> My recent experience with 4 stratum 1 servers (one of them an
> appliance from Meinberg - great box BTW) is that you will never have
> every server in sync with each other due to the latency and delays of
> the network (even LAN, same segment as is in this case). I don't depend
> on external NTP servers.

Lan will have delays of the order of .15ms (worse for gigabit) with
accuracy of the order of 10-20us. Direct PPS will be of order 2-5us, but
only if you have only one. If you have more than one, then they get in
the way of each other ( the interrupts all arrive at the same time, and
each has to wait for th other to finish which is about 10us. )

[Paul G]

On Saturday, May 25, 2013 1:06:48 AM UTC-4, unruh wrote:
> GPS receivers are not particularly good time sources UNLESS you use PPS.

All of my clocks use PPS and in fact none of them use GPS to number the seconds
except the one that uses refid GPS. That clock is a purpose built NTP-equivalent server called a Laureline which uses PPS and numbers
seconds via serial input.

Per the refclock document the PPS/NMEA (20/22) drivers use time1 to correct for
PPS offset while 22 uses time2 to correct for serial latency. Since the clocks
differ by O(100 microsec) without time1 correction I assumed that was internal rather than network latency.

> If you use PPS they should all agree to the 1micro second level.

They do per ntpq given resp. 124, 151 and 85 microsecond time1.

> grabbing times via the net from the those computers
> will be dominated by network issues.

I've been assuming that since I get fairly consistent offsets (fudged or not)
and jitter that network latency is not the culprit.

But I'm clearly confused about something.

[Paul G]

On Saturday, May 25, 2013 3:05:13 AM UTC-4, David Taylor wrote:
> What happens with the fudge set to zero?
> Please show the table for each machine.

Showing 192.168.0.2 and .244 with time1 = 0 as compared to the first post where
time1 was ~100 microsec.

These two have gigabit interfaces. 0.2 is driven by a Garmin 18x and 0.244 is
driven by a Firefly (I).

0.2

remote refid st t when poll reach delay offset jitter
==============================================================================

o127.127.22.0 .PPS. 0 l 1 8 377 0.000 -0.001 0.002
+192.168.0.244 .PPS. 1 u - 8 377 0.075 -0.128 0.001
*192.168.0.210 .GPS. 1 u 7 8 377 0.519 0.117 0.098

0.244

remote refid st t when poll reach delay offset jitter
==============================================================================

o127.127.22.0 .PPS. 0 l 5 8 377 0.000 0.000 0.002
+192.168.0.2 .PPS. 1 u 4 8 377 0.079 0.132 0.003
+192.168.0.210 .GPS. 1 u 4 8 377 0.547 0.253 0.135
*192.168.0.192 .PPS. 1 u 3 8 377 0.475 0.262 0.036

[David Taylor]

Well, that's interesting! Many thanks. 128/132 microseconds is way
more than I would expect a GPS/PPS signal to be off - it should be
nearer 1 microsecond (unless you're doing something unusual like a long
line driver, or triggering off the wrong edge of a 130 microsecond
pulse, perhaps?). I can't see a spec for the pulse width from the
Firefly, but I do see it has a programmeable offset. I presume that's
set to zero?

Purely for comparison, on the systems here:

- my FreeBSD/Intel Atom server sees my Windows XP system as -0.004, and
my Win-7 system as -0.077.

- my three Raspberry Pi cards see the FreeBSD server as -0.030 to +0.100.

- my three Raspberry Pi cards see the Windows XP system as -0.040 to +
0.070, and my Win-7 system as -0.020 to + 0.130. One RasPi is connected
via Wi-Fi.

All of these are triggered from PPS signals which are well within 0.1
microseconds of each other when viewed on a digital storage scope. The
18x LVC signal to the FreeBSD node may be a little delayed because of
the way it's being processed. I guess it says something about the
internal delays in each PC, and perhaps something about the difference
between network and serial/DCD processing. Each PC's clock is /not/
being set to UTC, but to some offset from UTC, which NTP measures as
near zero.

Cheers,
David

[E-Mail Sent to this address will be added to the BlackLists]

Paul G wrote:> Showing 192.168.0.2 and .244 with time1 = 0 as compared

> to the first post where time1 was ~100 microsec.
>
> These two have gigabit interfaces.

Is interrupt coalescing disabled on the gigabit ethernet adapters?


--
E-Mail Sent to this address <Blac...@Anitech-Systems.com>
will be added to the BlackLists.

[Doug Calvert]

unruh has a nice page about interrupt coalescing:

http://axion.physics.ubc.ca/scatter/rt.html

> _______________________________________________
> questions mailing list
> ques...@lists.ntp.org
> http://lists.ntp.org/listinfo/questions
>

[David Taylor]

On 06/06/2013 20:17, Doug Calvert wrote:
> unruh has a nice page about interrupt coalescing:
>
> http://axion.physics.ubc.ca/scatter/rt.html

This is a quite interesting note, unfortunately rendered rather less
useful by the graphs, which have no axis names or units marked, and
which appear to have inconsistent scales (although this may simply be
lack of legibility). For example:

http://axion.physics.ubc.ca/scatter/rtdt1.gif

The units on the vertical axis appear to be labelled:

-0.0001 -5x10-6 0 5x10-6 0.0001

which I interpret as -100us, -5 us, 0, +5us +100us, but it appears to
be a linear axis. Is the 5x10-6 supposed to be 5x10-5? The character
needs serious improvement as it looks like 5x10-6 to me. But in any
case, as microseconds are referred to in the text, would it not be a lot
clearer if the graphs were also in microseconds, rather than some small
fraction of seconds?

Similar problems in:

http://axion.physics.ubc.ca/scatter/gps.gif

where the -5 and -6 suffices are very difficult to distinguish. I do
hope that Bill can re-do these graphs some time as it would enhance the
value of this work.

[unruh]

On 2013-06-07, David Taylor <david-...@blueyonder.co.uk.invalid> wrote:
> On 06/06/2013 20:17, Doug Calvert wrote:
>> unruh has a nice page about interrupt coalescing:
>>
>> http://axion.physics.ubc.ca/scatter/rt.html
>
> This is a quite interesting note, unfortunately rendered rather less
> useful by the graphs, which have no axis names or units marked, and
> which appear to have inconsistent scales (although this may simply be
> lack of legibility). For example:
>
> http://axion.physics.ubc.ca/scatter/rtdt1.gif
>
> The units on the vertical axis appear to be labelled:
>
> -0.0001 -5x10-6 0 5x10-6 0.0001

Yes, supermongo, the plotting program, does tend to make 5 and 6 similar
at the small sizes used in superscripts. But as you say, it is a linear
scale, and thus that must be a 5 rather than a 6.

>
> which I interpret as -100us, -5 us, 0, +5us +100us, but it appears to

Since that is, as you point out, an interpretation that produces silly
results, it is probably the interpretation that is at fault.

> be a linear axis. Is the 5x10-6 supposed to be 5x10-5? The character
> needs serious improvement as it looks like 5x10-6 to me. But in any
> case, as microseconds are referred to in the text, would it not be a lot
> clearer if the graphs were also in microseconds, rather than some small
> fraction of seconds?
>
> Similar problems in:
>
> http://axion.physics.ubc.ca/scatter/gps.gif
>
> where the -5 and -6 suffices are very difficult to distinguish. I do
> hope that Bill can re-do these graphs some time as it would enhance the
> value of this work.

I am not at all sure that I can find the data anymore to redo the
graphs.
About all I could do is to use gimp to replace the axes labels and make
the 5 look a bit more like 5 rather than 6. The problem is at the small
scale of the superscripts, the little culicue up at the bottom of the 5
in the font used becomes a single pixel closing off the the loop, making
it look a lot like a 6. So I used gimp to open it up again, and also to
labels on the axes.

[David Taylor]

Thanks, Bill. Sounds like an update to "supermongo" is needed! Can it
label the axes? Not critical to regenerate the graphs, but perhaps if
you add a little to the text to clarify scales and axes it may help others.

Cheers,
David

[unruh]

I did use gimp to label the axes and scales and to make the 5's look
like 5's. I
think tht the problem is that the font used for 6 pixel high text makes
5 look like 6.
Have a look again to see if it is better.

>
> Cheers,
> David

[Rick Jones]

unruh <un...@invalid.ca> wrote:
> I did use gimp to label the axes and scales and to make the 5's look
> like 5's. I think tht the problem is that the font used for 6 pixel
> high text makes 5 look like 6. Have a look again to see if it is
> better.

At least one typo in the text:

Finally it was changed to 1000 Mbps with rx-users: 0 (no
interrupt coalescence.

That should be rx-usecs.

rick jones
--
I don't interest myself in "why." I think more often in terms of
"when," sometimes "where;" always "how much." - Joubert
these opinions are mine, all mine; HP might not want them anyway... :)
feel free to post, OR email to rick.jones2 in hp.com but NOT BOTH...

[unruh]

On 2013-06-07, Rick Jones <rick....@hp.com> wrote:
> unruh <un...@invalid.ca> wrote:
>> I did use gimp to label the axes and scales and to make the 5's look
>> like 5's. I think tht the problem is that the font used for 6 pixel
>> high text makes 5 look like 6. Have a look again to see if it is
>> better.
>
> At least one typo in the text:
>
> Finally it was changed to 1000 Mbps with rx-users: 0 (no
> interrupt coalescence.
>
> That should be rx-usecs.

Thanks. Fixed.
>
> rick jones

[David Taylor]

On 07/06/2013 19:02, unruh wrote:
> I did use gimp to label the axes and scales and to make the 5's look
> like 5's. I
> think tht the problem is that the font used for 6 pixel high text makes
> 5 look like 6.
> Have a look again to see if it is better.

Much improved, Bill, thanks very much.

[Paul G]

On Sunday, May 26, 2013 5:25:28 PM UTC-4, unruh wrote:
> Lan will have delays of the order of .15ms (worse for gigabit)

This not the delay reported by ntpq. Is it wrong or are you talking about another measurement?

[Paul G]

On Tuesday, June 4, 2013 3:44:19 PM UTC-4, E-Mail Sent to this address will be added to the BlackLists wrote:
> Is interrupt coalescing disabled on the gigabit ethernet adapters?

Not by me. I assume when ntpq reports symmetric delays proportional to interface speed that interrupts are not the issue for my question. Perhaps that's a poor assumption.

My question was -- given PPS clocks on the same network with ntpq offsets of:
0.021, 0.130, 0.064 and -0.046 -- any of which can be fudged away how do I determine the truth. That is how much is (consistent) hardware latency of any sort versus algorithmic restriction.

e.g. delay, offset and jitter from two sides of a gigE connection:
0.073 -0.008 0.002
0.064 0.011 0.006

Likewise fastE:
1.226 -0.059 0.076
1.104 0.109 0.068

[E-Mail Sent to this address will be added to the BlackLists]

Paul G wrote:
> My question was -- given PPS clocks on the same network with ntpq offsets of:
> 0.021, 0.130, 0.064 and -0.046 -- any of which can be fudged away how do
> I determine the truth. That is how much is (consistent) hardware latency
> of any sort versus algorithmic restriction.

enable stats
statistics loopstats peerstats
If NMEA, configure the GPS(S) to only output one sentence e.g. $GPZDA

On one system at a time,
set the GPS and PSP RefClock to noselect, with time1 0 and time2 0
have several other NTP servers configured {I usually shoot for 6 to 10}
with all involved systems continuously running for more than one day;
take an average of the offsets for the last day for 127.127.20.x,
{or whichever GPS driver you are using} set the time2 for the GPS
After running for more than a day with time2 seeming ok,
take an average of the offsets for the last day for 127.127.22.x, set the time1 for the PPS

Run for another day to see if the fudge times still seem fine,
change the Noselect(s) to Prefer and remove any excess servers that were added.

{If the offsets / fudge times seem too large perhaps the PPS signal is inverted?
set flag2 1, and run for more than a day (again).}


It seemed like David Taylor covered that on may 25th.
<https://groups.google.com/d/msg/comp.protocols.time.ntp/ftFngXfRtLU/MuHqVHuXJ_EJ>

--
E-Mail Sent to this address <Blac...@Anitech-Systems.com>
will be added to the BlackLists.

[David Lord]

I don't really understand your question.

When I have two pps sourced systems they will each be << 10us offset
but depending on network delay at that time ntpq from a different
system will show offset of > 100us. Ntpd tries to compensate for
delays but best it can do is assume equal delays and maximum delay
equal to total rtt delay. The interrupt delay is another variable
of possibly few us unless there are queued interrupts.

Loop_summary and peer_summary will produce daily averages but even
then I see big differences (relating to temperature/load).


David

[E-Mail Sent to this address will be added to the BlackLists]

Paul G wrote:

> BlackList wrote:
>> Is interrupt coalescing disabled on the gigabit ethernet adapters?
> Not by me. I assume when ntpq reports symmetric delays proportional
> to interface speed that interrupts are not the issue for my question.
> Perhaps that's a poor assumption.

Gigabit Ethernet adapter interrupt coalescing just makes
the packet latency less deterministic, if you don't turn it off.

--
E-Mail Sent to this address <Blac...@Anitech-Systems.com>
will be added to the BlackLists.

[Paul G]

On Thursday, June 20, 2013 5:51:50 PM UTC-4, E-Mail Sent to this address will be added to the BlackLists wrote:
>
>
>
> enable stats [etc. etc.]

As noted earlier I've done that or it's not applicable. E.g. I only use the
PPS driver and my seconds are numbered by an appliance that doesn't run ntpd.

>
> On one system at a time, ...

> have several other NTP servers configured {I usually shoot for 6 to 10}

I don't have six to ten stratum one servers (but maybe I should) and it doesn't
seem useful to compare my << 500 microsecond offsets to non-local clocks.

> with all involved systems continuously running for more than one day;

> take an average ...

In another thread (and some here) I explain how I've done that and I don't
really like the e.g. 124 microsecond time1 I derived. However it does result
in O(1) microsec. offsets between some of my clocks.

> perhaps the PPS signal is inverted?

It's not.

> It seemed like David Taylor covered that on may 25th.

Yes. While I appreciate the suggestions and the good will behind them they don't seem informed by my question/problem description.

My key point is that ntpq appears to be telling me odd things. E.g. my network
is low latency, symmetric and consistent but some of my offsets are one or two
orders of magnitude beyond other offsets.

So my question is how to find what I hope is hardware latency using the tools
at hand or the coverse given multiple S1 clocks with O(10) microsecond offsets
which one is right.

I expect I will move a set of them to a 10/100 switch and see if that makes a
difference.

Ideally all my clocks pairs would look like this (both have time1 0):

localhost oPPS(0) .PPS. ... 377 0.000 0.001 0.001
black +aster .PPS. ... 377 0.065 -0.003 0.004

localhost oPPS(0) .PPS. ... 377 0.000 0.000 0.004
aster +black .PPS. ... 377 0.066 0.007 0.010

But maybe some of them are just not up to the task:
aster *ntp1 .PPS. ... 377 0.526 0.129 0.166

[E-Mail Sent to this address will be added to the BlackLists]

Paul G wrote:
> E.g. my network is low latency, symmetric and consistent
> but some of my offsets are one or two orders of magnitude
> beyond other offsets.

Gigabit interrupt coalescing causes non deterministic
asymmetric latency.

> I expect I will move a set of them to a 10/100 switch and
> see if that makes a difference.

If it is due to gigabit interrupt coalescing it will make
it more deterministic (although higher latency).


--
E-Mail Sent to this address <Blac...@Anitech-Systems.com>
will be added to the BlackLists.

[Charles Elliott]

You may be making a mistake using stratum 1 servers; especially the
government servers are way over used, and there can be large, variable, and
indeterminate delays just pushing a packet thru them. Stick with stratum 2
servers; few people know about them apparently.

And this is not off-topic. You efforts may be aided by having one computer
connected to 9 or 10 external servers (using the huff puff option) and to
your two local clocks (using the noselect option). That way you can tell
which local clock is closer to the truth and how much they differ. For the
computer connected to the external servers, don't run anything else on it
while you are experimenting. A high load, shouldn't, but does affect how
fast NTPD can adjust to changing conditions. Also, watch the offsets and
jitter. When they become high, the router your ISP uses is jammed up with
movie downloads or commercial data. You should wait until the Internet is
quiescent before observing or taking measurements.

Best of luck!

Charles Elliott

[Paul G]

On Friday, June 21, 2013 9:16:28 PM UTC-4, Charles Elliott wrote:
> You may be making a mistake using stratum 1 servers

All the S1 servers are on my local network. I (temporarily) have six.

[unruh]

It is the time between the packet sent out and the packet received back,
as reported in the peerstats file. \no idea or care what ntpq reports.

[David Woolley]

Paul G wrote:
> I have four GPS receivers into four different computers. I've assumed that I
> should be able to get small offsets between them but to do so I need

Basically, if this is important to you, you will need special hardware
instrumentation. You will need a time standard that it known to be a
lot better than any of the normal sources, and which generates something
like a pulse per second output, probably a portable atomic clock. You
will need an output pin on the PC that can be controlled directly from
the PCI bus (no USB in between, even internally). You will need
software on the PC that has direct access the I/O port controlling the
pin (there must be no kernel code virtualising the device - you must
have IO permissions at the hardware level. The software also needs to
be able to read the interpolated time as quickly as possible, which
probably means running in the kernel.

Externally, you will need a gated counter and and a frequency source of
period several times less than the desired measurement accuracy and
accurate to better than one cycle over a second, possibly a 10MHz output
from the atomic clock.

Arrange the atomic clock to start the counter, on the second, and the
output from the PC to stop it. Either side of setting the output from
the PC read the interpolated software clock, and after the complete
cycle, output the fractional seconds part of the time. Repeat this
every second.

You should then have enough information to calculate the actual PC
software clock offset, from which you can decide which of your ntpd
sources is best.

Note, if you want to work to better than about 100ns, you will need to
look up the data the data sheets for some of the logic used, to correct
for the logic propagation delays.

Alternatively you can just accept that there will always be some
uncertainty in the measurements.