Syncing and peering for a multi-continent deployment

[Blair Zajac]

I'm working on deploying a large number of servers to data centers on
three continents, say US, Europe and India. Because each data center
will be writing data and timestamping the data, I want them all to think
they are at the same time, even if it isn't exactly the correct UTC
time. The data will be synced to other data centers, so relative
ordering of when something happened can be important.

To not ping stratum-1 and -2 level servers from a large number of boxes,
I'm setting up three stratum-3 NTP servers in each data center.

Here's what I've read:

1) Each ntp server should have 4 to 7 upstream clocks [1], [2], [3].

2) The three stratum-3 NTP servers in each data center should be peers
of each other [4]. So each NTP server would have at most 2+7=9 connections.

Questions:

1) Should each data center use nearby stratum-2 clocks or pick a set of
stratum-2 clocks that are network wise in the center between all data
centers (it may not be possible to get a true center)?

2) Should each stratum-3 server in a single data center use all 4-7
upstream clocks? Or should the 4-7 be split between the three stratum-3
servers? The diagram at [4] suggests they should be split? If that's
the case, then how are falsetickers identified?

3) Should I have all the stratum-3 servers in each data center be peers
of each other, so each would have 8 peers? This would ensure that all
my clocks think they are at the same time. If I do this, then I would
need to increase maxclocks to support up to 7 (upstream) + 8 (peers)=15
clocks?

4) Or, should I not peer data centers to each other and trust that the
stratum-2 clocks near each data center will be close to the other clocks?

Other random questions:

1) It doesn't appear that its necessary to set up symmetric keys for peers?

2) Is there a way to tell which peer is the "master" peer? Looking
through ntpdc and ntpq I didn't see anything.

Thanks,
Blair

[1] http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Secti
[2] http://support.ntp.org/bin/view/Support/SelectingOffsiteNTPServers#Secti
[3] http://support.ntp.org/bin/view/Support/StartingNTP4#Section_7.1.4.3.2.
[4] http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#AEN3101

[John Hasler]

Blair Zajac writes:
> 1) Each ntp server should have 4 to 7 upstream clocks [1], [2], [3].

Each NTP server should have a GPS receiver. Fall back to the Internet
if GPS goes down.
--
John Hasler
jha...@newsguy.com
Dancing Horse Hill
Elmwood, WI USA

[David Lord]

What level of time offsets would you find acceptable?

From peer_summary mean rms maximum
(us) (us) (us)

Internet - ntp3.lordynet.org 500 500 1000

Lan - ntp3.lordynet.org 50 250 700
(no temperature control)

PPS - me6000e.home.lordynet.org 0 5 33
(no temperature control)

Informed guesses:
PPS (constant ambient temperature) 0 4 15
PPS (OCXO system clock) 0.050 0.100

Most of variation on my lan is due to change of ambient
temperature whilst for my PPS synced pc it is from both changes
of system temperature under load and ambient temperature.


Below, for ntp3, the selected peer is shown from "ntpq -c rv" by
refid=81.187.61.78 and from "ntpq -p" by the "*" in the first
column.

For "me6000e", refid=PPSb and from "ntpq -p" by the "o" in the
first column whilst the preferred peer indicated by the "*" has a
large offset of 8.666 msec.


David

#----------------------------------------------------------------

# ntp3.lordynet.org
# ntpq -c rv -p ntp3.lordynet.org
associd=0 status=061d leap_none, sync_ntp, 1 event, kern,
version="ntpd 4.2.6p5-o Wed Feb 1 07:49:06 UTC 2012 (import)",
processor="i386", system="NetBSD/6.1_RC2", leap=00, stratum=2,
precision=-18, rootdelay=0.405, rootdisp=11.529, refid=81.187.61.78,
------------------
reftime=d50d3ca0.b65a4f90 Mon, Apr 8 2013 12:54:56.712,
clock=d50d3eab.e868dd3e Mon, Apr 8 2013 13:03:39.907, peer=1276, tc=6,
mintc=3, offset=0.204, frequency=-10.411, sys_jitter=0.079,
clk_jitter=0.029, clk_wander=0.004, tai=35, leapsec=201207010000,
expire=201306280000

Mon Apr 8 12:53:00 GMT 2013
remote refid st t when poll reach delay offset
jitter
==============================================================================
ns0.lordynet.or 81.187.61.65 3 u 41 64 376 0.640 -0.546
0.517
*ns2.lordynet.or .MSFa. 1 u 12 64 377 0.457 0.293
0.036
+??????????????? ?????????????? 2 u 124 256 377 15.147 0.131
0.179
+??????????????? ?????????????? 2 u 82 256 377 15.688 0.096
0.112
-??????????????? ?????????????? 2 u 44 256 377 21.732 -0.078
0.115
-??????????????? ?????????????? 2 u 19 256 377 23.610 0.048
0.164
-??????????????? ?????????????? 3 u 210 256 377 26.161 -1.134
1.266

#----------------------------------------------------------------

# me6000e
# peerstats.me6000e.20130407
ident cnt mean rms max delay dist disp
==========================================================================
127.127.20.2 1350 -0.256 17.910 68.071 0.000 2.564 1.742
127.127.22.2 1350 0.000 0.005 0.033 0.000 0.930 0.930
81.187.61.65 70 1.089 0.297 0.580 0.974 23.814 11.149
81.187.61.74 288 1.037 0.555 1.457 1.481 7.191 3.117
81.187.61.69 292 1.042 0.196 0.619 0.826 6.208 2.852
????????????? 69 0.775 0.269 0.702 16.096 30.449 11.337
81.187.61.78 75 1.035 0.227 0.515 0.749 23.093 11.335

# ntpq -c rv -p me6000e
associd=0 status=01fd leap_none, sync_pps, 15 events, kern,
version="ntpd 4.2.6p5-o Wed Feb 1 07:49:06 UTC 2012 (import)",
processor="i386", system="NetBSD/6.1_RC2", leap=00, stratum=1,
precision=-18, rootdelay=0.000, rootdisp=1.705, refid=PPSb,
reftime=d50d40c7.76bd4a0f Mon, Apr 8 2013 13:12:39.463,
clock=d50d40fa.eede15ee Mon, Apr 8 2013 13:13:30.933, peer=58635, tc=6,
mintc=3, offset=0.005, frequency=-35.769, sys_jitter=0.005,
clk_jitter=0.004, clk_wander=0.009, tai=35, leapsec=201207010000,
expire=201306280000

remote refid st t when poll reach delay offset
jitter
==============================================================================
*GPS_NMEA(2) .GPSb. 4 l 53 64 377 0.000 8.666
11.460
oPPS(2) .PPSb. 0 l 51 64 377 0.000 0.005
0.005
+ns0.lordynet.or 81.187.61.65 3 u 18 64 377 1.424 0.421
0.472
+ns2.lordynet.or .MSFa. 1 u 40 256 377 0.439 1.111
0.562
+ns1.lordynet.or 81.187.61.78 2 u 40 64 377 0.699 1.219
0.528
+ns3.lordynet.or 81.187.61.78 2 u 193 256 377 1.081 0.751
0.599
+??????????????? ????????????? 2 u 181 256 377 15.920 1.074
0.760
#----------------------------------------------------------------

[David Woolley]

David Lord wrote:

> What level of time offsets would you find acceptable?

He's probably interested in offset from his time reference, rather than
the ntpd "offset" measurement, which could be optimistic, if there are
asymmetric delays.

Intercontinentally he can probably tolerate several milliseconds or even
low tens of milliseconds without any risk that there is any doubt
that two events are not causally related.

[John Hasler]

David Woolley writes:
> Intercontinentally he can probably tolerate several milliseconds or
> even low tens of milliseconds without any risk that there is any doubt
> that two events are not causally related.

He might be using software that won't tolerate such offsets, though.

[David Woolley]

John Hasler wrote:
> David Woolley writes:
>> Intercontinentally he can probably tolerate several milliseconds or
>> even low tens of milliseconds without any risk that there is any doubt
>> that two events are not causally related.
>
> He might be using software that won't tolerate such offsets, though.

Unless he is doing common view observations on something
extra-terrestrial, I'm not sure how the software would not tolerate than
sort of offset.

[John Hasler]

David Wooley writes:
> Unless he is doing common view observations on something
> extra-terrestrial, I'm not sure how the software would not tolerate
> than sort of offset.

He might be using something like a distributed database that has an
arbitrary definition of simultaneity hard coded.

[Blair Zajac]

Thanks Charles,

OK. So unless I go for some GPS-based clocks in my servers, each data
center will use close stratum-2 servers and the stratum-3 clocks will
not peer with clocks in my other datacenters.

Regarding peering, then should each stratum-3 server in a single data

center use all 4-7 upstream clocks? Or should the 4-7 be split between

the three stratum-3 servers? The diagram at [1] suggests they should be

split? If that's the case, then how are falsetickers identified?

Blair

[1] http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#AEN3101

On 04/08/2013 05:45 PM, Charles Elliott wrote:
> I don't think you will ever achieve acceptable synchronization between the
> data centers without accurate external clocks, such as GPS-based clocks,
> attached to a time server in each location. Internet traffic acts as waves
> of high use and periods of quiescence. During these periods of high traffic
> your data centers will differ by 250 ms or more and will take some time to
> resynchronize during quiescent periods. This problem is greatly exacerbated
> by the large distances between your data center locations. NTPD cannot
> prevent that. Nor is it really NTPD's fault. The ISPs have installed very
> large buffers in front of their routers to prevent dropping packets and
> sending choke packets to the data originators, as the Internet was
> originally designed. Hence the Internet is not automatically rerouting
> packets when traffic patterns change, again as originally designed. Money
> is the root of all evil.
>
> Charles Elliott

>> _______________________________________________
>> questions mailing list
>> ques...@lists.ntp.org
>> http://lists.ntp.org/listinfo/questions
>

[David Taylor]

On 10/04/2013 03:07, Blair Zajac wrote:
> Thanks Charles,
>
> OK. So unless I go for some GPS-based clocks in my servers, each data
> center will use close stratum-2 servers and the stratum-3 clocks will
> not peer with clocks in my other datacenters.
>
> Regarding peering, then should each stratum-3 server in a single data
> center use all 4-7 upstream clocks? Or should the 4-7 be split between
> the three stratum-3 servers? The diagram at [1] suggests they should be
> split? If that's the case, then how are falsetickers identified?
>
> Blair
>
> [1] http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#AEN3101

That diagram (figure 5) shows each of the local prime servers being fed
from just two external sources, but as you say, 4-7 would be more
sensible. If it were me, I would go for one, two or three servers you
know you can trust being in common across the stratum-3 servers, and
fill in the rest with the "pool" directive. This will fill up the
servers with up to 10 external sources, which are dropped and an
alternative automatically selected if the source goes bad.

I have information on my Web page about low-cost stratum-1 servers using
Windows, FreeBSD and Raspberry Pi PCs.

http://www.satsignal.eu/ntp/index.html
--
Cheers,
David
Web: http://www.satsignal.eu

[unruh]

On 2013-04-17, David Taylor <david-...@blueyonder.co.uk.invalid> wrote:
> On 10/04/2013 03:07, Blair Zajac wrote:
>> Thanks Charles,
>>
>> OK. So unless I go for some GPS-based clocks in my servers, each data
>> center will use close stratum-2 servers and the stratum-3 clocks will
>> not peer with clocks in my other datacenters.
>>
>> Regarding peering, then should each stratum-3 server in a single data
>> center use all 4-7 upstream clocks? Or should the 4-7 be split between
>> the three stratum-3 servers? The diagram at [1] suggests they should be
>> split? If that's the case, then how are falsetickers identified?
>>
>> Blair
>>
>> [1] http://www.ntp.org/ntpfaq/NTP-s-config-adv.htm#AEN3101
>
> That diagram (figure 5) shows each of the local prime servers being fed
> from just two external sources, but as you say, 4-7 would be more
> sensible. If it were me, I would go for one, two or three servers you
> know you can trust being in common across the stratum-3 servers, and

Also make sure that those link back to different stratum 1 or 0 sources.
It does little good to have 5 servers, all of whom derive their time
from the same stratum 1 source, when that common source goes down.

(ntpq -p tells you the source (refid) of the source you are using, and you can
follow that back).

Of course now adays, almost all refer back to a source that is GPS. So
even if they have different gps receivers they are all back to a common
source, the gps system. But perhaps if the whole gps goes down, or goes
bad, we as a society will have bigger problems than our computer clocks.

[John Hasler]

unruh writes:
> But perhaps if the whole gps goes down, or goes bad, we as a society
> will have bigger problems than our computer clocks.

Well, it's run out of a single center by a single agency so things other
than a loss of satellites could compromise it. Fortunately there's
GLONASS and will soon be Galileo. Does anyone sell a timing receiver
with GLONASS capability?

[Uwe Klein]

John Hasler wrote:
> unruh writes:
>
>>But perhaps if the whole gps goes down, or goes bad, we as a society
>>will have bigger problems than our computer clocks.
>
>
> Well, it's run out of a single center by a single agency so things other
> than a loss of satellites could compromise it. Fortunately there's
> GLONASS and will soon be Galileo. Does anyone sell a timing receiver
> with GLONASS capability?

Well, you can buy GPS receivers that do NAVSTAR _and_ GLONASS.
Triple service receivers are around the corner.

uwe

[Harlan Stenn]

(found unsent in my drafts folder...)

John Hasler writes:
> Blair Zajac writes:
> > 1) Each ntp server should have 4 to 7 upstream clocks [1], [2], [3].
>
> Each NTP server should have a GPS receiver. Fall back to the Internet
> if GPS goes down.

These GPS receivers should be different brands.

H