NTP fails on 507 testing on Cloud hosting service.

[Steve M. Fabac, Jr.]

What's the difference between ntpdate and ntpd? (see further below
for ntpdate).

I've a client where we are testing moving their physical 5.0.7
server from physical hardware on-site at their office to a 507 VM
hosted by http://www.virtacore.com/

I've restored a backup of the physical machine to the Virtacore cloud
to create the 507 VM and brought it up.

During testing, I noted that the time is off and drifting further
away from wall-clock time.

Ntpd is running with the standard /etc/ntp.conf I set up for SCO 5.0.7
systems but ntp is not adjusting the clock.

I set up a check_date script that runs every 10 minutes from root's
crontab on the 507 VM and it shows that the 507 VM is loosing time:

507VM vs 507 Physical
Tue Feb 26 15:00:00 CST 2013
Tue Feb 26 15:09:19 CST 2013 -09:19

507VM vs 507 Physical
Tue Feb 26 15:10:00 CST 2013
Tue Feb 26 15:23:05 CST 2013 -13:05

507VM vs 507 Physical
Tue Feb 26 15:20:00 CST 2013
Tue Feb 26 15:36:50 CST 2013 -16:50

507VM vs 507 Physical
Tue Feb 26 15:30:00 CST 2013
Tue Feb 26 15:50:34 CST 2013 -20:34

507VM vs 507 Physical
Tue Feb 26 15:40:00 CST 2013
Tue Feb 26 16:04:20 CST 2013 -24:20

507VM vs 507 Physical
Tue Feb 26 15:50:00 CST 2013
Tue Feb 26 16:18:05 CST 2013 -28:05

So that after 24 hours I see:

507VM vs 507 Physical
Wed Feb 27 15:40:00 CST 2013
Wed Feb 27 17:41:08 CST 2013 -2:01:08

507VM vs 507 Physical
Wed Feb 27 15:50:00 CST 2013
Wed Feb 27 17:54:53 CST 2013 -2:04:53

507VM vs 507 Physical
Wed Feb 27 16:00:00 CST 2013
Wed Feb 27 18:08:39 CST 2013 -2:08:39

507VM vs 507 Physical
Wed Feb 27 16:10:00 CST 2013
Wed Feb 27 18:22:23 CST 2013 -2:12:23

507VM vs 507 Physical
Wed Feb 27 16:20:00 CST 2013
Wed Feb 27 18:36:08 CST 2013 -2:16:08

507VM vs 507 Physical
Wed Feb 27 16:30:00 CST 2013
Wed Feb 27 18:49:54 CST 2013 -2:19.54


A google search turned up "Timekeeping-In-VirtualMachines.pdf"
but the suggestions for /etc/ntp.conf did not resolve the issue:


> INFORMATION GUIDE / 2 3
> Timekeeping in VMware Virtual Machines
> VMware vSphere 5.0, Workstation 8.0, Fusion 4.0
> Using NTP in Linux and Other Guests
> The Network Time Protocol is usable in a virtual machine with proper configuration
> of the NTP daemon. The following points are important:
>
> --> Do not configure the virtual machine to synchronize to its own
> (virtual) hardware clock, not even as a fallback with a high stratum
> number. Some sample ntpd.conf files contain a section specifying the
> local clock as a potential time server, often marked with the comment
> �undisciplined local clock.� Delete any such server specification
> from your ntpd.conf file.
>
> --> Include the option tinker panic 0 at the top of your ntp.conf file.
> By default, the NTP daemon sometimes panics and exits if the underlying
> clock appears to be behaving erratically. This option causes the daemon
> to keep running instead of panicking.
> --> Follow standard best practices for NTP: Choose a set of servers to synchronize
> to that have accurate time and adequate redundancy. If you have many virtual
> or physical client machines to synchronize, set up some internal servers for
> them to use, so that all your clients are not directly accessing an external
> low-stratum NTP server and overloading it with requests.
>
> The following sample ntp.conf file is suitable if you have few enough clients that
> it makes sense for them to access an external NTP server directly. If you have many
> clients, adapt this file by changing the server names to reference your internal NTP
> servers.
>
> NOTE: Any tinker commands used must appear first.
>
> # ntpd.conf
> tinker panic 0
> restrict 127.0.0.1
> restrict default kod nomodify notrap
> server 0.vmware.pool.ntp.org
> server 1.vmware.pool.ntp.org
> server 2.vmware.pool.ntp.org
> server 3.vmware.pool.ntp.org
>
> Here is a sample /etc/ntp/step-tickers corresponding to the sample ntp.conf file above.
>
> # step-tickers
> 0.vmware.pool.ntp.org
> 1.vmware.pool.ntp.org
>
> Make sure that ntpd is configured to start at boot time. On some distributions
> this can be accomplished with the command chkconfig ntpd on, but consult your
> distribution�s documentation for details. On most distributions, you can start
> ntpd manually with the command /etc/init.d/ntpd start.


I applied the suggestions with modifications of my standard
/etc/ntp.conf:


#
# sample /etc/ntp.conf file CLIENT configuration
# Added restrict per sample from Timekeeping-In-VirtualMachines.pdf
# removed kod as /usr/adm/syslog message indicates it as unknown cmd
#
restrict 127.127.1.0
#restrict default kod nomodify notrap
restrict default nomodify notrap

server tick.usno.navy.mil

# ^ The IP address of the stratum 0 server machine that you would
# be asking for the time.

# Declare local clock as stratum 10 server in case we loose contact with
# Strata 1 servers and so that our machine will serve time to the network.

#server 127.127.1.0
#fudge 127.127.1.0 stratum 10

server 127.127.1.0 commented out per Timekeeping-In-VirtualMachines.pdf


The above did not work. In desperation, I set up a crontab entry
to run ntpdate one minute before common execute times for the client's
crontab scripts:

0,10,20,30,40,50 * * * * /usr/bin/checkdate > /dev/null
#* * * * * /usr/bin/checkdate > /dev/null
4,9,14,19,24,29,34,39,44,49,54,59 * * * * /etc/ntpdate -s tick.usno.navy.mil > /dev/null 2>&1

With the result in the /usr/adm/check_date.log:

507VM vs 507 Physical
Fri Apr 5 23:40:10 CDT 2013
Fri Apr 5 23:40:34 CDT 2013

Added
4,9,14,19,24,29,34,39,44,49,54,59 * * * * /etc/ntpdate -s tick.usno.navy.mil > /
to root's crontab and killed /etc/ntpd so that ntpdate will work.

507VM vs 507 Physical
Fri Apr 5 23:50:51 CDT 2013
Fri Apr 5 23:50:52 CDT 2013

507VM vs 507 Physical
Fri Apr 5 23:54:05 CDT 2013
Fri Apr 5 23:54:31 CDT 2013

507VM vs 507 Physical
Fri Apr 5 23:55:02 CDT 2013

507VM vs 507 Physical
Fri Apr 5 23:56:15 CDT 2013
Fri Apr 5 23:56:31 CDT 2013

...

Changed the checkdate logging to every 60 seconds to see the
result of the time drift and reset at 4, 9, 14 minutes etc...

The "-NN" times are normalized by subtracting VM time from "real"
time.

507VM vs 507 Physical
Sat Apr 6 00:00:00 CDT 2013
Sat Apr 6 00:00:05 CDT 2013 -5

507VM vs 507 Physical
Sat Apr 6 00:01:13 CDT 2013
Sat Apr 6 00:01:28 CDT 2013 -15

507VM vs 507 Physical
Sat Apr 6 00:02:12 CDT 2013
Sat Apr 6 00:02:33 CDT 2013 -21

507VM vs 507 Physical
Sat Apr 6 00:03:12 CDT 2013
Sat Apr 6 00:03:39 CDT 2013 -27

507VM vs 507 Physical
Sat Apr 6 00:04:11 CDT 2013
Sat Apr 6 00:04:44 CDT 2013 -33

507VM vs 507 Physical
Sat Apr 6 00:05:00 CDT 2013
Sat Apr 6 00:05:07 CDT 2013 -7

507VM vs 507 Physical
Sat Apr 6 00:06:13 CDT 2013
Sat Apr 6 00:06:28 CDT 2013 -15

507VM vs 507 Physical
Sat Apr 6 00:07:12 CDT 2013
Sat Apr 6 00:07:33 CDT 2013 -21

507VM vs 507 Physical
Sat Apr 6 00:08:12 CDT 2013
Sat Apr 6 00:08:39 CDT 2013 -27

507VM vs 507 Physical
Sat Apr 6 00:09:12 CDT 2013
Sat Apr 6 00:09:44 CDT 2013 -31

507VM vs 507 Physical
Sat Apr 6 00:10:00 CDT 2013
Sat Apr 6 00:10:06 CDT 2013 -6

507VM vs 507 Physical
Sat Apr 6 00:11:14 CDT 2013
Sat Apr 6 00:11:29 CDT 2013 -15

....

checkdate logging returned to every 10 minutes and ntpdate running
4,9,14,19,24,29,34,39,44,49,54,59 * * * * /etc/ntpdate -s tick.usno.navy.mil > /dev/null 2>&1
ntpd killed

507VM vs 507 Physical
Sat Apr 6 06:50:22 CDT 2013
Sat Apr 6 06:50:23 CDT 2013

507VM vs 507 Physical
Sat Apr 6 07:00:22 CDT 2013
Sat Apr 6 07:00:22 CDT 2013

507VM vs 507 Physical
Sat Apr 6 07:10:22 CDT 2013
Sat Apr 6 07:10:22 CDT 2013

507VM vs 507 Physical
Sat Apr 6 07:20:22 CDT 2013
Sat Apr 6 07:20:22 CDT 2013

507VM vs 507 Physical
Sat Apr 6 07:30:22 CDT 2013
Sat Apr 6 07:30:22 CDT 2013

507VM vs 507 Physical
Sat Apr 6 07:40:22 CDT 2013
Sat Apr 6 07:40:22 CDT 2013


So: Why does ntpdate work to change the system clock when nptd does not?

Apr 7 01:16:57 507vm ntpd[7764]: ntpd 4.0.98h Tue Feb 18 02:25:48 PST 2003 (1)
Apr 7 01:16:57 507vm ntpd[7764]: precision = 7 usec
Apr 7 01:16:57 507vm ntpd[7764]: kern_enable is 1
Apr 7 01:16:57 507vm ntpd[7764]: configure: keyword "tinker" unknown, line ignored
Apr 7 01:16:57 507vm ntpd[7764]: configure: keyword "kod" unknown, line ignored
Apr 7 01:16:57 507vm ntpd[7764]: frequency initialized -141.205 from /etc/driftfile

# ntpdc
ntpdc> peers
remote local st poll reach delay offset disp
=======================================================================
=tick.usno.navy. 172.16.254.100 1 64 1 0.16805 21.588063 7.93750
ntpdc> peers
remote local st poll reach delay offset disp
=======================================================================
=tick.usno.navy. 172.16.254.100 1 64 1 0.16805 21.588063 7.93750
ntpdc> peers
remote local st poll reach delay offset disp
=======================================================================
=tick.usno.navy. 172.16.254.100 1 64 1 0.16805 21.588063 7.93750
ntpdc> peers
remote local st poll reach delay offset disp
=======================================================================
=tick.usno.navy. 172.16.254.100 1 64 3 0.07915 45.495814 3.93773
ntpdc> peers
remote local st poll reach delay offset disp
=======================================================================
=tick.usno.navy. 172.16.254.100 1 64 7 0.07915 45.495814 1.93822
ntpdc> peers
remote local st poll reach delay offset disp
=======================================================================
=tick.usno.navy. 172.16.254.100 1 64 7 0.07915 45.495814 1.93822
ntpdc> peers
remote local st poll reach delay offset disp
=======================================================================
=tick.usno.navy. 172.16.254.100 1 64 7 0.07915 45.495814 1.93822
ntpdc> peers
remote local st poll reach delay offset disp
=======================================================================
=tick.usno.navy. 172.16.254.100 1 64 17 0.06685 94.261869 0.93829
ntpdc>


--
Steve Fabac
S.M. Fabac & Associates
816/765-1670

[Scot Jenkins]

Steve M. Fabac, Jr. wrote:
> What's the difference between ntpdate and ntpd? (see further below
> for ntpdate).

I don't have access to a 5.0.7 system, just SCO's online docs. My
experience has been with physical hardware servers, UnixWare and other
Unix-like systems. That said, here's what I know.


ntpdate is generally run _once_ during boot to set the system clock,
like this:

/usr/sbin/ntpdate -b -s time_server

-b
Force the time to be stepped using the settimeofday system call,
rather than slewed (default) using the adjtime system call. This option
should be used when called from a startup file at boot time.

-s
Divert logging output from the standard output (default) to the
system syslog(SLIB) facility. This is designed primarily for convenience
of cron scripts.

time_server

Best practice is to configure one host on your network to sync the time
with no more than three public stratum 2 ntp servers. Often an upstream
network device will provide ntp service, such as a firewall, router, or
even a local DNS server. Check with your network provider.

Otherwise, stratum 2 servers can be found here:
http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers

Next, configure all your local machines to sync to only your local
time server. This reduces bandwidth and load on the public ntp servers.
In your case, the local ntp server should probably be a physical machine
rather than a virtual one.

The ntpd daemon is then started and remains running to keep the system's
time in sync. The ntp daemon generally refuses to run if the time has
drifted too far (several minutes).

On UnixWare 7.1.4 ntpdate and in.xntpd are started on boot from
/etc/inet/config. I couldn't find in the SCO docs where these are
started on 5.0.7 systems.


If a box is out of time sync I generally do the following. Make
appropriate substitutions for locations of your files and daemons.

* kill any running [x]ntpd processes and make sure they are dead

* zero out the ntp drift file just to start fresh:
cat /dev/null > /etc/inet/ntp.drift

* run ntpdate _once_ with the -b option to set time:
/usr/sbin/ntpdate -b time_server

* start ntpd to keep time in sync:
/usr/sbin/in.xntpd

* After about 10 minutes, check status:

/usr/sbin/ntpq -pn

An entry prefixed with a '*' indicates it is the server your host is
currently synchronized with. See ntpq man page for more details.

I don't have an answer as to why the virtual servers won't keep time
correctly. I've had clocks run really fast when running other OSes
under emulators like bochs or qemu.

scot

echo myemailaddr | sed -e 's/nospam12//' to reach me

[Steve M. Fabac, Jr.]

Scot Jenkins wrote:
> Steve M. Fabac, Jr. wrote:
>> What's the difference between ntpdate and ntpd? (see further below
>> for ntpdate).

Scot,

What I was really looking for is an answer to the
question: "why does ntpdate work on the 5.0.7VM but ntpd does not?"

From man ntpdate:
Time adjustments are made by ntpdate in one of two ways. If
ntpdate determines the clock is in error more than 0.5 second it
will simply step the time by calling the system settimeofday(S)
routine. If the error is less than 0.5 seconds, it will slew the
time by calling the system adjtime(S) routine. The latter
technique is less disruptive and more accurate when the error is
small, and works quite well when ntpdate is run by cron every
hour or two. In the latter case, the slew adjustment is actually
50% larger than the measured offset, since this will tend to keep
a badly drifting clock more accurate.


And since the clock on the 5.0.7VM is always off by more then 0.5 seconds
"ntpdate -s server ntp1.kansas.net" will always step the clock and does
not require the -b directive.

>
> I don't have access to a 5.0.7 system, just SCO's online docs. My
> experience has been with physical hardware servers, UnixWare and other
> Unix-like systems. That said, here's what I know.
>
>
> ntpdate is generally run _once_ during boot to set the system clock,
> like this:
>
> /usr/sbin/ntpdate -b -s time_server

...

> Best practice is to configure one host on your network to sync the time
> with no more than three public stratum 2 ntp servers. Often an upstream
> network device will provide ntp service, such as a firewall, router, or
> even a local DNS server. Check with your network provider.
>
> Otherwise, stratum 2 servers can be found here:
> http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers

Thanks for the detailed answer. I've run ntpd on all my client's servers
for years with my basic /etc/ntp.conf file. They have always worked with
no trouble.

A special thanks for the link:
http://support.ntp.org/bin/view/Servers/StratumTwoTimeServers

Checking the primary servers link I see that tick.usno.navy.mil is
listed as "RestrictedAccess." Fancy that. I've never had a problem using
tick.usno.navy.mil restricted that it might be. But, being a good net citizen,
I'm updating all my client's systems and replaced tick.usno.navy.mil with
stratum 2 servers from the list that are geographically close to the client
machines.

Since I can't get ntpd to work on the 5.0.7 VM, I've had to continue using
ntpdate in crontab to slap the system back into some close approximation
of wall clock time.

Running ntpdate every five minutes and specifying a "RestrictedAccess"
stratum 1 server was a clumsy mistake.

>
> Next, configure all your local machines to sync to only your local
> time server. This reduces bandwidth and load on the public ntp servers.
> In your case, the local ntp server should probably be a physical machine
> rather than a virtual one.
>
> The ntpd daemon is then started and remains running to keep the system's
> time in sync. The ntp daemon generally refuses to run if the time has
> drifted too far (several minutes).

on 5.0.7 ntpd is started from /etc/tcp

# ls -lt /etc/rc2.d/*tcp*
... /etc/rc2.d/S85tcp -> /var/opt/K/SCO/tcp/2.1.1Hw/etc/tcp
# ls -lt /etc/tcp
... /etc/tcp -> /var/opt/K/SCO/tcp/2.1.1Hw/etc/tcp

I've always modified /etc/tcp to use ntpdate to set the clock before
starting ntpd so that ntpd does not choke because the clock is too
far from the correct time:

# ntpd - Don't run this at the same time as timed.
if [ -f /etc/ntp.conf ]; then
# Nb. tickadj is NOT necessary or recommended.
primary=`grep server /etc/ntp.conf | head -1 | awk ' { print $2 } ' `
ntpdate $primary
/etc/ntpd -g
echo "ntpd \c"
fi

#

And the result show up in /etc/rc2.d/messages/S85tcp.log:

+ [ -f /etc/ntp.conf ]
+ grep server /etc/ntp.conf
+ head -1
+ awk { print $2 }
primary=ntp1.kansas.net
+ ntpdate ntp1.kansas.net
10 Apr 16:37:06 ntpdate[413]: step time server 64.6.144.6 offset 6.189221 sec
+ /etc/ntpd -g
+ echo ntpd \c
ntpd

Showing that the clock was stepped 6 seconds on boot up.

That's interesting as I'd have expected the ntpdate on boot up to have a
larger error to deal with.

As a test I'll kill ntpdate in crontab, wait 10 minutes, note the
time difference and then command a reboot to see what ntpdate does on
the reboot.

507VM vs 507 Physical
Wed Apr 10 17:40:28 CDT 2013
Wed Apr 10 17:40:29 CDT 2013

507VM vs 507 Physical
Wed Apr 10 17:50:26 CDT 2013
Wed Apr 10 17:50:27 CDT 2013

Crontab edited to comment out ntpdate
-rw-rw---- 1 root cron 3246 Apr 10 17:55 /usr/spool/cron/crontabs/root

Machine left running to accumulate clock drift log.

507VM vs 507 Physical
Wed Apr 10 18:01:03 CDT 2013
Wed Apr 10 18:01:44 CDT 2013

507VM vs 507 Physical
Wed Apr 10 18:10:04 CDT 2013
Wed Apr 10 18:11:38 CDT 2013

507VM vs 507 Physical
Wed Apr 10 18:20:57 CDT 2013
Wed Apr 10 18:23:37 CDT 2013

So the last log entry for /usr/adm/check_date.log shows that the time is
off by -3:20 seconds (slow from wall clock time)

+ [ -f /etc/ntp.conf ]
+ grep server /etc/ntp.conf
+ head -1
+ awk { print $2 }
primary=ntp1.kansas.net
+ ntpdate ntp1.kansas.net
10 Apr 18:29:00 ntpdate[413]: step time server 64.6.144.6 offset 8.439303 sec
+ /etc/ntpd -g
+ echo ntpd \c
ntpd


# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
triangle.kansas navobs1.wustl.e 2 u 44 64 77 34.359 214340. 4000.00
ntp-2.gw.illino truechimer.cite 2 u 46 64 77 38.856 213721. 4000.00
clock.team-cymr usno.pa-x.dec.c 2 u 49 64 77 39.723 211847. 4000.00
#

But on boot up, ntpdate adjusts the clock by only 8.439 seconds?!?!?

Where is the expected three minutes 20 seconds?

>
> On UnixWare 7.1.4 ntpdate and in.xntpd are started on boot from
> /etc/inet/config. I couldn't find in the SCO docs where these are
> started on 5.0.7 systems.
>
>
> If a box is out of time sync I generally do the following. Make
> appropriate substitutions for locations of your files and daemons.
>
> * kill any running [x]ntpd processes and make sure they are dead
>
> * zero out the ntp drift file just to start fresh:
> cat /dev/null > /etc/inet/ntp.drift
>
> * run ntpdate _once_ with the -b option to set time:
> /usr/sbin/ntpdate -b time_server
>
> * start ntpd to keep time in sync:
> /usr/sbin/in.xntpd
>
> * After about 10 minutes, check status:
>
> /usr/sbin/ntpq -pn
>
> An entry prefixed with a '*' indicates it is the server your host is
> currently synchronized with. See ntpq man page for more details.
>
> I don't have an answer as to why the virtual servers won't keep time
> correctly. I've had clocks run really fast when running other OSes
> under emulators like bochs or qemu.
>
> scot
>
> echo myemailaddr | sed -e 's/nospam12//' to reach me
>
>

[Bela Lubkin]

> But on boot up, ntpdate adjusts the clock by only 8.439 seconds?!?!?
>
> Where is the expected three minutes 20 seconds?

3:20 was how much time the booted OSR507 VM kernel had lost out of its
own internal accounting of time. When you rebooted the VM, OSR507
kernel startup read a new starting point for time out of the VM virtual
hardware RTC. *That* clock appears to be 8s off (presumably due to the
cloud hosting provider not keeping the host environment NTP-sync'd). At
least I'm pretty sure the other boot instances I deleted out of the
quoted text showed a similar 8s offset at guest OS startup time...

Each guest boot reads the virtual RTC, which isn't losing time (or at
least is losing it at a very low rate).

As I remember, OSR507 loses time on ESX hosts because it programs the
PIT into a mode that ESX doesn't expect and can't really be expected to
emulate successfully.

========================================================================

I had a patch -- AFAIK still in use at some VMW customer(s) who received
it from me while I was there -- an OSR507 guest kernel patch which
caused it to use a better PIT mode which then didn't confuse ESX. But I
no longer have access to that and -- I apologize -- I should have
published that code when I had it. If I could have gotten permission
from VMW to do so.

The code in question was about 8 different runtime patches, provided in
source form as an OSR507 kernel driver embedded in its own
pack.d/$NAME/space.c file, thus compiled at kernel link time. I forget
the name of this driver. Oh yeah, it was "funi" because its first task
had to do with Fixing some UNI-processor madness in OSR507 kernel. I
never did get to a point where I felt I had 507 SMP running cleanly in a
VM: better to use a 1-CPU VM so that you do not light up the OSR507 SMP
code.

I recommend asking VMW support to let out the OSR507 guest "FUNI"
driver. They will not and should not be willing to *support* it, so I
doubt it would be released as a KB article or anything more formal than
that. Perhaps the "VMware Flings" route of publication would work.
Another possible route would be if they approached me, gave me
permission to independently publish it as my own work and not VMW's
responsibility -- they would also have to provide me with an actual copy
of it from which to do such publishing.

FUNI had fixes for 6-8 different OSR5-as-VMW-guest issues. Some
of these may already be fixed in either newer ESX releases than
I was working with (FUNI was developed on ESX 3.5 and eventually
partially-adapted to ESX 4.0); or in newer OSR5 releases (it was
developed to help OSR506 and 507-not-V, I believe at least some of
what it addressed was dealt with by SCO/TSG/UnXis by the time they
reached OSR507V3). Some may never have been fixed. The patch included
a control variable (OSR507 kernel boot parameter) by which you could
enable only specific sub-fixes; and all the fixes were designed to fail
gracefully, i.e. *not* patch the OSR5 kernel, if the expected code
wasn't detected.

As this is the first public mention, I will also more explicitly say: I
am no longer working at VMware. I've moved on to a virtualization-
related startup about which I will eventually say more. Although, for
purposes of this newsgroup, perhaps it would suffice to say that I doubt
the new company's work would ever be used to run any sort of SCO
anything...

>Bela<