time out of sync due to high processor load

10 views
Skip to first unread message

Tino Schwarze

unread,
May 8, 2002, 1:58:08 PM5/8/02
to
Hi there,
recently, we've stumbled across a problem we cannot solve. We have
installed RedHat 7.2 on test boxen (currently RedHat 6.2 is used but
we're going to upgrade all 1000+ Linux boxen).

ntp (4.1.0) is installed and activated by default because we use AFS
which needs synched clocks for Kerberos authentication.

The problem is: If a CPU intensive process is run for a long time
(e.g. a distributed.net client), the time starts to get out of sync.
(around 7 minutes in 7 hours) We have two stratum-2 servers and
several stratrum-3 servers at our network.

The box we're currently examining this problem at is an Athlon 950.

Has anybody an idea what could cause this and how to fix it?

Bye+Thanks! Tino.

--
Those who desire to give up Freedom in order to gain Security,
will not have, nor do they deserve, either one. (T. Jefferson)

those who know me have no need of my name

unread,
May 12, 2002, 7:24:27 PM5/12/02
to
<abbovg$pdp$1...@narses.hrz.tu-chemnitz.de> divulged:

>The problem is: If a CPU intensive process is run for a long time
>(e.g. a distributed.net client), the time starts to get out of sync.
>(around 7 minutes in 7 hours) We have two stratum-2 servers and
>several stratrum-3 servers at our network.

this shouldn't happen, since the clock is being disciplined by code in the
kernel which no amount of cpu-bound processing can prevent. if you really
believe this to the the cause perhaps you should run the client at a
priority of 20 (lowest possible), or use a different scheduler, or apply
the nanokernel patches.


--
bringing you boring signatures for 17 years

Tino Schwarze

unread,
May 14, 2002, 7:17:05 PM5/14/02
to
those who know me have no need of my name <not-a-rea...@usa.net> wrote:

>>The problem is: If a CPU intensive process is run for a long time
>>(e.g. a distributed.net client), the time starts to get out of sync.
>>(around 7 minutes in 7 hours) We have two stratum-2 servers and
>>several stratrum-3 servers at our network.

> this shouldn't happen, since the clock is being disciplined by code in the
> kernel which no amount of cpu-bound processing can prevent. if you really
> believe this to the the cause perhaps you should run the client at a
> priority of 20 (lowest possible), or use a different scheduler, or apply
> the nanokernel patches.

The phenomenon can be reproduced. No dnetc - time is in sync. If I
start dnetc, I get a 30 second derivation half an hour later.
Stopping dnetc fixes this after a short while. I observed the jitter
and offset values to raise a lot (jitter > 4000).

We are pretty helpless.

Bye, Tino.

Koos van den Hout

unread,
May 15, 2002, 3:59:20 AM5/15/02
to
Tino Schwarze <tino.s...@informatik.tu-chemnitz.de> wrote:
> The phenomenon can be reproduced. No dnetc - time is in sync. If I
> start dnetc, I get a 30 second derivation half an hour later.
> Stopping dnetc fixes this after a short while. I observed the jitter
> and offset values to raise a lot (jitter > 4000).

Hmm.. if this is modern PC or Sun hardware: try installing software that
can read the motherboard temperature sensors. I do constant measurements of
cpu and motherboard temperature, and found the result of running rc5 quite
noticable:

http://idefix.net/~koos/pics/temperature-with-rc5.png
(link won't work when referenced from other pages)

Such a temperature change might 'destabilize' your system enough to let ntp
run out of sync.

Just my theory,

Koos van den Hout

--
Koos van den Hout, PGP keyid 0x27513781
ko...@cs.uu.nl herding Suns and networks
+31-30-2534104 Visit my site about books with reviews
http://idefix.net/~koos/ http://www.virtualbookcase.com/

Tino Schwarze

unread,
May 15, 2002, 11:44:04 AM5/15/02
to
Koos van den Hout <ko...@cs.uu.nl> wrote:

>> The phenomenon can be reproduced. No dnetc - time is in sync. If I
>> start dnetc, I get a 30 second derivation half an hour later.
>> Stopping dnetc fixes this after a short while. I observed the jitter
>> and offset values to raise a lot (jitter > 4000).

> Hmm.. if this is modern PC or Sun hardware: try installing software that
> can read the motherboard temperature sensors. I do constant measurements of
> cpu and motherboard temperature, and found the result of running rc5 quite
> noticable:

Hm. I'm pretty sure that the machine gets noticeably hotter when
running dnetc - I'm currently testing with an Athlon 950, stepping 2.

> Such a temperature change might 'destabilize' your system enough to let ntp
> run out of sync.

Why should the clock run out of sync due to high temperature?

Koos van den Hout

unread,
May 15, 2002, 12:57:19 PM5/15/02
to
Tino Schwarze <tino.s...@informatik.tu-chemnitz.de> wrote:
>> Such a temperature change might 'destabilize' your system enough to let ntp
>> run out of sync.

> Why should the clock run out of sync due to high temperature?

Given all the discussion of temperature versus stability of clocks
(including graphs of clock stability where opening of a window showed in
the graph) I thought the serious raising of temperature (due to high
processing) could destabilize the clock completely.

Koos

Ulrich Windl

unread,
May 17, 2002, 6:57:10 AM5/17/02
to
Tino Schwarze <tino.s...@informatik.tu-chemnitz.de> writes:

> those who know me have no need of my name <not-a-rea...@usa.net> wrote:
>
> >>The problem is: If a CPU intensive process is run for a long time
> >>(e.g. a distributed.net client), the time starts to get out of sync.
> >>(around 7 minutes in 7 hours) We have two stratum-2 servers and
> >>several stratrum-3 servers at our network.
>
> > this shouldn't happen, since the clock is being disciplined by code in the
> > kernel which no amount of cpu-bound processing can prevent. if you really
> > believe this to the the cause perhaps you should run the client at a
> > priority of 20 (lowest possible), or use a different scheduler, or apply
> > the nanokernel patches.
>
> The phenomenon can be reproduced. No dnetc - time is in sync. If I
> start dnetc, I get a 30 second derivation half an hour later.

30 seconds clock offset cannot be because of temperature coefficients
of a quartz. The problem require further examination, but I suspect
lost timer interrupts, something NTP can do little about.

> Stopping dnetc fixes this after a short while. I observed the jitter
> and offset values to raise a lot (jitter > 4000).
>
> We are pretty helpless.

Consider something like this for debugging in Linux i386:

void update_wall_time(unsigned long ticks)
{
do_get_fast_time(t);
+ if (ticks > 1)
+ printk(KERN_INFO "update_wall_time: ticks is %lu\n", ticks);

Ulrich

Hal Murray

unread,
May 18, 2002, 12:56:30 AM5/18/02
to
>30 seconds clock offset cannot be because of temperature coefficients
>of a quartz. The problem require further examination, but I suspect
>lost timer interrupts, something NTP can do little about.

This brings up a question I've been meaning to ask...

Modern CPUs have cycle counters. Lost timer interrupts
shouldn't be a serious problem. You just have to treat
a timer interrupt as a occasion to read the cycle counter
and update the clock based on the number of cycles since
the last reading rather than so-many microseconds per
interrupt.

I've browsed in the Linux kernel and couldn't find any
code that looked like that. Is it there? What do other
kernels do? Is there some reason it won't work that way?

--
The suespammers.org mail server is located in California. So are all my
other mailboxes. Please do not send unsolicited bulk e-mail or unsolicited
commercial e-mail to my suespammers.org address or any of my other addresses.
These are my opinions, not necessarily my employer's. I hate spam.

David Schwartz

unread,
May 18, 2002, 3:37:00 AM5/18/02
to
Hal Murray wrote:

> Modern CPUs have cycle counters. Lost timer interrupts
> shouldn't be a serious problem. You just have to treat
> a timer interrupt as a occasion to read the cycle counter
> and update the clock based on the number of cycles since
> the last reading rather than so-many microseconds per
> interrupt.

Okay, but *how* many microseconds?



> I've browsed in the Linux kernel and couldn't find any
> code that looked like that. Is it there? What do other
> kernels do? Is there some reason it won't work that way?

Chicken and egg problem. How do you decide how much time has elapsed
based upon the number of cycles that have passed? The base system clock
is not particularly stable (medium to long term) since it's usually not
designed to be.

DS

Hal Murray

unread,
May 18, 2002, 6:38:10 PM5/18/02
to
[using cycle counter to drive time-of-day clock]

> Chicken and egg problem. How do you decide how much time has elapsed
>based upon the number of cycles that have passed? The base system clock
>is not particularly stable (medium to long term) since it's usually not
>designed to be.

How does the current code work?

The Linux code prints out the CPU clock speed at startup so there
isn't any problem getting pretty close.

I've been assuming that the "drift" logic is calibrating the CPU
clock. It might be working on the 32KHz crystal feeding the battery
backed clock. Is that used for interrupts?

But the same technology for calibration should work in either case.

Or are you trying to tell me that the 32KHz clock is much better
(temperature wise) than the CPU clock? [I guess I could write code
to compare the cycle counter with the system clock.]

David L. Mills

unread,
May 18, 2002, 9:08:28 PM5/18/02
to
Hal,

It's actually worse; consider SMP systems where each processor has its
own cycle counter. My code, which is in the stock kernels for Ultrix,
Alpha and in the nanokernel and (somewhat modified) for FreeBSD, the
clock discipline consists of several interlocking feedback loops, one
for the timer interrupt and one for each processor. The cycle counters
interpolate between timer interrupts; the NTP daemon actually
disciplines only the timer interrupt. The cycle counters are wrangled
(right word) about once a second to align with the timer. The real hard
thing is to avoid little discontinuities when, as you point out, all the
little oscillators act like a rom full of cats jumping a little one way
and then another.

Dave

David L. Mills

unread,
May 18, 2002, 9:08:10 PM5/18/02
to Hal Murray
Hal,

It's actually worse; consider SMP systems where each processor has its
own cycle counter. My code, which is in the stock kernels for Ultrix,
Alpha and in the nanokernel and (somewhat modified) for FreeBSD, the
clock discipline consists of several interlocking feedback loops, one
for the timer interrupt and one for each processor. The cycle counters
interpolate between timer interrupts; the NTP daemon actually
disciplines only the timer interrupt. The cycle counters are wrangled
(right word) about once a second to align with the timer. The real hard
thing is to avoid little discontinuities when, as you point out, all the
little oscillators act like a rom full of cats jumping a little one way
and then another.

Dave

David Schwartz

unread,
May 18, 2002, 10:36:46 PM5/18/02
to
Hal Murray wrote:
>
> [using cycle counter to drive time-of-day clock]
>
> > Chicken and egg problem. How do you decide how much time has elapsed
> >based upon the number of cycles that have passed? The base system clock
> >is not particularly stable (medium to long term) since it's usually not
> >designed to be.
>
> How does the current code work?
>
> The Linux code prints out the CPU clock speed at startup so there
> isn't any problem getting pretty close.

Pretty close is worthless in this case. As processor load goes up,
processor temperature goes up too.

> I've been assuming that the "drift" logic is calibrating the CPU
> clock. It might be working on the 32KHz crystal feeding the battery
> backed clock. Is that used for interrupts?
>
> But the same technology for calibration should work in either case.
>
> Or are you trying to tell me that the 32KHz clock is much better
> (temperature wise) than the CPU clock? [I guess I could write code
> to compare the cycle counter with the system clock.]

Exactly.

DS

David Schwartz

unread,
May 18, 2002, 11:54:53 PM5/18/02
to

Of course, you could use the TSC to detect a missed clock interrupt, if
that's a real problem.

DS

Tino Schwarze

unread,
May 21, 2002, 11:34:38 AM5/21/02
to
Ulrich Windl <wiu0...@rcips2.cip.uni-regensburg.de> wrote:

>> >>The problem is: If a CPU intensive process is run for a long time
>> >>(e.g. a distributed.net client), the time starts to get out of
>> >>sync. (around 7 minutes in 7 hours) We have two stratum-2
>> >>servers and several stratrum-3 servers at our network.
>>
>> > this shouldn't happen, since the clock is being disciplined by
>> > code in the kernel which no amount of cpu-bound processing can
>> > prevent. if you really believe this to the the cause perhaps you
>> > should run the client at a priority of 20 (lowest possible), or
>> > use a different scheduler, or apply the nanokernel patches.
>>
>> The phenomenon can be reproduced. No dnetc - time is in sync. If I
>> start dnetc, I get a 30 second derivation half an hour later.

> 30 seconds clock offset cannot be because of temperature
> coefficients of a quartz. The problem require further examination,
> but I suspect lost timer interrupts, something NTP can do little
> about.

This already has been suggested by somebody else. I performed a little
test: Take timer-int-counter from /proc/interrupts, wait 10 minutes
(on external host, triggered via semaphore-file in AFS), take
timer-int-counter again, subtract. Result: 59059 interrupts in 10
minutes. The machine should get 100 interrupts per second - I conclude
that around 900 got lost during 10 minutes.

I verified that there are in fact interrupts missing by running the
test again without processor load. I got 60113 this time - I account
the one second to filesystem latency.

> Consider something like this for debugging in Linux i386:

> void update_wall_time(unsigned long ticks)
> {
> do_get_fast_time(t);
> + if (ticks > 1)
> + printk(KERN_INFO "update_wall_time: ticks is %lu\n", ticks);

Okay, maybe we're losing timer interrupts - but what can we do about it?
The TSC has been mentioned before; a kernel patch would be nice in any
case which adjusts for lost timer interrupts. (They should be clearly
detectable using the TSC.)

Johan Swenker

unread,
May 22, 2002, 6:31:37 AM5/22/02
to
In article <acdpee$85s$1...@narses.hrz.tu-chemnitz.de>,
tino.s...@informatik.tu-chemnitz.de says...

>
>Okay, maybe we're losing timer interrupts - but what can we do about it?
Search for the activity that blocks the interrupts.
I once had a similar problem. In my case it was linked to an insmod and
rmmod which were performed by a kernel deamon.
In your case it could be a proces like the journal deamon or an other
filesystem helper which disables interrupts momentarily.

Reducing the amount of memory (with a boot option) might solve this
problem.

>The TSC has been mentioned before; a kernel patch would be nice in any
>case which adjusts for lost timer interrupts. (They should be clearly
>detectable using the TSC.)
>
>Bye, Tino.
>
>--
>Those who desire to give up Freedom in order to gain Security,
>will not have, nor do they deserve, either one. (T. Jefferson)

BTW ever played with notebooks. They loose time even faster.

Regards, Johan Swenker

Mark Turner

unread,
May 22, 2002, 7:22:53 AM5/22/02
to
Johan Swenker <J.B.S...@kpn.com> wrote:
>Search for the activity that blocks the interrupts.

In my case on two different machines the loss of interrupts is caused
by IDE tape drives. My "fix" was to restart ntpd after the daily backups.

Cheers,

Mark.

Tino Schwarze

unread,
May 22, 2002, 8:30:25 AM5/22/02
to
Johan Swenker <J.B.S...@kpn.com> wrote:

>>Okay, maybe we're losing timer interrupts - but what can we do about it?

> Search for the activity that blocks the interrupts.
> I once had a similar problem. In my case it was linked to an insmod and
> rmmod which were performed by a kernel deamon.
> In your case it could be a proces like the journal deamon or an other
> filesystem helper which disables interrupts momentarily.

Fact is that there _should_ occur zero I/O while the distributed.net
client is running. It is _pure_ processor load happening here.

Researching linux-kernel archives, I get the impression that it is
really hard to detect lost timer interrupts since Intel made the
TimeStepCounter go slower when SpeedStep is in use.

Maybe I'll try the low latency patches and see whether that helps.

Hal Murray

unread,
May 23, 2002, 5:19:28 AM5/23/02
to
Thank you David and David for enlightening me about the way things
actually work.

I wonder if the kernel code will make more sense now that I won't
be looking for something that isn't there.

This might also explain why my attempt at temperature correction
isn't working as well as I expected - I've got the sensor on the
wrong crystal. The main crystal is right next to the CPU. The
32KHz crystal is over in a far corner of the board.

Is there any data out there on the relative temperature sensitivity
of the two crystals in a typical PC?


> It's actually worse; consider SMP systems where each processor has its
> own cycle counter. My code, which is in the stock kernels for Ultrix,
> Alpha and in the nanokernel and (somewhat modified) for FreeBSD, the
> clock discipline consists of several interlocking feedback loops, one
> for the timer interrupt and one for each processor. The cycle counters
> interpolate between timer interrupts; the NTP daemon actually
> disciplines only the timer interrupt. The cycle counters are wrangled

...

How many SMP systems have more than one main crystal? Is it
reasonable to assume that the cycle counters in all CPUs
are running at the same frequency? If so, would that simplify
or speed up the code?

How much would it help if there was a shared IO device with
a cycle counter that all CPUs in an SMP system could read?
(Say as part of a FPGA as per a previous thread.)
IO devices are horribly slow relative to modern CPUs.
That could easily be a step in the wrong direction.


> Pretty close is worthless in this case. As processor load goes up,
> processor temperature goes up too.

I don't think the processor temperature matters. It's the temperature
of the crystal and it's temperature coefficient. Or rather the
relative temperature swing of two crystals scaled by their
temperature coeffiecients and the actual temperature swing.
Yes, the slow crystal probably stays cooler because it is placed
near the edge of the board, but that doesn't matter if the room
temperature is changing.


> > Or are you trying to tell me that the 32KHz clock is much better
> > (temperature wise) than the CPU clock? [I guess I could write code
> > to compare the cycle counter with the system clock.]

> Exactly.

I wrote a first pass at the code. Read clock and cycle counter.
Wait a while. Read again. Compute cycles per second. Convert
to PPM off of (hard coded) target speed. Print answer in loopstats
style so it's easy to plot. ... (need doubles rather than floats)

It's roughly the same size as the drift computed by NTP - call it
3 PPM swing vs 2 over yesterday. Opposite sign. But there is
also >1 ms of swing in the clock offset and that's roughly 10 ppm
so I'm not sure what the final answer is.


Just for fun... Most PCs FM modulate the main clock to reduce EMI.
Anybody know what the temperature coefficient of that is?

David L. Mills

unread,
May 24, 2002, 1:28:17 AM5/24/02
to Hal Murray
Hal,

There are a couple of papers cited at the NTP project page (linked from
www.ntp.org) that have graphs of typical oscillator frequenxcy
misbehavior. The things you might be looking for are heat spikes due to
CPU fans, etc., but we have not seen really bad performance from that
source. In fact, with a bit of care, you can detect a blast of
floating-point computations by carefully watching the disciplined
frequency. But, the frequency watch is much more useful in detecting fan
failures and air conditioning misadventures, etc.

Dave

David L. Mills

unread,
May 24, 2002, 1:28:49 AM5/24/02
to
Hal,

There are a couple of papers cited at the NTP project page (linked from
www.ntp.org) that have graphs of typical oscillator frequenxcy
misbehavior. The things you might be looking for are heat spikes due to
CPU fans, etc., but we have not seen really bad performance from that
source. In fact, with a bit of care, you can detect a blast of
floating-point computations by carefully watching the disciplined
frequency. But, the frequency watch is much more useful in detecting fan
failures and air conditioning misadventures, etc.

Dave

Reply all
Reply to author
Forward
0 new messages