Jenkins displays bogus Clock Difference
Martin B.
My master is runnning on a Windows 2003/32 server box (the service is
running as a domain user "builduser")
I have one Slave node that is Running on a Windows 2008R2/64 server box
(the slave service is also running under "builduser".
Both machines have exactly the same time/timezone settings, yet
seemingly arbitrarily Jenkins will display a Clock Difference of "59 min
behind" for this node.
Restarting the slave service doesn't help(or so I think).
When I restart the master service (leaving the slave running), the Clock
Difference warning is gone once the master has restarted and the slave
connected again.
The slave is connecting via JNLP.
Any ideas?
cheers,
Martin
Richard Mortimer
On 21/07/2011 14:04, Martin B. wrote:
> Hi!
>
> My master is runnning on a Windows 2003/32 server box (the service is
> running as a domain user "builduser")
>
> I have one Slave node that is Running on a Windows 2008R2/64 server box
> (the slave service is also running under "builduser".
>
> Both machines have exactly the same time/timezone settings, yet
> seemingly arbitrarily Jenkins will display a Clock Difference of "59 min
> behind" for this node.
>
One thought...
Is one or both of the instances started with a timezone set for the JVM.
This would override the the OS notion of timezone. Similarly one may be
configured to ignore DST or something similar.
I know that with the Sun jvm you can set timezones using
-Duser.timezone=UTC
or similar.
I've been bit by similar issues in the past.
Richard
Martin B.
> Hi,
>
> On 21/07/2011 14:04, Martin B. wrote:
>> Hi!
>>
>> My master is runnning on a Windows 2003/32 server box (the service is
>> running as a domain user "builduser")
>>
>> I have one Slave node that is Running on a Windows 2008R2/64 server box
>> (the slave service is also running under "builduser".
>>
>> Both machines have exactly the same time/timezone settings, yet
>> seemingly arbitrarily Jenkins will display a Clock Difference of "59 min
>> behind" for this node.
>>
> One thought...
>
> Is one or both of the instances started with a timezone set for the JVM.
> This would override the the OS notion of timezone. Similarly one may be
> configured to ignore DST or something similar.
>
> I know that with the Sun jvm you can set timezones using
>
> -Duser.timezone=UTC
>
> or similar.
>
> I've been bit by similar issues in the past.
>
At the moment, no Clock Difference is displayed, but I remeber these
values being the same the last time a Clock Diff was displayed and I
checked these.
The system properties are:
master - jenkins/systemInfo
* java.runtime.version = 1.6.0_24-b07
* os.arch = x86
* os.name = Windows 2003
* os.version = 5.2
* user.country AT
* user.language = de
* user.timezone = Europe/Berlin
slave - jenkins/computer/name1/systemInfo
* java.runtime.version = 1.6.0_26-b03
* os.arch = x86 (I suspect this is because it's the 32bit Java)
* os.name = Windows Server 2008 R2
* os.version = 6.1
* user.country AT
* user.language = de
* user.timezone = Europe/Berlin
So there isn't really any difference here as far as I can see ...
cheers,
Martin
Richard Mortimer
I agree. No obvious difference there.
>>> Restarting the slave service doesn't help(or so I think).
>>>
>>> When I restart the master service (leaving the slave running), the Clock
>>> Difference warning is gone once the master has restarted and the slave
>>> connected again.
>>>
>>> The slave is connecting via JNLP.
So if restarting the master makes it go away do you know what causes it
to appear?
Does it just appear after a period of normal running or is it related to
restarting the slave?
I don't really know how all the Clock Difference checking works but
having a clue as to when the problem appears might help someone who does
know.
Regards
Richard
Martin B.
> On 22/07/2011 12:15, Martin B. wrote:
>> On 21.07.2011 15:42, Richard Mortimer wrote:
>
>>> I know that with the Sun jvm you can set timezones using
>>>
>>> -Duser.timezone=UTC
>>>
>>> or similar.
>> At the moment, no Clock Difference is displayed, ...
I just checked now, after the weekend, and the "59 min behind" display
is back.
My first posting here was on Thursday 21st, and on Friday 22nd there
wasn't any Difference display. There was also no activity at all on
Jenkins during the weekend (since it's still in a test phase) but the
servers are utilized.
So *either* it's to do with the duration (that would be 4 days since
Thursday) *or* it's something that's done on one of the servers: Maybe
some script has some (temporary) side-effect wrt. to the clock settings
that's confusing Jenkins.
Come to think of it: we have one automated test that (temporarily)
modifies the system time by one hour for a few seconds. That one
definitely also ran from Thursday to Friday, but maybe it takes a few
runs and some race condition to confuse Jenkins and the multiple runs on
the weekend did it.
As it is, at the moment the clocks+settings are fully in sync and
Jenkins is still displaying a Clock Difference, but - aha! -
When I refreshed the page in the browser (with F5) the Clock Difference
was still there, but now I clicked the [Refresh Status] button and the
Difference Display is gone. (I missed that button in the past and
assumed refreshing the page would refresh the status somehow.)
Well, seems I'll have to monitor that further to see how it's going ...
## BTW: Is there *any* logging wrt. to this Clock Difference detection
that I could enable?? ##
cheers,
Martin
Simon Wiest
Jenkins is updating the node monitors for disk space, clock sync, ping
time, etc. once every hour - or when you press the 'refresh' button, of
course.
If you have a build that fiddles about the system clock, perhaps it's
just a unlucky coincidence that Jenkins did run the clock monitor
exactly while your build changed the system clock... ;O(?
You can examine the node monitor activity by adding a logger for this:
1. Goto Jenkins -> Manage Jenkins -> System Log.
2. Press "Add new log recorder..." button.
3. Name the new recorder as you like, e.g. "hudson.node_monitors"
4. Press "Add" next to "Logger".
5. Enter "hudson.node_monitors" in the Logger field, set Logging level
to "All", press "Save".
6. Refresh the page of this new logger to see logging messages from the
node monitors, like the following (sorry, the output contains some
German, but you'll get the idea...):
25.07.2011 16:38:59 hudson.node_monitors.SwapSpaceMonitor$1
FINE: Node monitoring Freier Swap Space completed in 78ms
25.07.2011 16:38:59 hudson.node_monitors.ClockMonitor$1
FINE: Node monitoring Zeitdifferenz completed in 31ms
25.07.2011 16:38:59 hudson.node_monitors.TemporarySpaceMonitor$1
FINE: Node monitoring Freier TEMP-Platz completed in 31ms
25.07.2011 16:38:59 hudson.node_monitors.DiskSpaceMonitor$1
FINE: Node monitoring Freier Plattenplatz completed in 31ms
25.07.2011 16:38:59 hudson.node_monitors.ResponseTimeMonitor$1
FINE: Node monitoring Antwortzeit completed in 16ms
25.07.2011 16:38:59 hudson.node_monitors.ArchitectureMonitor$DescriptorImpl
FINE: Node monitoring Architektur completed in 16ms
Hope that helps,
cheers,
Simon.
--
Martin B. (25.07.2011 09:00):
Martin B.
> Hi Martin,
>
> Jenkins is updating the node monitors for disk space, clock sync, ping
> time, etc. once every hour - or when you press the 'refresh' button, of
> course.
>
> If you have a build that fiddles about the system clock, perhaps it's
> just a unlucky coincidence that Jenkins did run the clock monitor
> exactly while your build changed the system clock... ;O(?
>
> You can examine the node monitor activity by adding a logger for this:
>
> 1. Goto Jenkins -> Manage Jenkins -> System Log.
> 2. Press "Add new log recorder..." button.
> 3. Name the new recorder as you like, e.g. "hudson.node_monitors"
> 4. Press "Add" next to "Logger".
> 5. Enter "hudson.node_monitors" in the Logger field, set Logging level
> to "All", press "Save".
> 6. Refresh the page of this new logger to see logging messages from the
> node monitors, like the following (sorry, the output contains some
> German, but you'll get the idea...):
>
Deutsch: Kein Problem! ;-)
This is excellent! Hopefully I'll get to the bottom of this using these
logs.
cheers,
Martin