idle jenkings but constantly very high cpu usage

[gucki]

Hi there,

I'm using a fresh debain squeeze system and just installed latest jenkings (1.473) using the official debian package.

It's working great so far, but even when idle jenkins constantly consumes 100 - 200% cpu (8 cores). No jobs are running, no polling i happening (nothing configured), no "special" plugins are used...

jenkins  11653  0.0  0.0  18196   540 ?        S    11:46   0:00 /usr/bin/daemon --name=jenkins --inherit --env=JENKINS_HOME=/var/lib/jenkins --output=/var/log/jenkins/jenkins.log --pidfile=/var/run/jenkins/jenkins.pid -- /usr/bin/java -Djava.net.preferIPv4Stack=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/run/jenkins/war --httpPort=8080 --ajp13Port=-1 --httpListenAddress=127.0.0.1

jenkins  11655  178 14.1 1040216 297320 ?      Sl   11:46  16:37 /usr/bin/java -Djava.net.preferIPv4Stack=true -jar /usr/share/jenkins/jenkins.war --webroot=/var/run/jenkins/war --httpPort=8080 --ajp13Port=-1 --httpListenAddress=127.0.0.1

To debug this a little I looked at jenkings threads and it seems that 3-4 threads are constantly running. Strace of any of these threads shows several

futex(0x7f11dc007c28, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0x7f11dc007c54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341568603, 555590000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

futex(0x7f11dc007c28, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0x7f11dc007c54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341568603, 555684000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

futex(0x7f11dc007c28, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0x7f11dc007c54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341568603, 555776000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

futex(0x7f11dc007c28, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0x7f11dc007c54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341568603, 555871000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

futex(0x7f11dc007c28, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0x7f11dc007c54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341568603, 555966000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

futex(0x7f11dc007c28, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0x7f11dc007c54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341568603, 556060000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

futex(0x7f11dc007c28, FUTEX_WAKE_PRIVATE, 1) = 0

futex(0x7f11dc007c54, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, {1341568603, 556153000}, ffffffff) = -1 ETIMEDOUT (Connection timed out)

per second.

Anybody knows how to debug this further and fix?

Corin

[Les Mikesell]

That sounds like the linux 'leap-second' bug if the system has been up
since June 30th and is running ntp. One way to fix it is (as root
from the command line):
date -s "`date`"

--
Les Mikesell
lesmi...@gmail.com

[Corin Langosch]

On 06.07.2012 at 14:56 +0200, Les Mikesell <lesmi...@gmail.com> wrote:
> On Fri, Jul 6, 2012 at 4:58 AM, gucki <corin.l...@netskin.com> wrote:

>> ....

>>
>> It's working great so far, but even when idle jenkins constantly consumes
>> 100 - 200% cpu (8 cores). No jobs are running, no polling i happening
>> (nothing configured), no "special" plugins are used...

> That sounds like the linux 'leap-second' bug if the system has been up
> since June 30th and is running ntp. One way to fix it is (as root
> from the command line):
> date -s "`date`"

Great, that fixed it. Going to update my kernels asap :). Thanks!

[Les Mikesell]

On Fri, Jul 6, 2012 at 8:07 AM, Corin Langosch
<corin.l...@netskin.com> wrote:
>
>>> It's working great so far, but even when idle jenkins constantly consumes
>>> 100 - 200% cpu (8 cores). No jobs are running, no polling i happening
>>> (nothing configured), no "special" plugins are used...
>>
>> That sounds like the linux 'leap-second' bug if the system has been up
>>
>> since June 30th and is running ntp. One way to fix it is (as root
>> from the command line):
>> date -s "`date`"
>
>
> Great, that fixed it. Going to update my kernels asap :). Thanks!

No rush - there shouldn't be another leap second for a few years.
Probably long enough to forget how it is supposed to work again...

--
Les Mikesell
lesmi...@gmail.com

[Denis Walrave]

THANKS!

I had been struggling with this (hudson in my case...) before I read your post...
Denis

[Les Mikesell]

On Mon, Jul 9, 2012 at 1:55 AM, Denis Walrave <denis....@gmail.com> wrote:
> THANKS!
>
> I had been struggling with this (hudson in my case...) before I read your post...

It is just coincidental that the problem shows up in java - it is
really the kernel's thread handling that was messed up by a bug in
handling the leap second update. Resetting the system time about any
way other than via ntp on the day of a leap second should fix it.


--
Les Mikesell
lesmi...@gmail.com