java process consumes up to 200% CPU while using the UI

[Brian J. Murrell]

I have an installation of 1.596.2 that I am trying to get working here
but I am finding that any interaction with the web UI, even by just a
single user, will have the Jenkins java process consuming anywhere up to
200% of CPU (i.e. two full cores). Accordingly interaction is very
sluggish.

Probably nobody knows offhand why this is happening but I'd appreciate
any advise on how to debug what is going on.

While I am a software developer, I am not a Java developer and
completely unfamiliar with the Java development ecosystem but I suppose
there must be some way to profile Java applications, yes? Any pointers
would be much appreciated.

Cheers,
b.

[Brent Atkinson]

Brian,

There is a tool called jconsole that ships with the JDK. You can use it to see detailed information about what is happening. What you want to know is where the running threads are spending their time when the system bogs down.

Hope this helps,

Brent

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1428672933.25665.34.camel%40interlinx.bc.ca.
For more options, visit https://groups.google.com/d/optout.

[Brian J. Murrell]

On Fri, 2015-04-10 at 09:39 -0400, Brent Atkinson wrote:
> Brian,

Hi Brent,

> There is a tool called jconsole that ships with the JDK. You can use it to
> see detailed information about what is happening. What you want to know is
> where the running threads are spending their time when the system bogs down.
>
> Hope this helps,

Yes, very much so. Gives me somewhere to start at least. I seem to be
having trouble starting jenkins with JMX though. It seems to start up
initially, with the following arguments:

/etc/alternatives/java -Dcom.sun.akuma.Daemon=daemonized -Xmx3G
-Dhttp.proxyHost=proxy.rr.intel.com -Dhttp.proxyPort=911
-DJENKINS_HOME=/var/lib/jenkins -jar /usr/lib/jenkins/jenkins.war
--logfile=/var/log/jenkins/jenkins.log --webroot=/var/cache/jenkins/war
--daemon -Dcom.sun.management.jmxremote
-Dcom.sun.management.jmxremote.port=9010
-Dcom.sun.management.jmxremote.local.only=false
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false --webroot=/var/lib/jenkins/war
--httpsPort=8080 --httpPort=-1 --ajp13Port=-1

But then dies pretty quickly with:

Running from: /usr/lib/jenkins/jenkins.war
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:622)
at Main._main(Main.java:293)
at Main.main(Main.java:98)
Caused by: java.lang.IllegalArgumentException: Multiple command line argument specified: -Dcom.sun.management.jmxremote.port=9010
at winstone.cmdline.CmdLineParser.parse(CmdLineParser.java:68)
at winstone.Launcher.getArgsFromCommandLine(Launcher.java:361)
at winstone.Launcher.main(Launcher.java:334)
... 6 more

Any ideas?

b.

[Brent Atkinson]

Caused by: java.lang.IllegalArgumentException: Multiple command line argument specified: -Dcom.sun.management.jmxremote.port=9010

This seems to suggest that somehow the underlying servlet container (java web server, named winstone) is getting multiple arguments for the JMX port. I don't know if it's an option, but if you could run jconsole locally on the host it's much much easier.

Brent

b.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1428675483.25665.41.camel%40interlinx.bc.ca.

[Brian J. Murrell]

On Fri, 2015-04-10 at 10:26 -0400, Brent Atkinson wrote:
>
> This seems to suggest that somehow the underlying servlet container (java
> web server, named winstone) is getting multiple arguments for the JMX port.

Indeed.

> I don't know if it's an option, but if you could run jconsole locally on
> the host it's much much easier.

I tried that too. It still requires JMX. When I try to select the
Jenkins java PID jconsole reports "Note: The management agent is not
enabled on this process" and the Connect button is greyed out.

Cheers,
b.

[Brent Atkinson]

I tried that too.  It still requires JMX.  When I try to select the
Jenkins java PID jconsole reports "Note: The management agent is not
enabled on this process" and the Connect button is greyed out.

 

You're right, but connecting locally should not require the remote ports bound, just the agent enabled. Anyone know why this doesn't work out-of-the-box?

At this point, you have options:

  * figure out how to enable remote JMX (why it's failing for you)

  * figure out how to enable local JMX (not sure what's involved)

  * go another route, such as the monitoring plugin (can't vouch for it)

Brent

b.

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1428677235.25665.44.camel%40interlinx.bc.ca.

[Brian J. Murrell]

On Fri, 2015-04-10 at 11:31 -0400, Brent Atkinson wrote:
>
> You're right, but connecting locally should not require the remote ports
> bound,

No. You are right about that. But the problem is that the complaint is
not isolated to the JMX report options. *Any* JMX option yields a
complaint about duplicate options being found.

> Anyone know why this doesn't work
> out-of-the-box?

Yes, indeed. Anyone at all? Please.

> At this point, you have options:
>
> * figure out how to enable remote JMX (why it's failing for you)
> * figure out how to enable local JMX (not sure what's involved)

Right. These seem like the most fruitful paths but not really knowing
anything at all about how Java applications are developed or deployed
(i.e. I have no clue what a Jar is or what Winstone is, etc.) I'm kind
of grasping in the dark here.

> * go another route, such as the monitoring plugin (can't vouch for it)

I suspect analysing it from outside (i.e. jconsole) will be more
fruitful that analysing from the inside.

I did manage to find out that performance is better, (but still not what
I would expect) with one of the following plugins disabled:

antisamy-markup-formatter.jpi.disabled
ant.jpi.disabled
cobertura.jpi.disabled
configurationslicing.jpi.disabled
credentials.jpi.disabled
cvs.jpi.disabled
external-monitor-job.jpi.disabled
git.jpi.disabled
javadoc.jpi.disabled
locks-and-latches.jpi.disabled
mapdb-api.jpi.disabled
scm-api.jpi.disabled
ssh-slaves.jpi.disabled
subversion.jpi.disabled
throttle-concurrents.jpi.disabled
translation.jpi.disabled
windows-slaves.jpi.disabled
xunit.jpi.disabled
xvnc.jpi.disabled

But brute-force is not really the most fruitful way to go about this,
particularly when disabling random plugins can actually prevent Jenkins
from starting up -- leading to a yak-shaving exercise in debugging
plugin dependencies.

Plus the fact that even with every single plugin we have installed
disabled, performance is still sluggish, just not as sluggish as with
one or more of the above plugins enabled. With one or more of the above
plugins enabled, job->configure is particularly slow.

In any case, maybe time for a new thread on this list about how to
enable JMX with Jenkins...

Cheers,
b.

[Christoph Kutzinski]

There's also the jstack tool which comes which each JDK which can take a
stack trace of Jenkins. That might also help to figure out the original
problem.