jenkins 100% cpu, many blocked threads

[Zetan Drableg]

Hello,

Recently upgraded to Jenkins 1.537. Performance of the app has slowed a lot.

It takes forever to load views, even with only 20 jobs in them.
Wall Display plugin shows frequent timeouts.

Today I attached visualVM and can see jenkins @ 100% cpu.

Thread dump reveals 147 threads BLOCKED waiting to acquire a lock.

This looks like a classic contention bug where many threads try to access a data structure that has synchronized access.

Lots of these -

"Handling GET /jenkins//job/autoDevAgentTest_824test/api/json : http-8080-199" daemon prio=10 tid=0x00007f8ad41fc800 nid=0x55c0 waiting for monitor entry [0x00007f8aa5794000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:849)
        - waiting to lock <0x000000074742bd58> (a java.lang.Object)
        at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:454)
        at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1627)
        at hudson.model.User.getProjects(User.java:468)
        at hudson.scm.MailAddressResolverImpl.findMailAddressFor(MailAddressResolverImpl.java:21)
        at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:112)
        at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:532)
        at sun.reflect.GeneratedMethodAccessor1084.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66)
        at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:182)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
        at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:182)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
        at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
        at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Model.writeTo(Model.java:154)
        at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:267)
        at hudson.model.Api.doJson(Api.java:196)

vs one holding the lock

"Handling GET /jenkins//job/autoScmBAT_suse10_x86_apollo/api/json : http-8080-101" daemon prio=10 tid=0x00007f8ad4128000 nid=0x5524 waiting for monitor entry [0x00007f8abb46d000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at hudson.model.AbstractBuild.getChangeSet(AbstractBuild.java:849)
        - locked <0x000000074742bd58> (a java.lang.Object)
        at hudson.model.AbstractBuild.hasParticipant(AbstractBuild.java:454)
        at hudson.model.AbstractProject.hasParticipant(AbstractProject.java:1627)
        at hudson.model.User.getProjects(User.java:468)
        at hudson.scm.MailAddressResolverImpl.findMailAddressFor(MailAddressResolverImpl.java:21)
        at hudson.tasks.MailAddressResolver.resolve(MailAddressResolver.java:112)
        at hudson.tasks.Mailer$UserProperty.getAddress(Mailer.java:532)
        at sun.reflect.GeneratedMethodAccessor1084.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.kohsuke.stapler.export.MethodProperty.getValue(MethodProperty.java:66)
        at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:182)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
        at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:182)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
        at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:232)
        at org.kohsuke.stapler.export.Property.writeValue(Property.java:137)
        at org.kohsuke.stapler.export.Property.writeTo(Property.java:114)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:187)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Model.writeNestedObjectTo(Model.java:182)
        at org.kohsuke.stapler.export.Model.writeTo(Model.java:154)
        at org.kohsuke.stapler.ResponseImpl.serveExposedBean(ResponseImpl.java:267)
        at hudson.model.Api.doJson(Api.java:196)

Is this a known bug? Any work around?

Thanks
-Nate

[Baptiste Mathus]

Hi,
As you already seem to have dug quite a bit on this, I would advise to create a jira entry with the information you gathered.

Btw, your subject seems to show two opposite issues: blocked threads won't burn CPU. If Jenkins is actually occupying 100% cpu for a long period, you should also find RUNNING threads.

My 2 cents

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.