Jenkins Master Goes to 100% (at random?)

[Eric Crahen]

Hi,

  We're running Jenkins master with a bunch of slaves. Typical load on the master is around 5-10% which is not very much. Several times now, we've gotten into spikes where the CPU goes to 100% and stays there until we restart it, and this usually only helps a little.

  We dump the stack, and typically we find a whole bunch of threads sitting there marshaling XML aggressively. Usually, we can connect this to a plugin via the stack trace, and we'll uninstall that plugin, and that tends to help. 

  So far, the plugins I can remember include:

* audit-trail

* build-timer

* log-parser

  That's just the most recent ones that I can remember. We must be going on the 5th or 6th plugin now. I'm afraid it's starting to shake peoples confidence in Jenkins, I'd rather not have to replace it with a different build system because it generally works, but when it doesn't - it's bad. We actually keep a CircleCI system around because we haven't been able to build enough confidence in Jenkins.

  I have a couple general questions:

  1) We're on Jenkins 2.18 - so, why does this keep happening? We've seen other bugs about threads getting stuck hammering the CPU. Seems like the issues were marked as resolved, but we end up having to uninstall these plugins all together.

  2) Is there a list of known "bad" plugins?

  3) Why not cache more and guard against this? It's easy to blame the plugins as the thing triggering this, but why not make Jenkins core more resilient to this, and assume people can't/won't write perfect plugins? Getting to be a fool me once type of situation here....

--

- Eric

[Daniel Beck]

You don't say what files get loaded. Assuming it's builds, I'd look into available heap memory. Install the monitoring plugin. With too little memory, Jenkins can get into situations where it needs to load builds, and to make space, discards already loaded build records, just to discard the new builds from memory again, …

For example the project stats plugin is known to cause this a lot.

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAPyj2qEQ8Ych%2B6%3DDd71QVwR%2B0nkPY%3D3t9r%3DyKWE6YeUVUd9Y8g%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[touseef]

you can increase the heap size in the jenkins config.xml file, that will increase the availability of heap size for jenkins ,this might help u.Other thing is u can ugrade to 2.7.4 latest stable version of jenkins