Loaded Classes Count grows over time: crashes Jenkins with Perm Gen

[grayaii]

We went from Jenkins 1.651.1 to 2.7.2 and our Jenkins master kept on crashing with:

java.lang.OutOfMemoryError: PermGen space

Where were not specifying -XX:MaxPermSize in our JENKINS_JAVA_OPTIONS before, so I added it, and kept on increasing the value after every crash.

We are now at 512m:

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xms3584m -Xmx3584m -XX:MaxPermSize=512m -Dhudson.model.ParametersAction.keepUndefinedParameters=true"

Using the Jenkins Monitoring plugin, we see that the Loaded Classes Count keeps growing without stopping:

How do you debug this? 

Something must be loading classes continuously.

Any ideas how we can debug this?

[grayaii]

I tried this tweaking the startup options, but things still grow:

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters=true"

We are using system groovy script and the python jenkinsapi pip module to do a lot of things, so I bet something changed in the way things load in groovy.

I just found a JIRA ticket associated with this:

https://issues.jenkins-ci.org/browse/JENKINS-33358

I ran the scripts here: https://wiki.jenkins-ci.org/display/JENKINS/Monitoring+Scripts and I see a bazillion things like this (Display heap histogram (object instances per class)):

sun.reflect.GeneratedMethodAccessor

sun.reflect.GeneratedSerializationConstructorAccessor

I'll see if adding this helps any:

-Dgroovy.use.classvalue=true

Alex

[Baptiste Mathus]

This is definitely not normal. You should take a heap dump and use typical tools for analyzing such dumps in Java. Eclipse MAT is one of them, developed [originally?] by SAP IIRC, but there are many others.

Also, I would definitely advise you to switch over to a JDK8, which hasn't permgen anymore. This is the JDK version we generally recommend. Note that the version of JDK used to run Jenkins can indeed be different than the one(s) used for your builds.

Cheers

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/025b4d0c-ce5a-4163-8447-6be523629cab%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Baptiste Mathus]

Correction: for the perm gen, this is not exactly a heap dump, by definition. IIRC, MAT or the likes should still be helpful. 

What you want to identify is the class FQN and so on that are the most present. Possibly, that'd help identiying which plugin or something is responsible for that leak.

[Alexander Gray II]

Thank you so much for this info.

I ended up upgrading my JDK from 7 to 8.

I also added: -Dgroovy.use.classvalue=true

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dgroovy.use.classvalue=true -Xms4096m -Xmx4096m -XX:MaxPermSize=1024m -XX:+CMSClassUnloadingEnabled -XX:+UseConcMarkSweepGC -Dhudson.model.ParametersAction.keepUndefinedParameters=true"

Which seemed to do the trick!

Now my classes count does not increase at all, and my permgen space is only around 100MB and it hasn't moved.

I also updated this ticket to say that by adding that command line arg, everything seems to work now:

https://issues.jenkins-ci.org/browse/JENKINS-33358

Thanks!!!

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/q7M2GJwArJ8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CANWgJS53VN_6FnLJ%2BQ58jzte1g2Zw00V1pCcH%3D4xxiSe-SiHNA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

-Alex-