Jenkins using a lot more resources after upgrade

[Ugo Bellavance]

Hi,

After some testing we upgraded Jenkins in production, from 1.617-1.1 to 1.656-1.1 on April 20. However, since the upgrade we're running into problems that we never saw before and it's now kind of out of control:

• We had to raise the nofile parameter in limits.conf for the jenkins user because we were getting "Too many open files" errors. We raised it to 8192 soft, 10240 hard on April 28th. By looking into /proc/pid/ld/ I found out that most of these files where inet sockets. Got these errors again on May 8th and 9th. I didn't change anything yet for that issue
• We had to raise the java heap memory. It was at -Xmx4096m, we raised it to -Xmx7168m, then to -Xmx10752m and adjusted the vRAM allocation for this VM accordingly, but it doesn't solve the problem, we're still getting "java.lang.OutOfMemoryError: Java heap space" errors.  When that starts happening, we have to restart jenkins because the web interface is not responding anymore (which makes it difficult to troubleshoot because we can't really tell what is running at this moment.

Note that we do run Selenium testing through IC, but nothing has changed drastically after the upgrade. RHEL updates were applied a few days before (Apr 6th) and that include a minor update to Firefox (38.0 => 38.7). We're using openjdk java JRE. Was updated to 1.7.0.99 (from 1.7.0.79) on April 6th, 1.7.0.101 on May 2nd.

Has anyone experienced something similar?  What are my best options? Our workaround now is to restart jenkins manually when needed.

• Would it be possible to rollback to 1.617 without breaking anything?
• Try the LTS version (that would mean a downgrade - would it break stuff?)
  
• Jump to version 2 (2.2-1.1 available)
• Inspect the JVM's memory
• Any other idea is welcome

Thanks in advance,

Ugo

[John Mellor]

Yeah, I’m seeing that too, and I am also running Jenkins 1.656 version.  I’m only running about 400 jobs in Jenkins with 11 slaves, so it’s definitely not the busiest build environment out there.  I’ve currently bumped the heap up to 5GB, and still do not have anywhere near enough to run the backup plugin for example.  I’m thinking of doubling it to see if that helps, but your experience seems to say that this is also not going to be enough.

 

This problem is also scaring me about whether unreasonable resource utilization has been corrected in a near-future upgrade to Jenkins 2.2…

--
You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/4e6f40ca-cbf7-41b4-874e-18df1c04f9ce%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ugo Bellavance]

On Wednesday, May 11, 2016 at 12:32:59 PM UTC-4, John Mellor wrote:

Yeah, I’m seeing that too, and I am also running Jenkins 1.656 version.  I’m only running about 400 jobs in Jenkins with 11 slaves, so it’s definitely not the busiest build environment out there.  I’ve currently bumped the heap up to 5GB, and still do not have anywhere near enough to run the backup plugin for example.  I’m thinking of doubling it to see if that helps, but your experience seems to say that this is also not going to be enough.

I could try increasing again, but it doesn't make much sense.

 

 

This problem is also scaring me about whether unreasonable resource utilization has been corrected in a near-future upgrade to Jenkins 2.2…

It's partly comforting, but also scary to see that we're not alone with this problem.  Yes, it would be interesting to know if it has been fixed in version 2, and it would also be nice to know when this problem was introduced and if we can downgrade to a know working version.  Should we open a bug in jenkins's JIRA? Have you seen https://issues.jenkins-ci.org/browse/JENKINS-34573? Have you generated a heap dump? I'm not too familiar with that (yet).

Ugo

[Stephen Connolly]

I am suspecting JENKINS-34213 may be your issue.

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/d8605af0-9c53-434c-9d10-6b82fd762cf7%40googlegroups.com.

[Ugo Bellavance]

On Thursday, May 12, 2016 at 4:31:37 AM UTC-4, Stephen Connolly wrote:

I am suspecting JENKINS-34213 may be your issue.

I have new information.  We installed the monitoring plugin and I started using VisualVM.  I can see that, while I use -Xmx10752m, it only uses a little less than 8 GB.  Is there a problem with my configuration? I also got a heap dump.

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dcom.sun.management.jmxremote -Xmx10752m"

(in /etc/sysconfig/jenkins on RHEL6).

[Ugo Bellavance]

Links for images:

 http://picpaste.com/pasted-1463060174-109834-vyo0x5qe.png

http://picpaste.com/usedMemory-YYL6OQhQ.png

[Raymond Accary]

Hi,

If it helps, you might avoid the crash by installing the monitoring plugin, and triggering garbage collection once the memory is approaching the maximum allocated heap size. This is a workaround until someone is able to diagnose the root cause. I have an open issue : https://issues.jenkins-ci.org/browse/JENKINS-34573 but thought I'd run the suggestion.

[Ugo Bellavance]

On Thursday, May 12, 2016 at 2:01:41 PM UTC-4, Raymond Accary wrote:

Hi,

If it helps, you might avoid the crash by installing the monitoring plugin, and triggering garbage collection once the memory is approaching the maximum allocated heap size. This is a workaround until someone is able to diagnose the root cause. I have an open issue : https://issues.jenkins-ci.org/browse/JENKINS-34573 but thought I'd run the suggestion.

Have anyone monitored the JVM with VisualVM? I have found that it looks like a specific memory pool may be filling up: Old Gen.  I'm trying now with:

-Djava.awt.headless=true -Dcom.sun.management.jmxremote -Xms10752m -Xmx10752m -XX:NewRatio=10

We'll see in a few days.  In the last few days, the problem have occured each morning at around 4:45.  I'm not sure if the same jobs run on the week-ends as well.

[Ugo Bellavance]

From what I can see, it will always fill the memory.  Probably a leak, but I don't know how to be 100% sure about it. When I try to load a heap dump in VisualVM, but it shows nothing except "Not supported for this JVM. 

[Stephen Connolly]

I will repeat:

> I am suspecting JENKINS-34213 may be your issue.

--

You received this message because you are subscribed to the Google Groups "Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-use...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/1b43cd1d-531b-4638-8e53-15d5f5670f91%40googlegroups.com.

[Ugo Bellavance]

Thanks, but I can see that this issue is fixed but I don't know how to update my Jenkins install so that I have the fix.  I'm using 1.656.

Ugo

[Stephen Connolly]

It is fixed, but in the latest version of remoting... which is not something you can upgrade without either building a custom build of Jenkins or upgrading Jenkins.

You could crack open your jenkins.war and replace the remoting.jar with the fixed version and seal it back up again and see if that fixes your issue... probably the quickest way to confirm if my theory as to the source of your leak is correct, but you would need to know what you are doing

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/145bf27e-4bcc-4444-b46c-9d8fe4efc1a2%40googlegroups.com.

[Stephen Connolly]

Jenkins 2.5 has the fix IIUC

[Ugo Bellavance]

Ok, I did analyse the heap dump with Eclipse's Memory Analyzer Tool and here's what I found:

One instance of "java.util.concurrent.ScheduledThreadPoolExecutor" loaded by "<system class loader>" occupies 6,801,517,856 (89.16%) bytes. The instance is referenced by org.tmatesoft.svn.core.wc.DefaultSVNRepositoryPool @ 0x561b36858 , loaded by "hudson.ClassicPluginStrategy$AntClassLoader2 @ 0x56057c428". The memory is accumulated in one instance of "java.util.concurrent.RunnableScheduledFuture[]" loaded by "<system class loader>".

Does it match JENKINS-34213?

[Stephen Connolly]

Hard to say for certain as that could still be references maintained by the unexported objects that are pending unexport. If you can get your instance into that state and then have it sit idle for a couple of hours... if it is JENKINS-34573 then you should see the retained memory reduce as one reference is cleared every minute. Alternatively disconnect and reconnect all your slaves and see if that frees the memory up - JENKINS-34573 is only an issue for long running busy slaves

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a7867041-c143-4d08-a7ee-22e4b78a4418%40googlegroups.com.

[Ugo Bellavance]

Hi Stephen,

We don't use slaves. We only have one master.

Should I create an issue in JIRA for my specific issue?

(is there a reason why you got the conservation out of the group?)

Thanks,

--
You received this message because you are subscribed to a topic in the Google Groups "Jenkins Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jenkinsci-users/qLwBFyQ84Z4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jenkinsci-use...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CA%2BnPnMyz55b4ppFhQ3STgH5fK7cwmanns8VR_yFmmDLZ5BVKeA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Ugo Bellavance (ug...@lubik.ca)

[Stephen Connolly]

If you have no slaves and only run builds on the master then we can definitively state that JENKINS-34573 is not the root cause. If you have slaves (even if idle or unused) then we cannot.

If you have a definitive answer to the above question then you should probably create a JIRA

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CACSSk%2BUNBm15yaMbhm85tcd%3D50aN2E8eKbf0BMvv0TEdoWS3Og%40mail.gmail.com.

[Ugo Bellavance]

We only have 1 Jenkins server so I'll create a JIRA,

Thanks a lot for your help :)!

[Ugo Bellavance]

https://issues.jenkins-ci.org/browse/JENKINS-34919

[Ugo Bellavance]

Any idea about how long it should take before I get at least an acknoledgement on my JIRA?

[Stephen Connolly]

This is a community project. If you need commercial support with corresponding SLAs then you might consider obtaining a support contract from a commercial supplier. I know my employers (CloudBees) are in this realm and I am aware that there are others.

Otherwise, unless you can find somebody who is interested in scratching this itch and has time to scratch it, you are just going to have to wait or try and scratch it yourself.

To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/a324b78c-1446-4778-aa77-d8b1cc49f1dd%40googlegroups.com.