jenkins running out of memory - heapspace - I have a memory leak ...

[Neil Tingley]

This is driving me slightly crazy. Jenkins keeps eating memory whether I give in 2g or 4g heapspace. Is there anything that anyone can think of I can do to figure out where the memory is going? I've got the monitoring plugin installed and have done some profiling but its not of much use.  I've got a heap dump if anyone would care to look at that? 

In three years of using hudson/jenkins this is the first time I've ever had such a problem. 

[Neil Tingley]

On Jenkins 4.1.3 - latest SVN plugin etc.

[Jesse Farinacci]

Greetings,

On Mon, Jul 11, 2011 at 3:12 AM, Neil Tingley <neilt...@gmail.com> wrote:
> Is there anything that anyone can think of I can
> do to figure out where the memory is going?

https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError

-Jesse

--
There are 10 types of people in this world, those
that can read binary and those that can not.

[Neil Tingley]

Thanks, looking at a heap dump now.  got about 50% of the heap as java.lang.StackTraceElement ...

[Nord, James]

Old verion of the subversion plugin?

 

/James

---

**************************************************************************************
This message is confidential and intended only for the addressee. If you have received this message in error, please immediately notify the postm...@nds.com and delete it from your system as well as any copies. The content of e-mails as well as traffic data may be monitored by NDS for employment and security purposes. To protect the environment please do not print this e-mail unless necessary.

NDS Limited. Registered Office: One London Road, Staines, Middlesex, TW18 4EX, United Kingdom. A company registered in England and Wales. Registered no. 3080780. VAT no. GB 603 8808 40-00
**************************************************************************************

[Neil Tingley]

Nope, saw that thead :-) Using v 1.28. Something somewhere is not finalizing properly.

[Maven User]

We have the same issue.

Just this morning I came into this:

com.google.common.collect.ComputationException:
java.lang.OutOfMemoryError: GC overhead limit exceeded

searching around, it seems related to a guava version (fixed in 09 I
guess). I can see that when I expand the descriptor element within
the monitoring app, I see this:

guava-r06.jar,

within the list. Also looking over the last few versions of jenkins,
I can see it's using 06.

Thoughts?

[Neil Tingley]

 Here's hoping its that then. Hopefully Abayer or kohsuke etc can chip in. 

[Maven User]

How many hits per minute are you serving up?

[Neil Tingley]

10-20 I guess. Not busy.

[Richard Geiger]

We're hitting a similar situation (see
http://groups.google.com/group/jenkinsci-users/browse_frm/thread/674ccc238dcac798
), so I'll hope along with you!

I'm not very experienced with javastuff, and have to wonder whether
it's possible to just somehow substitute the new (09) guava jar into
the Jenkins .war file, but I suspect there are build-time things that
need to be done, such that it will require a Jenkis rebuild. Do you
know if that's correct? Or *is* there a clever way to use the newer
guava release with the existing jenkins.war file?

[Maven User]

If you're lucky enough to have a staging build environment, I'd
suggest updating the jenkins war there, and trying it out for a few
days.

Have you looked at what is in the Hudson war? Just curious if they've
fixed the same thing already....


On Jul 13, 9:39 pm, Richard Geiger <r...@purestorage.com> wrote:
> We're hitting a similar situation (seehttp://groups.google.com/group/jenkinsci-users/browse_frm/thread/674c...

[Maven User]

I just looked at the Hudson war - it's shipping with guava 09.

Again - I have no idea of the implications of changing this jar and
quite honestly, your mileage may vary - but we're seeing the same
thing. Under load, we start leaking memory in a similar way that was
described in a different post that was fixed with the newer version of
guava.

IMHO - this has been my view on the Hudson/Jenkins split, as a
consumer of the product, that Jenkins offers a rapid release cycle/
newer features/robust plugin community vs the stability of Hudson.

I may have said this earlier, but we're serving up 2k to 4k hits per
min, I can't be restarting tomcat every third day because jenkins is
leaking memory and connections...

[Maven User]

Do you have the monitoring plugin installed?

If so, what does your memory consumption look like for a few weeks?
Mine looks like a saw blade where every point is where we had to
restart tomcat.

On my local jenkins instance, I can see that over time, the number of
active sessions keeps creeping skyward as well.

On Jul 13, 9:39 pm, Richard Geiger <r...@purestorage.com> wrote:

> We're hitting a similar situation (seehttp://groups.google.com/group/jenkinsci-users/browse_frm/thread/674c...

[Maven User]

I have dynatrace hooked up - getting some trending now....

I've set up 250 jobs that build at midnight (to do a "dir") and have
three people with tray trackers polling every second all 250 jobs.

I have an httpd caching proxy in front of jenkins (not sure if that's
recommended or not), and that seems to be keeping load down on the
tomcat server.

If there's nothing to report, I'll try dropping the caching proxy out
of the equation and continue to hammer this test server. The leak
should emerge.

I'll report back - most likely on Monday if there is anything exciting
(sooner if a better trend/details emerge).

[Maven User]

Anyone know what jrobin is?

[Maven User]

Apparently, org.jrobin is part of javamelody which is used in the
monitoring plugin for jenkins.

(this is probably NOT the issue, but part of the problem)

Continuing to trend...

[evernat]

I can take a look at the heap dump, in case I can see something.
Was it taken when the memory was almost full (>3G for example)?
Where can I download it from?

Emeric

[Maven User]

Sorry, we're building a trend right now.

I'll know more on Monday (hopefully)

[Vojtech Juranek]

> Thanks, looking at a heap dump now. got about 50% of the heap as
> java.lang.StackTraceElement ...

any change to see the heap dump? If not, it would be interesting to take a
look on gc roots of java.lang.StackTraceElement (i.e. which objects keep the
references to StackTraceElement objects)

[Maven User]

With apache proxying (and caching) requests, we've seen flat memory
usage.

So our memory usage and issues are clearly stemming from the high load
put on the server.

I'm going to try and drop away apache tonight and log for 24 - 48
hours again.

[Maven User]

Ok, I've again pulled apache httpd of the equation, three machines
asking for status on 400 jobs every second.

[Armin Häberling]

I think we suffer from the same memory leak as Neil does.

We run Jenkins 1.414 with the latest svn plugin. I see the same
symptoms, constantly growing memory usage and an increasing number of
StackTraceElement on the heap.

Looking at the heap dump is see that most of the StackTraceElement are
part of an ExportTable.
And almost all entries in the ExportTable contain a
hudson.scm.PerJobCredentialStore instance.

Armin

[Jerome Lacoste]

> Looking at the heap dump is see that most of the StackTraceElement are 
> part of an ExportTable. 
> And almost all entries in the ExportTable contain a 
> hudson.scm.PerJobCredentialStore instance. 

Do you see some XStream related instances as well ?

J

[Cees Bos]

Hi all, 

We also face very high heap usage with the latest versions.

We had a heap setting of 1.5 Gb for the master. We had to change that to 2Gb, but even then it reach out of heap. Now we have it on 3Gb and even now to reaches the limit, but not fail yet, but I expect it to fail today (but I cannot restart at this moment because of running jobs).

We upgrade from 1.413 to 1.417 to 1.429. In 1.413 we did not observe this issue.

Regards,
Cees

[Armin Häberling]

None in the Export table. There are some instances of
com.thoughtworks.xstream.core.util.ConcurrentWeakHashMap$HashEntry on
the heap, but that less 1% of all instances.
So I guess they are not related to the leak.

Armin

[Neil Tingley]

I am on 1.413 and raised the issue. Does anyone need another heap dump. I was restarting jenkins every day but thought I would leave it and let it soak up some memory.

[Christoph Kutzinski]

Have you already opened a JIRA issue for this?

BTW: could be that the leak is related to this commit
https://github.com/jenkinsci/remoting/commit/055bb4d644e775d4293439d203aacd40840142bc

-------- Original-Nachricht --------
> Datum: Wed, 20 Jul 2011 03:58:41 -0700 (PDT)
> Von: Neil Tingley <neilt...@gmail.com>
> An: jenkins...@googlegroups.com
> Betreff: Re: jenkins running out of memory - heapspace - I have a memory leak ...

[Maven User]

Couple of questions for everyone jumping on this thread -

1 - How are you running jenkins? Via a tomcat (or some other type
server) or standalone?
2 - What's the load on the server like?

We're running 1.418, had some similar issues to prior versions but can
confirm we had the same memory consumption issues in 1.417.

We ran dynatrace over night with +2K hpm average and I can see a
slight trend upward now, but it typically took me 2 - 3 days to hit
the oom errors...

Do people think this is directly related to the svn plugin? I'm not
even touching svn and seeing these errors/problems.

[Jerome Lacoste]

On Wednesday, July 20, 2011 2:26:00 PM UTC+2, kutzi wrote:

Have you already opened a JIRA issue for this?

BTW: could be that the leak is related to this commit
https://github.com/jenkinsci/remoting/commit/055bb4d644e775d4293439d203aacd40840142bc

It looks like the problem exists for some time:

http://jenkins.361315.n4.nabble.com/Severe-memory-leak-in-hudson-td3162254.html

The commit you point to remove auto-unexport (could explain a leak) but is from May.

One thing that is odd is the lack of synchronized on the PerJobCredentialStore#writeReplace, because of the use of the IS_SAVING ThreadLocal instance.

But I don't see the relation to the leak.

J

[Neil Tingley]

On Wednesday, 20 July 2011 13:41:56 UTC+1, Maven User wrote:

Couple of questions for everyone jumping on this thread -

1 - How are you running jenkins?  Via a tomcat (or some other type
server) or standalone?
2 - What's the load on the server like?

We're running 1.418, had some similar issues to prior versions but can
confirm we had the same memory consumption issues in 1.417.

1.413

Standalone.

I updated to guava-r09.jar to see if that would work

Will leave jenkins up until it gobbles 3gig and do another heap dump

 

 

[Maven User]

I'm curious - what jdks are people using?

Between the known memory leaks in tomcat 6.0.29 and the known memory
leaks in the jvm (some within the GC process) - the larger the system
the quicker we may be bleeding memory.

I have the same setup running locally now with jdk 1.6.0_22 and I'm
not bleeding memory anywhere close to the rate we are in production
using jdk 1.6.0_17.

On Jul 20, 4:00 am, Cees Bos <cbos...@gmail.com> wrote:

[Neil Tingley]

using 1.6.0_26 on linux RH 5.5. 

[Vojtech Juranek]

> I am on 1.413 and raised the issue. Does anyone need another heap dump.

if you can provide a heap dump, it would be nice and could make it more easy
to track (I haven't seen any link to any heap dump so far, just info that heap
is occupied by StackTraceElement which are refereed from ExportTable, which is
valueable information, but having whole heap dump would be better:-)

[Vojtech Juranek]

hm, we run 1.409.1 with SVN plugin 1.25 on Sun JDK 1.6.0_25 and don't observe
any memory problems (and have large enough instance that it has to appear if
there were this problem )

[Maven User]

Sadly, I also can't recreate this locally anymore.

My set up is:

jdk - 1.6.0_22
tomcat - 6.0.32
jenkins - 1.420

This is re-creatable with (our prod config)

jdk - 1.6.0_17
tomcat - 6.0.29
jenkins - 1.418

Both setups have 300+ jobs and a range of history for each. I was
able to have a few coworkers point their trackers at my setup,
decrease polling to 1 second and over night, saw it reach 25K hits per
minute. No appreciable memory leaking.

On our prod configuration however, with 2K - 3K hits per minute, I can
see the ramp effect in our memory consumption and I'm already planning
for a restart in the next three days.

Should I get a heap dump or something before it gets restarted? And
if so (because it will be MASSIVE), where's the best place to store if
someone wants to look?

[Maven User]

Heh - I tried that link and it crashed firefox - I'm wondering if I
could retrieve the contents via curl or something.

I saw it get way over #2K records there though.

As per the link below, it says "if there are too many" - how many is
too many?

What can we do about this?

My local testing I'm unable to have a master/agent configuration...

On Jul 11, 7:02 am, Jesse Farinacci <jie...@gmail.com> wrote:
> Greetings,
>
> On Mon, Jul 11, 2011 at 3:12 AM, Neil Tingley <neilting...@gmail.com> wrote:
> > Is there anything that anyone can think of I can
> > do to figure out where the memory is going?
>
> https://wiki.jenkins-ci.org/display/JENKINS/I'm+getting+OutOfMemoryError
>
> -Jesse
>
> --
> There are 10 types of people in this world, those
> that can read binary and those that can not.

[Maven User]

Ok, I did this against one agent:


curl -u <username>:<pass> -O http://<servername>/computer/<slavename>/dumpExportTable

Resulting file is 348 mb....

there are 4 million lines in there.

So is remoting the issue? What can I do to fix it? Additionally,
what should I look for in this file?

[Vojtech Juranek]

hm, I tried (just for sure that it's really not happening on our instance)
dump export tables on several slaves and observed what you described -
PerJobCredentialStore is not release from export table. On our instance it
doesn't cause any problems (acording to jmap StrackTrace occupies only about
30MB) as probably only very few jobs use SVN with credentials (AFAIK most of
the jobs use anonymous svn or git)

[Cees Bos]

The heap usage over the last month on our buildmaster.

I don't know why there are gaps, but this is the history we have based on the monitoring plugin.

It was working fine with 1.5 Gb memory, but even 3 Gb heap memory is not enough.

We are using CentOS 5.5 and OpenJDK 1.6.0_17.

[Maven User]

We have a similar graph - the gaps are the system being so pinned it
can't log is my guess.

On Jul 21, 1:07 pm, Cees Bos <cbos...@gmail.com> wrote:
> The heap usage over the last month on our buildmaster.
> I don't know why there are gaps, but this is the history we have based on
> the monitoring plugin.
>
> It was working fine with 1.5 Gb memory, but even 3 Gb heap memory is not
> enough.
>
> We are using CentOS 5.5 and OpenJDK 1.6.0_17.
>
> [image: usedMemory.png]
>

> On Thu, Jul 21, 2011 at 6:01 PM, Vojtech Juranek <vjura...@redhat.com>wrote:

[Vojtech Juranek]

I wasn't able to find a JIRA for it, so I created new one [1].
Still not sure about it, but IMHO it's caused by
PerJobCredentialStore.writeReplace() which increases reference count, but
later on it's not decreased and therefore it's kept in ExportTable

[1] https://issues.jenkins-ci.org/browse/JENKINS-10424

[Maven User]

Nice work - and thanks again!

I'll also keep my eyes peeled and if there's anything I can do, please
let me know.

[Maven User]

What release is this fix targeted for guys?

https://issues.jenkins-ci.org/browse/JENKINS-10424

(same as above)

[Vojtech Juranek]

> What release is this fix targeted for guys?

See Kohsuke's commnet under the JIRA:
"Currently the fix is targeted in 1.424."

[Maven User]

yep, I sat that, but yesterday it was listed here:

http://jenkins-ci.org/changelog

Today it isn't.

Not sure if that url is an accurate representation of the bugs
addressed in a given release or not.

[Vojtech Juranek]

> yep, I sat that, but yesterday it was listed here:
>
> http://jenkins-ci.org/changelog
>
> Today it isn't.

not sure why it disappears, but in jenkins rc branch it still depends on
remoting 2.2 where it was fixed, so IMHO should be fixed in upcomming release

[Maven User]

Can anyone confirm/deny this is corrected in 1.424?

Sorry - I seem unable to make a release note in jira that contains
just the changes for 1.424.

[Jesse Farinacci]

Greetings,

On Tue, Aug 2, 2011 at 9:02 AM, Maven User <maven....@gmail.com> wrote:
> Sorry - I seem unable to make a release note in jira that contains
> just the changes for 1.424.

Peek at the changelog:
https://github.com/jenkinsci/jenkins/blob/master/changelog.html#L77

or

$ git clone git://github.com/jenkinsci/jenkins.git
$ cd jenkins
$ git log jenkins-1.423..jenkins-1.424

[Jerome Lacoste]

On Tuesday, August 2, 2011 3:11:51 PM UTC+2, Jesse Farinacci wrote:
> Greetings,
> 
> On Tue, Aug 2, 2011 at 9:02 AM, Maven User <maven....@gmail.com> wrote:
>> Sorry - I seem unable to make a release note in jira that contains
>> just the changes for 1.424.

> Peek at the changelog:
> https://github.com/jenkinsci/jenkins/blob/master/changelog.html#L77
>
>or
> 
> $ git clone git://github.com/jenkinsci/jenkins.git
> $ cd jenkins
> $ git log jenkins-1.423..jenkins-1.424
>

The fix is indeed in 1.424 but the entry disappeared from the changelog page. We'll fix the page.

Jerome

[Maven User]

Thank you!

I think MANY of us are/were waiting for this change!

[Maven User]

For the record, I've upgraded tomcat to 6.0.32, jdk to 1.6.0_22 and
the memory consumption is FLAT.

I'm VERY excited to NOT have to worry about restarting everything
weekly now.

Long live Jenkins!

[Neil Tingley]

Hurrah for Jenkins! 

I'll test this fix on our setup.