Memory Leak in pipeline?

[Daniel Weber]

Hi all,

we keep running into memory issues with pipelines. I put together all information here:

   https://github.com/DanielWeber/jenkins-issues/tree/master/pipeline-memory-leak

including a small example I used to reproduce the problem. We're kind of desperate to 

find a solution, since we went "all-in" on pipelines and now we are in trouble.

Any pointers are welcome. We have a test machine running to reproduce this, I saved 

some heap dumps and can extract any information that could be of help.

Please help us, Jenkins Developers. You are our only hope... 

Thanks in advance!

Best Regards,

Daniel

[Daniel Beck]

Is this https://issues.jenkins-ci.org/browse/JENKINS-33358? Have you tried -Dgroovy.use.classvalue=true?

> --
> You received this message because you are subscribed to the Google Groups "Jenkins Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-de...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/2eb3e936-2d80-4a45-8077-bf1dfe39611f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Daniel Beck]

> On 07.11.2016, at 14:33, Daniel Beck <m...@beckweb.net> wrote:
>
> Is this https://issues.jenkins-ci.org/browse/JENKINS-33358? Have you tried -Dgroovy.use.classvalue=true?

Never mind, I just saw…

> • ✓ Set -Dgroovy.use.classvalue=true, as reported in JENKINS-33358. Did not help

FWIW I'd double check this is actually set at the correct location on the command line, and System.getProperty(…) reports the correct value. I've seen too many `java -jar jenkins.war -Dwhatever=something` in the past…

[Jesse Glick]

On Monday, November 7, 2016 at 7:58:35 AM UTC-5, Daniel Weber wrote:

we keep running into memory issues with pipelines

Already have a fix for a set of leaks in https://github.com/jenkinsci/workflow-cps-plugin/pull/83, which should be in the next release. Whether or not that is *your* leak is impossible to say without trying the update. Your GC root analysis shows only weak references, so it is not actually pointing to the problem at all.

[Daniel Weber]

Thanks. I double-checked. It was set correctly and really does not help here.

[Jesse Glick]

On Mon, Nov 7, 2016 at 9:21 AM, Jesse Glick <jgl...@cloudbees.com> wrote:
> should be in
> the next release.

Released as 2.23.

Do not use `groovy.use.classvalue`.

[Daniel Weber]

Hi Jesse,

I was about to start building the plugin on my machine when I saw your message about the release.

Updated it on our test machine. It is running for more than 10 hours now with 2.23, heap usage is at 

about 500MB, loaded classes below 18k. After more than 120k pipeline executions. 

More than 700k classes have been successfully *unloaded*.

Thanks a lot! This fixed our problem :-)

Best Regards,

Daniel

[Jesse Glick]

On Tue, Nov 8, 2016 at 3:27 AM, Daniel Weber <daniel.w...@gmail.com> wrote:
> Thanks a lot! This fixed our problem :-)

Good to hear.

There is test coverage demonstrating lack of leaks for basic
scenarios, which seems to have included yours, but there can always be
some surprises waiting in the wings. I found that when you have some
method defined outside the build, say in a plugin:

package some.permanent.pkg;
public class SomeClass {
@Whitelisted // go ahead, use me without worries
public static String checkTheFrobnitz(Object whatever) {…}
}

and you call it from Pipeline script with an argument whose *runtime
type* is defined in that script:

import some.permanent.pkg.*
stage('Checks') {
echo "Frobnitz verified: ${SomeClass.checkTheFrobnitz(this)}"
}

then Groovy will helpfully cache the fact that
`SomeClass.checkTheFrobnitz` on (this version of) `WorkflowScript`
should resolve to `checkTheFrobnitz(Object)` rather than some other
overload or weird dynamic invocation. And it will hold onto this cache
from `SomeClass`—so a new cache entry is created on every build, whose
“retained set” includes every class that build defined. The cache uses
a soft reference, so the entry will *eventually* get tossed out, but
not promptly after the end of the build (the VM argument
`-XX:SoftRefLRUPolicyMSPerMB=1` seems to ameliorate the problem). I
have not yet figured out a way to efficiently search for these entries
and blow them away, so for now I just hope this case is rare in
practice.

To make things more interesting, Groovy 1.x (used in Jenkins 1.x) and
2.x (used in Jenkins 2.x) each have their own set of bugs in this
area. The Java platform is not innocent either; there are poorly
conceived caches baked into JavaBeans and Serialization which do not
get cleared on time without some tortuous workarounds. Pipeline
implementation code tries to clean up everything it can—enough to make
the defined tests pass. Stay vigilant and report any reproducible
leaks in the future: classes not unloaded, `GroovyClassLoader`
instances never being collected.

/me sighs

[Karsten G.]

Hello Jesse,

Daniel and I are Jenkins pipeline fans so we will stay tuned and report any issues we find. Thanks again for the fast release, this saves us a lot of time and headaches.

Regards,

Karsten

[Ben Hines]

We are heavy pipeline users and have seen some improvement in memory use since updating to this version of the plugins, however, we are now seeing 'Compressed class space' OOMs fairly frequently. (running in Java 8 + Tomcat 8) Any tips on debugging it?

-Ben

[Jesse Glick]

Generally you want to look through heap histograms and check whether the number of `GroovyClassLoader`s is increasing (typically 4 per build IIRC) and never (hard) or rarely (soft) returning to a baseline even after explicit GC. If so, check root references in a memory analyzer (JVisualVM ~ NetBeans profiler, MAT, YourKit, etc.). A confirmed reference chain is enough to file a bug report, especially if steps to reproduce are known.