OOM error in submodule build; Jenkins says "SUCCESS"
Drury, Tim
I’ve posted this issue on StackOverflow with screenshots here: http://stackoverflow.com/questions/9672641/strange-build-status-for-jenkins-maven-submodule
This is Jenkins 1.450 building a Maven 3.0.4 project (as a maven project, not free form) with many submodules.
I was noticing some strange behavior that one artifact appeared to have stale classes. When I dug down into that artifact build I noticed it appeared to have several zombie builds running for days/weeks (see screenshot at link), yet no executors were consumed. I didn’t follow the “Jenkins hung” process, but restarted both slaves and the master (sorry!). After restarting the main project, which had appeared to be unstable (sadly this is normal for us), now showed all builds as failed! This is “strange issue #1”.
Digging into the logs I noticed that this particular submodule was throwing an OutOfMemoryError while building, but Jenkins (or Maven?) was labeling it as successful:
Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:574)
at java.lang.Shutdown.runHooks(Shutdown.java:128)
at java.lang.Shutdown.sequence(Shutdown.java:173)
at java.lang.Shutdown.exit(Shutdown.java:218)
at java.lang.Runtime.exit(Runtime.java:90)
at java.lang.System.exit(System.java:869)
at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:149)
at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:63)
Triggering a new build of foo-main-cobertura #345
Finished: SUCCESS
and was building downstream projects despite the fact that the build failed with an OOM error. This is “strange issue #2”.
I’ve increased the heap as far as I can with the 32-bit JVM so I’m switching to a 64-bit JVM trying to get a good build for QA.
Any ideas about these two issues:
1. Seemingly good/unstable builds switching to red (failed) after Jenkins restart
2. OOM error during build interpreted as successful
Thanks,
-tim
Tim Drury
Architect
SAP Manufacturing Execution (SAP ME)
Supply Chain Management
SAP Labs, LLC
Please consider the impact on the environment before printing this e-mail.
Drury, Tim
Since I’ve gotten no response on this issue, I have a follow-up “meta” question:
Does anyone think this is user error or a bug? Should I create a jira issue for it?
I don’t want to overdramatize my issue, but showing a build as good/unstable when, in fact, it’s bad is a serious issue for software that builds production-quality products.
-tim
Richard Mortimer
From what you have said there doesn't seem to be any obvious user error
and it does look like some form of bug to me.
Before reporting it in JIRA it would be a good idea to follow the advice
at https://wiki.jenkins-ci.org/display/JENKINS/Build+is+hanging to see
if there are any threads hanging around before you restart Jenkins.
There may also be some errors in the main jenkins log at the following
URL (replace <jenkins base url> the the URl of your Jenkins installation).
http://<jenkins base url>/log/all
Of course once you have that stuff a quick check using your favourite
search engine for similar reports would be good.
Regards
Richard
On 14/03/2012 13:45, Drury, Tim wrote:
> Since I�ve gotten no response on this issue, I have a follow-up �meta�
> question:
>
> Does anyone think this is user error or a bug? Should I create a jira
> issue for it?
>
> I don�t want to overdramatize my issue, but showing a build as
> good/unstable when, in fact, it�s bad is a serious issue for software
> that builds production-quality products.
>
> -tim
>
> *From:*jenkins...@googlegroups.com
> [mailto:jenkins...@googlegroups.com] *On Behalf Of *Drury, Tim
> *Sent:* Tuesday, March 13, 2012 9:40 AM
> *To:* jenkins...@googlegroups.com
> *Subject:* OOM error in submodule build; Jenkins says "SUCCESS"
>
> I�ve posted this issue on StackOverflow with screenshots here:
> http://stackoverflow.com/questions/9672641/strange-build-status-for-jenkins-maven-submodule
>
> This is Jenkins 1.450 building a Maven 3.0.4 project (as a maven
> project, not free form) with many submodules.
>
> I was noticing some strange behavior that one artifact appeared to have
> stale classes. When I dug down into that artifact build I noticed it
> appeared to have several zombie builds running for days/weeks (see
> screenshot at link), yet no executors were consumed. I didn�t follow the
> �Jenkins hung� process, but restarted both slaves and the master
> (sorry!). After restarting the main project, which had appeared to be
> unstable (sadly this is normal for us), now showed all builds as failed!
> This is �strange issue #1�.
>
> Digging into the logs I noticed that this particular submodule was
> throwing an OutOfMemoryError while building, but Jenkins (or Maven?) was
> labeling it as successful:
>
> Exception in thread "main" java.lang.OutOfMemoryError: unable to create
> new native thread
>
> at java.lang.Thread.start0(Native Method)
>
> at java.lang.Thread.start(Thread.java:574)
>
> at java.lang.Shutdown.runHooks(Shutdown.java:128)
>
> at java.lang.Shutdown.sequence(Shutdown.java:173)
>
> at java.lang.Shutdown.exit(Shutdown.java:218)
>
> at java.lang.Runtime.exit(Runtime.java:90)
>
> at java.lang.System.exit(System.java:869)
>
> at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:149)
>
> at org.jvnet.hudson.maven3.agent.Maven3Main.main(Maven3Main.java:63)
>
> Triggering a new build of foo-main-cobertura #345
>
> Finished: SUCCESS
>
> and was building downstream projects despite the fact that the build
> failed with an OOM error. This is �strange issue #2�.
>
> I�ve increased the heap as far as I can with the 32-bit JVM so I�m
> switching to a 64-bit JVM trying to get a good build for QA.
>
> Any ideas about these two issues:
>
> 1.Seemingly good/unstable builds switching to red (failed) after Jenkins
> restart
>
> 2.OOM error during build interpreted as successful
>
> Thanks,
>
> -tim
>
> *Tim Drury*
>
> Architect
>
> SAP Manufacturing Execution (SAP ME)
>
> Supply Chain Management
>
> *SAP Labs, LLC*
>
> T +1 404 943 2088
>
> F +1 610 492 2257
>
> mailto:t.d...@sap.com
>
> www.sap.com
>
> *Please consider the impact on the environment before printing this e-mail.*
>