[JIRA] [test-stability-plugin] (JENKINS-31660) StackOverflowError when maximum number of builds archived

9 views
Skip to first unread message

dave.hunt@gmail.com (JIRA)

unread,
Nov 19, 2015, 1:02:01 PM11/19/15
to jenkinsc...@googlegroups.com
Dave Hunt created an issue
 
Jenkins / Bug JENKINS-31660
StackOverflowError when maximum number of builds archived
Issue Type: Bug Bug
Assignee: kutzi
Components: test-stability-plugin
Created: 19/Nov/15 6:01 PM
Priority: Minor Minor
Reporter: Dave Hunt

We've been seeing a StackOverflowError with Test stability enabled and Discard Old Builds:

FATAL: null
java.lang.StackOverflowError
	at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:197)
	at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:143)
	at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:62)
	at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:247)
	at hudson.tasks.test.TestResult.getPreviousResult(TestResult.java:142)
	at hudson.tasks.junit.SuiteResult.getPreviousResult(SuiteResult.java:283)
	at hudson.tasks.junit.CaseResult.getPreviousResult(CaseResult.java:446)
	at hudson.tasks.junit.CaseResult.freeze(CaseResult.java:575)
	at hudson.tasks.junit.SuiteResult.freeze(SuiteResult.java:325)
	at hudson.tasks.junit.TestResult.freeze(TestResult.java:627)
	at hudson.tasks.junit.TestResultAction.load(TestResultAction.java:200)
	at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:143)
        ... repeated ...

Disabling the test stability resolves our issue.
Add Comment Add Comment
 
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
Atlassian logo

kutzi@gmx.de (JIRA)

unread,
Nov 19, 2015, 2:47:02 PM11/19/15
to jenkinsc...@googlegroups.com
kutzi commented on Bug JENKINS-31660
 
Re: StackOverflowError when maximum number of builds archived

Thanks for reporting this.
Do you have the bottom of the stack trace or the full stack trace? I cannot see in the include snippet where the test-stability-plugin itself is involved?

Also: what do you mean with "maximum number of builds archived"?

dave.hunt@gmail.com (JIRA)

unread,
Nov 20, 2015, 3:53:01 AM11/20/15
to jenkinsc...@googlegroups.com

The trace ends the way it starts, as it appears to be in a loop:

        ...
        at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:143)
	at hudson.tasks.junit.TestResultAction.getResult(TestResultAction.java:62)
	at hudson.tasks.test.AbstractTestResultAction.findCorrespondingResult(AbstractTestResultAction.java:247)
Finished: FAILURE

The plugin doesn't appear in the trace at all, but disabling it causes the problem to go away. This particular job is configured to discard old builds after 28 days. I can't be certain that the issue is related to the discarding of old builds or perhaps just an excessive number of builds. In fact, after checking, this job builds every 10 minutes, so it would start discarding after 4,032 builds. We're currently at 3,959 builds so although we're close, builds likely haven't started to be discarded just yet.

kutzi@gmx.de (JIRA)

unread,
Nov 21, 2015, 7:32:01 AM11/21/15
to jenkinsc...@googlegroups.com
kutzi commented on Bug JENKINS-31660

Strange, I don't understand how the plugin can be involved in this, if it's not in the stack trace at all.

BTW: which versions do you have in use (Jenkins and especially junit-plugin).
I wonder if it may be related to this:
https://github.com/jenkinsci/junit-plugin/commit/e8cf1377d1d42b3e1be2550ebf5d6a7bccef4634

kutzi@gmx.de (JIRA)

unread,
Nov 21, 2015, 7:33:01 AM11/21/15
to jenkinsc...@googlegroups.com
kutzi commented on Bug JENKINS-31660

BTW: please still attach the full stack trace!

dave.hunt@gmail.com (JIRA)

unread,
Nov 23, 2015, 5:39:03 AM11/23/15
to jenkinsc...@googlegroups.com
Dave Hunt updated an issue
 

Full console text attached.
Change By: Dave Hunt
Attachment: consoleText

dave.hunt@gmail.com (JIRA)

unread,
Nov 23, 2015, 5:42:02 AM11/23/15
to jenkinsc...@googlegroups.com

which versions do you have in use (Jenkins and especially junit-plugin)

Jenkins: 1.625.2
JUnit Plugin: 1.9
Test Stability History: 1.0

kutzi@gmx.de (JIRA)

unread,
Nov 23, 2015, 1:47:04 PM11/23/15
to jenkinsc...@googlegroups.com
kutzi updated an issue
 
Change By: kutzi
Component/s: junit-plugin

kutzi@gmx.de (JIRA)

unread,
Nov 23, 2015, 1:50:02 PM11/23/15
to jenkinsc...@googlegroups.com
kutzi updated an issue

kutzi@gmx.de (JIRA)

unread,
Nov 23, 2015, 2:03:02 PM11/23/15
to jenkinsc...@googlegroups.com
kutzi commented on Bug JENKINS-31660
 
Re: StackOverflowError when maximum number of builds archived

I wonder what is happening when you view the test results on this job even with test-stability disabled.

This looks like a bug in the junit-plugin itself, which is probably only triggered by test-stability, because it is 'looking' at the test results, and the error would also happen when you look at the results manually.

dbeck@cloudbees.com (JIRA)

unread,
Nov 23, 2015, 10:33:03 PM11/23/15
to jenkinsc...@googlegroups.com

Workaround would be to increase stack size using the -Xss JVM argument.

dave.hunt@gmail.com (JIRA)

unread,
Nov 24, 2015, 4:59:01 AM11/24/15
to jenkinsc...@googlegroups.com

I wonder what is happening when you view the test results on this job even with test-stability disabled.

If I recall correctly, we had a very similar stack trace attempting to view the test results. After disabling test stability on the job though, this has recovered.

Workaround would be to increase stack size using the -Xss JVM argument.

We're disabled the plugin for affected jobs, but if it starts happening in other jobs I will experiment with this. Is there a way to get an indication of what I should set this to? Is there a way for the plugin to detect this scenario and make the -Xss suggestion to the user?

kutzi@gmx.de (JIRA)

unread,
Nov 24, 2015, 4:23:01 PM11/24/15
to jenkinsc...@googlegroups.com
kutzi commented on Bug JENKINS-31660

Is there a way for the plugin to detect this scenario and make the -Xss suggestion to the user?

I don't think that is a feasible approach. Changing the -Xss size should be only a last resort when all other things fail and - if there is really an indefinite loop and not only the large number of jobs is the problem - it wouldn't even solve anything at all.

It would be great if you could make the jenkins builds folder of the job available, if it's possible. E.g. if it doesn't contain any sensitive data.
So we could try to reproduce this problem.

will-web-jenkinsci@harris.ch (JIRA)

unread,
Dec 1, 2015, 5:18:02 AM12/1/15
to jenkinsc...@googlegroups.com
Will Harris updated an issue
 

I was about to file a similar bug, but not related to the Test Stability plugin. I had a very similar situation, using Jenkins 1.639 and JUnit plugin 1.9, where on one of my jobs with ~2450 builds Jenkins suddenly started throwing a SOE when trying to read in my (python generated) junit.xml. Unfortunately I can't remember the exact circumstances, if anything had recently been upgraded etc.

I also had the problem when trying to display the Test Results for prior builds, even those where I know I had previously been able to see the test results.

As part of the debugging process for the JIRA ticket, I limited the number of old builds to 90 days, which brought my total builds for the job to ~780. I then went back to the very first build remaining, and was able to see the test results. I was still unable to see the test results for the last build that had recorded them however, so I started a manual binary search through my build history to see if I could pinpoint a particular build from which I could no longer see the results. At some point however, I was no longer seeing the problem.

Considering that the code in the stack trace refers to getPreviousResult I suspect somehow the history of JUnit results was corrupted, and by manually going through the results I somehow put things back in order.

I've attached an earlier stack trace for reference. Hope this experience report helps in some way.
Change By: Will Harris
Attachment: jenkins_stack_trace.txt

vitorg@yandex.ru (JIRA)

unread,
Feb 25, 2016, 7:44:01 AM2/25/16
to jenkinsc...@googlegroups.com
Vitaliy Pomazyonkov updated an issue

I don't even have Test stability plugin installed and still getting this error. Console log jenkins.log is attached.
Jenkins ver. 1.648
Change By: Vitaliy Pomazyonkov
Attachment: jenkins.log

vitorg@yandex.ru (JIRA)

unread,
Feb 29, 2016, 7:46:03 AM2/29/16
to jenkinsc...@googlegroups.com
Vitaliy Pomazyonkov updated an issue

Here is more informative exception log: jenkins-full-exception.log.
Change By: Vitaliy Pomazyonkov
Attachment: jenkins-full-exception.log

dave.hunt@gmail.com (JIRA)

unread,
Jul 21, 2016, 9:03:02 AM7/21/16
to jenkinsc...@googlegroups.com
Dave Hunt updated an issue
Change By: Dave Hunt
Attachment: console.log
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
Atlassian logo

dave.hunt@gmail.com (JIRA)

unread,
Jul 21, 2016, 9:07:01 AM7/21/16
to jenkinsc...@googlegroups.com
Dave Hunt commented on Bug JENKINS-31660
 
Re: StackOverflowError when maximum number of builds archived

Just seen this again after an upgrade to Jenkins 2.7.1. The first build failure after the upgrade was reported as expected, however the next build passed but hit this stack overflow. Disabling Test Stability History in the configuration allowed the build to pass without this exception.

I've attached the full console log including the exception: console.log

zbynek@geogebra.org (JIRA)

unread,
Apr 13, 2019, 4:21:03 AM4/13/19
to jenkinsc...@googlegroups.com

This happens often when there are skipped tests because of a bug in the Junit plugin, see https://github.com/jenkinsci/junit-plugin/pull/117
It's independent on Build Stability plugin.
This message was sent by Atlassian Jira (v7.11.2#711002-sha1:fdc329d)

fillermark@tutanota.com (JIRA)

unread,
Dec 8, 2019, 11:31:03 PM12/8/19
to jenkinsc...@googlegroups.com

A StackOverflowError is simply signals that there is no more memory available. It is to the stack what an OutOfMemoryError is to the heap: it simply signals that there is no more memory available. JVM has a given memory allocated for each stack of each thread, and if an attempt to call a method happens to fill this memory, JVM throws an error. Just like it would do if you were trying to write at index N of an array of length N. No memory corruption can happen. The stack can not write into the heap.

The common cause for a stackoverflow is a bad recursive call. Typically, this is caused when your recursive functions doesn't have the correct termination condition, so it ends up calling itself forever. Or when the termination condition is fine, it can be caused by requiring too many recursive calls before fulfilling it.

Here's an example:

public class Overflow {
public static final void main(String[] args)

{ main(args); }

}
That function calls itself repeatedly with no termination condition. Consequently, the stack fills up because each call has to push a return address on the stack, but the return addresses are never popped off the stack because the function never returns, it just keeps calling itself.

 
This message was sent by Atlassian Jira (v7.13.6#713006-sha1:cc4451f)
Atlassian logo
Reply all
Reply to author
Forward
0 new messages