[jvm] StackOverflowError when running > 50,000 steps

[mila.rodrig...@gmail.com]

I have a set of feature files that include a good mix of scenarios and scenario outlines. In all there are about 50,000 steps. I can run these features individually in IntelliJ and they all pass successfully. However when I try to run them from the command line, or on our Jenkins build server, it throws a StackOverflowError. 

The error happens after all tests finish, when it's trying to generate the report of how many scenarios and steps were run. On smaller projects, it prints out something that looks like this:

576 Scenarios (576 passed)

4417 Steps (4417 passed)

0m1.445s

In the current project, it fails with a StackOverflowError:

Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "main"

There is no stack trace attached to the StackOverflowError. It just hangs there. On my Jenkins server I ended up having to restart both the master and the slave to get the build to terminate.

I'm using cucumber-jvm version 1.2.2, cucumber-junit version 1.2.2, and Java 1.7. The VM is 64-bit and has the default 1024kb stack. The tests themselves take about 6-8 minutes to execute, and they all pass.

Is this a problem with the cucumber-junit code, or the JUnit framework itself? Short of increasing the stack size, is there another way I can refactor my tests or adjust my test setup to not cause this issue? (I don't want to rely on increasing the stack size because I'm only at 5% coverage of my business logic right now, so there's a lot more tests to be done.)

[Aslak Hellesøy]

On Friday, 8 May 2015 at 18:40, mila.rodrig...@gmail.com wrote:

I have a set of feature files that include a good mix of scenarios and scenario outlines. In all there are about 50,000 steps. I can run these features individually in IntelliJ and they all pass successfully. However when I try to run them from the command line, or on our Jenkins build server, it throws a StackOverflowError. 

The error happens after all tests finish, when it's trying to generate the report of how many scenarios and steps were run. On smaller projects, it prints out something that looks like this:

576 Scenarios (576 passed)

4417 Steps (4417 passed)

0m1.445s

In the current project, it fails with a StackOverflowError:

Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "main"

There is no stack trace attached to the StackOverflowError. It just hangs there. On my Jenkins server I ended up having to restart both the master and the slave to get the build to terminate.

I'm using cucumber-jvm version 1.2.2, cucumber-junit version 1.2.2, and Java 1.7. The VM is 64-bit and has the default 1024kb stack. The tests themselves take about 6-8 minutes to execute, and they all pass.

Is this a problem with the cucumber-junit code, or the JUnit framework itself?

I have never seen this error and it's impossible to tell what causes it without access to code that would let us reproduce it.

I understand you might not be able to share code, so my best advice is to tweak your vm or hack on the cucumber-jvm code to give you some more information. I can't give you any pointers since I have no information where this is happening. Your own code might just as well be the culprit here.

I think that's more likely than this being a bug in cucumber. StackOverflow usually means infinite recursion, and if that's happening in Cucumber I think we'd already have other people reporting that.

My guess is that one of your scenarios will trigger this in isolation, just figure out which one.

Aslak

Short of increasing the stack size, is there another way I can refactor my tests or adjust my test setup to not cause this issue? (I don't want to rely on increasing the stack size because I'm only at 5% coverage of my business logic right now, so there's a lot more tests to be done.)

--
Posting rules: http://cukes.info/posting-rules.html
---
You received this message because you are subscribed to the Google Groups "Cukes" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cukes+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[mila.rodrig...@gmail.com]

On Friday, May 8, 2015 at 2:45:45 PM UTC-6, Aslak Hellesøy wrote:

On Friday, 8 May 2015 at 18:40, mila.rodrig...@gmail.com wrote:

I have a set of feature files that include a good mix of scenarios and scenario outlines. In all there are about 50,000 steps. I can run these features individually in IntelliJ and they all pass successfully. However when I try to run them from the command line, or on our Jenkins build server, it throws a StackOverflowError. 

The error happens after all tests finish, when it's trying to generate the report of how many scenarios and steps were run. On smaller projects, it prints out something that looks like this:

576 Scenarios (576 passed)

4417 Steps (4417 passed)

0m1.445s

In the current project, it fails with a StackOverflowError:

Exception: java.lang.StackOverflowError thrown from the UncaughtExceptionHandler in thread "main"

There is no stack trace attached to the StackOverflowError. It just hangs there. On my Jenkins server I ended up having to restart both the master and the slave to get the build to terminate.

I'm using cucumber-jvm version 1.2.2, cucumber-junit version 1.2.2, and Java 1.7. The VM is 64-bit and has the default 1024kb stack. The tests themselves take about 6-8 minutes to execute, and they all pass.

Is this a problem with the cucumber-junit code, or the JUnit framework itself?

I have never seen this error and it's impossible to tell what causes it without access to code that would let us reproduce it.

I understand you might not be able to share code, so my best advice is to tweak your vm or hack on the cucumber-jvm code to give you some more information. I can't give you any pointers since I have no information where this is happening. Your own code might just as well be the culprit here.

I think that's more likely than this being a bug in cucumber. StackOverflow usually means infinite recursion, and if that's happening in Cucumber I think we'd already have other people reporting that.

My guess is that one of your scenarios will trigger this in isolation, just figure out which one.

Aslak

Thank you for your reply.

However I'm quite convinced that the problem lies in either cucumber-junit or junit itself rather than cucumber-jvm or my scenario implementations. I have observed the following:

• All features and scenarios terminate as expected and pass when run independently. (By this, I mean run individually instead of as part of the suite as a whole.)
  
• When I replaced JUnit with TestNG in my project (and swapped cucumber-testng for cucumber-junit), the StackOverflowError stopped occurring and the build terminated properly, all features and scenarios passing, after 8 minutes and 55 seconds runtime.

I'm slightly concerned that this may be a side-effect of Cucumber identifying each step as a JUnit test (rather than each scenario); maybe JUnit itself has a critical mass regarding how many tests can be run in a single suite? However I'm not entirely sure how to go about testing this theory, and my Google searches have not yielded any results about some kind of potential test count ceiling in JUnit. (Of course, 55,000 is something of a ridiculous number to begin with.)

I've begun preparing a project with a similar number of steps that reproduces the issue; I will post a link when it's complete.

Mila