[JVM] Cucumber json generation results in OutOfMemoryError: Java heap space

[Csaba Bejan]

Hello,

Recently as we were adding new tests to our suite we noticed that after the test execution finished we got an exception regarding Java heap space from GSON when it was trying to create the json report abut the test execution.

The obvious solution would be to change the jvm memory setting but this is only a temporary solution as the number of tests added to our suite is constantly growing. Also splittting the suite in smaller sets should solve the problem as it would result in smaller separate jsons but at the moment we don't have control over this aspect.

As I understand streaming should address this issue: https://sites.google.com/site/gson/streaming

My question is that whether anyone knows a workaround for this or will this be addressed in the future?

Thanks in advance,

Csaba

[Paolo Ambrosio]

On Wed, Oct 1, 2014 at 5:58 AM, Csaba Bejan <bejan...@gmail.com> wrote:

> Hello,
>
> Recently as we were adding new tests to our suite we noticed that after the
> test execution finished we got an exception regarding Java heap space from
> GSON when it was trying to create the json report abut the test execution.
> The obvious solution would be to change the jvm memory setting but this is
> only a temporary solution as the number of tests added to our suite is
> constantly growing. Also splittting the suite in smaller sets should solve
> the problem as it would result in smaller separate jsons but at the moment
> we don't have control over this aspect.
>
> As I understand streaming should address this issue:
> https://sites.google.com/site/gson/streaming

Did you profile the memory while running tests? If you didn't how can
you be sure that it's GSON the issue? It can be that the heap space is
used mainly by other classes and by chance GSON is the one that fills
it up completely. I don't see GSON used heavily inside Cucumber.

What were the memory settings used to run Cucumber? What formatter(s)
were you using? Did you start your application from inside the tests
or on a separate JVM?

> My question is that whether anyone knows a workaround for this or will this
> be addressed in the future?

Can you replicate the issue in the java-calculator example by copying
the same scenario from the example over and over to match the number
in your app's functional tests?


Paolo


> Thanks in advance,
> Csaba

[Csaba Bejan]

You are right I can't be sure that GSON is the issue, didn't do any profiling. What we know is that we can add hundreds of new scenarios and there is no issue. Everything works fine, however as soon as I'm adding back the json format option I'm getting this exception at the end of the test execution.

Answering your questions:

- Memory settings: -Xms512m -Xmx512m (when changed it to 1024 the problem was solved but this is only temporary given that new tests are added every day)

- I removed all the formatters, it made no difference, when json was added I got out of memory.

- We are executing the tests as part of our maven build and we are using the surefire plugin. That executes the unit tests and there is one empty test which we are using to execute all of our cucumber tests and we are configuring this through annotation.

- I can try to replicate the issue with the method you described and also will try to do the profile but unfortunately this week I won't have time to do it.

A little background, maybe it would be useful:

- We have one runner which executes all our cucumber tests so one huge json file is generated in the end

- The size of or json at the moment is ~58Mb (I'm not sure how much memory is used by GSON to create the json from the result object. By the way the json is 2.2 million lines long and we are adding new scenarios constantly

- We have ~1800 scenarios at the moment.

Thanks,

Csaba

[Paolo Ambrosio]

Hi Csaba,

Please read the group rules and reply inline.

On Wed, Oct 1, 2014 at 11:08 PM, Csaba Bejan <bejan...@gmail.com> wrote:

> You are right I can't be sure that GSON is the issue, didn't do any
> profiling. What we know is that we can add hundreds of new scenarios and
> there is no issue. Everything works fine, however as soon as I'm adding back
> the json format option I'm getting this exception at the end of the test
> execution.

From this email, you are probably right and technically you would
benefit from a JSON formatter using a streaming API, as you are
saying.

> Answering your questions:
> - Memory settings: -Xms512m -Xmx512m (when changed it to 1024 the problem
> was solved but this is only temporary given that new tests are added every
> day)

Current workstations come with at least 8GB and servers much more. I
believe there isn't a real need to improve the memory efficiency of
the formatter (see also my comments at the end).

You also need to consider if whatever software is going to parse that
massive JSON is using a streaming API, or you'll have the same issue
there.

> - I removed all the formatters, it made no difference, when json was added I
> got out of memory.
> - We are executing the tests as part of our maven build and we are using the
> surefire plugin. That executes the unit tests and there is one empty test
> which we are using to execute all of our cucumber tests and we are
> configuring this through annotation.
> - I can try to replicate the issue with the method you described and also
> will try to do the profile but unfortunately this week I won't have time to
> do it.
>
> A little background, maybe it would be useful:
> - We have one runner which executes all our cucumber tests so one huge json
> file is generated in the end
> - The size of or json at the moment is ~58Mb (I'm not sure how much memory
> is used by GSON to create the json from the result object. By the way the
> json is 2.2 million lines long and we are adding new scenarios constantly
> - We have ~1800 scenarios at the moment.

From a separate point of view, a monolithic application with 1800
scenarios is massive! Are you sure you still need all those scenarios?
Are they technical-facing or business-facing? Have you got a testing
pyramid or a testing ice-cream cone? Matt keeps saying that teams
should be brave and delete scenarios that are not providing much
value. You might find more information on this topic in this
presentation:

https://skillsmatter.com/skillscasts/4984-post-bdd-team


Paolo

[Csaba Bejan]

Hi Paolo,

Thanks for the feedback. We have to have a discussion internally regarding how we should proceed.

2014. október 2., csütörtök 10:19:59 UTC-5 időpontban Paolo Ambrosio a következőt írta:

Hi Csaba,

Please read the group rules and reply inline.

On Wed, Oct 1, 2014 at 11:08 PM, Csaba Bejan <bejan...@gmail.com> wrote:

> You are right I can't be sure that GSON is the issue, didn't do any
> profiling. What we know is that we can add hundreds of new scenarios and
> there is no issue. Everything works fine, however as soon as I'm adding back
> the json format option I'm getting this exception at the end of the test
> execution.

From this email, you are probably right and technically you would
benefit from a JSON formatter using a streaming API, as you are
saying. 

> Answering your questions:
> - Memory settings: -Xms512m -Xmx512m (when changed it to 1024 the problem
> was solved but this is only temporary given that new tests are added every
> day)

Current workstations come with at least 8GB and servers much more. I
believe there isn't a real need to improve the memory efficiency of
the formatter (see also my comments at the end).

You also need to consider if whatever software is going to parse that
massive JSON is using a streaming API, or you'll have the same issue
there.

 

We are using maven-cucumber-reporting which is slowing down heavily as we are adding new tests so we know that we have to make changes in our approach.

 

> - I removed all the formatters, it made no difference, when json was added I
> got out of memory.
> - We are executing the tests as part of our maven build and we are using the
> surefire plugin. That executes the unit tests and there is one empty test
> which we are using to execute all of our cucumber tests and we are
> configuring this through annotation.
> - I can try to replicate the issue with the method you described and also
> will try to do the profile but unfortunately this week I won't have time to
> do it.
>
> A little background, maybe it would be useful:
> - We have one runner which executes all our cucumber tests so one huge json
> file is generated in the end
> - The size of or json at the moment is ~58Mb (I'm not sure how much memory
> is used by GSON to create the json from the result object. By the way the
> json is 2.2 million lines long and we are adding new scenarios constantly
> - We have ~1800 scenarios at the moment.

From a separate point of view, a monolithic application with 1800
scenarios is massive! Are you sure you still need all those scenarios?

Actually we expect this number to be multiplied in a few months. All of the existing tests are told to be valid and only business has control over what should be executed. In short no test can be removed, new tests will be added in large numbers and it's really important to execute all tests at every build (and have meaningful reports of each execution).

 

Are they technical-facing or business-facing?

They are both.

Have you got a testing
pyramid or a testing ice-cream cone?

Our framework is responsible for the acceptance tests and every test is covering some kind of requirement. All of them are equally important we can't have a subset of them as "smoke" or "regression". I can't go into more details, I know it's strange but this is how it is. We have our unit test suite which is a separate thing but we are responsible only for the acceptance tests and all acceptance tests have to be automated and executed at every build.

This is where we are, we are working on separating our suite to different runners at least based on features but that is causing problems from business point of view and we need nicely aggregated reports in the end which is going back to the huge json problem again, we were just looking for alternatives in the mean time. 

Thanks again at least we are clear on what to expect and what to consider.

 

Matt keeps saying that teams
should be brave and delete scenarios that are not providing much
value. You might find more information on this topic in this
presentation:

https://skillsmatter.com/skillscasts/4984-post-bdd-team 

I will definitely check it out but as I said we don't have control over the number and the "attributes" of the tests executed by the framework.

Csaba