OutOfMemoryError during stress tests with TestNG

[crapp]

Hello,

we are using TestNG to do stress tests of a server component. So basically TestNG invokes a SOAP client over and over again.
We want to run one testcase a huge number of times (e.g. 1000 times). With the standard JVM parameters we ran into a OutOfMemoryError after 300 runs because the HeapSize was too low. So we increased that to 1024MByte (inital and max). So now we are able to run about 500 tests without getting this error.

Do you have any suggestion to reduce the memory consumption? Why doesn't the memory get released after each testrun? Does this have anything to do with the generation of the test report?
The SOAP message the client sends is fairly small; app. 2KByte.

Any help on this is appreciated.

Regards,
Christian
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=151996&messageID=224802#224802

[Cédric Beust ♔]

Hi Christian,

I'm having a hard time believing that 1000 tests would consume so much memory and I'm confused by your numbers since you say that increasing the heap allowed you to switch from 1000 to 500 tests (half).

There is obviously a memory problem but the only way to track these is to use a profiler and see exactly which objects are being retained.  Can you run such a tool and report the results you get?

Another side question:  are you using a @Factory?

--
Cédric

[Steve Loughran]

On 02/11/2007, Cédric Beust ♔ <cbe...@google.com> wrote:
> Hi Christian,
>
> I'm having a hard time believing that 1000 tests would consume so much
> memory and I'm confused by your numbers since you say that increasing the
> heap allowed you to switch from 1000 to 500 tests (half).
>
> There is obviously a memory problem but the only way to track these is to
> use a profiler and see exactly which objects are being retained. Can you
> run such a tool and report the results you get?

1. reach for "troubleshooting guide for Java SE 6 with Hotspot VM"
for OOM problems:

http://java.sun.com/javase/6/webnotes/trouble/TSG-VM/TSG-VM.pdf

lots of tips on getting memory dumps from running code

2. anything that plays games with classloaders can leak memory like a
broken sieve. Which SOAP stack are you using?

[crapp]

Thanks for your answers and the link.

Here me answers in short:

- We weren't able to run 1000 test cases yet. The maximum runs we completed successfully were 500 after increasing the heap size to 1GByte.
- We are using Axis2 1.2 as SOAP stack.
- Yes, we are using Factory. And I also played around with not using a factory but still the same behavior.

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=151996&messageID=226785#226785

[Steve Loughran]

On 05/11/2007, crapp <testng...@opensymphony.com> wrote:
>
> Thanks for your answers and the link.
>
> Here me answers in short:
>
> - We weren't able to run 1000 test cases yet. The maximum runs we completed successfully were 500 after increasing the heap size to 1GByte.
> - We are using Axis2 1.2 as SOAP stack.

I would discuss this with the axis2 team. It is very symptomatic of
classloader leaks or other big leaks of memory. I know that axis2 does
its own classloader games, which is always something I've been unhappy
about.

Look at the java-6 hotspot docs, grab a heap dump and find out what is
consuming all the memory. Something is holding on to stuff that shoud
be released.

[Cédric Beust ♔]

On Nov 4, 2007 11:34 PM, crapp <testng...@opensymphony.com> wrote:

Thanks for your answers and the link.

Here me answers in short:

- We weren't able to run 1000 test cases yet. The maximum runs we completed successfully were 500 after increasing the heap size to 1GByte.
- We are using Axis2 1.2 as SOAP stack.
- Yes, we are using Factory. And I also played around with not using a factory but still the same behavior.

Have you tried returning an Iterator instead of an Object[] in your factory?  This might solve your memory problem.

--
Cédric

[crapp]

I have written my own classes to run tests several times. The good news: it doesn't seem to be TestNG to be the problem. The same OutOfMemoryError problem occurs after a little bit more than 300 runs.
The bad news is: I haven't figured out yet, who is causing the memory leak.
I am stilling profiling my application...

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=151996&messageID=226997#226997

[Cédric Beust ♔]

Good to know.  Still, I think that returning an Iterator should solve your *testing* problem (but of course, it won't fix the fact that some objects seem to be retained).

--
Cedric

--
Cédric

[crapp]

Hello again,

I did replace the return type of my factory by 'Iterator'. But now I get this, which seems to be clear to me but does not correlate to the suggestion above. What did I do wrong?

Exception in thread "main" java.lang.ClassCastException: java.util.AbstractList$Itr
at org.testng.internal.FactoryMethod.invoke(FactoryMethod.java:71)
at org.testng.internal.TestNGClassFinder.<init>(TestNGClassFinder.java:121)
at org.testng.TestRunner.initMethods(TestRunner.java:265)
at org.testng.TestRunner.init(TestRunner.java:211)
at org.testng.TestRunner.init(TestRunner.java:173)
at org.testng.TestRunner.<init>(TestRunner.java:130)
at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:444)
at org.testng.SuiteRunner.privateRun(SuiteRunner.java:234)
at org.testng.SuiteRunner.run(SuiteRunner.java:190)
at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:792)
at org.testng.TestNG.runSuitesLocally(TestNG.java:765)
at org.testng.TestNG.run(TestNG.java:699)
at org.testng.TestNG.privateMain(TestNG.java:824)
at org.testng.TestNG.main(TestNG.java:802)

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=151996&messageID=227665#227665

[Cédric Beust ♔]

Looks like a TestNG bug, can you reproduce it on a simple test class?

--
Cedric

On Nov 5, 2007 11:16 PM, crapp <testng...@opensymphony.com > wrote:

Hello again,

I did replace the return type of my factory by 'Iterator'. But now I get this, which seems to be clear to me but does not correlate to the suggestion above. What did I do wrong?

Exception in thread "main" java.lang.ClassCastException: java.util.AbstractList$Itr
       at org.testng.internal.FactoryMethod.invoke(FactoryMethod.java:71)

       at org.testng.internal.TestNGClassFinder .<init>(TestNGClassFinder.java:121)

       at org.testng.TestRunner.initMethods(TestRunner.java:265)
       at org.testng.TestRunner.init(TestRunner.java:211)
       at org.testng.TestRunner.init(TestRunner.java :173)
       at org.testng.TestRunner.<init>(TestRunner.java:130)
       at org.testng.SuiteRunner$DefaultTestRunnerFactory.newTestRunner(SuiteRunner.java:444)

       at org.testng.SuiteRunner.privateRun (SuiteRunner.java:234)

       at org.testng.SuiteRunner.run(SuiteRunner.java:190)
       at org.testng.TestNG.createAndRunSuiteRunners(TestNG.java:792)
       at org.testng.TestNG.runSuitesLocally(TestNG.java:765)
       at org.testng.TestNG.run(TestNG.java:699)
       at org.testng.TestNG.privateMain(TestNG.java:824)
       at org.testng.TestNG.main(TestNG.java:802)
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=151996&messageID=227665#227665

--
Cédric

[Mark Derricutt]

Any chance of adding support to TestNG for DataProviders returning an Iterable rather than Object[][] ?  I could see that being quite handy and you'd get each iteration reported back to the IDEs and reports as well..  Would only work with @Test methods that take a single parameter thou..

On 11/6/07, Cédric Beust ♔ <cbe...@google.com> wrote:

Have you tried returning an Iterator instead of an Object[] in your factory?  This might solve your memory problem.

--

It`s not the tree that forsakes the flower, but the flower that forsakes the tree
Someday I`ll learn to love these scars - Bye Bye Beautiful

[Cédric Beust ♔]

On Nov 5, 2007 11:33 PM, Mark Derricutt <ma...@talios.com> wrote:

Any chance of adding support to TestNG for DataProviders returning an Iterable rather than Object[][] ?  I could see that being quite handy and you'd get each iteration reported back to the IDEs and reports as well..  Would only work with @Test methods that take a single parameter thou..

I think that would be too limiting, but maybe Iterator<Object[]>:  the parameters for one method invocation are created eagerly, but the sets of parameters for each invocation would be created lazily...

However, I'm not completely convinced it's that useful yet:  these objects have better odds to be big when they are test classes than when they are parameters to test methods, in my opinion.  Still, it would be nicely consistent with @Factory.

--
Cédric

[crapp]

So I finally got the reason for the high memory consumption. It's not a memory leak. We used Axis2 ina wrong way.
No I am able to run thousands of testruns without a problem.

The problem was that we were instanciating a new stub for each call. But this is not necessary since a stub can be used to do various calls.

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=151996&messageID=228803#228803

[Steve Loughran]

On 07/11/2007, crapp <testng...@opensymphony.com> wrote:
>
> So I finally got the reason for the high memory consumption. It's not a memory leak. We used Axis2 ina wrong way.
> No I am able to run thousands of testruns without a problem.
>
> The problem was that we were instanciating a new stub for each call. But this is not necessary since a stub can be used to do various calls.

maybe, but it still shouldnt leak so badly. A GC should force a
cleanup unless something was keeping lots of references around.