Profiling TestNG Tests in Eclipse

[Ivan A Krizsan]

Just wanted to share some experiences in regard to profiling TestNG tests:

Eclipse has a set of profiling tools named TPTP, please refer to this excellent introduction: http://www.eclipse.org/articles/Article-TPTP-Profiling-Tool/tptpProfilingArticle.html

Having installed TPTP, the only thing I needed to add was a small program to programatically start my test-suite, see: http://forums.opensymphony.com/thread.jspa?threadID=21238&tstart=0

With all things in place, I can now profile my starter-application using Profile As, instead of Run As, when right-clicking on the file.

Note that it is a very good idea to set up the Profiling Set (see TPTP article) to include only the packages in which the code you want to profile is located and exclude everything else. If this is not done, you will have the results from profiling TestNG mixed with the results from profiling your own code.

Despite excluding all but your own packages, profiling will take much longer than executing the tests in a normal way. There might be a remedy for this that I am unaware of.
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=21814&messageID=42065#42065

[Alexandru Popescu]

#: Ivan A Krizsan changed the world a bit at a time by saying (astral date: 3/10/2006 3:12 AM) :#

I have used from time to time YourKit for profiling. It was an excellent experience. I haven't
noticed big changes in the run time.
Maybe the price is prohibitive for personals (at least this is the case for myself), but I think it
is one of the best tools in this field.

BR,

./alex
--
.w( the_mindstorm )p.

[Ivan A Krizsan]

I have had further opportunity to use TestNG and Eclipse TPTP performance analysis tools together and the combination is only getting better... :-)

One thought struck me, however, when analyzing memory usage of an application:
It seems like TestNG retains references to the test classes even after having finished running a test suite(?). Perhaps this is nothing but a lack of explicitly setting the references to the tests to null?

I am using J2SE 1.4.2_08 and TestNG 5.0.0.

If this is not a local phenomena or a problem with the 1.4 SDK, would it be possible to explicitly set the references to test classes to null, once being done with them?
I am going to experiment a little with the TestRunner::afterTest (?) method, but this might be of interest to others.
I feel this would aid when using TestNG and analysing memory usage of your application.

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=21814&messageID=75234#75234

[Alexandru Popescu]

Hi Ivan!

Thanks for investigation. I would like to better understand what
exactly are you refering to when saying "retains references to the
test classes". Is it: TestNG still has references to the test
instances, or is it TestNG has references to the test classes. If the
later, than I think there is nothing we can do (moreover the classes
will be unloaded from the JVM only if the classloader used to load
them will be dismissed). But in case you are refering to test
instances, than I think we should try to remove those references, so
that the GC will be able to release them.

./alex
--
.w( the_mindstorm )p.

[Ivan A Krizsan]

Hello!
Ah, sorry, I expressed myself unclearly, or even wrongly.
Of course it is references to instances of the test classes that I was speaking about.

I run an entire suite of tests in a loop observing memory usage using TPTP, a total of 45 test classes, 85 tests.
The first and second time I run the test suite in the loop, no instances of the test classes are garbage collected.
The third time, one instance of each test class is garbage collected, with two live instances remaining etc.
So, it seems like there is at most 2 live instances of each test class after an execution of the suite.
If it would be possible to remove the references after having executed the test or suite, that would indeed be nice.

Regards,
Ivan

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=21814&messageID=75488#75488

[Alexandru Popescu]

On 7/25/06, Ivan A Krizsan <testng...@opensymphony.com> wrote:
>
> Hello!
> Ah, sorry, I expressed myself unclearly, or even wrongly.
> Of course it is references to instances of the test classes that I was speaking about.
>
> I run an entire suite of tests in a loop observing memory usage using TPTP, a total of 45 test classes, 85 tests.
> The first and second time I run the test suite in the loop, no instances of the test classes are garbage collected.
> The third time, one instance of each test class is garbage collected, with two live instances remaining etc.
> So, it seems like there is at most 2 live instances of each test class after an execution of the suite.
> If it would be possible to remove the references after having executed the test or suite, that would indeed be nice.
>

Thanks for clarifications Ivan. I will definitely look into fixing
this. I will keep you posted with any advance. Also, I would much
appreciate if you have any hints for now to where those instances are
living (at least to allow me to start digging).

TIA,

./alex
--
.w( the_mindstorm )p.

> Regards,

[Ivan A Krizsan]

Hello!

I guess the easiest way to find a starting point for digging is to create a test class with one test method and one default constructor. Then set a breakpoint in the constructor and debug. At least this is the way I used when having a quick peek...
This way I guess you can find where the test class instances are created and their references stored (they seem to be contained in an instance of ITestClass, if I am not mistaken).
Then one would need to clear these references at some place. A wild guess would be the TestRunner::afterRun method...

Sorry about not being more specific, this is all I got at the moment. :-P

Regards,
Ivan
---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=21814&messageID=75543#75543

[BenLau]

Be forwarned: Eclipse TPTP is very slow on Unix. I think the underlying
ProbeKit engine is from Rational.

[Ivan A Krizsan]

Hi!
I am yet to try TPTP on UNIX, but here are some advice regarding profiling and speed:

Try use the package filtering, it substantially speeds things up. You definitely want to avoid profiling the TestNG code.
Also, profiling for a limited time is a useful option, for instance having profiling switched off when launching the application in question and, later when reaching areas/functions of interest, switching on profiling.
Avoid collecting more than absolute minimum of data, for instance do not collect instance level information unless necessary.

I suspect that TPTP speed may also be related to the JVM you are using and its features.

Happy profiling!

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=21814&messageID=75782#75782

[Alexandru Popescu]

I've been using a couple of time YourKit and I've been happy with it.
I guess my next try will go the same direction ;-].

./alex
--
.w( the_mindstorm )p.

[Alexandru Popescu]

Oke, after a little digging I might have an idea what is happening.
So, here is the short story:

- TestNG is using the same instance for invoking the @Test methods
- TestNG is using the same instances for invoking any @Before/@After methods
- considering the lifecycle of a suite, test, group these instances
are kept alive so that we have the chance to run the required @After
(Group/Test/Suite) methods

Now, this is definitely not perfect :-], but trying to release them
after each end-lifecycle moment (@AfterClass if there are no other
@After; @AfterGroup if there are no other @AfterTest/@AfterSuite) is
quite a complex task.

Now, we may have a solution: when initially analysing the test class,
decide which is the best moment to release the instance and use a
"synthetic" @AfterX method to trigger this release.

However, I would like to hear about scenarios where releasing the test
instance would bring
more benefits than just releasing the heavy resources in that instance
using an @AfterX method.

HTH,

./alex
--
.w( the_mindstorm )p.

PS: YourKit helped me understand this ;-].

[Ivan A Krizsan]

YourKit is not free, TPTP is. :-)
TPTP is also more than just profiling tools, it is an open source platform for building test and performance monitoring tools on.

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=21814&messageID=75856#75856

[Ivan A Krizsan]

Hi!
Thanks for your efforts!
It seems we have another case of memory management, Java style. :-P
Not much to do about it, I guess (this is not a TestNG problem, it is a Java problem).

> However, I would like to hear about scenarios where
> releasing the test
> instance would bring
> more benefits than just releasing the heavy resources
> in that instance
> using an @AfterX method.

Sorry, cannot come to think of any such scenarios at the moment.
Already modified my tests to "clean up" after themselves.

Regards,
Ivan

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=21814&messageID=75858#75858