Performance issue running code in TestNG?

[Rich Taylor]

We have run into a severe performance issue running some of our code in TestNG - happens in JUnit also.

We have a library that can convert MS Word documents to PDF.  With some documents there is a severe
slowdown when the code is run via TestNG or JUnit.  Timings look like this:

     Call library directly                          5-6 seconds

     Call library from TestNG         220-280 seconds

     Call library from JUnit                  ~300 seconds

I have tried with different JRE versions (1.7, 1.7), architectures (32-bit, 64-bit) and memory sizes

(default -> -Xmx3072m).  No differences.

This doesn't happen with all of our code - just with this library and some specific files. (We've been happily
using TestNG for regression testing for a while now.)  But, the slowdown is severe and is the difference
between 2 minutes or 2 hours for testing a set of files.

Based on the symptoms our code has something to do with the problem.  I've spent a few days chasing
that without success (grrr).

Can anyone illuminate how TestNG may be interacting with our code to cause this performance issue?

Thanks for any help!

Rich Taylor

Sr. Developer

Qoppa Software (www.qoppa.com)

[Cédric Beust ♔]

Hi Rich,

Well, since the problem happens with both TestNG and JUnit, the problem seems to be with your code somewhere.

Can you run the tests with a profiler and see if anything comes up?

Such a difference in time should be easy to track down.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users+unsubscribe@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at https://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Rich Taylor]

Well...  ultimately I expect to identify some part of our code that is a factor for the problem.

However, since it _only_ happens in the context of a unit testing framework...  I expect to find
some of the problem there too...

We're just running our usual code - where we aren't calling into TestNG or JUnit.

Yet, simply the context of running as a TestNG test has slowed down our code
by an incredible factor of 60x (or so).

Can you shine light on what is different in the environment when running 

code in a TestNG test?

-= Rich

-- 

Cédric

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.

[Cédric Beust ♔]

You are definitely right that the fact this is only happening when run from a test environment is mystifying, and why I'm really curious to get to the bottom of this. I think the answer will make us all chuckle.

As far as TestNG is concerned, test methods are invoked through simple reflection so it's hard for me to imagine that a piece of code that's invoked from a `main()` would run more slowly when invoked by TestNG through a `@Test`.

Can you do this profiler run? It would help a lot.

If not, you could try to do some binary searching and progressively disable test methods until you find the one causing the problem?

-- 

Cédric

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users+unsubscribe@googlegroups.com.

[Rich Taylor]

Do you have a preferred/recommended profiler?

Fortunately I haven't needed to become intimate with such a tool recently...

-= Rich

[Rich Taylor]

I'm now working with VisualVM for profiling.

Most of the delay is happening in SocketInputStream.read(byte b[], int off, int length, int timeout)
which is at the bottom (top?) of a stack started from BaseMessageSender$ReaderThread.run()

I am not net debugging w/TestNG source, so I can't say much.  But, looks like network
communication being used for something is getting slowed down radically.

Does that tell us anything?  (while I continue digging...)

-= Rich

[Cédric Beust ♔]

Sounds like you're running TestNG from your IDE if these classes are involved.

Can you try running your tests with TestNG from the command line?

-- 

Cédric

To unsubscribe from this group and stop receiving emails from it, send an email to testng-users+unsubscribe@googlegroups.com.

[Rich Taylor]

I ran from command line & (quickly) attached the VisualVM profiler.

Same results - almost all time is in SocketInputStream.socketRead() in a thread named "RMI TCP Connection"

In Eclipse I set a breakpoint in socketRead() and I see it getting called from 

REF_WordImages.REF_wordToImage_embed [TestNG]

org.testng.remote.RemoteTestNG at localhost:63360

Thread [main] (Running)

Thread [ReaderThread] (Suspended (breakpoint at line 116 in SocketInputStream))

owns: InputStreamReader  (id=18)

SocketInputStream.socketRead(FileDescriptor, byte[], int, int, int) line: 116

SocketInputStream.read(byte[], int, int, int) line: 170

SocketInputStream.read(byte[], int, int) line: 141

StreamDecoder.readBytes() line: 284

StreamDecoder.implRead(char[], int, int) line: 326

StreamDecoder.read(char[], int, int) line: 178

InputStreamReader.read(char[], int, int) line: 184

BufferedReader.fill() line: 161

BufferedReader.readLine(boolean) line: 324

BufferedReader.readLine() line: 389

BaseMessageSender$ReaderThread.run() line: 232

C:\Program Files\Java\jdk1.8.0_92\bin\javaw.exe (Jun 23, 2017, 4:31:01 PM)

When running command line & with profiling I see massive delays - so, not related to Eclipse I assume.

I'm not sure where to look next.

-= Rich Taylor