controlling data-driven test method order, and forcing tests to skip

[Curtis Rueden]

Hi,

I am using TestNG to run some data-driven tests on my group's software, particularly our Bio-Formats file formats library (http://www.loci.wisc.edu/ome/formats.html). TestNG is really useful because of its data-driven testing capabilities. So first of all, thank you for producing this great testing framework.

However, there are some things I cannot figure out how to do.

First, the easy, less important question:

1) Is there a way for a test to flag itself as skipped (rather than passed or failed) if the test is inapplicable for a particular piece of input data? Right now I just make the test pass in such a situation. But there are places where it would be more accurate for the test to declare itself skipped -- maybe it could throw a specially-handled TestSkipException or something?

And the more involved question:

2) How can I effectively control the test method execution order at the per-data-object level?

To explain what I mean, allow me to give some background on how I am using TestNG.

My goal is to use TestNG to drive a test suite for the Bio-Formats file format readers. I have a large collection of image files, and I want to feed them to my tests one at a time as input data, calling each test method once for each input data object.

The first challenge is that there is a very large amount of image data -- about 140 GB and growing. Some of the readers need to do a significant amount of parsing on each file up front.

Our original approach was to have a data provider method that instantiates one format reader object per data file, and initializes them, but the amount of memory required to maintain the program state on so many files at once is prohibitive. I had to abandon use of TestNG's data provider functionality.

Fortunately, I discovered that TestNG's factory-driven test generation logic works for what I want to do, which is run all tests on a given data object, then move on to the next data object and repeat (dumping state from the previous data object, as it is no longer needed). However, I saw the note in the TestNG documentation that barring any dependencies, test order is not guaranteed. I fear that though TestNG is running the tests in my desired order, it is just an implementation detail.

In addition, I cannot use any hard dependencies with this scheme. Ideally, I would like my first test for each data object to be a "testInitialization" test, which opens the file and does the initial parsing. If that fails, there is no need to run the other tests. But as soon as I put a depends clause on any such method, TestNG rearranges my test order to run testInitialization on ALL files before calling any other tests on ANY of the files. While this behavior makes sense to me, I am wondering if there is a way to instruct TestNG to treat each data object as its own encapsulated test sub-suite, and process them in order.

Possible workarounds:
* Allow the tests can run out of order, and reinitialize each file with every test. This works, but makes everything run orders of magnitude slower, which makes testing the entire 140 GB repository extremely tedious (50+ hours of execution time versus <12 hours).
* For each file, bundle everything into one big test. But then we lose the modularity that TestNG normally provides.

If you want to see our code, you can find it here:
* https://skyking.microscopy.wisc.edu/trac/java/browser/trunk/loci/tests/testng/
* https://skyking.microscopy.wisc.edu/trac/java/browser/trunk/build.xml#L465

Relevant files are FormatReaderTest, which encapsulates the tests for a given file (String id), and FormatReaderTestFactory, which recursively generates the list of FormatReaderTests from a root folder containing image data. Oh, and DotTestListener is a modified version of TestNG's DotTestListener that fixes some bugs (output of "." instead of "F" and "S", off-by-one newlines).

We invoke via Ant without an explicit XML file (which is really convenient, btw):
ant -Dtestng.directory=/path/to/my/data test-all

Lastly, we are not making very good use of TestNG's reporting facilities yet. If you do look at the code, you'll see we actually have our own logging mechanism in addition to TestNG's output. I plan to improve this, but first I would like to get the testing process itself running more smoothly.

Sorry for the length, and thanks in advance for any thoughts.

-Curtis
---------------------------------------------------------------------
Posted via Jive Forums
http://forums.opensymphony.com/thread.jspa?threadID=237865&messageID=317715#317715

[Cédric Beust ♔]

Hi Curtis,

On Tue, Feb 26, 2008 at 10:44 AM, Curtis Rueden <testng...@opensymphony.com> wrote:

Hi,

I am using TestNG to run some data-driven tests on my group's software, particularly our Bio-Formats file formats library (http://www.loci.wisc.edu/ome/formats.html). TestNG is really useful because of its data-driven testing capabilities. So first of all, thank you for producing this great testing framework.

However, there are some things I cannot figure out how to do.

First, the easy, less important question:

1) Is there a way for a test to flag itself as skipped (rather than passed or failed) if the test is inapplicable for a particular piece of input data? Right now I just make the test pass in such a situation. But there are places where it would be more accurate for the test to declare itself skipped -- maybe it could throw a specially-handled TestSkipException or something?

Good guess, it's actually called SkipException :-)
 

And the more involved question:

2) How can I effectively control the test method execution order at the per-data-object level?

To explain what I mean, allow me to give some background on how I am using TestNG.

My goal is to use TestNG to drive a test suite for the Bio-Formats file format readers. I have a large collection of image files, and I want to feed them to my tests one at a time as input data, calling each test method once for each input data object.

The first challenge is that there is a very large amount of image data -- about 140 GB and growing. Some of the readers need to do a significant amount of parsing on each file up front.

Our original approach was to have a data provider method that instantiates one format reader object per data file, and initializes them, but the amount of memory required to maintain the program state on so many files at once is prohibitive. I had to abandon use of TestNG's data provider functionality.

Did you ever consider using "lazy" data providers?  When your data provider returns an Iterator<Object[]> instead of an Object[][], TestNG will only instantiate the returned values one at a time, which should fix your problems.

I'll address your other points if you tell me that this won't work for you...

--
Cédric

 

[Curtis Rueden]

Hi Cedric,

Thanks for your reply.

> Good guess, it's actually called SkipException :-)

Great, that is exactly what I was looking for. I didn't see it on the documentation-main page, but I should have checked the Javadocs.

> Did you ever consider using "lazy" data providers?

> When your data provider eturns an Iterator<Object[]> instead of an

> Object[][], TestNG will only instantiate the returned values one at a time, which
> should fix your problems.

All right, I reworked my code to use a lazy data provider that returns an Iterator, rather than the static data provider or factory approaches I was using before. However, the method execution order is still not what I want.

To boil down my situation a bit:

I have three test methods: testA, testB and testC. They share a common data provider.

If I run the test suite on a directory containing three files -- file1, file2 and file3 -- with the Iterator approach, the following order of execution happens:

testA(file1)
testA(file2)
testA(file3)
testB(file1)
testB(file2)
testB(file3)
testC(file1)
testC(file2)
testC(file3)

Unfortunately, the order I need is:

testA(file1)
testB(file1)
testC(file1)
testA(file2)
testB(file2)
testC(file2)
testA(file3)
testB(file3)
testC(file3)

I think the problem is related to the fact that the three tests share the same data provider. TestNG calls the data provider method for each test, so there is a separate FormatReaderTestIterator instance for each test method. They each have an independent position in the iteration, and TestNG freely takes advantage of that. What I would like is to have a single Iterator instance for the entire test class, and execute all the tests on a particular data object (respecting the dependency chain, but only within the context of the current data object), before moving on to the next object in the iteration.

Maybe I'm being naive, but it seems like there could be an execution parameter that controls whether multiple tests with a shared data provider behave this way. So I guess this is a feature request, unless there is another way to accomplish what I'm trying to do.

I can mock up this example with source code, if that would be helpful.

-Curtis
---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=237865&messageID=318004#318004

[Cédric Beust ♔]

Hi Curtis,

You can't do that at the moment, unfortunately.  It's a limitation that has been bugging me for a while, but it requires some deep modifications in TestNG's engine, so I haven't had the time to look into it so far...

--
Cédric

[Curtis Rueden]

Hi Cédric,

> You can't do that at the moment, unfortunately. It's a limitation that has
> been bugging me for a while, but it requires some deep modifications in
> TestNG's engine, so I haven't had the time to look into it so far...

Ok, that's helpful to know. If it is something you ever get around to doing, I would be glad to test it by applying it to my use case. In the mean time, the limitation can be worked around by using the factory approach I described in my first message. With the factory, I do get the method execution order that I want -- as long as I don't make use of any dependencies.

If you like, I can file an issue about this on the Google Code project page.

-Curtis
---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=237865&messageID=318123#318123

[Cédric Beust ♔]

Hi Curtis,

On Tue, Feb 26, 2008 at 5:48 PM, Curtis Rueden <testng...@opensymphony.com> wrote:

Hi Cédric,

> You can't do that at the moment, unfortunately.  It's a limitation that has
> been bugging me for a while, but it requires some deep modifications in
> TestNG's engine, so I haven't had the time to look into it so far...

Ok, that's helpful to know. If it is something you ever get around to doing, I would be glad to test it by applying it to my use case. In the mean time, the limitation can be worked around by using the factory approach I described in my first message. With the factory, I do get the method execution order that I want -- as long as I don't make use of any dependencies.

If you like, I can file an issue about this on the Google Code project page.

That would be great, thanks!

--
Cédric

[Curtis Rueden]

OK, I have filed Issue 19 describing the problem, and included some source code that demonstrate the issue in a simple manner. Please let me know if anything is not clear, or if there is anything else I can do to help.

http://code.google.com/p/testng/issues/detail?id=19

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=237865&messageID=319162#319162

[Cédric Beust ♔]

Curtis, we are using Jira to track our issues :

http://jira.opensymphony.com/browse/TESTNG

Sorry about that, I wish I could just shut down the issue tracker on Google Code so people don't accidentally file bugs there...

--
Cédric

[Curtis Rueden]

> Curtis, we are using Jira to track our issues :
>
> http://jira.opensymphony.com/browse/TESTNG
>
> Sorry about that, I wish I could just shut down the
> issue tracker on Google
> Code so people don't accidentally file bugs there...

Ok, I have refiled on Jira:

http://jira.opensymphony.com/browse/TESTNG-212

I labeled the issue as "New Feature," but there is also a bug report buried in there regarding method dependencies with factory classes. Lastly, if you didn't notice the mention in my original post, I fixed some bugs in org.testng.reporters.DotTestListener:

https://skyking.microscopy.wisc.edu/trac/java/browser/trunk/loci/tests/testng/DotTestListener.java

-Curtis

---------------------------------------------------------------------
Posted via Jive Forums

http://forums.opensymphony.com/thread.jspa?threadID=237865&messageID=319222#319222

[Cédric Beust ♔]

On Wed, Feb 27, 2008 at 1:45 PM, Curtis Rueden <testng...@opensymphony.com> wrote:

> Curtis, we are using Jira to track our issues :
>
> http://jira.opensymphony.com/browse/TESTNG
>
> Sorry about that, I wish I could just shut down the
> issue tracker on Google
> Code so people don't accidentally file bugs there...

Ok, I have refiled on Jira:

http://jira.opensymphony.com/browse/TESTNG-212

I labeled the issue as "New Feature," but there is also a bug report buried in there regarding method dependencies with factory classes. Lastly, if you didn't notice the mention in my original post, I fixed some bugs in org.testng.reporters.DotTestListener:

https://skyking.microscopy.wisc.edu/trac/java/browser/trunk/loci/tests/testng/DotTestListener.java

Thanks!  I had fixed it in the documentation but not in the code...

--
Cédric