invocationCount and failures

[Manik Surtani]

I currently have a test that stresses some code, and needs to run ~ 1000 times.  I've annotated the test method as @Test(invocationCount = 1000) and it works fine, but when there is a failure, say on iteration 10, TestNG will still attempt to run the remaining 990 invocations.

How can I configure it to stop on success OR first failure?

Cheers,
Manik

[Cédric Beust ♔]

It's not possible at the moment, but it's a reasonable request.

We probably need to make it configurable, though, or we might break existing code.

--
Cedric

--
Cédric

[Cédric Beust ♔]

Hi Manik,

I have a first version of this new functionality, which you can test by downloading:

http://testng.org/testng-5.7b.zip

You need to enable this behavior, which you can do with either of the following:

Command line:
java org.testng.TestNG -skipfailedinvocationcounts...

ant:
<testng skipfailedinvocationcounts="true" ...>

testng.xml:
<suite skipfailedinvocationcounts="true"...>
or
<test skipfailedinvocationcounts="true"...>

I only have basic tests in place at this point, so let me know if you encounter any problem...

--
Cédric

[Andreas Guther]

Cedric,

I think using camel case for the attribute/parameter makes it more readable.

skipFailedInvocationCounts

Andreas

[Cédric Beust ♔]

Hi Andreas,

On Sep 16, 2007 6:36 AM, Andreas Guther <andreas...@gmail.com> wrote:

Cedric,

I think using camel case for the attribute/parameter makes it more readable.

skipFailedInvocationCounts

It's not an annotation attribute, it's only used on the command line, testng.xml and build.xml, where our convention seems to be all lowercase...

--
Cédric

[Manik Surtani]

Thanks Cedric, appreciate the quick response.

Is this version available on a maven2 repo - as a snapshot, perhaps?  My test suite is driven by maven2. 

As for configuration, and implementation, is this not something best set on a per test basis, alongside invocationCount?

E.g., @Test (invocationCount=100, skipFailedInvocationCounts=true) ... ?

It is concievable that one may not want this setting for all tests in a test suite, so making this an annotation attribute may make more sense...

Cheers,
Manik

[Cédric Beust ♔]

You're right, an attribute makes more sense.  I'll make the change.

--
Cédric

[Manik Surtani]

Thanks.  Like I said just let me know when a snapshot is available on maven2 and I'll give it a go.  :-)

I've been moving the JBoss Cache test suite from JUnit to TestNG and so far things have been great.  Helps me lose a lot of "framework" code like running tests in a loop, different threads, etc etc.

[Cédric Beust ♔]

Well, I'm not very familiar with the Maven integration, so someone else will have to take care of it, but at least, you should be able to do a quick sanity check on my fix on a small example...

Glad to hear your conversion efforts are going well!  Feel free to share any tips you might have on the process.

I'll keep you posted.

--
Cédric

[Cédric Beust ♔]

Manik,

I implemented this change as an annotation attribute in the HEAD, please let me know if you need a jar file and I'll create one for you.

--
Cédric

[Manik Surtani]

Thanks - I'll build it myself from srcs.

BTW do you know who maintains TestNG's maven2 repo on ibiblio, etc.?

[Cédric Beust ♔]

On 9/24/07, Manik Surtani <manik....@gmail.com> wrote:

Thanks - I'll build it myself from srcs.

BTW do you know who maintains TestNG's maven2 repo on ibiblio, etc.?

I'm not sure any more, I have to say...  Alexandru?

--
Cédric

[Alexandru Popescu ☀]

Hmmm... unfortunately I have to say not me. The only thing I have
tried was to push some maven2 surefire plugin code for working with
TestNG. However, I don't have commit rights there, so everything is
peered through Brett.

./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator

> --
> Cédric
> >
>

[Mark Derricutt]

Has anything further happened on the maven2 side of TestNG?

[Alexandru Popescu ☀]

On 10/1/07, Mark Derricutt <ma...@talios.com> wrote:
> Has anything further happened on the maven2 side of TestNG?
>

Well, the story so far is that I got some code committed on the
plugin, and provided some more... but never heard back about this
second one.

I have tried to get commit rights so that I can quickly change things
in there, but it looks like due to internal discussions this is not
possible.

Concluding, I have no idea what is the current status right now :-(.

./alex
--
.w( the_mindstorm )p.
TestNG co-founder
EclipseTestNG Creator

> On 9/30/07, Alexandru Popescu ☀

[Mark Derricutt]

Doh :( Time to slap those maven guys around a bit more then.

Who was it that working on their own non-surefire based TestNG plugin (I don't think it was open source thou?)?  Even if its not integrated into surefire a dedicated plugin would at least let maven2 people actually use the current TestNG with maven.

[Manik Surtani]

FYI, at JBoss, we've got our own version of the Surefire plugin as a part of the JBoss Build project.  It is a fork, but for reasons to do with the plugin being hardcoded into Maven2's lilfecycle, we can't change the artifactId or groupId.  We can, however, change the version we release.

http://anonsvn.labs.jboss.com/labs/jbossbuild/maven-surefire-jboss/trunk/

A snapshot with SUREFIRE-315 and SUREFIRE-349 fixed are here:

http://snapshots.jboss.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/

To use the JBoss version of Surefire, make sure you

1) Have http://repository.jboss.org/maven2 and http://snapshots.jboss.org/maven2 in your repository list
2) Specify 2.4-jboss-1-SNAPSHOT as the version of Surefire you wish to use.

Also, feel free to contribute fixes/enhancements to this source tree.  Details will be on http://labs.jboss.org/jbossbuild shortly.

Cheers,
Manik