2 questions about invocation-numbers

[danielmoennich]

Hi all,

I have two questions concerning the invocation-numbers used in the
testng-failed.xml to repeat only the failed tests of test methods that
use data provider.

1) Is it possible to assign those invocation-numbers to the related
ITestResult and to access this information with testNG listener/
reporter?

2) If for a test run a testng-failed.xml file containing invocation-
numbers is used, the following testng-failed.xml file(s) will contain
wrong invocation-numbers. Here's an example:
- first test run: 3 Tests fail with invocation numbers 8,17,45
--> testng-failed.xml with <include name="testmethod" invocation-
numbers="8 17 45"/> gets generated
- second test run using testng-failed.xml: 2 tests (with invocation-
numbers 17 and 45) fail
--> testng-failed.xml with <include name="testmethod" invocation-
numbers="1 2"/> gets generated
- third test run using second testng-failed.xml would repeat the wrong
tests

Is this behavior already known and is there an alternative to this
approach?

Thanks much,
Daniel

[Cédric Beust ♔]

Hi Daniel,

On Sun, Jul 25, 2010 at 7:09 AM, danielmoennich <daniel....@googlemail.com> wrote:

Hi all,

I have two questions concerning the invocation-numbers used in the
testng-failed.xml to repeat only the failed tests of test methods that
use data provider.

1) Is it possible to assign those invocation-numbers to the related
ITestResult and to access this information with testNG listener/
reporter?

Can't you get this information from ITestResult#getMethod()#getFailedInvocationNumbers()?

 

2) If for a test run a testng-failed.xml file containing invocation-
numbers is used, the following testng-failed.xml file(s) will contain
wrong invocation-numbers. Here's an example:
- first test run: 3 Tests fail with invocation numbers 8,17,45
--> testng-failed.xml with <include name="testmethod" invocation-
numbers="8 17 45"/> gets generated
- second test run using testng-failed.xml:  2 tests (with invocation-
numbers 17 and 45) fail
--> testng-failed.xml with <include name="testmethod" invocation-
numbers="1 2"/> gets generated
- third test run using second testng-failed.xml would repeat the wrong
tests

Is this behavior already known and is there an alternative to this
approach?

This sounds like a bug, do you think you could send me a short test case allowing me to reproduce it?

--
Cédric

[Cédric Beust ♔]

2010/7/25 Cédric Beust ♔ <ced...@beust.com>

 

2) If for a test run a testng-failed.xml file containing invocation-
numbers is used, the following testng-failed.xml file(s) will contain
wrong invocation-numbers. Here's an example:
- first test run: 3 Tests fail with invocation numbers 8,17,45
--> testng-failed.xml with <include name="testmethod" invocation-
numbers="8 17 45"/> gets generated
- second test run using testng-failed.xml:  2 tests (with invocation-
numbers 17 and 45) fail
--> testng-failed.xml with <include name="testmethod" invocation-
numbers="1 2"/> gets generated
- third test run using second testng-failed.xml would repeat the wrong
tests

Is this behavior already known and is there an alternative to this
approach?

This sounds like a bug, do you think you could send me a short test case allowing me to reproduce it?

Actually, thinking about this more, this sounds like the normal behavior.

When you run testng-failed.xml with only invocation 17 and 45, they are going to fail again, but since this time only two tests were run, the testng-failed.xml will be overwritten with the values 1 and 2...

--
Cédric

[danielmoennich]

Hello Cedric,

On 26 Jul., 02:45, Cédric Beust ♔ <ced...@beust.com> wrote:
>
> Actually, thinking about this more, this sounds like the normal behavior.
>
> When you run testng-failed.xml with only invocation 17 and 45, they are
> going to fail again, but since this time only two tests were run, the
> testng-failed.xml will be overwritten with the values 1 and 2...
>

yes, it seems like this is just how it works, no bug. Why I'm askig
this is because I
was thinking of rerunning failed tests of a test suite several times
to reduce the number of
false positives caused by instable test environment...

On 25 Jul., 16:37, Cédric Beust ♔ <ced...@beust.com> wrote:
>
> Can't you get this information from
> ITestResult#getMethod()#getFailedInvocationNumbers()?
>

I tried using ITestResult#getMethod()#getFailedInvocationNumbers() and
it gives me
a list with numbers of invocations that failed so far. But to me it
seems like this list
is out of sync when accessing it with listeners (I used
IInvokedMethodListener and TestListenerAdapter)
and running the tests in parallel. E.g., as I called the
getFailedInvocationNumbers()
within afterInvocation() or OnTestFailure(), the list contained for
example 11,13,8 - but the current invocation
number was actually 4. So I don't know how to correctly assign the
invocation-numbers to the test results.

Thanks so far,
Daniel

[Steven Erat]

Like the original poster, I have a script used in Jenkins to perform successive rounds of retries to help eliminate false failures.  Upon failure in the first round, ant executes the test suite with the found testng-failed.xml as input.   I observe that for a dataprovider test that has only 2 rows of data, when the second test iteration fails but not the first, that the testng-failed.xml refers to invocation-count=0, which upon retry runs the first test iteration condition but not the second.  Hence the result is the test is marked as passed since the second condition is not the one run in the retry.

Here's my dataprovider csv input file with headers:

criteria,media,minTotalResults
sheep,Photos,51000
garçon,Videos,100

 
The test performs a search using the specified criteria.  It is clear in the first round (and manually reproduced) that the criteria at index 0 (criteria=sheep) passes and criteria at index 1 (criteria=garçon) fails.  The generated testng-failed.xml contains under test>classes the following:

      <class name="tests.search.SearchAnonymousBrowse">
        <methods>
          <include name="anonBrowse" invocation-numbers="0"/>
        </methods>
      </class> <!-- tests.search.SearchAnonymousBrowse -->

The observed result of this test case is that the retry round executes the test with the first row of input data at index 0 where criteria=sheep.  This produces a PASS result and the failure that occurred when criteria=garçon is omitted from the retry.

By contrast, from IntelliJ where the test is executed from a Run Configuration of type method, the first round shows the first iteration passes and the second iteration fails. Using IntelliJ's "Rerun failed tests" then executes only the failed condition for the second data row.   (There was an issue in IntelliJ where a Rerun failed tests would run all invocation numbers but fixed in v15 to run only the specified failures).

Is there any other information I can provide?

Thank you,
Steven Erat

[Steven Erat]

Edit regarding contents of the testng-failed.xml files.  Note that there was 1 original round, plus 2 retry rounds for a total of 3 test rounds

Here's a more careful review of the various testng-failed.xml files. 

test-output /
     > Failed suite [Failed suite [My TestSuite]] 
         - testng-failed.xml <----- Results from round 3: anonBrowse not included in this retry round
     > Failed suite [My TestSuite]
         - testng-failed.xml <----- Results from round 2: <include name="anonBrowse" invocation-numbers="0"/> (criteria=sheep was run and passed)
    >My TestSuite 
        - testng-failed.xml <----- Results from round 1: <include name="anonBrowse" invocation-numbers="1"/> (criteria=garcon was run and failed)