Problem with @Factory, @BeforeClass failure and skipped tests

[jfpoilpret]

Hello,

going further on my factory experiments (in order to proceed with
converting my JUnit3 tests to TestNG) I have discovered a strange
behavior that I believe is probably a bug. I am using TestNG 5.7 (from
maven2).

Let's consider the following snippet:

public class TestNGProblemFactoryTest
{
@Factory static public Object[] create()
{
Object[] tests = new Object[5];
for(int i = 0; i < tests.length; i++)
{
tests[i] = new TestNGProblemTest(i);
}
return tests;
}

@Test(groups = {"utest"})
static public class TestNGProblemTest implements ITest
{
public String getTestName()
{
return "" + _num;
}

public TestNGProblemTest(int num)
{
_num = num;
}

@BeforeClass public void init()
{
System.out.println("init() #" + _num);
if (_num % 2 == 0)
{
fail("init() failed");
}
}

@Test public void test()
{
System.out.println("test() #" + _num);
}

private final int _num;
}
}

As you can see, on the 5 TestNGProblemTest instances created by the
create() factory method, 3 will have their @BeforeClass init() method
failing, while for the 2 other instances, that method will be OK.

Now if I run this test with TestNG, here is the results I get
(excerpted from testng-results.xml):

test-method status="FAIL" signature="init()" name="init (4)"
test-method status="SKIP" signature="init()" name="init (2)"
test-method status="SKIP" signature="init()" name="init (0)"
test-method status="SKIP" signature="test()" name="test (1)"
test-method status="SKIP" signature="test()" name="test (4)"
test-method status="SKIP" signature="init()" name="init (1)"
test-method status="SKIP" signature="test()" name="test (2)"
test-method status="SKIP" signature="test()" name="test (0)"
test-method status="SKIP" signature="init()" name="init (3)"
test-method status="SKIP" signature="test()" name="test (3)"

It turns out that TestNG as started executing instance #4 first, for
that instance the @BeforeMethod init() fails. This failure enforces
skipping test() of instance #4 (which is expected behavior) but also
skips ALL other methods of all other 4 instances, which is not
expected at all: @BeforeClass is supposed to work at instance level
(according to the doc), hence it should have an impact only on the
instance #4 on the example above, and TestNG should still try to run
at least init() on each instance and test() of all instances which
init() did not fail!

The expected result for the above snippet should look as follows
(maybe in a different order since we cannot determine the order in
which TestNG will run each instance methods, but the important info
below is the status value):

test-method status="FAIL" signature="init()" name="init (4)"
test-method status="SKIP" signature="test()" name="test (4)"
test-method status="FAIL" signature="init()" name="init (2)"
test-method status="SKIP" signature="test()" name="test (2)"
test-method status="FAIL" signature="init()" name="init (0)"
test-method status="SKIP" signature="test()" name="test (0)"
test-method status="PASS" signature="test()" name="test (1)"
test-method status="PASS" signature="init()" name="init (1)"
test-method status="PASS" signature="init()" name="init (3)"
test-method status="PASS" signature="test()" name="test (3)"

I.e. TestNG tries to run all init() methods, and test() only for
instances where init() has failed.

Is this a known problem (I could not find any JIRA for a similar
issue)?
Does TestNG 5.8 solves this issue? (I could not check since 5.8 is not
available on any maven2 repository, but the 5.8 changes don't seem to
cover this issue anyway).

This problem seems quite serious to me because my understanding is
that all factory-created instances of a test class are independent of
each other in terms of failure/skipping behavior.

Is there any workaround for this?

Cheerrs

Jean-Francois

[Alexandru Popescu ☀]

As far as I know this is a limitation of the current versions.

./alex
--
.w( the_mindstorm )p.
Alexandru Popescu
TestNG co-founder
EclipseTestNG Creator

[jfpoilpret]

Is there any plan to fix this some day?
Without such a fix, this makes the whole existence of @Factory quite
useless (except in the case where all tests run perfectly of course,
which is the ideal case, but not always possible, in particular at the
times those tests are first written or later extended).

Jean-Francois

On Apr 9, 11:17 pm, "Alexandru Popescu ☀"
<the.mindstorm.mailingl...@gmail.com> wrote:

> EclipseTestNG Creator- Hide quoted text -
>
> - Show quoted text -

[Alexandru Popescu ☀]

On Wed, Apr 9, 2008 at 7:21 PM, jfpoilpret <jfpoi...@yahoo.fr> wrote:
>
> Is there any plan to fix this some day?

For sure :-).

[Nadav Nagel]

Hi.

I am looking at this post thread, and I am having the same problem (using TestNG 6.9.11)

Is this issue was fixed?, and if not there are any planes to fix it?

This really blocking me, maybe you can give me some help on how to fix it?

[santhu ravikanti]

> is this issue fixed in latest version of test NG ?we are facing same in 
> >
>

[santhu ravikanti]

HI All,

we are facing same issue ,is this issue fixed in latest test ng version?