TestNG fail if there was some exception in test class initialization

[vitaliy.pomazyonkov]

Hi!

There is a bug in TestNG 5.13.1 (earler versions seems also affected).
In case if test class initialization failed, TestNG shows error like:
[testng] [ERROR]:
[testng] Cannot instantiate class xx.xx.XxTest
[testng] The tests failed.
And not running any other tests.

Expected behaviour is to mark this test as failed, record
initialization exception and continue running all other tests in
suite.

Cedric, please, take a look.

[Cédric Beust ♔]

Vitaliy,

This is the expected behavior: if one of your test classes can't be instantiated, there's something wrong with your environment so TestNG aborts.

You should remove this class from your testng.xml if it's not valid.

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Konstantin]

Cédric,

I have the question on this:
Why test class initialization exceptions and exceptions in test
methods have different priorities? It is not necessary that
initialization exceptions connected with environment, usually these
part are placed in the configuration methods.
And I absolutely agree with Vitaliy that it would be extremely useful
to have the exception message in output.

> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > .

[vitaliy.pomazyonkov]

I think that initialization exception like any other unexpected
exception in test class must be processed by TestNG in the same way.
And anyway TestNG must NOT stop running all other tests.

For example, if test initializes some property in constructor and
there is something wrong, then we will get and fix it next morning
(our suite takes about 7 hours), but in current implementation we will
got only report about 1 wrong test of hundreds... so we have lost many
test results because of one wrong test, which is not expected.

> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > .

[Nalin Makar]

Why do you want to run tests when the environment or the test suite xmls are not setup correctly?

The old behavior was that the test classes were initialized just before execution. This wasn't implemented correctly and the whole execution would stop if there was a class initialization exception. We have changed this behavior to initialize all the classes right at the beginning of test run and the test run doesn't proceed with any test if the XMLs are not setup correctly.

Any nonexistent classes should anyways not be part of your suites.

-nalin

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

[vitaliy.pomazyonkov]

Here is our workflow: every day many commiters submits their tests
code to SVN, then every night whole big test suite executed and at the
morning we are cheking results. So if someone will commit wrong test
(that throws exception in constructor), then we will not get results
of other correct tests, we will not get results at all, which means
that we lost 1 day of autotesting.
But if TestNG will mark that fail-initialized test as failed and
continue execution of other tests, everything will be OK - we will fix
incorrect test and check results of all other (correct) tests.
I doesn't mean situation when TestNG XML is not correct or something
wrong in environment, but only that situation when because of 1 wrong
test we lost results of all other tests.

On 27 сен, 19:57, Nalin Makar <nalin.ma...@gmail.com> wrote:
> Why do you want to run tests when the environment or the test suite xmls are
> not setup correctly?
>
> The old behavior was that the test classes were initialized just before
> execution. This wasn't implemented correctly and the whole execution would
> stop if there was a class initialization exception. We have changed this
> behavior to initialize all the classes right at the beginning of test run
> and the test run doesn't proceed with any test if the XMLs are not setup
> correctly.
>
> Any nonexistent classes should anyways not be part of your suites.
>
> -nalin
>
> On Mon, Sep 27, 2010 at 2:50 AM, vitaliy.pomazyonkov <
>

> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>

[Cédric Beust ♔]

On Mon, Sep 27, 2010 at 10:58 AM, vitaliy.pomazyonkov <vitaliy.p...@gmail.com> wrote:

Here is our workflow: every day many commiters submits their tests
code to SVN, then every night whole big test suite executed and at the
morning we are cheking results. So if someone will commit wrong test
(that throws exception in constructor), then we will not get results
of other correct tests, we will not get results at all, which means
that we lost 1 day of autotesting.
But if TestNG will mark that fail-initialized test as failed and
continue execution of other tests, everything will be OK - we will fix
incorrect test and check results of all other (correct) tests.
I doesn't mean situation when TestNG XML is not correct or something
wrong in environment, but only that situation when because of 1 wrong
test we lost results of all other tests.

You only lose the results of tests whose results you couldn't have trusted anyway because the method that configures them crashed.

--

Cédric

[vitaliy.pomazyonkov]

No, we are losing results of all tests including that we can trust.
Here is example of two tests:

public class InitFailTest {
public InitFailTest() {
// Here we have some nasty error.
throw new RuntimeException("Nasty error!");
}

@Test(groups = {"smoke", "read"})
public void testSomething() {
// Just do nothing.
}
}

public class OtherImportantTest {
@Test(groups = {"smoke", "read"})
public void testImportantFunctionality() {
// This test doesn't depends on InitFailTest and we need it
results anyway, doesn't matter if InitFailTest failed to init or not.
}
}

<suite name="Whole big suite">
<test name="InitFailTest">
<classes>
<class name="InitFailTest"/>
</classes>
</test>

<test name="OtherImportantTest">
<classes>
<class name="OtherImportantTest"/>
</classes>
</test>
</suite>

Result:
[testng] [ERROR]:
[testng] Cannot instantiate class InitFailTest
[testng] The tests failed.

On 27 сен, 22:00, Cédric Beust ♔ <ced...@beust.com> wrote:
> On Mon, Sep 27, 2010 at 10:58 AM, vitaliy.pomazyonkov <
>
>

[Cédric Beust ♔]

This is a different error from a configuration failing: one of your test classes cannot be instantiated.

You should either fix this or remove that class from your run, but I think TestNG's reaction to abort everything when this happens is correct in this case (it's like a compilation error).

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[vitaliy.pomazyonkov]

Why do you think that it is correct TestNG's reaction? It is more user-
friendly to just log exception like all other exceptions and show it
in report. Any method can throw exception, but constructor can't...
what is the logic in this case?
Maybe it is different to implement in current TestNG architecture?

[Cédric Beust ♔]

Hi Vitaliy,

On Tue, Sep 28, 2010 at 8:35 AM, vitaliy.pomazyonkov <vitaliy.p...@gmail.com> wrote:

Why do you think that it is correct TestNG's reaction? It is more user-
friendly to just log exception like all other exceptions and show it
in report. Any method can throw exception, but constructor can't...
what is the logic in this case?
Maybe it is different to implement in current TestNG architecture?

I think that exceptions in a test constructor are the symptom of something unexpected (just like a configuration method failing). It should simply never happen and I think it's important for TestNG to get the user's attention by aborting if such a thing happens.

Once you see this error, you have several options:

• Something is indeed wrong and you get a chance to fix it.
• The exception is not unexpected (e.g. transient failure that your environment can cause) and since you know it's not going to impact the tests negatively, you can simply catch it yourself so TestNG will never see it. This also gives you a chance to set up your environment in a way that will make your tests still mean something ("the database is down, let's just use an in-memory database then").
  

As you can see, this approach gives you the most flexibility and it guarantees that you won't be deceived by false positives (tests passing not because they actually pass but because the environment is unstable).

Doing it the other way is basically letting TestNG assume that all failures are acceptable and in my experience, it's easy to overlook logged exceptions when you start having hundreds of tests.

Back to your particular problem: if this is exception can happen, why not catch it yourself so TestNG never sees it?

--
Cédric

[vitaliy.pomazyonkov]

Cédric, thanks for detailed answer.
I can't catch all such exceptions because them can be also trown on
class field definition, for example:

public class SomeTest {
private String s = "abc".substring(5, 10);
}

Creating of new instance of such class will throw
java.lang.StringIndexOutOfBoundsException: String index out of range:
10. And it can't be catched.

[Jason McSwain]

Hello Vitaliy,

I don't mean to jump in the middle here, but can you explain why "StringIndexOutOfBoundsException" is not catchable?

if the problem is that it's in the constructor, and you don't have access to the actual constructor call,  then why not move the logic into an @BeforeClass method ?

i.e.

public class SomeTest {
   private String s = null;

    @BeforeClass
    public void classSetup() throws Exception {
        try{

            s = "abc".substring(5, 10);
        }

        catch (StringIndexOutOfBoundsException e){
            //deal with the error here.
        }
    }
}

I hope this helps.
-Jason-

--

[Cédric Beust ♔]

On Tue, Sep 28, 2010 at 12:37 PM, Jason McSwain <jasonw...@gmail.com> wrote:

I don't mean to jump in the middle here, but can you explain why "StringIndexOutOfBoundsException" is not catchable?

Probably because Vitaliy chose to initialize this field at declaration time instead of doing it in the constructor (not a good idea, as I already pointed out, but hopefully that was just for illustration purposes :-)).

--
Cédric

[Cédric Beust ♔]

On Tue, Sep 28, 2010 at 12:17 PM, vitaliy.pomazyonkov <vitaliy.p...@gmail.com> wrote:

Cédric, thanks for detailed answer.
I can't catch all such exceptions because them can be also trown on
class field definition, for example:

public class SomeTest {
   private String s = "abc".substring(5, 10);
}

Creating of new instance of such class will throw
java.lang.StringIndexOutOfBoundsException: String index out of range:
10. And it can't be catched.

You should definitely not define literals that you know will not construct :-)

Put dangerous initializations in your constructor and catch them there. But more importantly: ask yourself if you can't make your test classes more robust in this respect.

--
Cédric

[vitaliy.pomazyonkov]

Of cource it is just example. As i told before, we have many different
commiters and initialization of fields may be implemented using many
different ways. For now i have two choices: say to everyone to not
initialize fields at definition because in this case we may lose all
test results, or find a way to fix it in TestNG.

On Sep 28, 11:41 pm, Cédric Beust ♔ <ced...@beust.com> wrote:

[vitaliy.pomazyonkov]

Yep, it was choosen to init class field on definition and it is normal
in Java. But in some cases when programmer was not fully careful, it
may cause exception.

[Bill Michell]

So what you want is for TestNG to ignore "bad" classes when collecting candidate tests, but carry on the test run and run tests in classes that do initialise correctly, because you work in and environment where you can't always guarantee that all classes will initialise cleanly?

Presumably you would want the overall result to be a Fail, but to receive detailed results for any tests that *did* run...

--
Bill Michell
billm...@gmail.com

[Carsten]

I think ignoring "bad" classes (actually: let them fail) is a good
idea. Testng as a testing framework should never crash (abort), no
matter if there are test classes which cannot be loaded due to
exception in the constructor or other class load issues.

In case testng aborts the reporters aren't called anymore and log
information might get lost - which is in particular for integration
tests a problem.

And as Vitaliy already mentioned, it is always possible that a
developer is not careful and adds test which cause exceptions.

Carsten

> billmich...@gmail.com

[Todd Wells]

Vitaliy, I think you may want to look at the configfailurepolicy setting: http://groups.google.com/group/testng-users/msg/fb0fce19dff65c0e?pli=1

I'm not sure if this is the exact scenario because it sounds like you're failing in class initialization rather than in a config method, but it's worth looking at. If you could move whatever logic is in your class initialization into a config method, you could take advantage of this.

> >>> A quick review on how to control this setting: 
> >>> Command line:  -configfailurepolicy continue|skip 
> >>> testng.xml:  <suite configfailurepolicy="continue|skip" ...> 
> >>> Ant task:  <testng configfailurepolicy="continue|skip" ...> 

> >>> Feature recap: When configfailurepolicy is set to 'continue', TestNG 
> >>> will continue to run any remaining tests that do not depend on the 
> >>> "instance" of a configuration method that fails or encounters a 
> >>> SkipException.  For example, if a @BeforeClass method in a base class 
> >>> with 2 subclasses fails when running for subclass 1, subclass 2 will 
> >>> not be automatically skipped.  (Subclass 1 will be skipped as 
> >>> expected.)  In a similar way, if a local @BeforeMethod method fails 
> >>> for test 1, test 2 will not be automatically skipped.  (Test 1 will 
> >>> though.) 

[Konstantin]

Guys, I still do not catch the idea of TestNGs behavior in this case.

e.g. I have two suites, and two tests (two test classes) per suit
without any dependencies between tests. One of the test can't be
initialized (there is an exception in the constructor).
In the situation described above I've got only this:
[testng] [ERROR]:

[testng] Cannot instantiate class xx.xx.XxTest
[testng] The tests failed.

I absolutely agree that all the test methods from test class with
error should be skipped (the same idea as for configuration methods
fail) but why should second test be skipped and, a fortiori, second
suite?
And why it is better to show only message "Cannot instantiate class
xx.xx.XxTest" and do not show the original cause of this.

Cédric, Nalin please explain if I'm wrong in my thoughts.
Thanks

On Sep 28, 11:30 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> On Tue, Sep 28, 2010 at 12:17 PM, vitaliy.pomazyonkov <
>

[Nalin Makar]

You are not wrong and neither is the way TestNG works currently. Taking your example, consider two suites S1 and S2 and that of the 50 tests in S2, 1 is specified using a class can not be initialized.

The previous behavior of TestNG was that it would execute S1, and all the tests in S2 up to the point where it encountered the rouge class and then just fail and quit. This way no reports were generated. I fixed this to move all the class initialization code (including dataprovider related tests) right to the beginning of test run initialization. This way, we first check all the classes in S1 and S2 and only proceed if we are able to ensure that everything is in order.

I made this change to work this way because, frankly, I found this to be the easier fix :-). We could argue on two different philosophies at play here:

a. All tests specified in test suites should be correct. If not, you haven't configured your test run correctly. This is good as it forces the tester to fix the problems upfront.
b. We should skip, the tests specified using classes that fail to initialize. This is good as it allows us to get the results at least on the rest of the tests.

I am fine either way. I don't know how much effort is required to change the behavior to option 'b'. I am still working on understanding TestNG internals.

-nalin

p.s. Can you file a bug in JIRA for "why it is better to show only message "Cannot instantiate class xx.xx.XxTest" and do not show the original cause of this." I believe we don't know the real reason why the initialization failed, but if it's available, we can expose that.

-nalin

[Konstantin]

Nalin,
thank you for detailed answer. I've created JIRA task: TESTNG-426 for
initialization error message.

My suggestion according the issue with skipped tests is not to choose
between two philosophies, but provide the choice to users. Make fix by
analogy with 'configfailurepolicy' for configuration errors.

Konstantin.

> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > .

[Martin van Dijken]

I'd like you opinions on the setup we created that is not really
working the way I want it to. I think this is a nice case for option
b. but please correct me if it's not 100% connected.

We have a bunch of Selenium tests. Each test has a @beforeClass method
that tells Selenium to open up a page. When the page is opened, there
are several @test annotated methods that check the page for
correctness.

The problem I'm running into is that when one of the @beforeClass
methods fails, no more tests are executed. My desire is that all the
tests in that class are not executed any more, but that the rest of
the test classes still continue.

Some more details:

<suite name="Pages for comparer.fr">
<parameter name="label" value="frFR"/>
<test name="All page tests">
<packages>
<package name="eu.comparegroup.selenium.page">
</package>
</packages>
</test>
</suite>

TestNG 5.14
Selenium client driver 1.0.1

Look forward to hearing from you,

Martin

[Cédric Beust ♔]

Hi Martin,

When a @BeforeClass fails, the class that it's run before should indeed be skipped from that point on, but it shouldn't impact the rest of your tests, so the behavior you are describing is not correct.

Can you post a short test class showing this problem?

Thanks.

-- 

Cédric

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Nalin Makar]

Also, Martin in your case, try setting -configfailurepoilcy=true while running tests and see if that helps. I am guessing that you might have a single @BeforeClass method in a parent class and the test classes extend from this class.

-nalin

2010/10/20 Cédric Beust ♔ <ced...@beust.com>

[Cédric Beust ♔]

On Wed, Oct 20, 2010 at 8:53 AM, Nalin Makar <nalin...@gmail.com> wrote:

Also, Martin in your case, try setting -configfailurepoilcy=true while running tests and see if that helps. I am guessing that you might have a single @BeforeClass method in a parent class and the test classes extend from this class.

Yes, that was my guess as well. I'm guessing your @BeforeClass is actually being used in more classes than you think, and therefore, causing all these classes to be skipped as well.

--
Cédric

[Martin van Dijken]

The problem I am encountering is indeed because I have inheritance in
my tests. I can't really post the entire test class, but let me try to
explain with pseudocode:

abstract class SeleniumTestBase{

@BeforeClass
configureAndOpenSelenium(){
configure(selenium);
selenium.open(getPageUrl());
}

abstract getPageUrl();
}

class TestTheReviewPage extends SeleniumTestBase{
getPageUrl(){
return "/users/review.xhtml";
}
@Test
validatePageHeader(){
assertTrue(selenium.isElementPresent("css=.logo"));
}
}

All of the classes inherit from this SeleniumTestBase and it might be
quite possible that this is the reason for all of them failing if
Selenium fails to open. Is there anything you can advise me here?

Thanks so far,

Martin

On Oct 20, 4:42 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> Hi Martin,
>
> When a @BeforeClass fails, the class that it's run before should indeed be
> skipped from that point on, but it shouldn't impact the rest of your tests,
> so the behavior you are describing is not correct.
>
> Can you post a short test class showing this problem?
>
> Thanks.
>
> --
> Cédric
>

> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>

>
> > > > > .
> > > > > For more options, visit this group at
> > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > .
> > For more options, visit this group at
> >http://groups.google.com/group/testng-users?hl=en.
>
> --

> Cédric

[Cédric Beust ♔]

Hi Martin,

So it looks like you have a @BeforeClass that's being inherited by n classes but if it fails, it should only impact 1 class but the other n-1 classes should keep running?

In this case, I would determine what part of the code can fail and make sure it can only impact one class by running this code only in this class's @BeforeClass, and not the superclass's @BeforeClass.

Does this make sense?  (I haven't had my coffee yet, I won't feel offended if you say "no" :-))

-- 

Cédric

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Martin van Dijken]

Hi Cédric,

You hit the nail on the head there, the situation and behavior are as
you state. The problem I see with moving the potentially failing code
to the inheriting classes is that it isn't quite as clean. I can make
it work of course, but it's not as elegant.

Is there a reason for doing it the way it works now?

Thanks so far,

Martin

On Oct 22, 4:55 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> Hi Martin,
>

> So it looks like you have a @BeforeClass that's being inherited by n classes
> but if it fails, it should only impact 1 class but the other n-1 classes
> should keep running?
>
> In this case, I would determine what part of the code can fail and make sure
> it can only impact one class by running this code only in this class's
> @BeforeClass, and not the superclass's @BeforeClass.
>
> Does this make sense?  (I haven't had my coffee yet, I won't feel offended
> if you say "no" :-))
>
> --
> Cédric
>

> > > > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
> > <testng-users%252Buns...@googlegroups.com<testng-users%25252Bun...@googlegroups.com>

>
> > > > > > > .
> > > > > > > For more options, visit this group at
> > > > > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > > --
> > > > You received this message because you are subscribed to the Google
> > Groups
> > > > "testng-users" group.
> > > > To post to this group, send email to testng...@googlegroups.com.
> > > > To unsubscribe from this group, send email to
> > > > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>
> > <testng-users%2Bunsu...@googlegroups.com<testng-users%252Buns...@googlegroups.com>
>
> > > > .
> > > > For more options, visit this group at
> > > >http://groups.google.com/group/testng-users?hl=en.
>
> > > --

> > > Cédric
>
> > --
> > You received this message because you are subscribed to the Google Groups
> > "testng-users" group.
> > To post to this group, send email to testng...@googlegroups.com.
> > To unsubscribe from this group, send email to
> > testng-users...@googlegroups.com<testng-users%2Bunsu...@googlegroups.com>

[Cédric Beust ♔]

Hi Martin,

Well, to be honest, I don't see how it should work any other way :-)

How do you expect code that is inherited by n classes from a base class to magically act differently when it's being run in a specific subclass? The only way to make this happen is for this specific subclass to have some extra logic that will make it work differently from the other n-1 classes.

Am I missing something?

-- 

Cédric

To unsubscribe from this group, send email to testng-users...@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

--
Cédric

[Martin van Dijken]

Hey Cedric,

Completely lost now...

Inheritance takes care of having the same code executed in different
classes, but potentially with different parameters. The internal state
of the resulting object is different per subclass when it gets run.
This means that the fact that the inherited method fails in one
subclass says nothing about the possibility of that method failing in
other classes.

You are calling the annotation @BeforeClass, but I think it should be
running right before an instance of a class actually starts it's
tests. The instances are all of different classes though, so I don't
really see how it is the same?

Hope you're not offended or anything, I don't completely see what you
mean here.

Thanks so far,

Martin

On Oct 25, 6:26 pm, Cédric Beust ♔ <ced...@beust.com> wrote:
> Hi Martin,
>

> Well, to be honest, I don't see how it should work any other way :-)
>
> How do you expect code that is inherited by n classes from a base class to
> magically act differently when it's being run in a specific subclass? The
> only way to make this happen is for this specific subclass to have some
> extra logic that will make it work differently from the other n-1 classes.
>
> Am I missing something?
>
> --
> Cédric
>

> ...
>
> read more »