Running each @Test in a separate ClassLoader

[Gili Tzabari]

Hi,

How can I tell TestNG to run @Test methods in parallel with each
method running in a separate Classloader for better isolation? The
existing mechanism of running methods in the same Classloader results
in race-conditions in slf4j, Guice and Grizzly because of the use of
static variables. I'm working with the authors on getting this fixed
but ultimately running in separate Classloaders is the best way to go
because it guarantees isolation. Any ideas?

Thanks,
Gili

[Cédric Beust ♔]

Hi Gili,

This is currently not possible. One way you could do this is to launch one TestNG instance per classloader. You could also automate this by writing a wrapper that will turn each <test> tag into its own suite and launch one TestNG on each of these suite files. The downside is that you will get different set of reports at the end, which you will have to put together (or you could write your own IReporter to support that).

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To post to this group, send email to testng...@googlegroups.com.
To unsubscribe from this group, send email to testng-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/testng-users?hl=en.

[cowwoc]

Hi Cedric,

    To be clear, I'm talking about each *method* having its own ClassLoader. Is it even possible to ask TestNG to run a single method and exit (even if others are annotated with @Test)?

    Would it be possible to add this functionality natively into TestNG?

Thanks,
Gili

[Cédric Beust ♔]

Wow, running each method in its own class loader is a huge waste of resources and it's probably going to significantly increase the duration of your test run, are you sure there is really no way around that?

At any rate, you can implement an IObjectFactory listener, which will make TestNG let you create all your objects yourself. It's only at the class level, though, so still not quite what you need, I'm afraid...

-- 

Cédric

[cowwoc]

Hi Cedric,

1. Just curious, did you actually measure this? It might not be as bad as you think. What overhead do you foresee aside from reloading the class files multiple times?
   
2. I think it's worth noting that scalability often comes at the price of absolute per-core performance, but that's fine. All we care about is that when we scale these tests out over 4, 8, ..., 1000 cores they'll run faster than under a single core. If we're adding even more isolation than we had before then that's icing on the cake.
   

    Anyway, I'd love to measure the actual overhead but I can't get started unless I find a way to run separate test methods in separate classloaders.

Gili

[Thomas Fišer]

You know, Cedric, this might actually be possible using the
IObjectFactory if you accept a bit of workaround. Stay with me for
this: One would have to overload the standard class loader to ignore
wether a class is already loaded or not. Let's say, for each Test
class I create a proxy class that instances such a class loader and
reflexively runs the respective test class. Then I use IObjectFactory
to instance the proxies on which I run the proxying method using the
@Test annotation. The test results would simply be handed back to
TestNG by the Proxies. Do you think that this could work?

On 27 Jun., 20:24, Cédric Beust ♔ <ced...@beust.com> wrote:
> Wow, running each method in its own class loader is a huge waste of
> resources and it's probably going to significantly increase the duration of
> your test run, are you sure there is really no way around that?
>
> At any rate, you can implement an IObjectFactory listener, which will make
> TestNG let you create all your objects yourself. It's only at the class
> level, though, so still not quite what you need, I'm afraid...
>
> --
> Cédric
>
> On Sat, Jun 25, 2011 at 1:47 PM, cowwoc <cow...@bbs.darktech.org> wrote:

> > **

[Gili Tzabari]

... No reply?

I think what Thomas is proposing would work, though it would be much
more efficient to add this functionality natively in TestNG (without
needing to resort to proxies).

Gili

[Cédric Beust ♔]

On Sun, Jul 3, 2011 at 10:33 AM, Gili Tzabari <cow...@bbs.darktech.org> wrote:

... No reply?

I think what Thomas is proposing would work, though it would be much
more efficient to add this functionality natively in TestNG (without
needing to resort to proxies).

I don't think there is enough need for such a feature to justify adding it to TestNG, personally...

-- 

Cédric

[Cédric Beust ♔]

On Tue, Mar 12, 2013 at 2:59 AM, Zeba Ahmad <cool...@gmail.com> wrote:

Whewps! Was there any solution to this? I'm surprised there hasn't been enough user need expressed for this? We are migrating our tests from JUnit to TestNG. Though the actual migration was automated, many of the tests are failing, since they were written with the assumption that each test is run in its own classloader!! :(

JUnit doesn't guarantee this either (actually, it pretty much never happens unless you implement your tests this way). Even if you run your tests in a separate JVM (which is pretty common), they are still going to run in the same classloader, whether you're using TestNG or JUnit.

-- 

Cédric

-- 

Cédric

[Zeba Ahmad]

Thanks for the reply Cedric. Here are the exact problems I'm facing. Not sure if I'm missing something here, or got the basics wrong!

1) My JUnit test suites have static variables at the test class level. I converted the class to TestNG
    - testA updates the value of the static variable during the test
    - testB runs, expecting the value to be as before, but fails since the value is now different!
2) My JUnit test refers to framework classes that internally use singletons. The createSingleton method will create different objects based on test scenario. I converted the class to TestNG
   - testA initialized the singleton based on testScenario1
   - testB creates testScenario2. However, the singleton simply returns the previously created object, instead of creating it over again! Test fails :(
3) My JUnit creates new MBeanServer with different domain names, to mimic different containers like "geronimo","tomcat","DefaultDomain", etc. I convert the test to TestNG
   - testA initialized testScenario by setting MBeanServer for container "Geronimo". The underlying MBeanServerFactory of javax stores these in a static final list.
   - testB initialized testScenario by setting MBeanServer for container tomcat. The underlying MBeanServerFactory of javax adds this to the same static final list. My test fails sporadically, as it sometimes detects enviroment as "tomcat" and sometimes as "geronimo"

These are some samples. It seems strange to me that nobody else faces the same problems?? It seems natural that test classes would attempt to use different values for static and final variables across tests.

Do you know at what level does testng group tests by classloader? (e.g. All tests in one class, all tests per package, all tests for any testng run)

Thanks in advance!
Zeba

[Cédric Beust ♔]

Hi Zeba,

I can't comment without seeing your exact code, but as a general rule, you don't need to use statics any more with TestNG. And I'm guessing that whatever the reason why it's not working has nothing to do with class loaders

-- 

Cédric

[Zeba Ahmad]

Hi Cedric,

Is there any way at all, through maven surefire plugin or otherwise to make TestNG use separate classloaders per test? How does the classloading work? Is it per test package/project/any other?

Here is a sample code:

public class MySingleton

{

private static synchronized instance;

public static synchronized MySingleton getInstance(){

     if(instance!=null) return instance;

     init();

}

private void init(){

     if(System.getProperty("os.name").contains("Win"){

                instance = new MyWindowsSingleton();

     }else{

                instance = new MyDefaultSingleton();

     }

}

}

Below are the test classes. The test methods obviously expect two different instances to be returned. However, TestNG initializes the singleton only once & returns the same object each time .. 
public class MySingletonTest{

   @Test

    public void testWindowsSingleton(){

             // First set System property "os.name" to "Windows 2008"

             // Now call getInstance

             MySingleton singleton = MySingleton.getInstance();

             // Now perform windows specific test

    }

   @Test

    public void testLinuxSingleton(){

             // First set System property "os.name" to "Linux"

             // Now call getInstance

             MySingleton singleton = MySingleton.getInstance();

             // Now perform linux specific test

    }

}

Thanks,

Zeba

[Cédric Beust ♔]

It's a singleton, it's expected to return the same value throughout the life of your application. Besides, system properties are set at JVM launch time and cannot be modified after that, so you must be launching two of them to achieve this effect, class loaders can't change that.

How were you launching your tests with JUnit? Just do the same thing with TestNG.