Announcing Distributed TestNG

[Cédric Beust ♔]

Hi everyone,

I am happy to announce the imminent release of a major new feature for TestNG:  Distributed TestNG.

More and more people have asked me for this feature these past months and I finally found some time to take a serious shot at it, and I was quite surprised to see that it came along very well.  Here is how it works.

Please keep in mind that there are many open issues (listed below), and I'm emailing testng-dev first in order to gather some feedback on these issues before announcing it on testng-users, so a lot of this is still in flux, but the good news is:  the implementation works, and it's now just a matter of packaging it nicely.

Overview

The idea is to be able to distribute your tests across several slave machines in order to accelerate the overall execution time.  In order to achieve this, you can now launch TestNG in "slave" mode on a remote machine by specifying a port:

java org.testng.TestNG -slave 5150

At which point, TestNG will just sit and wait for incoming connections.

Once you have launched all the slaves that you need, you declare them in a properties file:

# hosts.properties
testng.hosts=terra:5150 arkonis:5151

And you launch the "master" version of TestNG by passing it this host file:

java org.testng.TestNG -hostfile hosts.properties testng1.xml testng2.xml ...

The tests will then be dispatched randomly to the various hosts until they have all run.  All the results will be collected and presented in the usual HTML format (with the addition that these results will include the remote host they ran on).

Right now, two dispatch strategies are supported:  per-test or per-suite (the default).

By default, each suite (each testng.xml file) will be sent to a remote host in its entirety.  If you need a finer granularity, you can add the following to your hosts.properties:

# hosts.properties
testng.strategy=test

In this case, TestNG will parse all your testng.xml files, collect all the <test> stanzas and they will be sent individually to each remote host and then returned after they have run.

Great, what do I need to do to use this?

If you want to take advantage of distribution, there is only one little gotcha you need to be aware of:  your tests need to be static-friendly.

Since the slaves run in the same JVM for maximum performance, you need to make sure that the static part of your tests (if any) is correctly initialized.  For example, if you run this test twice in a row remotely:

public class T {
  public static int m_count = 0;

  @Test
  public void f() {
    m_count++;
    assertEquals(1, m_count);
  }
}

... the second time will fail because m_count will have kept its value after the first run.

Instead, you need to move the initialization in a @Configuration method:

public class T {
  public static int m_count = 0;

  @Configuration(beforeSuite = true) // or beforeTest, or after
  public void init() {
    m_count = 0;
  }

  @Test
  public void f() {
    m_count++;
    assertEquals(1, m_count);
  }
}

The current code in CVS implements everything explained above.  You can find an example hosts.properties in test/ and also master.bat and slave.bat that are convenience scripts to launch the various instances of Distributed TestNG.  Please try it and let me know what you think.

Here are some of the open issues that I'd like to get some feedback on:

• Terminology:  master / slaves?  Any other suggestion?  (not a big fan of these names right now)
  
  
• Specify the list of slaves in a .properties file?  In testng.xml?  In another XML file?  Allow for an ISlaveProvider so that hosts can be dynamically added with a plug-in?
  
  
• Slaves can only receive one connection right now, but I am thinking of moving to nio and allow multiple connections so they can be shared by several developers.  The idea is for an entire team to share the same hosts.properties and so potentially, different masters could hit the same slave.
  
  
• Follow-up question:  should multi-connection slaves be sequential or multi-threaded?  Should this be configurable per-slave?
  
  
• A master could be restricted to certain slaves.  All masters share the same properties file but only access a subset of them (would minimize slave-thrashing because of developers on the same team running all the tests at the same time).  How could we specify this?  Regular expression?  ISlaveFilter?
  

--
Cédric

[Eran]

Hi Cedric,

This is indeed a big step for TestNG and is looks very cool!

On 1/1/06, Cédric Beust ♔ <cbe...@google.com > wrote:

Hi everyone,

I am happy to announce the imminent release of a major new feature for TestNG:  Distributed TestNG.

More and more people have asked me for this feature these past months and I finally found some time to take a serious shot at it, and I was quite surprised to see that it came along very well.  Here is how it works.

Please keep in mind that there are many open issues (listed below), and I'm emailing testng-dev first in order to gather some feedback on these issues before announcing it on testng-users, so a lot of this is still in flux, but the good news is:  the implementation works, and it's now just a matter of packaging it nicely.

Overview

The idea is to be able to distribute your tests across several slave machines in order to accelerate the overall execution time.  In order to achieve this, you can now launch TestNG in "slave" mode on a remote machine by specifying a port:

java org.testng.TestNG -slave 5150

At which point, TestNG will just sit and wait for incoming connections.

Once you have launched all the slaves that you need, you declare them in a properties file:

# hosts.properties
testng.hosts=terra:5150 arkonis:5151

In my opinion, this has a drawback of having to maintain this file and keeping it up to date. As it common for computers to be inserted to and removed from the network or have their hostname changed, it would be difficult to keep track of the changes, especially for the target audience of TestNG which is made mostly of developers and not System / IT people (who know of such changes).

A possible solution I can think of is an auto discovering of the slaves. Since it's likely that developers teams are sitting together, they are probably sharing the same class C IP range. Therefore, the master can scan his class C (or do a broadcast) and discover the slaves automatically.
A further enhancement is to specify the IP range to be scanned in the hosts file. Something like

# hosts.properties
testng.hosts.ip.range=192.168.1.1-192.168.1.254, 192.168.2.50-192.168.2.100
testng.hosts.ports=5150-5160 , 5170

It might overload the network a little for a period depending on the IP range and number of possible ports for each host, but this method is more immune to changes as network topology changes are not very common.
The original testng.hosts property can still be set in the properties file as a secondary method but I believe it's mostly suitable for small networks.

And you launch the "master" version of TestNG by passing it this host file:

java org.testng.TestNG -hostfile hosts.properties testng1.xml testng2.xml ...

If you're going to have many suite files (which is probably why you would want to use the distributed testing feature, given the dispatch strategies) it would be a good idea to allow regexp for suite files (testng*.xml).

The tests will then be dispatched randomly to the various hosts until they have all run.  All the results will be collected and presented in the usual HTML format (with the addition that these results will include the remote host they ran on).

Right now, two dispatch strategies are supported:  per-test or per-suite (the default).

By default, each suite (each testng.xml file) will be sent to a remote host in its entirety.  If you need a finer granularity, you can add the following to your hosts.properties:

# hosts.properties
testng.strategy=test

In this case, TestNG will parse all your testng.xml files, collect all the <test> stanzas and they will be sent individually to each remote host and then returned after they have run.

 Why is per-suite the default strategy? Wouldn't per-test increase the "test utilization" or am I missing here something?

Great, what do I need to do to use this?

If you want to take advantage of distribution, there is only one little gotcha you need to be aware of:  your tests need to be static-friendly.

Since the slaves run in the same JVM for maximum performance, you need to make sure that the static part of your tests (if any) is correctly initialized.  For example, if you run this test twice in a row remotely:

public class T {
  public static int m_count = 0;

  @Test
  public void f() {
    m_count++;
    assertEquals(1, m_count);
  }
}

... the second time will fail because m_count will have kept its value after the first run.

Instead, you need to move the initialization in a @Configuration method:

public class T {
  public static int m_count = 0;

  @Configuration(beforeSuite = true) // or beforeTest, or after
  public void init() {
    m_count = 0;
  }

  @Test
  public void f() {
    m_count++;
    assertEquals(1, m_count);
  }
}

The current code in CVS implements everything explained above.  You can find an example hosts.properties in test/ and also master.bat and slave.bat that are convenience scripts to launch the various instances of Distributed TestNG.  Please try it and let me know what you think.

Here are some of the open issues that I'd like to get some feedback on:

• Terminology:  master / slaves?  Any other suggestion?  (not a big fan of these names right now)

I actually think they fit well. Maybe agent <=> slave?

• Specify the list of slaves in a .properties file?  In testng.xml?  In another XML file?  Allow for an ISlaveProvider so that hosts can be dynamically added with a plug-in?

I've already commented above about specifying slaves list. As to where specify it, I prefer the properties file approach but a different XML file can also be appropriate. However, I don't like the idea of specifying it in testng.xml for several reasons:

- Maintenance (every change is multiplied by the number of tesng.xml files).
- Inability to share the same (fairly constant) slaves list among several suites (well, you can copy&paste but...).
- Error prone (when having more than 1 testng.xml).

Of course, it has the advantage of specifying different slave lists for different suites but I'm not sure how useful it is.

Adding hosts dynamically is a good idea but for the near future, it's probably not a priority.

• Slaves can only receive one connection right now, but I am thinking of moving to nio and allow multiple connections so they can be shared by several developers.  The idea is for an entire team to share the same hosts.properties and so potentially, different masters could hit the same slave.

Completely agree.

• Follow-up question:  should multi-connection slaves be sequential or multi-threaded?  Should this be configurable per-slave?

Interesting question.
I would start with sequential slaves to simplify things and add multi-threaded slaves support in the future if there's a demand for it.
As whether to allow it to be configured per slave: configuration can always help optimize performance / resources utilization in specific situations but for the time being I think you can safely ignore it (let the code mature and get a momentum first).

• A master could be restricted to certain slaves.  All masters share the same properties file but only access a subset of them (would minimize slave-thrashing because of developers on the same team running all the tests at the same time).  How could we specify this?  Regular expression?  ISlaveFilter?

I don't think we should specify it at all. The problem here is that developers are not aware of each other so they can't distribute the tests evenly among the slaves (and besides, do you know someone who would limit himself to some slaves instead of using them all...?)

My suggestion is to let TestNG optimize the load in the following way:

Every slave would keep a load rate of itself (based on number of open / established connections, number of threads, memory usage, CPU time, etc.) and update it say every 1 sec. When a master wants to assign slave a test, it polls all the slaves and send the test to the one with the lowest load rate.


What do you think?

Regards,
Eran
 

[Eran]

One more idea I though about (not for any foreseeable release but still might be worth considering while the design is still fresh):
Providing QOS based on role (team leader, developer, SQA, etc.) or priority (urgent, high, normal, low, idle).
This of course will be relevant only when the slaves can handle more than one connection.

Regards,
Eran

[Alexandru Popescu]

I have finally got time to (at least read this). Excellent idea Cedric!
My main concern regarding it is the classloading mechanism. This sure
have in mind the idea that somebody using this testing env will
finally want to use an agent for multiple connections. At this moment,
we should start thinking about classloader hierarchies, moment when we
can solve also the re-entrant problem you are mentioning.
Now about comments:
- names: i would probably go with master and agents
- properties file: for the moment I guess this is enough. Somebody who
will start using it, will provide any drawbacks about using a
properties file vs his expectation
- the problem with multiple-connection per agent is quite tricky (and
i've written above my reasons to see it this way)
- multi-threaded: definitely; i cannot see any good reason to be sequential

And now I am getting the feeling we are starting to build a TestNG
server. Are we or have I missed something?

cheers,

./alex
--
.w( the_mindstorm )p.

On 1/1/06, Cédric Beust ♔ <cbe...@google.com> wrote:

[Cédric Beust ♔]

On 1/5/06, Alexandru Popescu <the.mindstor...@gmail.com> wrote:

I have finally got time to (at least read this). Excellent idea Cedric!
My main concern regarding it is the classloading mechanism. This sure
have in mind the idea that somebody using this testing env will
finally want to use an agent for multiple connections. At this moment,
we should start thinking about classloader hierarchies, moment when we
can solve also the re-entrant problem you are mentioning.

I don't think it's worth it.

First of all, classloaders are tricky.  Very tricky.

Second, I was initially reluctant to modify my tests so I could run them remotely, but I changed my mind.   My tests were not clean to start with and I violated the very rules that I created for @Configuration methods :  initialize your tests in @Configuration methods, nowhere else.

Now about comments:
- names: i would probably go with master and agents
- properties file: for the moment I guess this is enough. Somebody who
will start using it, will provide any drawbacks about using a
properties file vs his expectation

There are other options:

- Slaves could announce themselves by broadcasting
- We could also reverse the connection:  slaves connect to the master when they come up instead of the master knowing about the slaves.  It would make dynamic addition and removal of agents easier to implement but this solution comes with its own drawback as well

- the problem with multiple-connection per agent is quite tricky (and
i've written above my reasons to see it this way)

Why?  It's standard stuff:  listen on a socket, create a server socket by new bind, spawn it in a new thread or store the request in a queue and process it.

- multi-threaded: definitely; i cannot see any good reason to be sequential

Thred thrashing.  If your machine is single processor, running tests sequentially is almost guaranteed to be faster than running them in different threads.

And now I am getting the feeling we are starting to build a TestNG
server. Are we or have I missed something?

That's actually an idea I had floated around a few months ago (remember the web server discussion?). 

--
Cédric

[Alexandru Popescu]

First of all, I need to say that most probably I don't have enough details about this scenario, so
some of my assumptions can be completely wrong.

#: Cédric Beust ♔ changed the world a bit at a time by saying on 1/5/2006 11:24 PM :#

> On 1/5/06, Alexandru Popescu <the.mindstor...@gmail.com> wrote:
>>
>> I have finally got time to (at least read this). Excellent idea Cedric!
>> My main concern regarding it is the classloading mechanism. This sure
>> have in mind the idea that somebody using this testing env will
>> finally want to use an agent for multiple connections. At this moment,
>> we should start thinking about classloader hierarchies, moment when we
>> can solve also the re-entrant problem you are mentioning.
>
>
> I don't think it's worth it.
>
> First of all, classloaders are tricky. Very tricky.
>
> Second, I was initially reluctant to modify my tests so I could run them
> remotely, but I changed my mind. My tests were not clean to start with and
> I violated the very rules that I created for @Configuration methods :
> initialize your tests in @Configuration methods, nowhere else.
>

I was bringing the classloaders into discussion, cause I cannot see another good way to deal with
test redeployment, or different deployments (in a multi-threaded env). In my imagination the agent
would support different deployment and redeployments (see I am talking already of deployments, so I
am already seeing it like a server). What if we would build it upon an embedded Jetty web server?

> Now about comments:
>> - names: i would probably go with master and agents
>> - properties file: for the moment I guess this is enough. Somebody who
>> will start using it, will provide any drawbacks about using a
>> properties file vs his expectation
>
>
> There are other options:
>
> - Slaves could announce themselves by broadcasting
> - We could also reverse the connection: slaves connect to the master when
> they come up instead of the master knowing about the slaves. It would make
> dynamic addition and removal of agents easier to implement but this solution
> comes with its own drawback as well
>

This is a good reason to see what the user really wants, instead of spending time creating something
that is not really usefull. Maybe you can get more details for this part.

> - the problem with multiple-connection per agent is quite tricky (and
>> i've written above my reasons to see it this way)
>
>
> Why? It's standard stuff: listen on a socket, create a server socket by
> new bind, spawn it in a new thread or store the request in a queue and
> process it.
>
> - multi-threaded: definitely; i cannot see any good reason to be sequential
>
>

He he, I was not talking about this simple part, but rather thinking in class loading re-loading.

> Thred thrashing. If your machine is single processor, running tests
> sequentially is almost guaranteed to be faster than running them in
> different threads.
>
> And now I am getting the feeling we are starting to build a TestNG
>> server. Are we or have I missed something?
>
>
> That's actually an idea I had floated around a few months ago (remember the
> web server discussion?).
>
> --
> Cédric
>

cheers,

[Cédric Beust ♔]

On 1/5/06, Alexandru Popescu <the.mindstor...@gmail.com> wrote:

> I don't think it's worth it.
>
> First of all, classloaders are tricky.  Very tricky.
>
> Second, I was initially reluctant to modify my tests so I could run them
> remotely, but I changed my mind.   My tests were not clean to start with and
> I violated the very rules that I created for @Configuration methods :
> initialize your tests in @Configuration methods, nowhere else.
>

I was bringing the classloaders into discussion, cause I cannot see another good way to deal with
test redeployment, or different deployments (in a multi-threaded env). In my imagination the agent
would support different deployment and redeployments (see I am talking already of deployments, so I
am already seeing it like a server).

If the tests are static-friendly, you can use the same agent in the same JVM over and over again.  Right now, I can't think of any test that couldn't be made static-friendy using beforeSuite methods.

Actually, maybe I have an example.

Here is an interesting anecdote from when I was making our tests static-friendly:  once in a while, the Triangle test would fail.  It always worked when not distributed, and sometimes works in distributed and other times failed.  Until I realized that this test was actually spread over two <test> stanzas.  Sure enough, it would work when these two <test> happen to be sent to the same slave but fails when the two <tests> get split.

I need to see if I can rewrite this test to account for that.

--
Cédric

[Alexandru Popescu]

#: Cédric Beust ♔ changed the world a bit at a time by saying on 1/6/2006 12:19 AM :#

But how do you handle reloading of tests? Stop, deploy, restart? What if somebody else is running
his code at that moment?

[Cédric Beust ♔]

On 1/5/06, Alexandru Popescu <the.mindstor...@gmail.com> wrote:

But how do you handle reloading of tests? Stop, deploy, restart? What if somebody else is running
his code at that moment?

Ah ok I (finally) see what you are saying.  What if the classes have changed between two runs and an agent wants to run the latest version?

Good point.

Then we need to do what you say:  either classload, or start a new JVM.   This is not supported right now (remember, it's a very early version) but in the future, I envision agents having the capability to do that.   Not sure which way to go, but right now, it's good enough for the users who requested this change because they already save a lot of time just by distributing the tests, even if each agent is launched in a new JVM each time...

But I agree, this needs to be improved.

--
Cédric

[Alexandru Popescu]

#: Cédric Beust ♔ changed the world a bit at a time by saying on 1/6/2006 12:33 AM :#

Glad we got on the same page :-). After some more thinking, I really think that the small embedded
Jetty would do it quite smooth.

[Cédric Beust ♔]

On 1/5/06, Alexandru Popescu <the.mindstor...@gmail.com> wrote:

Glad we got on the same page :-). After some more thinking, I really think that the small embedded
Jetty would do it quite smooth.

Jetty, really?

It's a servlet container, how would it help with classloading test classes?

Also, not that classloading will require some additions to the protocol, so that an agent can be informed when they need to dump their current classes and load new ones.

--
Cédric

[Alexandru Popescu]

#: Cédric Beust ♔ changed the world a bit at a time by saying on 1/6/2006 12:49 AM :#

> On 1/5/06, Alexandru Popescu <the.mindstor...@gmail.com> wrote:
>>
>>
>> Glad we got on the same page :-). After some more thinking, I really think
>> that the small embedded
>> Jetty would do it quite smooth.
>
>
> Jetty, really?
>
> It's a servlet container, how would it help with classloading test classes?
>

Are you kidding me? Every servlet container has a very good specification for classloading. (even if
some of the implementation failed from time to time to do it correctly ;-) ).

> Also, not that classloading will require some additions to the protocol, so
> that an agent can be informed when they need to dump their current classes
> and load new ones.
>
> --
> Cédric
>

Indeed, the protocol would need some more things, but wouldn't be so difficult to do it.

[Cédric Beust ♔]

On 1/5/06, Alexandru Popescu <the.mindstor...@gmail.com> wrote:

> Jetty, really?
>
> It's a servlet container, how would it help with classloading test classes?
>

Are you kidding me? Every servlet container has a very good specification for classloading. (even if
some of the implementation failed from time to time to do it correctly ;-) ).

Well, sure, as do hundreds of other products :-)

My question is more:  why would we embed a servlet container when all we need is classloading stuff?  Or are you saying it's easy to extract this part from Jetty?

--
Cédric

[Alexandru Popescu]

Probably, because I wouldn't like to re-write such thing :-).

./alex
--
.w( the_mindstorm )p.