How to increase thread count via maven-surefire-plugin?

[Jeff]

I thought I had this worked out but I can't seem to get TestNG to run more than 10 threads.

Here is my test method:

@Test(

     dataProvider = "myDataProvider",

     threadPoolSize = 30

    )

public void test_BulkProcessing_SingleRequest(Map<String, String> requestInfo) throws HttpStatusException, IOException {

     System.out.println(

         String.format(

            "[tId:%d] Request: %s", 

            Thread.currentThread().getId(),

            request_id

         )

      );

     ...

}

In my POM file:

<build>

    <plugins>

        <plugin>

            <groupId>org.apache.maven.plugins</groupId>

            <artifactId>maven-surefire-plugin</artifactId>

            <configuration>

                <parallel>methods</parallel>

                <threadCount>30</threadCount>

            </configuration>

        </plugin>

    </plugins>

</build>

It doesn't seem to honor But it only runs 10 threads.  I can't see what else I'm missing and I don't want to use an XML file if I can help it since all the rest of the test groupings/dependencies are controlled via the annotations.

What else would I look for?

Thanks!

--

Jeff Vincent
See my LinkedIn profile at:
http://www.linkedin.com/in/rjeffreyvincent

[Jeff]

As a next attempt, I tried getting the XmlSuite object and doing a setThreadCount(30) as part of the @BeforeClass method as such:

 public void setup(ITestContext ctx) throws InitializationException {

        suite = ctx.getSuite().getXmlSuite();

        suite.setThreadCount(30);

        ...

   }

But that didn't help.  I then created a SuiteListener and in the onStart() method:

@Override

    public void onStart(ISuite suite) {

        XmlSuite mySuite = suite.getXmlSuite();

        mySuite.setThreadCount(30);

    }

Still nothing.  I'm not sure where it is getting the thread count of 10 as it is.  The value  before I set it to 30 is '1', so it clearly hasn't used the values from the POM nor the 'threadPoolSize' in the annotation.  I don't understand why the annotation is not having an effect.

What other options are there?

[Jeff]

FWIW, I have a very simple sample project that illustrates what I'm seeing if anyone wants to take a gander and tell me what I'm doing wrong.

Also FWIW, I saw a thread that said that if the annotation parameter 'threadPoolSize' is used, then 'invocationCount' must also be included.

When I did the following:

     @Test(dataProvider = "dataProvider", threadPoolSize = 50, invocationCount = 1)

    

I got 10 unique threads...again.  When I did:

     @Test(dataProvider = "dataProvider", threadPoolSize = 50, invocationCount = 10)

I saw 100 unique threads.  My gut feeling (unless I am misunderstanding) is that threadPoolSize is not being honored although I don't understand why increasing the invocationCount would also increase the number of unique threadId's I see by the same factor.

Sample attached...

[Cédric Beust ♔]

It's possible that the flag is not being passed to Surefire, someone recently fixed a similar bug (different flag though).

-- 

Cédric

--
You received this message because you are subscribed to the Google Groups "testng-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to testng-users...@googlegroups.com.
To post to this group, send email to testng...@googlegroups.com.
Visit this group at http://groups.google.com/group/testng-users.
For more options, visit https://groups.google.com/d/optout.

[Jeff]

Are you talking about the @Test 'threadPoolSize' value being sent to surefire or the threadCount from the POM?

Should either one work the same?

[Jeff]

I entered an ISSUE against surefire, but they say they tried it a different way (see the link below) and thinks there is an issue with TestNG. 

     https://issues.apache.org/jira/browse/SUREFIRE-1179

I made an attempt to run directly from the command-line without Maven to verify.  I modified my test project to generate a JAR with Dependencies (didn't include test classes though).  I then ran it as below from the <project>\target\ folder:

     java -classpath testng-test-1.0-SNAPSHOT-jar-with-dependencies.jar;test-classes org.testng.TestNG -testclass debug.ParallelTest.class

The method is fed by a parallel dataProvider that generates 10,000 iteration ID's.  The class uses a ConcurrentMap<Long, Long> to track the number of times each thread runs the test method.  The method has the annotation:

    @Test(dataProvider = "dataProvider", threadPoolSize = 30, invocationCount = 1)

When it runs, you can see the threads are running as expected, with interleaved results between threads as expected.

...

[tId:10] ParallelTest.testParallelDataProvider(ID_9981)

[tId:18] ParallelTest.testParallelDataProvider(ID_9980)

[tId:12] ParallelTest.testParallelDataProvider(ID_9979)

[tId:11] ParallelTest.testParallelDataProvider(ID_9978)

[tId:19] ParallelTest.testParallelDataProvider(ID_9977)

[tId:17] ParallelTest.testParallelDataProvider(ID_9976)

[tId:13] ParallelTest.testParallelDataProvider(ID_9975)

[tId:16] ParallelTest.testParallelDataProvider(ID_9974)

[tId:13] ParallelTest.testParallelDataProvider(ID_9991)

[tId:17] ParallelTest.testParallelDataProvider(ID_9990)

[tId:19] ParallelTest.testParallelDataProvider(ID_9989)

[tId:11] ParallelTest.testParallelDataProvider(ID_9988)

[tId:12] ParallelTest.testParallelDataProvider(ID_9987)

[tId:18] ParallelTest.testParallelDataProvider(ID_9986)

[tId:10] ParallelTest.testParallelDataProvider(ID_9985)

[tId:15] ParallelTest.testParallelDataProvider(ID_9984)

[tId:14] ParallelTest.testParallelDataProvider(ID_9983)

[tId:10] ParallelTest.testParallelDataProvider(ID_9999)

[tId:18] ParallelTest.testParallelDataProvider(ID_9998)

[tId:12] ParallelTest.testParallelDataProvider(ID_9997)

[tId:11] ParallelTest.testParallelDataProvider(ID_9996)

[tId:19] ParallelTest.testParallelDataProvider(ID_9995)

[tId:17] ParallelTest.testParallelDataProvider(ID_9994)

[tId:13] ParallelTest.testParallelDataProvider(ID_9993)

[tId:16] ParallelTest.testParallelDataProvider(ID_9992)

...

Assuming I did it properly the final stats are:

Total threads: 10

  Thread[16] : 700 iterations

  Thread[17] : 1100 iterations

  Thread[18] : 1129 iterations

  Thread[19] : 1127 iterations

  Thread[10] : 1064 iterations

  Thread[11] : 1025 iterations

  Thread[12] : 984 iterations

  Thread[13] : 640 iterations

  Thread[14] : 1097 iterations

  Thread[15] : 1134 iterations

Total iterations: 10000

I then added the -threadcount option to the TestNG command-line:

     java -classpath testng-test-1.0-SNAPSHOT-jar-with-dependencies.jar;test-classes org.testng.TestNG -testclass debug.ParallelTest.class -threadcount 30

In the end result is the same:

Total threads: 10

  Thread[16] : 591 iterations

  Thread[17] : 1145 iterations

  Thread[18] : 1109 iterations

  Thread[19] : 1079 iterations

  Thread[10] : 1095 iterations

  Thread[11] : 1101 iterations

  Thread[12] : 1106 iterations

  Thread[13] : 1120 iterations

  Thread[14] : 887 iterations

  Thread[15] : 767 iterations

Total iterations: 10000

Thoughts?

[Jeff]

Okay,  I just noticed via that command-line that there is an option called 'dataproviderthreadcount'.  This seems to be one of the missing links.  This is never discussed in ANY examples related to running tests in parallel.

When I changed this to 30 and got 30 test threads as expected.

So more questions:

• Why doesn't the 'threadPoolSize' apply to tests using parallel data providers?  
• Is there a different annotation parameter I'm missing for configuring the dataprovider thread pool/count?
• Should I be able to set the 'dataproviderthreadcount' via a setting in the Maven POM?

Thanks.

[Cédric Beust ♔]

Correct, data providers use a different thread pool, sorry for not catching this earlier.

-- 

Cédric

[Jeff]

Thanks for the response.  Would it be possible to add a parameter on the @DataProvider annotation to allow setting the dataProviderThreadCount?  

FWIW, the documentation is *very* unclear that DataProviders are different in how they run in parallel and I've burned a couple of days working through this :/.

For the sake of completeness, I've verified that I can programatically change the dataproviderthreadcount in the XmlSuite using a SuiteListener as well as via the suite from the ITestContext in my @BeforeClass method.

SuiteListener Example:

    @Override

    public void onStart(ISuite suite) {

        XmlSuite mySuite = suite.getXmlSuite();

        mySuite.setDataProviderThreadCount(30);

    }

ITestContext Example:

    @BeforeClass

    public void setup(ITestContext ctx)  {

      XmlSuite suite = ctx.getSuite().getXmlSuite();

      suite.setDataProviderThreadCount(20);

    }

[Cédric Beust ♔]

I think setting the thread count at the @DataProvider annotation level is a bit overkill and it's fine to let all data providers share one thread pool.

Note also that this is documented:

-dataproviderthreadcount

The default number of threads to use for data providers when running tests in parallel.

This sets the default maximum number of threads to use for data providers when running tests in parallel. It will only take effect if the parallel mode has been selected (for example, with the -parallel option). This can be overridden in the suite definition.

-- 

Cédric

[Jeff]

It is not sharing a pool, it seems to be creating a new pool for each @DataProvider. 

I have a test class with 3 data providers.  When I set the dataproviderthreadcount to 20 and ran it, I get 60 total threads.

[Cédric Beust ♔]

Ah yes, that could be.

-- 

Cédric