Running Serenity tests in parallel batches not working

[Strugariu Florin]

Tried to split tests in batches fallowing the examples provided in http://thucydides.info/docs/serenity-staging/#_running_serenity_tests_in_parallel_batches

The runner is not splitting the in separate batches.

Using:

serenity.batch.strategy=DIVIDE_BY_TEST_COUNT

serenity.batch.number=1

serenity.batch.size=2


We get an empty test run:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
[main] INFO net.thucydides.core.util.PropertiesFileLocalPreferences - serenity.batch.number=1
[main] INFO net.thucydides.core.util.PropertiesFileLocalPreferences - serenity.batch.strategy=DIVIDE_BY_TEST_COUNT
[main] INFO net.thucydides.core.util.PropertiesFileLocalPreferences - serenity.batch.size=2
Running net.serenity.samples.batch.junit.features.registration.LoginUserTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - in net.serenity.samples.batch.junit.features.registration.LoginUserTest
Running net.serenity.samples.batch.junit.features.registration.RegisterUserTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in net.serenity.samples.batch.junit.features.registration.RegisterUserTest
Running net.serenity.samples.batch.junit.features.registration.UserActivationProcessTest
Tests run: 0, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec - in net.serenity.samples.batch.junit.features.registration.UserActivationProcessTest

Results :

Tests run: 0, Failures: 0, Errors: 0, Skipped: 0

[INFO]
[INFO] --- serenity-maven-plugin:1.1.42:aggregate (serenity-reports) @ serenity-junit-webtests ---
[INFO] serenity.batch.number=1
[INFO] serenity.batch.strategy=DIVIDE_BY_TEST_COUNT
[INFO] serenity.batch.size=2
[INFO] current_project.base.dir: D:\web_ser_try\serenity-demos-master\junit-batch-configuration-example
Generating HTML Story Reports from D:\web_ser_try\serenity-demos-master\junit-batch-configuration-example\target\site\serenity
Generating HTML Story Reports to D:\web_ser_try\serenity-demos-master\junit-batch-configuration-example\target\site\serenity
[WARNING] To generate correct requirements coverage reports you need to set the 'serenity.test.root' property to the package representing the top of your requirements hierarchy.
GENERATE CUSTOM REPORTS
[INFO]
[INFO] --- maven-failsafe-plugin:2.18.1:verify (default) @ serenity-junit-webtests ---
[INFO] Failsafe report directory: D:\web_ser_try\serenity-demos-master\junit-batch-configuration-example\target\failsafe-reports
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------


If we use:
serenity.batch.strategy=DIVIDE_BY_TEST_COUNT

serenity.batch.number=2

serenity.batch.size=2

We run all the tests:  https://pastebin.mozilla.org/8920127

Is there something wrong with batch testing? Am I doing something wrong?

Here is the project: https://github.com/bebef1987/serenity_batches

Thanks,
Bebe

[adi....@outlook.com]

Hi Florin,

Looking at the documentation here: https://github.com/serenity-bdd/serenity-documentation/blob/master/src/asciidoc/system-props.adoc only the serenity.batch.count variable is being documented and not the batch.size one. Maybe try with that. The thucydides.info link seems old and in the jenkins setup displays the properties as thucydides.batch.strategy not serenity.batch.strategy and so on. Also you might want to try to send the props as command line props: -Dserenity.batch.count=4 -Dserenity.batch.number=1. I have tried it once on an older serenity version but gave up quickly with this and moved to another solution - see below.

Another solution you might like to use is using the meta.filter option. Setting a meta tag in the test file as "@suite suite1" and then running it with the -Dmeta.filter="+suite suite1" option and managing yourself how the tests are being split.

Thanks,

Adrian

[Strugariu Florin]

Tested with -Dserenity.batch.count=4 -Dserenity.batch.number=1 from the command line:

-Dserenity.batch.count=4 -Dserenity.batch.number=1 => No tests run
-Dserenity.batch.count=4 -Dserenity.batch.number=2 => all tests are run
-Dserenity.batch.count=4 -Dserenity.batch.number=3 => no tests run
-Dserenity.batch.count=4 -Dserenity.batch.number=4 => no tests run

I will investigate you proposed  solution but I would expect this to be handled automatically by the test runner.

Florin

[Liviu Carausu]

Hi Florin,

For the moment try to completely remove the maven-surefire-plugin from your pom.xml and run your batches again, let me know how it goes please.

[Gerhard Schuster]

Hi Florin,

in case this is still an issue for you.

Using batch files is a very clumsy way to paralelize test in my oppinion.

I want to have a dynamic test load balancing.

I use Serenity with cucumber feature files.

I use the maven-failsafe-plugin to do a dynamic test parallelisation.

a) My POM.xml contains this

            <plugin>

                <artifactId>maven-failsafe-plugin</artifactId>

                <version>2.18.1</version>

                <configuration>

                    <includes>

                        <include>**/*.java</include>

                    </includes>

                    <parallel>classes</parallel>

                    <threadCount>20</threadCount>

                    <argLine>-Xmx2048m</argLine>

                    <systemPropertyVariables>

                        <webdriver.driver>${webdriver.driver}</webdriver.driver>

                    </systemPropertyVariables>

                </configuration>

                <executions>

                    <execution>

                        <goals>

                            <goal>integration-test</goal>

                            <goal>verify</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

b) For each feature file I hava a test runner class

@RunWith(CucumberWithSerenity.class)

@CucumberOptions(features = "src/test/resources/features/myTests1")

public class MyTestSuite1 {

}

c) I start the Tests via

mvn clean verify

d) I've set up a Selenium Grid with 6 nodes, running 5 Browsers each

[kxramas]

Hi Gerhard,

Does your Serenity report show correct number of tests each time with this parallel approach ?

I am using Serenity and JBehave. I have configured multiple runners and using maven-failsafe-plugin to run them in parallel. However, with same set of tests Serenity report is not consistent. Out of 6 scenarios it sometimes shows 3 scenarios executed and sometimes all 6. Not sure what is wrong.

Appreciate your help!

Thanks.