Serenity parallel execution

[Sampath Gelli]

Hello does anybody tried parallel execution with serenity bdd. I am using serenity in integration with bamboo. I am using following properties to execute tests in parallel

(using thucydides.batch.size and thucydides.batch.number)?

Are these correct properties to use?does serenity have different properties than these?


Appreciate quick response

Regards
Sam

[Adrian Mitev]

I'm using a combination of Thucydides + JBehave + Maven failsafe plugin to achieve parallel execution. More info can be found here - http://mdolinin.github.io/blog/2014/01/17/thucydides-plus-jbehave-plus-maven-run-tests-in-parallel/

[Divya Jatain]

I am also looking for a solution to execute serenity tests in parallel. Any solution/suggestion will be appreciated.

[Николай Колев]

Same problem here ... 

We just moved from thucydides to serenity. On thucydides everything worked fine and we were running tests in parallel using maven surefire plugin with threadCount, but this doesn't work for serenity. Also the explanation here 

https://weblogs.java.net/blog/johnsmart/archive/2010/07/06/running-junit-tests-parallel-maven

does not work either. Can anyone help with this one? 

[Ilyas Patel]

Is it possible you can share your Pom file or just the part where you have defined the sure fire plugin?

I think we are doing something quite similar but I didn't have any issues moving to serenity.

[Henry Floyd]

I had the same problem.  What goal did you use to execute your tests?

I have found that "mvn clean integration-test -Dit.test=Test1,Test2  -Dwebdriver.remote.url=http://192.168.1.2:4444/wd/hub" would run the tests in parallel.

I did have the test before where i used "mvn clean integration-test -Dit.test=MyTestCodeSuite  -Dwebdriver.remote.url=http://192.168.1.2:4444/wd/hub" where is a file like the following

import net.serenitybdd.jbehave.SerenityStories;

public class CleanUpJobSuite extends SerenityStories {   
    public  CleanUpJobSuite() {   
        findStoriesCalled("Test1.story;Test2.story");
    }
}

This no longer works for me.

Does anyone else have a similar experience?

[Varadharajan Srinivasan]

I am also looking solution for the same problem. Could you please help?

[80Vikram]

I'm also trying to run test cases in parallel on same machine but without any success

Can someone please share details , if they could able to achieve this 

1. any annotation/s need to added to class file ?

I tried solutions mentioned in below blogs but without any success

1. http://mdolinin.github.io/blog/2014/01/17/thucydides-plus-jbehave-plus-maven-run-tests-in-parallel/ 

2. https://weblogs.java.net/blog/johnsmart/archive/2010/07/06/running-junit-tests-parallel-maven

Thanks & Regards,

Vikram

[80Vikram]

Hi John,

can you please clarify equivalent of @RunWith(ParallelRunner.class) in Serenity ?

Thanks in advance.

Regards,

Vikram

[80Vikram]

Hi All,

I could run test cases from single class file parallel with help of below configuration

https://github.com/serenity-bdd/serenity-articles/blob/3d487983afcab658318c1f286c7d2c251349b00b/introduction-to-serenity-with-junit/src/samples/etsy-tester/pom.xml

But still need to find answers to below

1. It open same number of browser as number of @Test I've. Is there a way to ask to open only 2 or 3 browsers ?

2. Post test execution all browsers are in open state. How to close this ?

In case someone know answer please clarify 

Thanks,

Vikram

[80Vikram]

Also during parallel execution screen shots are not getting captured and index.html doesn't show any info.

Any idea what is missing in either pom.xml or test cases file ?

Thanks,

Vikram

[80Vikram]

Hi All,

I again tried parallel execution with latest core version 1.1.3 but there faced below issues

It only creates individual reports under target > site > surefire-reports but under target > site > serenity there is no index.html file which gets generated.

I have test cases spread across individual classes and I don't use stories approach. 

Any help with this will be useful 

<plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-surefire-plugin</artifactId>

                <version>2.18</version>

                <configuration>

                    <parallel>classes</parallel>

                    <threadCount>5</threadCount>

                </configuration>

            </plugin>

Thanks & Regards,

Vikram

[80Vikram]

Hi All,

Quick update with the help of John of course :)

One needs to follow above approach and run mvn serenity:aggregate.

thanks & regards,

Vikram

[Ramu Vairavan]

I see parallel executions mentioned for JUnit and JBehave. Did anyone do parallel execution using Cucumber-JVM with Serenity?

If yes, it will be great if you could shed some insight on that. Thanks in advance!!

[Sampath S]

I am also getting errors in reporting with Serenity 1.1.X

error1: RuntimeException while executing runnable net.thucydides.core.reports.ReportService$2@d1176 with executor com.google.common.util.concurrent.MoreExecutors$DirectExecutorService@1f56650

error2: Error executing FreeMarker template

Tried with serenity:aggregate as well

Can anyone help on this.

[John Smart]

Are you getting a stack dump? Can you confirm that it works with Serenity 1.0.x and doesn't with Serenity 1.1.x? What was the last version that worked?

--
You received this message because you are subscribed to the Google Groups "Serenity BDD Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.
To post to this group, send email to thucydid...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

___________________________________________________
John Smart | Wakaleo Consulting  |  +61 407 247 642
Optimizing your software development process
http://www.wakaleo.com  |  john....@wakaleo.com
___________________________________________________

We love breaking down silos and helping smart teams collaborate better! Ask about our tailored on-site workshops in Agile Requirements Discovery, Behaviour Driven Development, Agile Development Practices, and Test Automation!

___________________________________________________

[John Smart]

I usually just have several test runners and use gradle or the Maven Failsafe plugin to run the test runners in parallel.

--

You received this message because you are subscribed to the Google Groups "Serenity BDD Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.

To post to this group, send email to thucydid...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Padmanaban b v]

Hi,

In my project, I am running parallel execution like this,

1) Created the multiple Test Suite Files with different set of stories.

2) In serenity.properties, I am passing driver capabilities

serenity.driver.capabilities = <If any thing need to pass>

3) In pom.xml, by using maven failesafe plugin 

<plugin>

<artifactId>maven-failsafe-plugin</artifactId>

<version>2.17</version>

<configuration>

<includes>

<include>**/*TestSuite.java</include>

</includes>

<forkCount>6</forkCount>

<reuseForks>true</reuseForks>

</configuration>

<executions>

<execution>

<goals>

<goal>integration-test</goal>

<goal>verify</goal>

</goals>

</execution>

</executions>

</plugin>

4) In maven execution, passing the -Dwebdriver.remote.url=http://<server name or ip>:4444/wd/hub -Dwebdriver.remote.driver=firefox

Note: I am not using Serenity inbuilt property, serenity.batch.size and serenity.batch.number.

Thanks,

Padmanaban

[80Vikram]

Hi All,

I'm running test cases in parallel with below setup

<plugin>

                <groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-failsafe-plugin</artifactId> <!-- http://maven.apache.org/surefire/maven-failsafe-plugin/ -->

                <version>2.18.1</version>

                <configuration>

                    <includes>

                        <include>**/*Test.java</include>

                        <!-- <include>**/Test*.java</include>

                        <include>**/When*.java</include> -->

                    </includes>

                    <!--  <argLine>-Xmx512m</argLine> -->

                    <!--  https://weblogs.java.net/blog/johnsmart/archive/2010/07/06/running-junit-tests-parallel-maven , http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html -->

                    <parallel>methods</parallel>

                    <!--  <threadCount>1</threadCount> -->

                    <threadCountMethods>1</threadCountMethods>

                    <!-- <percorethreadcount>false</percorethreadcount> -->

                    <useunlimitedthreads>false</useunlimitedthreads>

                    <!-- <forkCount>3</forkCount>

    <reuseForks>true</reuseForks> -->

                    <systemPropertyVariables>

                        <webdriver.driver>${webdriver.driver}</webdriver.driver>

                    </systemPropertyVariables>

                </configuration>

                <executions>

                    <execution>

                        <goals>

                            <goal>integration-test</goal>

                            <goal>verify</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

But it spawns more than 10+ browsers , how can I control to spawn only X number of browsers ?   I have single class with multiple test cases. I tried reading https://maven.apache.org/surefire/maven-failsafe-plugin/examples/fork-options-and-parallel-execution.html but couldn't find the answers

Thanks in advance.

Regards,

Vikram

[80Vikram]

Hi All,

I found out that JUnit + maven-failsafe-plugin doesn't really work well wrt parallelization. 

Either I should be using custom test runner or TestNG.

Anybody else found solution kindly share

Thanks & Regards,

Vikram

[80Vikram]

Hi All,

Please find below updates on this thread

1>      I could get parallel execution successfully implemented with below setup

<groupId>org.apache.maven.plugins</groupId>

                <artifactId>maven-surefire-plugin</artifactId> <!-- http://maven.apache.org/surefire/maven-surefire-plugin/ -->

                <version>2.18.1</version>

                <configuration>

                    <parallel>classes</parallel>

                    <threadCount>1</threadCount>

With this setup for each java file one browser will be launched.

But problem with this approach is there are test cases failures which won't happen with single m/c + single browser approach.

2> For now I have kept aside parallel approach and look for Grid to speed up automation execution.

Any other ideas for faster automation scripts executions are welcome other than cloud based service usage like Saucelabs / browserstack etc.

Thanks in advance.

Regards,

Vikram

[John Smart]

I find the problem with parallel tests is that it massively increases the risk of random Selenium/Browser issues. SauceLabs or BrowserStack is OK but they are slow, so you don't get much benefit if you don't have 10 or more parallel threads.

--

You received this message because you are subscribed to the Google Groups "Serenity BDD Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.
To post to this group, send email to thucydid...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[80Vikram]

Hi John,

Thanks for insights.

Per you; what will be best approach to speed up test cases ?

With Serenity; can I use Grid approach to distribute test cases ?

Thanks & Regards,
Vikram

[Harry King]

Hi Vikram,

I run my tests via Jenkins.  We have a few VMs setup with different versions of IE & Windows which I have the Jenkins agent installed on.  I've taken the poor man's multi threading approach:

• The Acceptance Test Suite is set up to take the story or folder as an argument (via a System Property).
• I've created a multi configuration job (which allows you to run the same job against several environments).
• I have a user defined axis which contains each story I want to run.
• A second job is triggered afterwards which combines the report data and generates the HTML.

There's currently a bit of manual overhead when a new story is written, but it got us executing our tests in parallel.  If you're interested, maybe I'll write a company blog post to demonstrate our setup.

As to Grid, I've found it to be too much trouble currently for our use cases.  Grid on its own doesn't get you parallel execution, its slower, and we need to be able to upload files, which means storing created files on a universally accessible network share.

Kind regards,

Harry

[Ilyas Patel]

Hi Harry

Would be great if you could write up your setup in a blog post. I've been looking at the multi-configuration jobs in Jenkins recently but can;t get my head around setting up the second job to create the final report.

 

[80Vikram]

Hi All,

I'm using below approach for parallel execution on grid

<plugin>
                    <groupId>org.apache.maven.

plugins</groupId>
                    <artifactId>maven-surefire-plugin</artifactId> <!-- http://maven.apache.org/surefire/maven-surefire-plugin/ -->
                    <version>2.18.1</version>
                    <configuration>
                        <parallel>classes</parallel>
                        <threadCount>1</threadCount>

                        <skip>${skip.parallel}</skip>
                        <!-- 1> If skip=true then parallel will not work  2>  What is this flag ?  -->
                        <!--  for exact search in google user "<skip>true<skip>"-->
                        <!-- https://issues.apache.org/jira/browse/SUREFIRE-319 -->
                        <!-- https://maven.apache.org/plugins/maven-deploy-plugin/faq.html Answer skip meaning -->
                        <!--  https://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html -->
                    </configuration>
            </plugin>

but post execution , I don't see index.html getting generated even though I've below in pom.xml

<plugin>
                    <groupId>net.serenity-bdd.

maven.plugins</groupId>
                    <artifactId>serenity-maven-plugin</artifactId> <!-- http://wakaleo.com/thucydides-one-page/_thucydides_reporting.html -->
                    <version>${serenity.maven.version}</version>         <!-- https://github.com/thucydides-webtests/thucydides/wiki/Getting-Started -->
                    <executions>
                        <execution>
                            <id>serenity-reports</id>
                            <phase>post-integration-test</phase>
                            <goals>
                                <goal>aggregate</goal>
                            </goals>
                        </execution>
                    </executions>
            </plugin>

Am I doing something wrong here ?

Each time , I had to execute mvn serenity:aggregate explicitly to generate index.html

thanks in advance.

Regards,
Vikram

[Yek]

Hi Ramu, did you find solution for your cucumber JVM parallel execution in Serenity BDD? If so could you please share the approach?

Thanks & Regards,
Yek.

[Tony Tubbs]

I am using maven-failsafe-plugin to run Serenity and Cucumber tests in parallel if that would help.

Sample app is here: https://github.com/tbbstny/SerenityCucumberStudy

[Ashish Uniyal]

Hi @Sampath

Can you please help me how to integrate Serenity with Bamboo?

Can you please let me know the steps.

Thanks

Ashish Uniyal

[80Vikram]

Hi John,

I'm trying to optimize parallel execution ( distribution of all test cases to available nodes ),  currently I'm using same approach but it has below issues

1. In case a node gets freed, it doesn't get assigned new job
2. In case a node crashes; its test class/ cases are not assigned to another node/s

did you face these issues and how do you handle it ?

Thanks & Regards,
Vikram