How to execute tests on one Vm using one job in Jenkins on different threads?

[Ali Ostadsaraie]

Hi All,

Just wondering, if there is any plugin for jenkins or a technique to run a set of test cases on a single virtual machine in parallel.

In our test framework, we can test 2 websites with 2 languages against any desired environment (e.g. Tes01, Test02, .., production) by running a batch file.

Testing 150 test cases on only one brand(website) and one language takes 4-5 hours. It looks a long time for executing 150 test cases especially when you want to run them against both websites and both languages that will take 4 times more, roughly. 

To speed up, in our fist step we decided run a subset of test suites in parallel against two websites and two languages by creating 4 jenkins jobs. So, instead of running them sequentially, they will be executed in parallel. I'm wondering if I can do this step better using any technique/plugin/configuring.

In second step, we are going assign more VMs which obviously will be helpful. 

Any idea/suggestion/experience on step 1 ?

Thanks,

Ali

[John Smart]

Hi Ali,

You might find this approach useful: https://weblogs.java.net/blog/johnsmart/archive/2011/12/26/faster-web-tests-parallel-batches-thucydides

There is also this approach if you are using JBehave: https://weblogs.java.net/blog/johnsmart/archive/2013/08/18/running-parallel-acceptance-tests-using-jbehave-thucydides-and-bamboo

- John

--
You received this message because you are subscribed to the Google Groups "Thucydides Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.
To post to this group, send an email to thucydid...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--

___________________________________________________
John Smart | Wakaleo Consulting  |  +61 407 247 642
Optimizing your software development process
http://www.wakaleo.com  |  john....@wakaleo.com
___________________________________________________

Check out our upcoming BDD/TDD Master classes, coming 

soon to Sydney, Melbourne, Canberra, Brisbane and Perth

___________________________________________________

[Ali Ostadsaraie]

Thanks for the links. I've used both of them to make it work in my framework.

Just wondering if there is any API that returns the number of test scenarios. Something like storyPaths() that returns number of story files. Distributing test cases among agents/batches is better than distributing story files.

Thanks,
Ali

[Ali Ostadsaraie]

Hi John,

I've noticed Jbehave supports multi-thrading and concurrent execution of story files. Just wondering, if it can be done in thucydides by configuring number of threads instead of batching story files as you've described in your blog. For example, we can set the number of threads to 10. That means if we run 10 story files, each will be executed on one thread concurrently.

Any idea how this can be done in Tucydides?

Thanks,
Ali

[Ali Ostadsaraie]

BTW, since I am not using Junit, I look for multi-threading configuration for jbehave (story files).

[John Smart]

The code in question is in the jbehave-plugin module - pull requests are welcome ;-)

Ali Ostadsaraie wrote:

BTW, since I am not using Junit, I look for multi-threading
configuration for jbehave (story files).


On Fri, Nov 1, 2013 at 10:38 AM, Ali Ostadsaraie
<aostad...@gmail.com <mailto:aostad...@gmail.com>> wrote:

    Hi John,

    I've noticed Jbehave supports multi-thrading and concurrent
    execution of story files. Just wondering, if it can be done in
    thucydides by configuring number of threads instead of batching
    story files as you've described in your blog. For example, we can
    set the number of threads to 10. That means if we run 10 story
    files, each will be executed on one thread concurrently.

    Any idea how this can be done in Tucydides?

    Thanks,
    Ali


    On Wed, Oct 30, 2013 at 1:12 PM, Ali Ostadsaraie

    <aostad...@gmail.com <mailto:aostad...@gmail.com>> wrote:

        Thanks for the links. I've used both of them to make it work
        in my framework.

        Just wondering if there is any API that returns the number of
        test scenarios. Something like storyPaths() that returns
        number of story files. Distributing test cases among
        agents/batches is better than distributing story files.

        Thanks,
        Ali


        On Fri, Oct 25, 2013 at 2:06 PM, John Smart

        <john....@wakaleo.com <mailto:john....@wakaleo.com>> wrote:

            Hi Ali,

            You might find this approach useful:
            https://weblogs.java.net/blog/johnsmart/archive/2011/12/26/faster-web-tests-parallel-batches-thucydides

            There is also this approach if you are using JBehave:
            https://weblogs.java.net/blog/johnsmart/archive/2013/08/18/running-parallel-acceptance-tests-using-jbehave-thucydides-and-bamboo

            - John


            On 26 October 2013 03:18, Ali Ostadsaraie

            <aostad...@gmail.com <mailto:aostad...@gmail.com>>

                <mailto:thucydides-users%2Bunsu...@googlegroups.com>.

                To post to this group, send an email to
                thucydid...@googlegroups.com

                <mailto:thucydid...@googlegroups.com>.

                For more options, visit
                https://groups.google.com/groups/opt_out.




            --
            ___________________________________________________
            John Smart | Wakaleo Consulting  | +61 407 247 642

            <tel:%2B61%20407%20247%20642>

            Optimizing your software development process
            http://www.wakaleo.com  | john....@wakaleo.com

            <mailto:john....@wakaleo.com>___________________________________________________

            Check out our upcoming *BDD/TDD Master classes*
            <http://wakaleo.com/training/testing-and-tdd-for-java-developers>,
            coming
            soon to *Sydney*
            <http://wakaleo.com/training/testing-and-tdd-for-java-developers/public-courses>,
            *Melbourne*
            <http://wakaleo.com/training/testing-and-tdd-for-java-developers/public-courses>,
            *Canberra*
            <http://wakaleo.com/training/testing-and-tdd-for-java-developers/public-courses>,
            *Brisbane*
            <http://wakaleo.com/training/testing-and-tdd-for-java-developers/public-courses>
            and *Perth*
            <http://wakaleo.com/training/testing-and-tdd-for-java-developers/public-courses>
            ___________________________________________________

[Ali Ostadsaraie]

I think this feature is very important for those who want to have real test automation system. It is not possible to sell them the test framework without multi-threading feature. 

A test framework without multi-threading feature may be good for exploring test automation or just for fun but definitely cannot be used in production level with a couple thousands of test cases.

Running multiple test cases on multiple threads makes a huge difference. I know Thucydides supports it in Junit, but many companies prefer Story-files method as it is more user friendly.

Due to importance of this feature, I'm wondering if you and Thucydides community can put multi-threading feature (in story-file method) in high priority. Since the feature is already implemented in Jbehave, you may need only some modification in Thucydides to support this great Jbehave feature. 

Unfortunately, my lack of knowledge in Thucydides code and my basic knowledge in Java doesn't make me a good candidate to work on this task, otherwise I would be more than happy to work on it.

Thanks,

Ali

On Mon, Nov 4, 2013 at 6:40 PM, John Smart <john....@wakaleo.com> wrote:

Yes, that is currently the case - I couldn't figure out how to do it in JBehave at the time of implementation. The JBehave API may have evolved since then.

On 5 November 2013 10:05, Ali Ostadsaraie <aostad...@gmail.com> wrote:

HI John,

I just want to double check that based on you thucydides doesn't support multi-thread feature as suggested in Jbejave. It only supports it for Junit.

Is that true?

Thanks,

Ali

On Sat, Nov 2, 2013 at 10:19 PM, Ali Ostadsaraie <aostad...@gmail.com> wrote:

Hi John,

I'll be more than happy to start working on this issue, however, I should say my knowledge of Java programming language is basic as I used to be a C/C++ programmer. Therefore, I need a little bit help in preparing my development environment for making changes in the code as well as I need to know which source code I start looking at and where to find them. As first step, I,ve created a git account to access the project but did not go further yet. Your help in this regard is much appreciated.

Based on suggestion in Jbehave, I thought all I need to do is changing the pom.xml. I have done that but no luck yet. I think the source code needs to be changed to. This is what I have added to the pom.xml:
            <plugin>
            <groupId>org.jbehave</groupId>
            <artifactId>jbehave-maven-plugin</artifactId>
            <version>${jbehave_core.version}</version>
            <executions>
              <execution>
                <id>embeddable-stories</id>
                <phase>integration-test</phase>
                <configuration>
                  <includes>
                    <include>**/*Stories.java</include>
                  </includes>
                  <ignoreFailureInStories>true</ignoreFailureInStories>
                  <ignoreFailureInView>false</ignoreFailureInView>
                  <threads>5</threads>
                  <executorsClass>org.jbehave.core.embedder.executors.SameThreadExecutors</executorsClass>
                </configuration>
                <goals>
                  <goal>run-stories-as-embeddables</goal>
                </goals>
              </execution>
            </executions>
            <dependencies>
              <dependency>
                <groupId>com.google.guava</groupId>
                <artifactId>guava</artifactId>
                <version>${guava.version}</version>
              </dependency>
            </dependencies>
          </plugin>

To unsubscribe from this group and stop receiving emails from it, send an email to thucydides-use...@googlegroups.com.
To post to this group, send an email to thucydid...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--

___________________________________________________
John Smart | Wakaleo Consulting  |  +61 407 247 642

Optimizing your software development process
http://www.wakaleo.com  |  john....@wakaleo.com

___________________________________________________

[Dez Za]

Absolutely agree.  We're at a point where we need this feature, but may have to remove Thucyddides out of the picture as it is preventing us from leveraging the power of Selenium Grid.

Derek

[John Smart]

I've added the "jbehave.threads" switch to version 0.9.222, but I haven't been able to test it comprehensively yet.

As a related comment, I've found running parallel web tests on a single machine (particularly with firefox) to be a bit unreliable - the more threads you have, the less stable the browser becomes.

Dez Za wrote:

[Ali Ostadsaraie]

Thanks, John. Looking forward to experiment the feature in 0.9.222 version

Cheers,

Ali

[Ali Ostadsaraie]

Just one more question. Since I've had to create multiple jobs in Jenkins and run them in parallel on a slave/node VM, I get multiple test reports from jobs. 

I'm looking for technique to make one report that includes reports of other jobs (aggregate them).

I could not use multi-configuration plugin in jenkins to simulate multi-threading per your suggestion. The reason is that I get error from TFS for trying to map the workspace to multiple jobs. In my case the problem could be having the test run on a slave node, while in your case it is on Jenkins . 

To get rid of the TFS "workspace is already mapped" error, I've had to created multiple free-style independent jobs and trigger them by a no_build job all on the salve machine. It works fine, however, I don't know how to aggregate test results and show them in parent job.

Although this is not related to Thucydides, I've thought somebody in the community may have experienced this and have solution for it.

Thanks,

Ali

 

[Derek]

Thanks for making the change.  I've pulled the latest code from github but can't seem to see any changes related to the 9.222 build, the pom history shows them going from 9.221 to 9.223

We're hoping to get around the browser stability issue by running a Selenium Grid hub on our build machine and running multiple PhantomJS nodes on disparate local servers.  I've got the grid working nicely, just need to run the tests in parallel to get the execution time down to a more manageable size, so developers get feedback as quickly as possible.

Derek

[John Smart]

HI Derek,

That sound's like it could work. The change is in thucydides-jbehave-plugin version 0.9.222 - set the jbehave.threads property.

[Ali Ostadsaraie]

Hi John,

It's not clear to me how to configure the system. Could you please light some shed on this.

Thanks,
Ali

[Ali Ostadsaraie]

What changes are needed in pom.xml, thucydides.properties, or in java code? How to use multi-threading feature?

Thanks.

[Ali Ostadsaraie]

Hi John,

Could you please provide information in how to use the new feature (multi-threading) in Thucydides-jbehave-plugin 0.9.222. I'm kind of stuck and don't know how to use it.

Thanks,
Ali

[John Smart]

Hi Ali,

Just put jbehave.threads=4 (for example) in your thucydides.properties file,
or pass it in as a command-line option (mvn verify -Djbehave.threads=4).

Note that I have made sure this does the same thing as the JBehave multi-threaded example code but I have not idea if it works...

- John.

Ali Ostadsaraie wrote:

Hi John,
Could you please provide information in how to use the new feature
(multi-threading) in Thucydides-jbehave-plugin 0.9.222. I'm kind of
stuck and don't know how to use it.

Thanks,
Ali


On Fri, Nov 8, 2013 at 10:24 PM, Ali Ostadsaraie
<aostad...@gmail.com <mailto:aostad...@gmail.com>> wrote:

    What changes are needed in pom.xml, thucydides.properties, or in
    java code? How to use multi-threading feature?

    Thanks.


    On Fri, Nov 8, 2013 at 6:49 PM, Ali Ostadsaraie

    <aostad...@gmail.com <mailto:aostad...@gmail.com>> wrote:

        Hi John,
        It's not clear to me how to configure the system. Could you
        please light some shed on this.
        Thanks,
        Ali


        On Fri, Nov 8, 2013 at 2:04 PM, Derek <derekd...@gmail.com

                an email to thucydides-use...@__googlegroups.com.

                To post to this group, send an email to

                thucydid...@googlegroups.__com.

                For more options, visit
                https://groups.google.com/__groups/opt_out
                <https://groups.google.com/groups/opt_out>.

            --
            You received this message because you are subscribed to
            the Google Groups "Thucydides Users Group" group.
            To unsubscribe from this group and stop receiving emails
            from it, send an email to
            thucydides-use...@googlegroups.com

            <mailto:thucydides-users%2Bunsu...@googlegroups.com>.

            To post to this group, send an email to
            thucydid...@googlegroups.com

            <mailto:thucydid...@googlegroups.com>.

[Derek]

Hi John,

I've tried using the -Djbehave.threads=4 option and including the setting in the thucydides.properties but it doesn't appear to alter the behaviour.  The output on start-up is as follows:

Processing system properties {}

Using controls EmbedderControls[batch=false,skip=false,generateViewAfterStories=true,ignoreFailureInStories=true,ignoreFailureInView=true,verboseFailures=false,verboseFiltering=false,storyTimeoutInSecs=300,threads=1]

Before storyBeforeStories

Any ideas?

Thanks,

Derek

[Derek]

Hi,

I got to the bottom of this, there is a small bug in ThucydidesReportingRunner.java on line 139, it reads:

int threadCount = environmentVariables.getPropertyAsInteger(ThucydidesJBehaveSystemProperties.JBEHAVE_THREADS, 1);

should be

int threadCount = environmentVariables.getPropertyAsInteger(ThucydidesJBehaveSystemProperties.JBEHAVE_THREADS.getName(), 1);

The work around for the moment is to run the code with the following command line parameter:

-DJBEHAVE_THREADS=4

Cheers,

Derek

On Saturday, October 26, 2013 5:18:03 AM UTC+13, aostad wrote:

[John Smart]

Hi Derek,

Thanks, I spotted the same issue this morning :-). However I am getting inconsistent JBehave behaviour in the JBehave/Thucydides integration tests (basically, it doesn't wait until all of the threads finish before generating the reports, it seems). Are your tests running alright in parallel?

Regards,

John.

Derek wrote:

[John Smart]

An update on this - it turns out that this feature is impossible to implement (I say again, impossible) for Thucydides with JBehave, because of the way JBehave event notification is handled in multi-thread mode. In fact, JBehave sends out notification messages about story events (start story, start scenario, start step, etc) after the tests have finished, not as they happen. As a result, it is impossible for Thucydides to know where the @Step methods are being called. In single thread mode, events are sent as the tests are executed, which results in normal behaviour.

The work-around is just to create JUnitStory tests for each JBehave story and configure surefire or failsafe to run in parallel mode. This approach works well.

- John.

[Derek]

Thanks John.  I'll give that approach a try.  As you say the screenshots in reports are all getting over-ridden and the output is not a true reflection on the tests.

Cheers,

Derek

On Saturday, October 26, 2013 5:18:03 AM UTC+13, aostad wrote:

[m.dolinin]

Hi John and others,

I allready try work-around that you mention. 
Create JUnitStory test for each JBehave story an it is work well.
I make some experiments with maven failsafe plugin configuration and define that works only when use forks for each test.
This is my configuration:

```xml

             <plugin>

                <artifactId>maven-failsafe-plugin</artifactId>

                <version>2.12</version>

                <configuration>

                    <systemPropertyVariables>

                        <webdriver.driver>${webdriver.driver}</webdriver.driver>

                        <maven.test.failure.ignore>true</maven.test.failure.ignore>

                    </systemPropertyVariables>

                    <includes>

                        <include>**/acceptancetest/*.java</include>

                    </includes>

                    <forkMode>perthread</forkMode>

                    <threadCount>4</threadCount>

                </configuration>

                <executions>

                    <execution>

                        <goals>

                            <goal>integration-test</goal>

                            <goal>verify</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

```

Also i create maven plugin for generate JUnitStory tests from each JBehave story file.
Plugin is placed here https://github.com/mdolinin/maven-thucydides-jbehave-plugin
To use it  please add this configuration to your pom.xml

```xml

             <plugin>

                <groupId>net.thucydides.maven.plugin</groupId>

                <artifactId>maven-thucydides-jbehave-plugin</artifactId>

                <version>0.9.223-SNAPSHOT</version>

                <executions>

                    <execution>

                        <goals>

                            <goal>generate-sources</goal>

                        </goals>

                    </execution>

                </executions>

            </plugin>

```

Also you should define in what package put new java classes.
For this use maven properties, below is present my configuration:

```xml

    <properties>

        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>

        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>

        <webdriver.driver>firefox</webdriver.driver>

        <project.junit.stories.package>m.dolinin.acceptancetest</project.junit.stories.package>

    </properties>

```
I hope this will be usefull for someone.

вторник, 12 ноября 2013 г., 12:25:56 UTC+2 пользователь John Smart написал:

[Ali Ostadsaraie]

Hi Michael,

Thanks for detailed explanation.

After applying all changes to my project and running "mvn integration-test" it tries to download it but cannot:
[ERROR] Plugin net.thucydides.maven.plugin:maven-thucydides-jbehave-plugin:0.9.223-SNAPSHOT or one of its dependencies could not be resolved: Could
 not find artifact net.thucydides.maven.plugin:maven-thucydides-jbehave-plugin:jar:0.9.223-SNAPSHOT in ....

Thanks,
Ali

[Mikhail Dolinin]

Hi Ali,

This plugin is created as experiment.

And it is not present in central maven repository.

To use it, you should download source code from https://github.com/mdolinin/maven-thucydides-jbehave-plugin

And than run mvn install to push assembled jar and pom to your local maven repo(or may be corporate artifactory).

If this plugin would be useful for others, then we should put it to thucydides-jbehave project.

But now it is like some experiment.

Parallel execution  with failsafe plugin works, but run not all of tests.

Now i am making some investigation about this.

[Ali Ostadsaraie]

Thanks. I've built it and applied all changes in POM and run the program. However, I don't see any change. I expect to see one java process is running with multiple browsers (1 per thread). What am I missing?

[Михаил Долинин]

Did maven plugin generate Junit story classes?

If yes please share your pom.xml and all parent pom files if they present. I think it could be some configuration issue.

—
Sent from Mailbox for iPhone

[Ali Ostadsaraie]

Hi John,

I'm still struggling with this issue. I have used the plugin that Michail provided (thanks) and generated one junitStory class per story file. I've made changes to POM file based on Michail suggestion too. Unfortunately, couldn't make it work.

Could you please be a little bit more specific and detailed about your suggested work-around. E.g.

Changes in POM ?

Thucydides.properties?

Any logic that need to be implemented (e.g. distributing data/test cases among threads)?

Do we still need Thucydides.batch.number and Thucidydes.batch.count?

...

Thanks,

Ali

On Thu, Nov 14, 2013 at 5:52 AM, m.dolinin <dmo.b...@gmail.com> wrote:

[Ali Ostadsaraie]

Any reply?

[martin f]

Awesome work, the plugin seems to do the trick nicely. I had to upgrade to maven-failsafe-plugin v. 2.16 and use the parameter <forkcount>. Also, the plugin requires <project.junit.stories.package> to be declared, as there is no default value. Other than that this doesn't seem like such an awful strategy for creating parallelization in with jbehave.  

[Mikhail Dolinin]

Thanks Martin for your feedback.

It is easy to add default value for <project.junit.stories.package> .

But i have no idea what value to setup.

Do you have any suggestions?

[martin f]

I think its best to leave out a default and force the user to make a decision on where these generated files should go. This way there are no surprises when files start showing up in your project in places you didn't expect. 

However, if default is nicer than the archetype generates the default acceptancetestsuite.java file in groupid.artifactid.jbehave, so if there is a way to match this then it would be consistent with the thucydides convention. 

[Mik B]

november 15 2013.18:32:19 UTC+1 m.dolinin:

Parallel execution  with failsafe plugin works, but run not all of tests.

Now i am making some investigation about this.

 

 Thanks for the plugin and your blog post about it, m.dolonin! Very useful. 

Did you have any progress on the issue you mention here: "run not all of tests"?

[m.dolinin]

Hi Mike,

Yes fix all known issues. And know you can run all your test in parallel mode 

or some test in fixed order and other in parallel (using "execution" section in maven failsafe configuration).

I create post with example how to do this http://mdolinin.github.io/blog/2014/01/19/thucydides-plus-jbehave-plus-maven-run-tests-in-fixed-order/

вторник, 21 января 2014 г., 21:58:50 UTC+2 пользователь Mik B написал:

[Mik B]

Den tisdagen den 21:e januari 2014 kl. 21:21:58 UTC+1 skrev m.dolinin:

Hi Mike,

Yes fix all known issues. And know you can run all your test in parallel mode 

or some test in fixed order and other in parallel (using "execution" section in maven failsafe configuration).

I create post with example how to do this http://mdolinin.github.io/blog/2014/01/19/thucydides-plus-jbehave-plus-maven-run-tests-in-fixed-order/

That sounds great. However, I couldn't install the current v. 0.9.230-SNAPSHOT of the plugin as there are 4 test failures and 1 skip. I'm trying a previous version instead   

[Mik B]

Den tisdagen den 21:e januari 2014 kl. 22:59:08 UTC+1 skrev Mik B:

Den tisdagen den 21:e januari 2014 kl. 21:21:58 UTC+1 skrev m.dolinin:

Hi Mike,

Yes fix all known issues. And know you can run all your test in parallel mode 

or some test in fixed order and other in parallel (using "execution" section in maven failsafe configuration).

I create post with example how to do this http://mdolinin.github.io/blog/2014/01/19/thucydides-plus-jbehave-plus-maven-run-tests-in-fixed-order/

That sounds great. However, I couldn't install the current v. 0.9.230-SNAPSHOT of the plugin as there are 4 test failures and 1 skip. I'm trying a previous version instead   

Unfortunately v.0.9.229 also displays the same issues. I'll have to make do without this plugin until I have time to investigate. Thanks anyway. I'm keen to see examples of jbehave and thucydides to learn from.

 

[Joey Robonaut]

I don't think it's impossible, just a bit awkward.  You could create a cross-referencing Format whose methods are called-back on the test threads at runtime.  It's registered as a reporter, but you can hook behavior with it.

CrossReference xref = new CrossReference().withThreadSafeDelegateFormat(

        new Format("XREF") {

            

            private final ThreadLocal<Story> curStory = new ThreadLocal<>();

            private final ThreadLocal<String> curScenario = new ThreadLocal<>();

            private final ThreadLocal<Map<String, String>> curTableRow = new ThreadLocal<>();

            

            @Override

            public StoryReporter createStoryReporter(FilePrintStreamFactory factory,

                    StoryReporterBuilder storyReporterBuilder) {

                

                return new NullStoryReporter() {

                    

                    @Override

                    public void beforeStory(Story story, boolean givenStory) {

                        curStory.set(story);

                    }

                    

                    @Override

                    public void beforeScenario(String title) {

                        curScenario.set(title);

                    }

                    

                    @Override

                    public void example(Map<String, String> tableRow) {

                        curTableRow.set(tableRow);

                    }

                };

            }

        });

configuration().useStoryReporterBuilder(

        new StoryReporterBuilder().withCrossReference(xref));