Re: [Cucumber] Cucumber JVM Parallel testing

653 views
Skip to first unread message

aslak hellesoy

unread,
Dec 10, 2012, 11:28:47 AM12/10/12
to Cucumber Users



On Fri, Dec 7, 2012 at 9:31 PM, snomys <snom...@gmail.com> wrote:
Hi Aslak


If you address the group more people might be inclined to answer. I haven't tried to run scenarios in parallel yet, but I know a few others have. There are also a few tickets on github about this.

Aslak
 
I met you briefly at the Skills Matter presentation earlier this year and have adopted the use of Cucumber JVM with Java.

I have been really struggling getting the parallelisation of the tests in my framework to run, so I downloaded the example parrallel-java and this seems to work as expected with surefire version: 2.12.4

The output as such:
Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
given0.0
given0.0
when5.0
when1.0
then1.0
then5.0
given5.0
given15.0
when1.0
when1.0
then16.0
then6.0

However when I update to version 2.4.3 I get completely different results, following presumably a sequential run:
Running cucumber.examples.java.parallel.PersonalGrowth1Test
given0.0
when1.0
then1.0
given5.0
when1.0
then6.0

Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.157 sec
Running cucumber.examples.java.parallel.StandardTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec <<< FAILURE!
Running cucumber.examples.java.parallel.PersonalGrowth2Test
given0.0
when5.0
then5.0
given15.0
when1.0
then16.0

Also notice the junit Tests run ending comes out twice for just a single test run.

Of course Ive spent the day googling, etc etc, our framework is more complex and opens browsers, instantiates configurations etc, so would ideally open 2 browsers and execute the tests in parallel.

The varying result of changing the surefire version in the pom worries me. I cant be sure either of the routines above are parallel, but presumably the 2.12.4 version would be. 

Do you have any advice or pointers to help me on, my plan would be to first emulate the simple printing out of numbers in my own framework, and then gradually extend on this to open browsers and do the setups etc, because I just want to ensure the tests run in parallel before the bells and whistles are applied.

One of the constant issues I get when trying to apply the same pom etc in the examples into my own project, and using version 2.12.4 of surefire is this:

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Concurrency config is parallel='classes', perCoreThreadCount=true, threadCount=2, useUnlimitedThreads=false
Scenario: Register successfullyorg.picocontainer.PicoCompositionException: Duplicate Keys not allowed. Duplicate for 'class com.qa.chimera.cucumber.junit.registration.RegistrationSteps'
at org.picocontainer.DefaultPicoContainer.addAdapterInternal(DefaultPicoContainer.java:381)
at org.picocontainer.DefaultPicoContainer.addComponent(DefaultPicoContainer.java:490)
at org.picocontainer.DefaultPicoContainer.addComponent(DefaultPicoContainer.java:443)
at org.picocontainer.DefaultPicoContainer.addComponent(DefaultPicoContainer.java:430)
at cucumber.runtime.java.picocontainer.PicoFactory.createInstances(PicoFactory.java:17)
at cucumber.runtime.java.JavaBackend.buildWorld(JavaBackend.java:41)
at cucumber.runtime.Runtime.buildBackendWorlds(Runtime.java:124)
at cucumber.runtime.World.buildBackendWorldsAndRunBeforeHooks(World.java:34)
at cucumber.runtime.model.CucumberScenario.buildWorldAndRunBeforeHooks(CucumberScenario.java:30)
at cucumber.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:51)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at cucumber.junit.Cucumber.run(Cucumber.java:111)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Scenario: Deposit successfullyjava.lang.IndexOutOfBoundsException: Index: 0, Size: 0
at java.util.ArrayList.RangeCheck(ArrayList.java:547)
at java.util.ArrayList.remove(ArrayList.java:387)
at cucumber.junit.JUnitReporter.match(JUnitReporter.java:32)
at cucumber.runtime.World.runStep(World.java:88)
at cucumber.runtime.model.CucumberScenario.runStep(CucumberScenario.java:72)
at cucumber.junit.ExecutionUnitRunner.runChild(ExecutionUnitRunner.java:78)
at cucumber.junit.ExecutionUnitRunner.runChild(ExecutionUnitRunner.java:18)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at cucumber.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:68)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at cucumber.junit.Cucumber.run(Cucumber.java:111)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)
Scenario: Deposit successfully 2java.lang.IllegalStateException: Cannot stop.  Current container state was: DISPOSED
at org.picocontainer.lifecycle.DefaultLifecycleState.stopping(DefaultLifecycleState.java:72)
at org.picocontainer.DefaultPicoContainer.stop(DefaultPicoContainer.java:699)
at cucumber.runtime.java.picocontainer.PicoFactory.disposeInstances(PicoFactory.java:23)
at cucumber.runtime.java.JavaBackend.disposeWorld(JavaBackend.java:46)
at cucumber.runtime.Runtime.disposeBackendWorlds(Runtime.java:130)
at cucumber.runtime.World.runAfterHooksAndDisposeBackendWorlds(World.java:63)
at cucumber.runtime.model.CucumberScenario.runAfterHooksAndDisposeWorld(CucumberScenario.java:76)
at cucumber.junit.ExecutionUnitRunner.run(ExecutionUnitRunner.java:70)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222)
at org.junit.runners.ParentRunner.run(ParentRunner.java:300)
at cucumber.junit.Cucumber.run(Cucumber.java:111)
at org.junit.runners.Suite.runChild(Suite.java:128)
at org.junit.runners.Suite.runChild(Suite.java:24)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:680)


I notice that in the examples, all Tests and Steps are in the same package, but our are broken into 2, 1 for registration and 1 for depositing. Would this matter?

I saw something when traversing the various postings online, that cucumber scenarios need to give a fail, to allow junit to do its concurrent reporting, but I didnt actually understand if this was first, true, and secondly, how to apply it in terms of the codebase. Would that require a change to the cucumber.java file, and if so what?

I would have posted this question onto a forum, but I was unsure where to do so, but if you would prefer, please point me to the best place to post it, and I will happily do so.

Im not sure what you will make of this question, today has been pretty intense trying to get to grips with it and understanding how it works, so if you need any further information please dont hesitate to write back and ask, and Ill update you accordingly.

Any help on this very much appreciated. Thanks in advance

--
-- Rules --
 
1) Please prefix the subject with [Ruby], [JVM] or [JS].
2) Please use interleaved answers http://en.wikipedia.org/wiki/Posting_style#Interleaved_style
3) If you have a question, don't reply to an existing message. Start a new topic instead.
 
You received this message because you are subscribed to the Google Groups Cukes group. To post to this group, send email to cu...@googlegroups.com. To unsubscribe from this group, send email to cukes+un...@googlegroups.com. For more options, visit this group at https://groups.google.com/d/forum/cukes?hl=en
 
 

Reply all
Reply to author
Forward
0 new messages