Substeps thoughts for parallel test execution?

[Keith Hayes]

Having used Substeps now for over a year on a project with rapid growth, our dev team has been in a constant fight to keep the CI build time down. I'm sure this is due to a number of issues in our implementation of Substeps and maybe down to mistakes in our testing strategy along the way, but whatever the cause, we're in a situation where we need to carefully balance the tests we're creating, aiming to minimise the increase in build time while not reducing test quality. 

I was thinking that it'd be really handy if we could run Substeps in parallel (Possibly similar to Selenium Grid). This would then introduce a scalable solution to our problem.

Is there are plans for a parallel framework within Substeps? 

What issues do you foresee with implementing one?

 

[Stu]

Hi Keith,

Parallel execution of features is actually something we looked at very early on in Substeps development - perhaps a little too early. At the time, the Substeps Maven plugin hadn't been developed, and so we were using the Surefire plugin, which has parallel execution capabilities (http://maven.apache.org/surefire/maven-surefire-plugin/examples/fork-options-and-parallel-execution.html).

Unfortunately, this had some serious issues for us, the most prominent being that Surefire wasn't failing the build when tests failed. These false-positives caused us to abandon the idea of parallel execution at the time, and the idea hasn't been re-visited since.

In general, I have no problem with the idea of introducing parallel execution to the Substeps framework. However, there are a number of challenges that will arise from this. For example, how do you mitigate race conditions where once scenario tests the rendering of a particular piece of data on the page, and another tests its deletion?

Additionally, there are a number of technical considerations that need to be taken into account. There are reports of some WebDriver implementations that are not thread safe for example (although I am yet to see this).

Parallel execution would certainly be a great feature, but it's safe to say we've a fair way to go yet. While it's not at the top of our priorities right now, we're always pleased to receive contributions if you'd like to help out.