Test execution is slow

[Xerxes]

Hi guys - finally got a chance to try SpecRun....

Despite having a series of problems in getting it to work (which I
won't table here) I eventually got to a situation where I was able to
run SpecRun.exe with the most simple srprofile.

However after testing it, i was sad to see that the runner was quite
slow compared to just a straight NUnit test run. To give you an idea
of run times, here are some stats i've gathered:
* My computer is a 2011 MBP running Parallels 7 with 4 cores and 4GB
ram assigned
* SpecRun reports that there are 773 tests in our Features project
* When running with NUnit, the tests take 1m 33s to execute
* When running with SpecRun (1 thread), the tests take 2m 20s to
execute
* When running with SpecRun (2 threads), the tests take 1m 57s to
execute
* When running with SpecRun (4 threads), the tests take 2m 59s to
execute
* When using 4 threads my CPUs never seem to run hard at all while
running SpecRun. The start of the test run sees ~80% CPU utilisation
but then drops off pretty quickly to about 20%
* These figures are reproducible, so it's not like I have outliers or
anything odd happening during a single test-run, and the results are
similarly sad for other developers on my team.

Parallel execution is our key feature - we're frustrated by long test-
times and will try any way to reduce execution time. Unfortunately it
seems SpecRun is too slow for us to use.

Are there settings I could tweak to speed it up, or any other
variables I haven't accounted for which would show this behaviour? We
don't need the detailed test execution report so if turning it off
could improve performance that would be awesome.

Cheers,
Xerx

[Gáspár Nagy]

Thanks for the feedback.

We haven't seen such case so far, so it is definitely interesting to
look into this deeply. What I see is that you have many, but relative
fast (~0.1s) tests, so maybe this is a case where SpecRun is not tuned
enough yet. We will have a look (maybe contact per mail for further
details) and try to sort it out.

Br,
Gaspar

[Xerxes]

sounds good. You can contact me on this address :)

Cheers,
xerx

[Codeifye]

I have been evaluating SpecRun as we want to speed up our tests suite by running tests in parallel. Though this post is rather old, I am having the exact same problem. Using an Intel i7 core running at 3.6GHz with 8 cores and 16GB RAM, I can run a suite of 2525 specflow tests in just over 4 minutes using NUnit.  Using SpecRun and running 7 threads in parallel the same tests took 14 minutes. Running in one thread the time would have been a great deal longer.  In both cases I ran the tests from within Visual Studio. I did try running SpecRun from the command line but performance did not improve. I am using Specflow 2.1.0, SpecRun.Runner 1.5.2 and SpecRun.Specflow 1.5.2.  

This is disappointing, as we were hoping to get some extra speed boost from running tests in parallel.

[Stephen McCafferty]

Can you maybe tell us a little bit more about your setup? The numbers are somewhat suprising as you point out yourself, so we'd be interested in finding out more about what is going on.

Could you tell us a little bit more about the application you are testing and what kind of tests you are performing? Each of your tests seems to have a pretty short individual execution time (2500 in 4 minutes is roughly 10 per second).

Could you also provide us with a log file and your .srprofile file to take a look at?