Re: [nsubstitute] Performance compared with Rhino?

[David Tchepak]

Hi Ben,

Thanks a lot for raising this. I'm wondering if a specific feature addition has caused NSub to slow down over all. Are you just using basic Returns() and Received() for these tests? Or doing a lot of stuff with When..Do?

It would be great if you could run a few experiments on your test suite:

* Benchmark any tests you convert before and after, and send through an example of a few with the biggest slow down.

* Profile a test run and see if there is an obvious place the hold up is

There are quite a few contributions queued up so I'll need to do a release soon, but straight after that I'll start looking at performance again and see what we can do for subsequent releases.

Cheers,

David

On Fri, Jan 11, 2013 at 5:09 AM, Ben Moxon <glen...@gmail.com> wrote:

I'm looking at NSubstitute for a fairly large project I am working on where currently we have used Rhino for our mocking framework. Converting tests over to NSubstitute is fairly easy and the code is much easier to read and understand ( not that we are planning to refactor them all- we just need to compare the frameworks ) but I am noticing that the test runs are taking a good deal longer - usually twice as long, sometimes several orders of magnitude. Although this is not a massive deal when the time per test is still well under a second, across thousands of tests it is going to start to make a difference and another part of the current evaluation we are doing on the project is trying to reduce our build time- slower tests would impede this goal.

Can anyone offer any performance tips for NSubstitute that might help the tests run a little quicker? I'd certainly be happy to move over to it in terms of writing and maintaining tests, but I am a little anxious about these test durations.

--
You received this message because you are subscribed to the Google Groups "NSubstitute" group.
To view this discussion on the web visit https://groups.google.com/d/msg/nsubstitute/-/qweqTHrIQtAJ.
To post to this group, send email to nsubs...@googlegroups.com.
To unsubscribe from this group, send email to nsubstitute...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/nsubstitute?hl=en.

[Ben Moxon]

They are fairly simple tests- I am actually creating copies of existing ones at the moment and then NSubstituting them, if that is the thing one does, so I have direct side by side comparisons.

In one case the Rhino version weighed in at 0.0008640 seconds, the NSubstitute takes 0.2250987 according to the MSTest result pane.

However when I took them out and ran them in isolation just picking out individual pairs of tests, I realised that NSubstitute was in fact slightly faster in every case. So for example the one that took 0.0008640 as part of the complete test suite runs in 0.1814 when asked to run alone, while without all the other tests in the background the NSubstitute version runs in 0.1732. It looks as though the speed difference I was seeing was more about something else in the environment ( MSTest? The CLR? ) being able to optimise for the large number of similar tests, while the NSubstitute test times were being affected by the time required to load in the dependencies for NSubstitute. With that balanced out NSubstitute seems to be coming in around 10% faster across the fairly limited set of tests I have implemented.

With an entire test suite based on it, I guess our tests would run a tiny bit faster in addition to being easier to read and write, so that looks like a justified change of direction!

Many thanks for your time,

-ben

[David Tchepak]

It may be just time taken to load a couple of dependencies Nsub has (a few more than RM IIRC) .

That said, I've spiked out a change today that speeds up a large test suite from another project by 5 to 10%, so I'm really glad you raised this. :)

To view this discussion on the web visit https://groups.google.com/d/msg/nsubstitute/-/EnRVh97RU1MJ.