Very slow Setup() performance - am I doing something wrong?

[andy]

I'm a new Moq user, and I'm finding it great so far. However I've
noticed that a lot of my tests are surprisingly slow. In particular
there is a class with around 20 (very simple) methods that I mock, and
just setting up those methods takes around 250ms for each test. I've
tried the same thing with RhinoMocks and it's much faster - < 10ms.
I'm willing to dig into the code to find out what's going on, but I
thought I'd ask here first, in case there's a simple explanation.

Here's a sample piece of code that shows the problem. When I run
this, Rhino takes 40ms for the first iteration, then 1ms for each
subsequent iteration. Moq is 250ms for the first iteration, then 70ms
for each subsequent iteration. (BTW I'm not too familar with RM, so
I'm not 100% sure if what I've done is equivalent to the Moq code)


using System;
using System.Diagnostics;
using Moq;
using Rhino.Mocks;

namespace testmoqperf
{
class Program
{
static void Main(string[] args)
{
Mock<Real> moqMock = new Mock<Real>();
Real rhinoMock = MockRepository.GenerateStub<Real>();

bool bRet = true;
int x = 10, y = -1;
int counter = 0;

Stopwatch sw = new Stopwatch();
while (counter++ < 10)
{
x++; y++;

sw.Reset(); sw.Start();
rhinoMock.Stub(p => p.a(x, y)).Return(bRet);
rhinoMock.Stub(p => p.b(x, y)).Return(bRet);
rhinoMock.Stub(p => p.c(x, y)).Return(bRet);
rhinoMock.Stub(p => p.d(x, y)).Return(bRet);
sw.Stop();
Console.WriteLine("RHINO: " + sw.ElapsedMilliseconds);

sw.Reset(); sw.Start();
moqMock.Setup(p => p.a(x, y)).Returns(bRet);
moqMock.Setup(p => p.b(x, y)).Returns(bRet);
moqMock.Setup(p => p.c(x, y)).Returns(bRet);
moqMock.Setup(p => p.d(x, y)).Returns(bRet);
sw.Stop();
Console.WriteLine("MOQ: " + sw.ElapsedMilliseconds);
}
}
}

public class Real
{
public virtual bool a(int x, int y) {return false;}
public virtual bool b(int x, int y) {return false;}
public virtual bool c(int x, int y) {return false;}
public virtual bool d(int x, int y) {return false;}
}

}

[andy]

It seems the problem is specific to using variables in the arguments.
So this is fine:

mock.Setup(p => f(10,20)).Returns(bRet)

but this is slow:

int x = 10, y = 20;
mock.Setup(p => f(x,y)).Returns(bRet)

The overhead increases linearly with each extra parameter.

A lot of the time seems to be taken up in MatcherFactory.CreateMatcher
- ExpressionType.Constant is fast, ExpressionType.MemberAccess is
slow. However it's not obvious to me how
MatcherFactory.CreateMatcher could/should be improved for the
ExpressionType.MemberAccess case.

I might investigate a bit more when I get some time. In the
meantime, I suppose I'll try to use hard-coded values where
possible.

[Daniel Cazzulino]

thanks a lot for the follow-up!

yes, lazily evaluating variables has a non-trivial cost. we could do eager evaluation as an alternative, basically capturing the value of the variables at setup time, but this may prevent more interesting uses of the variable values.

which one do you think is more important? is the performance a showstopper for using variables at this point?

/kzu

--
Daniel Cazzulino | Developer Lead | XML MVP | Clarius Consulting | +1 425.329.3471

[andy]

At the moment I have approx 100 tests which take around 40 seconds to
run. From some rough manual profiling, well over 50% of this 40
seconds is in Moq Setup() calls. I'm expecting this test suite to
grow substantially from here, so yes, I think the current performance
is a show-stopper, unless I can come up with a reasonable workaround.

Perhaps what's slightly unusual about my tests is that I'm making
heavy use of NUnit's new parameterised test features, so I'm using
variables in lots of places that others might use simple values.
Also my mocked infrastructure is fairly large, so I'm making quite a
few Setup() calls (10s, not 100s) for each test. Having said all
that, I don't think what I'm doing is *that* unusual, and I'm sure
others will hit this problem sooner or later.

It's probably my lack of understanding of lambda expressions,
expression trees etc, but I'm slightly confused with what you say
about changing Moq to "capturing the values ... at setup time",
because I believe that's what Moq currently does, albeit
indirectly through partial evaluation of the Expression. Specifically
the 'capture' seems to happen in
SubtreeEvaluator.Evaluate, where the MemberExpression is converted to
a ConstantExpression. As far as I can tell, it's the lambda.Compile
that's the slow part.

Moq actually does the lambda.Compile twice during Setup, once in
MatcherFactory.CreateMatcher
(Expression.Lambda<Action>(access).Compile().Invoke()), and once in
SubtreeEvaluator.Evalulate. Perhaps one of those could be
eliminated? But fundamentally I think the performance will always be
limited if there's a lambda.Compile in there anywhere.

Is there some way we can optimise this simple scenario to avoid using
Expressions containing MemberExpressions? For example, as a very
crude hack, I wrote this method that builds a 'fast' expression
(containing ConstantExpressions rather than MemberExpressions) to send
to Setup():

/// <summary>
/// Instead of this:
/// mock.Setup(p => p.f(x, y)).Returns(z);
/// Do this:
/// mock.Setup(FastExpr<Real,int>("f", x, y)).Returns
(z);
/// </summary>
static Expression<Func<T, TRes>> FastExpr<T, TRes>
(string
methodName, params object[] args)
{
MethodInfo method = typeof(T).GetMethod
(methodName);
ParameterExpression p = Expression.Parameter
(typeof(T), "p");

List<Expression> constantArgs = new
List<Expression>();
foreach (object arg in args)
constantArgs.Add(Expression.Constant
(arg));

MethodCallExpression call = Expression.Call(p,
method, constantArgs);
Expression<Func<T, TRes>> lambda =
Expression.Lambda<Func<T, TRes>>(call, p);

return lambda;
}


Obviously this is a horrible hack, but is there a way we could
somehow integrate this kind of optimisation into Moq Setup()?

Andy

[andreister]

"..capturing the values at setup time" means caching them as lambdas,
instead of evaluating the real values.


"...I'm making heavy use of NUnit's new parameterised test features,

so I'm using variables in lots of places that others might use simple

values..."

Can you elaborate more on this?

[Olof Bjarnason]

2009/3/24 andreister <andre...@gmail.com>

"..capturing the values at setup time" means caching them as lambdas,
instead of evaluating the real values.


"...I'm making heavy use of NUnit's new parameterised test features,

so I'm using variables in lots of places that others might use simple

values..."

Can you elaborate more on this?

I can do if for him ;)

Instead of writing this style:

[Test]
public void Test1() {
  var sut = new MyClass();
  var result = sut.Double(5); // literal 5
  Assert.That(result, Is.EqualTo(10)); // literal 10
}
 
[Test]
public void Test2() {
  var sut = new MyClass();
  var result = sut.Double(7); // literal 7
  Assert.That(result, Is.EqualTo(14)); // literal 14
}

.. he is using the TestCase syntax:

[TestCase(5, 10)]
[TestCase(7, 14)]
public void TestSoe(int input, int expectedOutput) {
  var sut = new MyClass();
  var result = sut.Double(input); // variable 'input' instead of literal
  Assert.That(result, Is.EqualTo(expectedOutput)); // variable 'expectedOutput' instead of literal
}

--
Min blogg:
http://olofb.wordpress.com
[My blog, in Swedish]

[andy]

> .. he is using the TestCase syntax:

Yes, and also the Values & Range attributes to do combinatorial
tests. Couple of examples here:

http://www.andymcm.com/blog/2009/03/some-cool-new-ish-nunit-features.html

[andreister]

wow :)

> http://www.andymcm.com/blog/2009/03/some-cool-new-ish-nunit-features....

[andy]

> Obviously this is a horrible hack ...

For the record the solution I've settled on is to wrap up my 'horrible
hack' as an extension method, which makes it a bit less horrible. So
this normal Setup() call:

myMock.Setup(p => p.DoSomething(x,y)).Returns(z);

becomes this:

myMock.FastSetup(p => p.DoSomething(x,y), x, y).Returns(z);

Notice that the variables are passed in addition to the lambda. You
can mix variables with non-variables, so there's no loss of
functionality over normal Setup(), e.g.:

myMock.FastSetup(p => p.DoSomething(It.IsAny<int>(), y), y).Returns
(z);

Applying this to my most frequently called Setup() methods has reduced
the time to run my tests from 56 seconds to 27 seconds, which I
consider very significant.

Obviously having to specify the variables twice like this is far from
ideal - it would be good if someone can think of a better solution,
but this will do me for the meantime.

If anyone wants a copy of the code, just email me.

[andy]

Though I've got a workaround for the slow Setup() problem, on
reflection I'm wondering if speeding up Setup() is missing the point a
bit. I'm repeating all these Setup calls for very test, which is just
pointless redundant work. Once the hard work of doing all the setup
on a mock is done, why can't I reuse it across tests?

The answer of course is that I can't reuse the mock because the test
might have modified it. But what if I could save the state of a mock
once initialised, then restore the state at the end of each test.
From a quick play with the Moq source, it seems this is very doable,
at least for simple cases. I added Save/Restore methods on the
Interceptor class, which just save and restore copies of the calls and
orderedCalls collections, and then added Save/Restore methods on the
Mock class which forward to the Interceptor methods.

Any reason why this is a bad idea? Is anyone else interested in this
functionality, or is it just me who would have a use for this?

Here's an example:

var mock = new Mock<MyClass>();

// Do some default setup and save it.
mock.Setup(p => p.f()).Returns(3);
Debug.Assert(mock.Object.f() == 3);
mock.SaveSetup();

// Overide the default setup
mock.Setup(p => p.f()).Returns(7);
Debug.Assert(mock.Object.f() == 7);

// Restore the default setup
mock.RestoreSetup();
Debug.Assert(mock.Object.f() == 3);

[Daniel Cazzulino]

I think it might be a useful feature.

SaveState()
RestoreState()

kind of thing?

as you say, it's actually quite easy to do...
would this be useful with some kind of integration perhaps with the rowtests?
is this something you just put in your test setup somehow?
I can't clearly visualize how you'd use this....

/kzu

--
Daniel Cazzulino | Developer Lead | XML MVP | Clarius Consulting | +1 425.329.3471

[Andy McMullan]

> I can't clearly visualize how you'd use this....

Currently I do this for almost every test:

1) Core mock setup - 5 or 6 mocks, approx 30-40 mock.Setup() calls.
2) Test-specific mock setup (typically 1 or 2 mock.Setup() calls)
3) Perform test

My idea is that instead of this I'd do the core mock setup once at the
start of the run, calling mock.SaveState() on each mock, and then each
test would become:

1) Reset core mock setup to default state, i.e. call
mock.RestoreState() on all mocks
2) Test-specific mock setup
3) Perform test

(Actually it's a little more complex than this because my core mock
setup is parameterised, so I'd need to manage a collection of core
mock setups, but the same basic principle applies)

An alternative would be to have a Clone() method on the mock - step 1
above would become "Clone default mocks". This might lend itself
better to parallelism of tests, because each test would be working on
its own separate mock object, rather than re-using the one from the
previous test.

[Andy McMullan]

> An alternative would be to have a Clone() method on the mock

I coded the Clone() method and it was straightforward to get it
working for simple Setup calls.  (I didn't test anything else like
Verify() or recursive mocks)

So I think Clone() is probably the way to go.   I guess we can wait
and see if many others get bitten by the performance issue that I
encountered - certainly doesn't seem like anyone else is complaining,
so maybe my scenario is quite unusual.

For the time being I'm happy to stick with my FastSetup workaround,
which has the advantage of not requiring any changes to Moq itself.