CreateMock runs two to four times slower when debugging (vs. just run)

[Karl Lew]

Oren,

We've hit a nasty problem. CreateMock and CreateMultiMock run SLOWER
when debugging in VS2005. We can no longer debug some unit tests
interactively and have resorted to Debug.WriteLine(). At first we
thought it might be Resharper, but we've disabled that and the problem
persists. Here are some sample run times with Rhino v3.3:

RUN TestPaycheckView: 30.500 seconds (mostly in CreateMock)
DEBUG TestPaycheckView: (no time) VS2005 crashes after 10m

A simple solution might exist for us--most of our interfaces being
mocked never change. We're guessing that the slowness might be in the
dynamic assembly creation. Any possibility of just generating the
assembly on demand (vs. every run)?

Thanks,
--Karl

Here's a simple test that reveals this slowness while debugging. Just
compare the Warmup times:

public interface I10
{
string S1 { get; set; }
string S2 { get; set; }
string S3 { get; set; }
string S4 { get; set; }
string S5 { get; set; }
string S6 { get; set; }
string S7 { get; set; }
string S8 { get; set; }
string S9 { get; set; }
string S10 { get; set; }
}


/// <summary>
/// Where does Rhino spend its time?
/// Notice that Debug vs Run numbers differ substantially in Warmup.
/// </summary>
[Test]
public void TestCreateMock()
{
long msStart = Environment.TickCount;

//Initialize to "warm up" test
object dummy;
dummy = m_Mocks.CreateMock<I10>();
dummy = m_Mocks.CreateMock<I100>();
long msElapsedWarmUp = Environment.TickCount - msStart;

msStart = Environment.TickCount;
for (int i = 0; i < 1000; i++)
{
I10 test = m_Mocks.CreateMock<I10>();
Assert.IsNotNull(test);
}

long msElapsedI10 = Environment.TickCount - msStart;

Debug.WriteLine(String.Format("Warmup:{0} I10:{1}", msElapsedWarmUp,
msElapsedI10,));
}

[Ayende Rahien]

The assembles that Rhino Mocks generate are globally cached, this means that even from two different mock repositories, you are only paying for the proxy construction once.

What is        dummy = m_Mocks.CreateMock<I100>(); ?

The results that I get:

Normal: Warmup:282 I10:62
During debug: Warmup:344 I10:62

Trying that with an interface with 30 members:

Warmup:328 I30:63
Warmup:484 I30:63

[Karl Lew]

Sorry, I forgot to delete that line, but you guessed correctly that
the I100 was a dummy interface with more (100) properties.
I'm glad you could reproduce the results that debug is slower than
run. Notice that increasing interface complexity strongly affects the
debug time (484/344 vs 328/282 time).
We're generating mocks for many more interfaces, so the problem we're
experiencing is much worse (VS2005 hangs and dies eventually).

Thanks for looking into this,
--Karl

[Ayende Rahien]

There is a direct correlation between the number of methods and the amount of work done by Rhino Mocks.
May I ask why you have interfaces with so many methods / properties. In general, they are very hard to use.
If this is a serious issue, I would gladly accept a patch that wou make Rhino Mocks support Persistent Proxies.

[Karl Lew]

Oren,

We are using mocks for legacy COM interfaces accumulated for over a
decade of major application development. 8(
As a result, we have no direct control over these interfaces and are
actually quite grateful for Rhino's ability to generate mock interface
implementations.
How difficult would it be to explore the PersistentProxies?

--Karl

On Jan 31, 4:42 pm, "Ayende Rahien" <aye...@ayende.com> wrote:
> There is a direct correlation between the number of methods and the amount
> of work done by Rhino Mocks.
> May I ask why you have interfaces with so many methods / properties. In
> general, they are very hard to use.
> If this is a serious issue, I would gladly accept a patch that wou make
> Rhino Mocks support Persistent Proxies.
>

> On 2/1/08, Karl Lew <well...@gmail.com> wrote:
>
>
>
>
>
> > Sorry, I forgot to delete that line, but you guessed correctly that
> > the I100 was a dummy interface with more (100) properties.
> > I'm glad you could reproduce the results that debug is slower than
> > run. Notice that increasing interface complexity strongly affects the
> > debug time (484/344 vs 328/282 time).
> > We're generating mocks for many more interfaces, so the problem we're
> > experiencing is much worse (VS2005 hangs and dies eventually).
>
> > Thanks for looking into this,

> > --Karl- Hide quoted text -
>
> - Show quoted text -

[Ayende Rahien]

Not very, you would need to change the ProxyBuilder implementation, give it all the interfaces that you want, it would generate an assembly, then you read from that assembly

On 2/1/08, Karl Lew <wel...@gmail.com> wrote:

[Karl Lew]

Cool!

Thanks for the tip. We'll download the source and tackle the issue.

8) Karl

On Jan 31, 8:57 pm, "Ayende Rahien" <aye...@ayende.com> wrote:
> Not very, you would need to change the ProxyBuilder implementation, give it
> all the interfaces that you want, it would generate an assembly, then you
> read from that assembly
>

> > > - Show quoted text -- Hide quoted text -

[Karl Lew]

Oren,

It's not looking good. I can change Rhino.Mocks to use
PersistentProxyBuilder to save the assembly, but:
* The saved Assembly is only for the most recently generated type
* PersistentProxyBuilder has no provision for loading in the generated
assembly.

I poked around in the Castle group. Looks like there are many other
folks who have run into this and that Castle may need to provide a
richer API for Rhino.Mocks to use.

http://support.castleproject.org//browse/DYNPROXY-55
http://support.castleproject.org/browse/DYNPROXY-72

At this point, our current solution is to use hand-written mocks for
key classes that cause performance issues. This is very tedious, since
it's the complicated interfaces with hundreds of methods which are the
ones we need to mock by hand. However, we are limping by on this
approach and are hoping that Castle may have a solution in the near
future. If you have any suggestions, they'd be most welcome.

Sadly,
--Karl

[Fabian Schmied]

> It's not looking good. I can change Rhino.Mocks to use
> PersistentProxyBuilder to save the assembly, but:
> * The saved Assembly is only for the most recently generated type
> * PersistentProxyBuilder has no provision for loading in the generated
> assembly.
>
> I poked around in the Castle group. Looks like there are many other
> folks who have run into this and that Castle may need to provide a
> richer API for Rhino.Mocks to use.
>
> http://support.castleproject.org//browse/DYNPROXY-55

Karl, I was the one who created that JIRA issue. Since reporting it,
I've improved the DP API quite a bit.

In fact, in the current DP trunk revisions, PersistentProxyBuilder
doesn't automatically save any longer (it did that previously, which
resulted in the assemblies holding only the last generated type);
instead, you need to manually call
PersistentProxyBuilder.SaveAssembly, which will save all the types
generated with that builder so far.

I haven't taken the time to implement support for loading the
assemblies until now simply because it's a lot of work. However, with
these performance issues popping up, I'll look into it again.

I'm not sure how often Ayende updates Rhino.Mocks to the latest Castle
trunk, though.

> http://support.castleproject.org/browse/DYNPROXY-72

Hm, it would be expected that CreateMock runs slower with a debugger
attached than without, but I wouldn't expect it to be that slow.
However, I don't think there's anything we (= Castle) can do there -
it seems to be a Reflection.Emit/debugger issue, right? (Maybe you
could do some profiling with a debugger attached to check where
exactly all the time is spent? I'd suspect one or several of the
Reflection.Emit methods.)

However, once you have save/load support, you can work-around that issue, right?

Fabian

[Fabian Schmied]

> > http://support.castleproject.org//browse/DYNPROXY-55

[...]

I have now implemented support for loading proxies from disk in Castle
DynamicProxy 2 (trunk revision 4800). As soon as Rhino.Mocks is
upgraded to the latest version of the Castle trunk, you should be able
to use that API (ModuleScope.LoadAssemblyIntoCache, access via
IProxyBuilder.ModuleScope).

Fabian

[Karl Lew]

Fabian,

Vielen Dank!

I should go to sleep more often. This is a fine present!

I will proceed with the updated Castle and see if I can update Rhino
with Oren's permission.

8) Karl

[Karl Lew]

Oren,

We've confirmed that Castle 4800 has the required functionality to
support cached mocks (thank you, Fabian!). Although we're still
thinking through the implications, we'd like to propose the following
design and would like your feedback.

Thanks,
--Karl & Raja

/// <summary>
/// PersistentMockRepository extends MockRepository with the
capability
/// to generate persistent assemblies for mock objects that change
rarely.
/// </summary>
public class PersistentMockRepository : MockRepository
{
/// <summary>
/// Create a new PersistentMockRepository, loading it from a
previously
/// saved assembly if present.
/// <see cref="isAssemblyLoaded"/>
/// </summary>
public PersistentMockRepository() ...

/// <summary>
/// Save the currently compiled assembly for re-use as an assembly
with the
/// returned path name.
/// </summary>
public string SaveAssembly() ...

/// <summary>
/// Return true if this assembly was loaded from the location
/// specified by a previous call to SaveAssembly. A
PersistentMockRepository
/// that has been loaded cannot be extended to create proxies that
are not
/// contained within the loaded assembly (i.e., a loaded
PersistentMockRepostiory
/// is frozen.)
/// </summary>
public bool IsAssemblyLoaded {get;}
}

[Ayende Rahien]

How would you specify what mocks you want to generate?

[Karl Lew]

I think there would be two types of uses:

USE CASE #1: Standard MockRepository that we all use regularly and
will continue to use.
USE CASE #2: PersistentMockRepository for mocking legacy types that
don't change.

Here's the detail of USECASE #2:

PersistentMockRepository persistMocks = new
PersistentMockRepository();

IFoo foo = persistMocks.CreateMock<IFoo>();
...etc....

persistMocks.ReplayAll();
persistMocks.SaveAssembly();

The ctor for PersistentMockRepository would load the cached assembly
if it's there. This means that the unit test code doesn't really need
to worry about the caching. If the mocked types need to be extended or
changed, the developer will need to remove the cached file, perhaps in
response to a MSVS Clean Solution event.

--Karl

On Feb 14, 12:13 am, "Ayende Rahien" <aye...@ayende.com> wrote:
> How would you specify what mocks you want to generate?
>

[Shane Courtrille]

Putting the requirement that the developer carry out an action when they make changes is a huge impediment to usage.  I personally have a big problem with how long mocking takes but I can't see myself using something that requires people to remember to clean a solution.

[Ayende Rahien]

I agree.
It should probably try to do a file timestamp check for it.

[Karl Lew]

Hmm. Any suggestions on how to check the file timestamps of all the
mocked types in a PersistentMockRepository()?

--Karl

On Feb 14, 9:51 am, "Ayende Rahien" <aye...@ayende.com> wrote:
> I agree.
> It should probably try to do a file timestamp check for it.
>

> On 2/14/08, Shane Courtrille <shanecourtri...@gmail.com> wrote:
>
>
>
>
>
> > Putting the requirement that the developer carry out an action when they
> > make changes is a huge impediment to usage.  I personally have a big problem
> > with how long mocking takes but I can't see myself using something that
> > requires people to remember to clean a solution.
>

> > > > - Show quoted text -- Hide quoted text -

[Fabian Schmied]

> Hmm. Any suggestions on how to check the file timestamps of all the
> mocked types in a PersistentMockRepository()?

You could check whether there is an assembly in the application base
directory that is newer than the generated file (if one exists) and
use this as a heuristic for proxy regeneration, for example.

Fabian

[Karl Lew]

Okay. That makes sense...
Hmm. I wonder if I can check the cachedAssembly.ReferencedAssemblies?
I think that could work because MSVS should have compiled the directly
referenced assemblies.
(testing...)

Thanks, Fabian!

8) Karl

On Feb 14, 10:55 am, "Fabian Schmied" <fabian.schm...@gmail.com>
wrote:

[Richard...@quixotecorp.com]

Fabian suggested:

> You could check whether there is an assembly in the application base
directory
> that is newer than the generated file (if one exists) and use this as
a
> heuristic for proxy regeneration, for example.

I think I'd probably try and work out some way to embed metadata e.g.
the assembly timestamp when the proxy was generated in the proxy
assembly. It's always possible, for example, that a referenced assembly
may be reverted to an earlier version in which case the timestamp test
would succeed when the desired result would be a failure (so the proxy
gets regenerated).

Just my humble opinion.

Regards,
Richard

* C O N F I D E N T I A L I T Y N O T I C E *
-----------------------------------------------------------
The content of this e-mail is intended solely for the use of the individual or entity to whom it is addressed. If you have received this communication in error, be aware that forwarding it, copying it, or in any way disclosing its content to any other person, is strictly prohibited. Quixote Traffic Corporation is neither liable for the contents, nor for the proper, complete and timely transmission of (the information contained in) this communication. If you have received this communication in error, please notify the author by replying to this e-mail immediately and delete the material from any computer.

[Karl Lew]

Richard,

Ho! I didn't think of that. Nice catch.

It will be easiest to save the referenced assemby codebase names and
time stamps in PersistentMockRepository.cache. This cache file will be
read and can be used to determine if the cache is stale (i.e., if
there is ANY change to ANY timestamp).

(testing...)

Thanks,
--Karl

[Fabian Schmied]

> It will be easiest to save the referenced assemby codebase names and
> time stamps in PersistentMockRepository.cache. This cache file will be
> read and can be used to determine if the cache is stale (i.e., if
> there is ANY change to ANY timestamp).

You can also embed information in the generated assembly file, e.g. by
defining a custom attribute on the proxy builder's AssemblyBuilder.
You can access the AssemblyBuilder via
IProxyBuilder.ModuleScope.StrongNamedModule/WeakNamedModule.Assembly,
I think. (Take a look at how DP stores the cache metadata in
ModuleScope.SaveAssembly(bool):
http://svn.castleproject.org:8080/svn/castle/trunk/Tools/Castle.DynamicProxy2/Castle.DynamicProxy/ModuleScope.cs)

Regards,
Fabian

[karl lew]

Fabian,

 

I'm unsure about using assembly metadata, because examining the metadata might (?) require that the cached assembly to be loaded. If the cached assembly is stale and loaded, unloading is a bit of a pain requiring loading into separate AppDomain, etc.

 

--Karl 

 

[Fabian Schmied]

> I'm unsure about using assembly metadata, because examining the metadata
> might (?) require that the cached assembly to be loaded. If the cached
> assembly is stale and loaded, unloading is a bit of a pain requiring loading
> into separate AppDomain, etc.

You're right, inspecting metadata via Reflection requires the assembly
to be loaded. If you want to avoid loading stale assemblies you'd need
to inspect it in a separate AppDomain (that's not so problematic, look
at the test cases in
http://svn.castleproject.org:8080/svn/castle/trunk/Tools/Castle.DynamicProxy2/Castle.DynamicProxy.Tests/ModuleScopeTestCase.cs
(bottom) and at the CrossAppDomainCaller class in
http://svn.castleproject.org:8080/svn/castle/trunk/Tools/Castle.DynamicProxy2/Castle.DynamicProxy.Tests/CrossAppDomainCaller.cs).

However, if you prefer the .cache file approach, just go ahead, it's
probably simpler :)

Fabian

[Patrick Steele]

Don't forget that .NET 2.0 added the "ReflectionOnlyLoad" method on
Assembly. This lets you load up an assembly just for reflection. You
can't execute any code and dependencies are not loaded either.

[Karl Lew]

Oren,

Here's the updated proposal for PersistentMockRepository with the
feedback incorporated.

--Karl

p.s., Fabian/Patrick, thanks for your help and guidance!
p.p.s., MbUnit can't run/debug the PersistentMockRepositoryTests with
Resharper. I had to switch to NUnit2.4.1 to develop the unit tests.
For example, here's just one of many problems encountered:
"The process cannot access the file 'C:\Program Files\JetBrains
\ReSharper\v3.1\vs8.0\Bin\DevExpress.Data.v7.1.dll' because it is
being used by another process."

/// <summary>
/// PersistentMockRepository extends MockRepository with the
capability
/// to generate persistent assemblies for mock objects that change
rarely.
/// </summary>
public class PersistentMockRepository : MockRepository
{
/// <summary>

/// Return the singleton PersistentMockRepository, loading it from a

previously
/// saved assembly if present.

/// <see cref="m_IsAssemblyLoadedFromCache"/>
/// </summary>
public static PersistentMockRepository GetInstance {get; }

/// <summary>
/// Save the currently compiled assembly for re-use as an assembly
with the

/// returned path name (e.g., "{some path}/CastleDynProxy2.dll").
This call can
/// only be executed ONCE.

/// </summary>
public string SaveAssembly() ...

/// <summary>

/// Attempt to delete the cached assembly and return true if the
cached assembly
/// was indeed found and deleted.
/// </summary>
/// <returns></returns>
public static bool DeleteAssembly() ...

/// <summary>
/// Return true if this assembly was loaded from the location
/// specified by a previous call to SaveAssembly. A
PersistentMockRepository
/// that has been loaded cannot be extended to create proxies that
are not
/// contained within the loaded assembly (i.e., a loaded
PersistentMockRepostiory
/// is frozen.)
/// </summary>

public bool IsAssemblyLoadedFromCache {get;}

/// <summary>
/// Return file path to be used for cached assembly
/// </summary>
/// <returns></returns>
public static string CachedAssemblyFilePath {get;}

/// <summary>
/// Returns path to cache metadata xml file which is used to
determine
/// timestamps of referenced assemblies in order to invalidate the
cache
/// when they change.
/// </summary>
public static string PersistentMockRepositoryXmlPath {get;}

[Ayende Rahien]

Do I need to do anything special (aside from using the persistent mock repository to use it?

[Karl Lew]

Well, yes and no. Regarding client usage, I think that there are no
external dependencies or special actions. However, one issue to
consider is that building Rhino will now require NUnit 2.4.1 as well
as MBUnit. I believe we can simply add an NUnit.2.4.1 directory to
SharedLibs to address this new requirement.

Today I will integrate the updated RhinoMocks into our own application
production code unit tests to evaluate the new
PersistentMockRepository api and implementation. Stay tuned!

8) Karl

p.s., I uploaded the current versions of the files to the group. If
you have a moment, please review: --Thanks!
http://rhinomocks.googlegroups.com/web/PersistentMockRepositoryTests.cs
http://rhinomocks.googlegroups.com/web/PersistentMockRepository.cs

p.s., attached

On Feb 16, 12:20 am, "Ayende Rahien" <aye...@ayende.com> wrote:
> Do I need to do anything special (aside from using the persistent mock
> repository to use it?
>

> >         public static string PersistentMockRepositoryXmlPath {get;}- Hide quoted text -

[Ayende Rahien]

In general, Rhino Mocks cannot have a dependency on a testing framework.

The _tests_ can do whatever they want, but the Rhino.Mocks.dll assumes only CLR 2.0

 

I'll look at the patches in detail in the next few days, no time at the moment

 

[Karl Lew]

On Feb 18, 4:08 pm, "Ayende Rahien" <aye...@ayende.com> wrote:
> In general, Rhino Mocks cannot have a dependency on a testing framework.
> The _tests_ can do whatever they want, but the Rhino.Mocks.dll assumes only
> CLR 2.0
>

Absolutely. We only require a change to the Rhino.Mocks.Tests 2.0
project. The production Rhino.Mocks is untouched and has no dependency
on nunit.

I've updated the wiki with information on PersistentMockRepository:
http://www.ayende.com/Wiki/(S(gx4imde23krdise4vch42zv2))/PersistentMockRepository.ashx

Initial tests with our application are promising. We are actually able
to debug our unit tests without crashing MSVS2005!!! WOOHOO!!!
This is a tremendous win for us and frees us from having to debug unit
tests by using WriteLine() statements.

8) Karl

p.s. A prerequisite for PersistentMockRepository is Castle-trunk 4800
with Fabian's changes. Although I could attempt to update the Rhino
trunk with Castle-trunk 4800, I'm nervous about doing that myself and
afraid I'd botch it. Would you mind doing that?

[Ayende Rahien]

I am pretty ashamed of myself, for letting this lie aside for so long.

Reviewed and committed, thanks

[Ayende Rahien]

I have reverted this patch, the tests are not running in a consistent manner.