mocking across tests

225 views
Skip to first unread message

Bryan Donovan

unread,
Dec 19, 2011, 9:33:55 PM12/19/11
to nod...@googlegroups.com
I'm currently using Mocha with Mockery for unit testing.  I can't figure out a good way to deal with this situation:

Test A requires a real module.
Test B and mocks that same module.

Running Test A in isolation passes.
Running Test B in isolation passes.
Running both at once: Test B fails every time.  

In the third scenario, Test B fails because the real module has already been loaded (even if Test A hasn't run yet). 

This occurred with Gently as well, and with nodeunit instead of Mocha, so I don't think it's an issue with Mockery or Mocha.  But I'm still guessing there's an easy solution to what is a trivial problem in other languages, and I'm just missing it.  I'm fine with using a different mocking library if that's the solution, as long as it's simple to use and doesn't require me to pollute source files with weird statements like Gently does.

I have a repeatable set of source code and tests here: https://github.com/BryanDonovan/nodejs-mock-test

Any advice would be greatly appreciated.


Thanks,

Bryan

Martin Cooper

unread,
Dec 19, 2011, 11:07:38 PM12/19/11
to nod...@googlegroups.com

You're right, it's not any of the libraries you're using. It's just
the Node module loader tripping you up by doing its usual caching.

You can easily work around this with Mockery by taking advantage of
the unhooking feature. What you do is modify the *non* mock test case
to register 'foo' as allowable and request that it be unhooked. That
way, 'foo' will be loaded independently by each test case. (It's
probably safest to also modify the allowable registration in the mock
test to unhook, so that test order doesn't matter.)

You could also modify only the mocking test case, but you'd have to
mess with the require cache manually, which is a bit messy because you
need to call a module loader function to resolve the path to 'foo'
first.

I've posted a diff in the Mockery ticket you filed:

https://github.com/mfncooper/mockery/issues/4

--
Martin Cooper


> Thanks,
>
> Bryan
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en

Nuno Job

unread,
Dec 20, 2011, 6:02:20 AM12/20/11
to nod...@googlegroups.com
I had this problem because I was doing HTTP. If you are doing HTTP however the solution already exists:


Nuno

Bryan Donovan

unread,
Dec 20, 2011, 12:26:08 PM12/20/11
to nodejs
Thanks Martin, that does indeed work. However, it really points to a
problem with node's require caching IMO. There really needs to be a
way to mock anything in a simpler fashion. This solution will require
me to add mockery statements to a whole bunch of tests that aren't
even doing any mocking, which is confusing to the next developer who
tries to read the tests, to say the least. But thanks.. I'll try this
out in our production code.. maybe it won't be too bad.

Maybe there could be (or is?) a way in node or a test framework to say
"load each file in its own mini require cache" -- just as if we ran
one test file at a time instead of loading all test files at once.


On Dec 19, 8:07 pm, Martin Cooper <mfncoo...@gmail.com> wrote:


> On Mon, Dec 19, 2011 at 6:33 PM, Bryan Donovan <brdono...@gmail.com> wrote:
> > I'm currently using Mocha with Mockery for unit testing.  I can't figure out
> > a good way to deal with this situation:
>
> > Test A requires a real module.
> > Test B and mocks that same module.
>
> > Running Test A in isolation passes.
> > Running Test B in isolation passes.
> > Running both at once: Test B fails every time.
>
> > In the third scenario, Test B fails because the real module has already been
> > loaded (even if Test A hasn't run yet).
>
> > This occurred with Gently as well, and with nodeunit instead of Mocha, so I
> > don't think it's an issue with Mockery or Mocha.  But I'm still guessing
> > there's an easy solution to what is a trivial problem in other languages,

> > and I'm just missing it.  I'm fine with using a differentmockinglibrary if


> > that's the solution, as long as it's simple to use and doesn't require me to
> > pollute source files with weird statements like Gently does.
>
> > I have a repeatable set of source code and tests
> > here: https://github.com/BryanDonovan/nodejs-mock-test
>
> > Any advice would be greatly appreciated.
>
> You're right, it's not any of the libraries you're using. It's just
> the Node module loader tripping you up by doing its usual caching.
>
> You can easily work around this with Mockery by taking advantage of
> the unhooking feature. What you do is modify the *non* mock test case
> to register 'foo' as allowable and request that it be unhooked. That
> way, 'foo' will be loaded independently by each test case. (It's
> probably safest to also modify the allowable registration in the mock
> test to unhook, so that test order doesn't matter.)
>

> You could also modify only themockingtest case, but you'd have to

Martin Cooper

unread,
Dec 21, 2011, 1:15:05 AM12/21/11
to nod...@googlegroups.com
On Tue, Dec 20, 2011 at 9:26 AM, Bryan Donovan <brdo...@gmail.com> wrote:
> Thanks Martin, that does indeed work. However, it really points to a
> problem with node's require caching IMO.  There really needs to be a
> way to mock anything in a simpler fashion.  This solution will require
> me to add mockery statements to a whole bunch of tests that aren't
> even doing any mocking, which is confusing to the next developer who
> tries to read the tests, to say the least. But thanks.. I'll try this
> out in our production code.. maybe it won't be too bad.

Yeah, I did say "work around". :-)

> Maybe there could be (or is?) a way in node or a test framework to say
> "load each file in its own mini require cache" -- just as if we ran
> one test file at a time instead of loading all test files at once.

There isn't a built-in way of doing this in Node. Given the narrow
context, though, you may be able to get away with giving each test a
clean cache to work with, and then resetting it at the end of the
test. I say "may" because, while it works for the test case you posted
(I tried it out), it's possible that you could run into issues with
more complex code, especially given how Mocha works. I haven't given
too much thought to all the potential repercussions yet.

If you want to try it, modify your with_mock test to add this at the
beginning of 'before':

cache = m._cache; m._cache = {};

and this at the end of 'after':

m._cache = cache;

This assumes 'm' is the result of require('module') and 'cache' is
declared in 'describe' scope.

I quickly tried simplifying this with a little bracketing function,
but Mocha got very unhappy, so I didn't pursue it.

I'll contemplate incorporating something along these lines in Mockery
once I have a chance to think about it, and the possible implications,
some more.

--
Martin Cooper

arunoda.s...@gmail.com

unread,
Dec 21, 2011, 2:10:09 AM12/21/11
to nod...@googlegroups.com
You can use horaa for this scenario.

When using the both.
1. Hijack the method on your first test case.
2. Reset after the test ran.
3. Then in the other test, real modules methods available for you.

cheers.

Dominic Tarr

unread,
Dec 24, 2011, 5:54:18 AM12/24/11
to nod...@googlegroups.com
perhaps the overall-simpler way to do this is just wrap your module in a closure so that you can inject the dependency/mock.

another 2 lines of code.

I use normally inject references to configuration, state, global event emitters...

Rob Ashton

unread,
Dec 24, 2011, 6:07:22 AM12/24/11
to nod...@googlegroups.com
Indeed - gotta say, relying on the dependency resolution mechanism to provide your fakes sounds brittle and prone to all the problems that a classic service location based mechanism comes with.

Passing in external components as part of the bootstrap mechanism in your process is flexible and leaves control of who gets what up to you or the consumers of the code, classic inversion of control stuff  really.

Martin Cooper

unread,
Dec 24, 2011, 10:53:48 AM12/24/11
to nod...@googlegroups.com
On Sat, Dec 24, 2011 at 3:07 AM, Rob Ashton <roba...@codeofrob.com> wrote:
> Indeed - gotta say, relying on the dependency resolution mechanism to
> provide your fakes sounds brittle and prone to all the problems that a
> classic service location based mechanism comes with.
>
> Passing in external components as part of the bootstrap mechanism in your
> process is flexible and leaves control of who gets what up to you or the
> consumers of the code, classic inversion of control stuff  really.

Perhaps I'm misunderstanding what you're saying. Let's say I'm
building a reusable library. One of my implementation decisions is to
use the file system for storage. I might change that decision in the
future, since it's not an inherent part of the interface to my
library. Tomorrow I might use redis or something. Today, though, I
"require('fs')" within the library and go about the job of using it
for storage. My unit tests must necessarily mock out 'fs', so I use a
library such as Mockery to do that.

In your model, how does my library get the 'fs' module? One answer is
that clients have to pass it in, as part of configuration, but that
would expose my current implementation to the client. Another answer
might be that I have to use a DI / IoC framework to get it, but that
limits the reusability of my library, because every user has to use
that framework now. Also, what if my library needs to make use of
other libraries that I didn't write (say, graceful-fs, for example)
that do their own "require('fs')"?

Could you perhaps elucidate your approach to this? I'd like to
understand how it would work in this kind of scenario. (I do
understand how DI / IoC works, and have used it effectively for years,
so no need to explain that part. :)

--
Martin Cooper

Rob Ashton

unread,
Dec 24, 2011, 12:05:57 PM12/24/11
to nod...@googlegroups.com
Well, I would probably start by saying that you shouldn't be mocking out other people's interfaces (fs) if we want to take the purist approach to testing.

But that's generally nonsense - sure, in an ideal world, the fact that you have fs is an implementation decision and the code you wished to write would take as part of its construction some object which does the job of persisting data. You wouldn't require 'fs', you would be given An object that has some methods that perform persistence tasks for you and the code responsible for running this would pass something in tondo this task. You would probably be working at a higher level of abstraction than just 'fs' and you would be performing your tests against this interface.

That's largely not how we operate in the real world, sure, but at this point we either accept that we can't isolate the fs and just perform an end to end get around that or  we let the above abstraction form and then pull it out.

Generally speaking though, I don't like implicit dependencies and that is what a require call is - if I have a require call I don't want to 'mock' it out because I am working at the level of abstraction (or lack of)  that goes with the use of whatever that require call is giving me.

So, if I am working with fs, I don't want to fake fs, I want to test directly against fs and make sure my code works as expected - at higher levels in my code I would probably fake the object that uses fs directly rather than fs itself. The alternative is to write a fake of fs that acts exactly like fs does, but that almost means writing an in memory representation of fs - that can have some value in some circumstances but I don't think this is the kind of thing most of us are looking for.

In short, require calls to me are an indication that the code I am writing is at a level of abstraction above the items I am requiring, and I want to test that integration directly. If I want to fake out things, I'll do it with explicit dependencies and the code consuming those dependencies doesn't care where it got them from  - classic dependency inversion.

Apologies for the brevity, typing on a borrowed iPad and I am non too good at it - I might try to blog these thoughts more coherently to get sensible feedback after Xmas. :)

Regards,

Rob

On Saturday, December 24, 2011, Martin y thougCooper <mfnc...@gmail.com> wrote: don
> On Sat, Dec 24, 2011 at 3:07 AM, Rob y fAke Ashton <roba...@codeofrob.com> wrote:
>> Indeed - gotta say, relying on the dependencyy. Or resolution mechanism to

>> provide your fakes sounds brittle and prone to all the problems that a
>> classic service location basedtyped  mechanism comes with.

Glenn Block

unread,
Dec 24, 2011, 1:42:21 PM12/24/11
to Rob Ashton, nod...@googlegroups.com
Wouldn't it be easy enough to just introduce a fileReader and fileWriter abstraction one implementation of which uses fs?

Sent from my Windows Phone

From: Rob Ashton
Sent: 12/24/2011 9:06 AM
To: nod...@googlegroups.com
Subject: Re: [nodejs] mocking across tests

Glenn Block

unread,
Dec 24, 2011, 2:14:35 PM12/24/11
to Rob Ashton, nod...@googlegroups.com
Forget the naming,  and no I was not suggesting mirroring fs, rather providing a pluggable abstraction for what fs is providing. Simple DI scenario.


Sent from my Windows Phone

From: Rob Ashton
Sent: 12/24/2011 11:10 AM
To: Glenn Block

Subject: Re: [nodejs] mocking across tests

I think when you call it file reader and filewriter you have already lost the battle (pedantry I know and I doubt it is what you meant) - if you are just mirroring the underlying api then there is no point to the abstraction - but yeah essentially as soon as you give the responsibility to the external process then what you pass in is up to you so you can do this

you could go so far as to just pass in fs from outside or a fake fs, but I don't see what you really gain by this. When I say inversion what I mean is that your code says 'I need something to give foos to, and get foos from and we'll call it a foo store, and you'll write unit tests against this relationship, and one of the implementations of this relationship will use fs, and if we test that implementation then the will be no need to fake fs as we want to make sure that implementation works as expected.

The summary of all of this is I think that trying to work round the require system in node is the wrong approach, if you want isolation and control over dependencies then this should happen in user land.

Rob Ashton

unread,
Dec 24, 2011, 2:20:30 PM12/24/11
to Glenn Block, Rob Ashton, nod...@googlegroups.com
Ofc not, I know you better than that. ;)

Glenn Block

unread,
Dec 24, 2011, 4:19:30 PM12/24/11
to Rob Ashton, nod...@googlegroups.com
Ha ha you do, but others don"t :-)


Sent from my Windows Phone

From: Rob Ashton
Sent: 12/24/2011 11:20 AM
To: Glenn Block
Cc: Rob Ashton; nod...@googlegroups.com

Bryan Donovan

unread,
Dec 27, 2011, 3:20:07 PM12/27/11
to nodejs
Thanks Arunoda. I tried using horaa in the past but ran into
problems. I can't remember what the problems were, so maybe I'll try
it again.

On Dec 20, 11:10 pm, "arunoda.susirip...@gmail.com"
<arunoda.susirip...@gmail.com> wrote:
> You can use horaa for this scenario.https://github.com/arunoda/horaa
>
> When using the both.
> 1. Hijack the method on your first test case.
> 2. Reset after the test ran.
> 3. Then in the other test, real modules methods available for you.
>
> cheers.
>
>
>
>
>
>
>
>
>
> On Wed, Dec 21, 2011 at 11:45 AM, Martin Cooper <mfncoo...@gmail.com> wrote:
> > On Tue, Dec 20, 2011 at 9:26 AM, Bryan Donovan <brdono...@gmail.com>
> @arunoda <http://twitter.com/arunoda>
> <http://gplus.to/arunoda>https://github.com/arunodahttp://www.linkedin.com/in/arunoda

Bryan Donovan

unread,
Dec 27, 2011, 3:21:03 PM12/27/11
to nodejs
Thanks Dominic. Can you provide an example of how you do this? I'm
not quite following.

On Dec 24, 2:54 am, Dominic Tarr <dominic.t...@gmail.com> wrote:
> perhaps the overall-simpler way to do this is just wrap your module in a
> closure so that you can inject the dependency/mock.
>
> another 2 lines of code.
>
> I use normally inject references to configuration, state, global event
> emitters...
>
> On Wed, Dec 21, 2011 at 8:10 PM, arunoda.susirip...@gmail.com <
>
>
>
>
>
>
>
> arunoda.susirip...@gmail.com> wrote:
> > You can use horaa for this scenario.
> >https://github.com/arunoda/horaa
>
> > When using the both.
> > 1. Hijack the method on your first test case.
> > 2. Reset after the test ran.
> > 3. Then in the other test, real modules methods available for you.
>
> > cheers.
>
> > On Wed, Dec 21, 2011 at 11:45 AM, Martin Cooper <mfncoo...@gmail.com>wrote:
>
> >> On Tue, Dec 20, 2011 at 9:26 AM, Bryan Donovan <brdono...@gmail.com>
> > @arunoda <http://twitter.com/arunoda>
> > <http://gplus.to/arunoda>https://github.com/arunoda

Bryan Donovan

unread,
Dec 27, 2011, 3:33:32 PM12/27/11
to nodejs
Thanks for all the replies.

It seems to me that perhaps the simplest solution is for the test
runner (mocha in this case) to run each test in isolation.

The problem here isn't really dependency inversion, etc., it's an
issue of being able to do "require('foo')" in one file and in a later
file say "foo is now mocked and replace by bar". If I write a module
called file_tools that abstracts file manipulation and requires 'fs',
and I write tests for this module, the problem still exists that I
won't be able to mock out file_tools later, regardless of whether I
can mock out fs.

Having to specify what files can be unhooked (in mockery) is painful,
and running each test in isolation is slow and surely makes it
difficult to hook in coverage tools, reporting tools, etc.

Thanks,
Bryan

On Dec 24, 1:19 pm, Glenn Block <glenn.bl...@gmail.com> wrote:
> Ha ha you do, but others don"t :-)
>
> Sent from my Windows Phone
> ------------------------------
> From: Rob Ashton
> Sent: 12/24/2011 11:20 AM
> To: Glenn Block
> Cc: Rob Ashton; nod...@googlegroups.com
> Subject: Re: [nodejs] mocking across tests
>
> Ofc not, I know you better than that. ;)
>
> >> On Saturday, December 24, 2011, Martin y thougCooper <mfncoo...@gmail.com>

Bryan Donovan

unread,
Dec 27, 2011, 5:36:31 PM12/27/11
to nod...@googlegroups.com
Arunoda, I forked horaa and wrote several tests in mocha, and it all seems to work as I'd like it to.  I believe I had issues in the past with relative vs absolute paths, but it was surely my lack of understanding of how it worked, or I might have assumed that it worked in a way that is similar to Gently.


You can run npm install and 'make test' to run the tests, which are written in bdd style.  I ported your tests over as well.

I'll try horaa again in real application to see if it works there.


Bryan

Martin Cooper

unread,
Dec 27, 2011, 8:50:42 PM12/27/11
to nod...@googlegroups.com
On Tue, Dec 27, 2011 at 12:33 PM, Bryan Donovan <brdo...@gmail.com> wrote:
> Thanks for all the replies.
>
> It seems to me that perhaps the simplest solution is for the test
> runner (mocha in this case) to run each test in isolation.
>
> The problem here isn't really dependency inversion, etc., it's an
> issue of being able to do "require('foo')" in one file and in a later
> file say "foo is now mocked and replace by bar".  If I write a module
> called file_tools that abstracts file manipulation and requires 'fs',
> and I write tests for this module, the problem still exists that I
> won't be able to mock out file_tools later, regardless of whether I
> can mock out fs.
>
> Having to specify what files can be unhooked (in mockery) is painful,
> and running each test in isolation is slow and surely makes it
> difficult to hook in coverage tools, reporting tools, etc.

Just curious - did you try out the clean cache trick I mentioned? If
that's something that works for your use case, I'd consider wrapping
it up inside Mockery, but it'd be nice to know if it's solving the
problem before I do that.

--
Martin Cooper

Martin Cooper

unread,
Dec 27, 2011, 8:50:49 PM12/27/11
to nod...@googlegroups.com
On Tue, Dec 27, 2011 at 2:36 PM, Bryan Donovan <brdo...@gmail.com> wrote:
> Arunoda, I forked horaa and wrote several tests in mocha, and it all seems
> to work as I'd like it to.  I believe I had issues in the past with relative
> vs absolute paths, but it was surely my lack of understanding of how it
> worked, or I might have assumed that it worked in a way that is similar to
> Gently.

When I tried horaa a while ago, I ran into a couple of issues that may
or may not be the same ones you ran into. I mention them just in case
they may jog your memory. :-)

First, the per-method hijacking can lead to problems if you're not
mocking all of the functions that you end up calling, directly or
indirectly. An example is a module that maintains state; mocking some
functions that would normally change the state may lead to non-mocked
functions that use that state now having / returning incorrect data.
This is why I prefer to mock on a per-module basis rather than a
per-function one.

Second, since horaa calls 'require' directly, the context of its
invocation is not the same as it would be without horaa; the parent
module, for example, is different. Thus relative paths and dependent
modules may not be resolved correctly.

--
Martin Cooper

Bryan Donovan

unread,
Dec 28, 2011, 1:50:47 PM12/28/11
to nod...@googlegroups.com
I haven't tried it yet in a real application, but hopefully can today.

Bryan Donovan

unread,
Dec 28, 2011, 1:59:13 PM12/28/11
to nod...@googlegroups.com

On Dec 27, 2011, at 5:50 PM, Martin Cooper wrote:

> On Tue, Dec 27, 2011 at 2:36 PM, Bryan Donovan <brdo...@gmail.com> wrote:
>> Arunoda, I forked horaa and wrote several tests in mocha, and it all seems
>> to work as I'd like it to. I believe I had issues in the past with relative
>> vs absolute paths, but it was surely my lack of understanding of how it
>> worked, or I might have assumed that it worked in a way that is similar to
>> Gently.
>
> When I tried horaa a while ago, I ran into a couple of issues that may
> or may not be the same ones you ran into. I mention them just in case
> they may jog your memory. :-)
>
> First, the per-method hijacking can lead to problems if you're not
> mocking all of the functions that you end up calling, directly or
> indirectly. An example is a module that maintains state; mocking some
> functions that would normally change the state may lead to non-mocked
> functions that use that state now having / returning incorrect data.
> This is why I prefer to mock on a per-module basis rather than a
> per-function one.

I tend to agree. Ideally we could mock either an entire module or just a method. I just tried horaa in a real application and it seems to be working ok, but I've only implemented it for a few test suites. Since I was using it in place of mockery where I had already built up mock modules that have mocked out versions of all the methods I'm using, it did require me to jump through some hoops to mock and restore all those methods.

>
> Second, since horaa calls 'require' directly, the context of its
> invocation is not the same as it would be without horaa; the parent
> module, for example, is different. Thus relative paths and dependent
> modules may not be resolved correctly.

Definitely a factor. I've found that I need to pass horaa the full module path for it to work, which I think is one of the issues I didn't understand the first time I tried it.

arunoda.s...@gmail.com

unread,
Dec 29, 2011, 2:30:17 AM12/29/11
to nod...@googlegroups.com
Seems like I missed some great discussion here :)
 
When I tried horaa a while ago, I ran into a couple of issues that may
or may not be the same ones you ran into. I mention them just in case
they may jog your memory. :-)

Honestly there may exists some errors :) And of course it would take up some memory otherwise if you not restore that.
BTW: here we are testing here. personally I don't care much on performance :) 
 
First, the per-method hijacking can lead to problems if you're not
mocking all of the functions that you end up calling, directly or
indirectly. An example is a module that maintains state; mocking some
functions that would normally change the state may lead to non-mocked
functions that use that state now having / returning incorrect data.
This is why I prefer to mock on a per-module basis rather than a
per-function one.

You got a point here. 
Unfortunately If you need to mock the whole module you've to come up with a totally new require()  system.
This is a only way I could found out fix this without worrying too much :)


Second, since horaa calls 'require' directly, the context of its
invocation is not the same as it would be without horaa; the parent
module, for example, is different. Thus relative paths and dependent
modules may not be resolved correctly.

Actually this is how node's module system works.

1. module will load to the memory when first time it saw the module
2. after that every require() takes the module from the memory
3. What horaa does is replacing the public methods in the memory directly
4. All you have to do is load the module you want to test relatively
5. It just work

And Horaa has limitations too..
horaa cannot be used to mock modules as a whole (which has no any public methods)
So as a cure I never design such modules :)



--
Arunoda Susiripala


Jeff Barczewski

unread,
Dec 29, 2011, 11:23:02 AM12/29/11
to nod...@googlegroups.com
Bryan,

I took a quick look at Horaa, and it looks nice, but I was wondering if there is a way to mock out a module that actually returns a function.

So for instance 

var foo = require('foo');

If foo is a function (or constructor), is there a way to mock it out with Horaa?

Thanks,

Jeff

arunoda.s...@gmail.com

unread,
Dec 29, 2011, 11:44:10 AM12/29/11
to nod...@googlegroups.com
Unfortunately with current design.

It is not possible.  

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en



--

Bryan Donovan

unread,
Dec 29, 2011, 12:55:53 PM12/29/11
to nod...@googlegroups.com
Oh.. this is what I was trying to remember.. I ran into this issue and believe this is why I didn't use Horaa.

Glenn Block

unread,
Dec 29, 2011, 5:18:31 PM12/29/11
to nod...@googlegroups.com
I am still having a hard time grocking WHY I need to mock a module. I believe I can design any API I can create in such a way that it offers me the ability to plug in fake dependencies. I want to cleanly be able to test the module then I just inject those fakes to isolate the code I am testing don't I? Granted I could be missing something big, but I don't see this issue has any different constraints than if I developed in say Ruby, Java, or C#.

Glenn

--

Rob Ashton

unread,
Dec 29, 2011, 5:32:51 PM12/29/11
to nod...@googlegroups.com
+1 glen

I think mocking modules is solving th wrong problem

Martin Cooper

unread,
Dec 29, 2011, 6:02:53 PM12/29/11
to nod...@googlegroups.com
On Thu, Dec 29, 2011 at 2:18 PM, Glenn Block <glenn...@gmail.com> wrote:
> I am still having a hard time grocking WHY I need to mock a module. I
> believe I can design any API I can create in such a way that it offers me
> the ability to plug in fake dependencies. I want to cleanly be able to test
> the module then I just inject those fakes to isolate the code I am testing
> don't I? Granted I could be missing something big, but I don't see this
> issue has any different constraints than if I developed in say Ruby, Java,
> or C#.

Right. And Java has popular solutions such as EasyMock for this. I've
no doubt Ruby and C# have similar solutions.

You said "I can design any API I can create in such a way that it
offers me the ability to plug in fake dependencies". Isn't that fake
dependency also a mock? I think perhaps we are really only "arguing"
over how mocks are set in place for testing, rather than the need to
mock dependencies.

More importantly, though, once I have that API, and I've tested the
dependent code, how do I test the real implementation of the
dependency? Say, for example, I have my API implementation that itself
depends on the file system. How do I test that it correctly handles
all manner of file system errors, or permissions issues? Or if it
relies on HTTP, how do I test that it reacts correctly to all the
different response codes, or malformed response data, or other errors?

At some point, regardless of the abstractions that you build on top
that help you test the upper layers, you still need to test the code
"at the bottom" that depends directly on, for example, the file
system, or HTTP, or sockets. I'm not seeing how you can test all the
error cases without mocking those dependencies, but I'm all ears if
there's a good way to do that.

--
Martin Cooper


Martin Cooper

unread,
Dec 29, 2011, 6:16:48 PM12/29/11
to nod...@googlegroups.com
On Wed, Dec 28, 2011 at 11:30 PM, arunoda.s...@gmail.com
<arunoda.s...@gmail.com> wrote:
> Seems like I missed some great discussion here :)
>
>>
>> When I tried horaa a while ago, I ran into a couple of issues that may
>> or may not be the same ones you ran into. I mention them just in case
>> they may jog your memory. :-)
>
>
> Honestly there may exists some errors :) And of course it would take up some
> memory otherwise if you not restore that.
> BTW: here we are testing here. personally I don't care much on performance
> :)

I don't believe anyone mentioned performance, actually.


>> First, the per-method hijacking can lead to problems if you're not
>> mocking all of the functions that you end up calling, directly or
>> indirectly. An example is a module that maintains state; mocking some
>> functions that would normally change the state may lead to non-mocked
>> functions that use that state now having / returning incorrect data.
>> This is why I prefer to mock on a per-module basis rather than a
>> per-function one.
>
>
> You got a point here.
> Unfortunately If you need to mock the whole module you've to come up with a
> totally new require()  system.
> This is a only way I could found out fix this without worrying too much :)

Not a new require system, no. But this is one reason I ended up writing Mockery.


>> Second, since horaa calls 'require' directly, the context of its
>> invocation is not the same as it would be without horaa; the parent
>> module, for example, is different. Thus relative paths and dependent
>> modules may not be resolved correctly.
>
>
> Actually this is how node's module system works.
>
> 1. module will load to the memory when first time it saw the module
> 2. after that every require() takes the module from the memory
> 3. What horaa does is replacing the public methods in the memory directly
> 4. All you have to do is load the module you want to test relatively
> 5. It just work

My point, though, is that every module gets its own 'require' function
that operates in the context of that module. Since horaa invokes
'require' directly, it is doing so in the horaa context, and not in
the context of the module that is using horaa. Module M calling
'require' is not the same as module M calling horaa and horaa then
calling 'require'.

It's also fairly easy to see that a relative path passed to horaa may
resolve to a different absolute path than the author intended, because
it will depend on the resolution as applied within horaa and not the
resolution as applied within the originating module.

--
Martin Cooper

Glenn Block

unread,
Dec 29, 2011, 6:16:46 PM12/29/11
to nod...@googlegroups.com
I am not arguing mocks or not, I am a big fan of using mocks/fakes. I am arguing that trying to mock a module itself is the wrong. Modules themselves should be designed to allow the functionality to be isolated/testable.

As for testing the real functionality, end to end / itntegration acceptance testing is a long standing and proven way to test the big picture. I don't need mocks then, i just actually do it. There's no substitute IMO.

Glenn

Martin Cooper

unread,
Dec 29, 2011, 6:29:36 PM12/29/11
to nod...@googlegroups.com
On Thu, Dec 29, 2011 at 3:16 PM, Glenn Block <glenn...@gmail.com> wrote:
> I am not arguing mocks or not, I am a big fan of using mocks/fakes. I am
> arguing that trying to mock a module itself is the wrong. Modules themselves
> should be designed to allow the functionality to be isolated/testable.

Fair enough. Unfortunately, though, we don't always control all of the
interfaces we need to work with, and they're not always amenable to
such methods. We still need to test against those too.

> As for testing the real functionality, end to end / itntegration acceptance
> testing is a long standing and proven way to test the big picture. I don't
> need mocks then, i just actually do it. There's no substitute IMO.

Sorry, I guess I am still missing something. Suppose my app is running
against some service I don't control, such as AWS or a Google service
or something. Or perhaps just Couch or Mongo. How do I test that my
code "does the right thing" in the face of errors from such services?
Those errors may only show up once in a blue moon, so I can't depend
on that happening during my testing. But I still need to know that my
app isn't going to fall over when it does happen, so I want to be able
to test the error cases before I go live.

Glenn Block

unread,
Dec 29, 2011, 6:37:27 PM12/29/11
to nod...@googlegroups.com
At some point my code has to call couch or mongo to make the http calls. At that point where that call happens, I would make the couch/mongo API a dependency that can be overridden. Once i do that I can now run tests all day long that simulate the error condition all day long to see how my system reacts.

I am not faking an entire module, I am faking just the parts necessary to make the test succeed.

Martin Cooper

unread,
Dec 29, 2011, 6:49:21 PM12/29/11
to nod...@googlegroups.com
On Thu, Dec 29, 2011 at 3:37 PM, Glenn Block <glenn...@gmail.com> wrote:
> At some point my code has to call couch or mongo to make the http calls. At
> that point where that call happens, I would make the couch/mongo API a
> dependency that can be overridden. Once i do that I can now run tests all
> day long that simulate the error condition all day long to see how my system
> reacts.

I think we're going in circles now. :-)

In the above scenario, suppose the couch/mongo API is your own, and
that both fake and real implementations are yours. How will you test
that the code in your real implementation (sitting on top of 'http')
handles all the different kinds of errors and response codes you might
get from the database, so that your API behaves as expected? I
understand that your fake implementation lets you test the
higher-level code, but what about testing the lower-level code?

--
Martin Cooper

Glenn Block

unread,
Dec 29, 2011, 9:31:29 PM12/29/11
to nod...@googlegroups.com
I see. IF had my druthers I would have a way to inject a fake into the http module that allows me to capture requests and fake responses. In Web API the last framework I worked on, we allow you to do just that in our HttpClient. You basically can throw a fake handler in the middle and intercept all requests. I haven't look to see if nodes http stack supports this today.

Regardless I would still look at this as something specific to the needs around the HTTP client classes rather than a general purpose module faking mechanism.

Isaac Schlueter

unread,
Dec 29, 2011, 10:14:37 PM12/29/11
to nod...@googlegroups.com
On Thu, Dec 29, 2011 at 18:31, Glenn Block <glenn...@gmail.com> wrote:
> I see. IF had my druthers I would have a way to inject a fake into the http
> module that allows me to capture requests and fake responses.

If you're testing an http client app, why not just make an http
server, and make requests to it? What's the difference between a
"fake" http module, and the real one, just listening on localhost?

Glenn Block

unread,
Dec 29, 2011, 10:21:34 PM12/29/11
to nod...@googlegroups.com
Test run ALOT faster with no network traffic and it is less infrastructure for the test environment. This will especially hit you if you are doing a large number of tests.

Martin Cooper

unread,
Dec 30, 2011, 2:28:31 PM12/30/11
to nod...@googlegroups.com
On Thu, Dec 29, 2011 at 6:31 PM, Glenn Block <glenn...@gmail.com> wrote:
> I see. IF had my druthers I would have a way to inject a fake into the http
> module that allows me to capture requests and fake responses. In Web API the
> last framework I worked on, we allow you to do just that in our HttpClient.
> You basically can throw a fake handler in the middle and intercept all
> requests. I haven't look to see if nodes http stack supports this today.

There isn't built-in support, but Nock looks interesting for this kind of thing:

https://github.com/pgte/nock

--
Martin Cooper

Glenn Block

unread,
Dec 30, 2011, 5:45:33 PM12/30/11
to nod...@googlegroups.com
That looks exactly like what I was talking about and it uses the example that you brought up. Now need to take it for a test run.

Bryan Donovan

unread,
Dec 30, 2011, 6:43:46 PM12/30/11
to nod...@googlegroups.com
I looked at nock before but didn't really get the point. Now I get it.. it actually overrides the node http library and intercepts the specified http requests. Cool indeed.

I prefer to mock the methods that do the http requests if possible though.

Glenn Block

unread,
Dec 30, 2011, 7:01:47 PM12/30/11
to nod...@googlegroups.com
I agree this not the end all solution. However it does help to address a whole category of issues like your validating that a system responds well when it receives a response that is due to some error on the server.

I'd love to understand more of the specific scenarios where you need to mock a module beyond relating to HTTP.

I do accept the fact that even though APIs CAN be designed to allow them to be easier to inject mocks, that is often not the case which is where these other solutions could be attractive.

secoif

unread,
Dec 31, 2011, 12:00:00 AM12/31/11
to nod...@googlegroups.com
We recently put together kin, a little factory library for generating test objects. Perhaps this would be useful to you. You can write rules to generate deeply nested object heirarchies, useful for creating seeds and mocks.

https://github.com/groupdock/kin

Feedback welcome.

Cliffano Subagio

unread,
Dec 31, 2011, 5:54:12 AM12/31/11
to nod...@googlegroups.com
AFAIK, not currently possible with Horaa, but I've been using Sandboxed Module to do what you described.
https://github.com/felixge/node-sandboxed-module

Cheers,
Cliff.

--

Martin Cooper

unread,
Dec 31, 2011, 10:48:19 PM12/31/11
to nod...@googlegroups.com
On Sat, Dec 31, 2011 at 2:54 AM, Cliffano Subagio <clif...@gmail.com> wrote:
> AFAIK, not currently possible with Horaa, but I've been using Sandboxed
> Module to do what you described.
> https://github.com/felixge/node-sandboxed-module

Mockery also enables this (using a different approach from that of
node-sandboxed-module).
https://github.com/mfncooper/mockery

--
Martin Cooper

Nuno Job

unread,
Jan 1, 2012, 10:51:35 AM1/1/12
to nod...@googlegroups.com

You guys should really check nock. Really. :)

Glenn Block

unread,
Jan 1, 2012, 2:24:29 PM1/1/12
to Nuno Job, nod...@googlegroups.com
Yep, i did. It is awesome.


Sent from my Windows Phone

From: Nuno Job
Sent: 1/1/2012 7:51 AM

To: nod...@googlegroups.com
Subject: Re: [nodejs] mocking across tests

Aneil Mallavarapu

unread,
Mar 12, 2012, 3:12:03 AM3/12/12
to nod...@googlegroups.com
Hi Bryan - 
The way mockery suggests you handle this issue is by calling 
    
     mockery.registerAllowable("modulePath",true); 

Where the second argument is "unhook", a signal which tells 

     mockery.deregisterAll()

to remove the module from the node module cache.

Unfortunately, there's actually a bug in mockery.js which prevents the node module cache from being cleared.  

I've committed a fix to my fork: https://github.com/amallavarapu/mockery

Details of the bug are in this pull request:  https://github.com/mfncooper/mockery/pull/6

I've also added functions to ensure ALL modules are cleared from the module cache after tests are run in this commit.  Just call:

     mockery.registerAllAllowableStart(true) 

In the before() block before you load the modules required for the test.  Then call:

     mockery.registerAllAllowableEnd();
     mockery.deregisterAll();

In the after() block.  This removes from the cache all the modules loaded between the start and end calls.

Aneil
On Monday, December 19, 2011 6:33:55 PM UTC-8, Bryan Donovan wrote:
I'm currently using Mocha with Mockery for unit testing.  I can't figure out a good way to deal with this situation:

Test A requires a real module.
Test B and mocks that same module.

Running Test A in isolation passes.
Running Test B in isolation passes.
Running both at once: Test B fails every time.  

In the third scenario, Test B fails because the real module has already been loaded (even if Test A hasn't run yet). 

This occurred with Gently as well, and with nodeunit instead of Mocha, so I don't think it's an issue with Mockery or Mocha.  But I'm still guessing there's an easy solution to what is a trivial problem in other languages, and I'm just missing it.  I'm fine with using a different mocking library if that's the solution, as long as it's simple to use and doesn't require me to pollute source files with weird statements like Gently does.

I have a repeatable set of source code and tests here: https://github.com/BryanDonovan/nodejs-mock-test

Any advice would be greatly appreciated.


Thanks,

Bryan

Martin Cooper

unread,
Apr 7, 2012, 10:52:01 PM4/7/12
to nod...@googlegroups.com
> Unfortunately, there's actually a bug in mockery.js which prevents the node
> module cache from being cleared.

The bug has been fixed in Mockery 1.1.2, now available via npm. The
cause was a change in the Node module loader starting in Node v0.6.10.
The latest Mockery now works with both older and newer versions of
Node.

--
Martin Cooper

Reply all
Reply to author
Forward
0 new messages