You're right, it's not any of the libraries you're using. It's just
the Node module loader tripping you up by doing its usual caching.
You can easily work around this with Mockery by taking advantage of
the unhooking feature. What you do is modify the *non* mock test case
to register 'foo' as allowable and request that it be unhooked. That
way, 'foo' will be loaded independently by each test case. (It's
probably safest to also modify the allowable registration in the mock
test to unhook, so that test order doesn't matter.)
You could also modify only the mocking test case, but you'd have to
mess with the require cache manually, which is a bit messy because you
need to call a module loader function to resolve the path to 'foo'
first.
I've posted a diff in the Mockery ticket you filed:
https://github.com/mfncooper/mockery/issues/4
--
Martin Cooper
> Thanks,
>
> Bryan
>
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
Maybe there could be (or is?) a way in node or a test framework to say
"load each file in its own mini require cache" -- just as if we ran
one test file at a time instead of loading all test files at once.
On Dec 19, 8:07 pm, Martin Cooper <mfncoo...@gmail.com> wrote:
> On Mon, Dec 19, 2011 at 6:33 PM, Bryan Donovan <brdono...@gmail.com> wrote:
> > I'm currently using Mocha with Mockery for unit testing. I can't figure out
> > a good way to deal with this situation:
>
> > Test A requires a real module.
> > Test B and mocks that same module.
>
> > Running Test A in isolation passes.
> > Running Test B in isolation passes.
> > Running both at once: Test B fails every time.
>
> > In the third scenario, Test B fails because the real module has already been
> > loaded (even if Test A hasn't run yet).
>
> > This occurred with Gently as well, and with nodeunit instead of Mocha, so I
> > don't think it's an issue with Mockery or Mocha. But I'm still guessing
> > there's an easy solution to what is a trivial problem in other languages,
> > and I'm just missing it. I'm fine with using a differentmockinglibrary if
> > that's the solution, as long as it's simple to use and doesn't require me to
> > pollute source files with weird statements like Gently does.
>
> > I have a repeatable set of source code and tests
> > here: https://github.com/BryanDonovan/nodejs-mock-test
>
> > Any advice would be greatly appreciated.
>
> You're right, it's not any of the libraries you're using. It's just
> the Node module loader tripping you up by doing its usual caching.
>
> You can easily work around this with Mockery by taking advantage of
> the unhooking feature. What you do is modify the *non* mock test case
> to register 'foo' as allowable and request that it be unhooked. That
> way, 'foo' will be loaded independently by each test case. (It's
> probably safest to also modify the allowable registration in the mock
> test to unhook, so that test order doesn't matter.)
>
> You could also modify only themockingtest case, but you'd have to
Yeah, I did say "work around". :-)
> Maybe there could be (or is?) a way in node or a test framework to say
> "load each file in its own mini require cache" -- just as if we ran
> one test file at a time instead of loading all test files at once.
There isn't a built-in way of doing this in Node. Given the narrow
context, though, you may be able to get away with giving each test a
clean cache to work with, and then resetting it at the end of the
test. I say "may" because, while it works for the test case you posted
(I tried it out), it's possible that you could run into issues with
more complex code, especially given how Mocha works. I haven't given
too much thought to all the potential repercussions yet.
If you want to try it, modify your with_mock test to add this at the
beginning of 'before':
cache = m._cache; m._cache = {};
and this at the end of 'after':
m._cache = cache;
This assumes 'm' is the result of require('module') and 'cache' is
declared in 'describe' scope.
I quickly tried simplifying this with a little bracketing function,
but Mocha got very unhappy, so I didn't pursue it.
I'll contemplate incorporating something along these lines in Mockery
once I have a chance to think about it, and the possible implications,
some more.
--
Martin Cooper
Perhaps I'm misunderstanding what you're saying. Let's say I'm
building a reusable library. One of my implementation decisions is to
use the file system for storage. I might change that decision in the
future, since it's not an inherent part of the interface to my
library. Tomorrow I might use redis or something. Today, though, I
"require('fs')" within the library and go about the job of using it
for storage. My unit tests must necessarily mock out 'fs', so I use a
library such as Mockery to do that.
In your model, how does my library get the 'fs' module? One answer is
that clients have to pass it in, as part of configuration, but that
would expose my current implementation to the client. Another answer
might be that I have to use a DI / IoC framework to get it, but that
limits the reusability of my library, because every user has to use
that framework now. Also, what if my library needs to make use of
other libraries that I didn't write (say, graceful-fs, for example)
that do their own "require('fs')"?
Could you perhaps elucidate your approach to this? I'd like to
understand how it would work in this kind of scenario. (I do
understand how DI / IoC works, and have used it effectively for years,
so no need to explain that part. :)
--
Martin Cooper
Just curious - did you try out the clean cache trick I mentioned? If
that's something that works for your use case, I'd consider wrapping
it up inside Mockery, but it'd be nice to know if it's solving the
problem before I do that.
--
Martin Cooper
When I tried horaa a while ago, I ran into a couple of issues that may
or may not be the same ones you ran into. I mention them just in case
they may jog your memory. :-)
First, the per-method hijacking can lead to problems if you're not
mocking all of the functions that you end up calling, directly or
indirectly. An example is a module that maintains state; mocking some
functions that would normally change the state may lead to non-mocked
functions that use that state now having / returning incorrect data.
This is why I prefer to mock on a per-module basis rather than a
per-function one.
Second, since horaa calls 'require' directly, the context of its
invocation is not the same as it would be without horaa; the parent
module, for example, is different. Thus relative paths and dependent
modules may not be resolved correctly.
--
Martin Cooper
> On Tue, Dec 27, 2011 at 2:36 PM, Bryan Donovan <brdo...@gmail.com> wrote:
>> Arunoda, I forked horaa and wrote several tests in mocha, and it all seems
>> to work as I'd like it to. I believe I had issues in the past with relative
>> vs absolute paths, but it was surely my lack of understanding of how it
>> worked, or I might have assumed that it worked in a way that is similar to
>> Gently.
>
> When I tried horaa a while ago, I ran into a couple of issues that may
> or may not be the same ones you ran into. I mention them just in case
> they may jog your memory. :-)
>
> First, the per-method hijacking can lead to problems if you're not
> mocking all of the functions that you end up calling, directly or
> indirectly. An example is a module that maintains state; mocking some
> functions that would normally change the state may lead to non-mocked
> functions that use that state now having / returning incorrect data.
> This is why I prefer to mock on a per-module basis rather than a
> per-function one.
I tend to agree. Ideally we could mock either an entire module or just a method. I just tried horaa in a real application and it seems to be working ok, but I've only implemented it for a few test suites. Since I was using it in place of mockery where I had already built up mock modules that have mocked out versions of all the methods I'm using, it did require me to jump through some hoops to mock and restore all those methods.
>
> Second, since horaa calls 'require' directly, the context of its
> invocation is not the same as it would be without horaa; the parent
> module, for example, is different. Thus relative paths and dependent
> modules may not be resolved correctly.
Definitely a factor. I've found that I need to pass horaa the full module path for it to work, which I think is one of the issues I didn't understand the first time I tried it.
When I tried horaa a while ago, I ran into a couple of issues that mayor may not be the same ones you ran into. I mention them just in case
they may jog your memory. :-)
First, the per-method hijacking can lead to problems if you're not
mocking all of the functions that you end up calling, directly or
indirectly. An example is a module that maintains state; mocking some
functions that would normally change the state may lead to non-mocked
functions that use that state now having / returning incorrect data.
This is why I prefer to mock on a per-module basis rather than a
per-function one.
Second, since horaa calls 'require' directly, the context of its
invocation is not the same as it would be without horaa; the parent
module, for example, is different. Thus relative paths and dependent
modules may not be resolved correctly.
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
--
Right. And Java has popular solutions such as EasyMock for this. I've
no doubt Ruby and C# have similar solutions.
You said "I can design any API I can create in such a way that it
offers me the ability to plug in fake dependencies". Isn't that fake
dependency also a mock? I think perhaps we are really only "arguing"
over how mocks are set in place for testing, rather than the need to
mock dependencies.
More importantly, though, once I have that API, and I've tested the
dependent code, how do I test the real implementation of the
dependency? Say, for example, I have my API implementation that itself
depends on the file system. How do I test that it correctly handles
all manner of file system errors, or permissions issues? Or if it
relies on HTTP, how do I test that it reacts correctly to all the
different response codes, or malformed response data, or other errors?
At some point, regardless of the abstractions that you build on top
that help you test the upper layers, you still need to test the code
"at the bottom" that depends directly on, for example, the file
system, or HTTP, or sockets. I'm not seeing how you can test all the
error cases without mocking those dependencies, but I'm all ears if
there's a good way to do that.
--
Martin Cooper
I don't believe anyone mentioned performance, actually.
>> First, the per-method hijacking can lead to problems if you're not
>> mocking all of the functions that you end up calling, directly or
>> indirectly. An example is a module that maintains state; mocking some
>> functions that would normally change the state may lead to non-mocked
>> functions that use that state now having / returning incorrect data.
>> This is why I prefer to mock on a per-module basis rather than a
>> per-function one.
>
>
> You got a point here.
> Unfortunately If you need to mock the whole module you've to come up with a
> totally new require() system.
> This is a only way I could found out fix this without worrying too much :)
Not a new require system, no. But this is one reason I ended up writing Mockery.
>> Second, since horaa calls 'require' directly, the context of its
>> invocation is not the same as it would be without horaa; the parent
>> module, for example, is different. Thus relative paths and dependent
>> modules may not be resolved correctly.
>
>
> Actually this is how node's module system works.
>
> 1. module will load to the memory when first time it saw the module
> 2. after that every require() takes the module from the memory
> 3. What horaa does is replacing the public methods in the memory directly
> 4. All you have to do is load the module you want to test relatively
> 5. It just work
My point, though, is that every module gets its own 'require' function
that operates in the context of that module. Since horaa invokes
'require' directly, it is doing so in the horaa context, and not in
the context of the module that is using horaa. Module M calling
'require' is not the same as module M calling horaa and horaa then
calling 'require'.
It's also fairly easy to see that a relative path passed to horaa may
resolve to a different absolute path than the author intended, because
it will depend on the resolution as applied within horaa and not the
resolution as applied within the originating module.
--
Martin Cooper
Fair enough. Unfortunately, though, we don't always control all of the
interfaces we need to work with, and they're not always amenable to
such methods. We still need to test against those too.
> As for testing the real functionality, end to end / itntegration acceptance
> testing is a long standing and proven way to test the big picture. I don't
> need mocks then, i just actually do it. There's no substitute IMO.
Sorry, I guess I am still missing something. Suppose my app is running
against some service I don't control, such as AWS or a Google service
or something. Or perhaps just Couch or Mongo. How do I test that my
code "does the right thing" in the face of errors from such services?
Those errors may only show up once in a blue moon, so I can't depend
on that happening during my testing. But I still need to know that my
app isn't going to fall over when it does happen, so I want to be able
to test the error cases before I go live.
I think we're going in circles now. :-)
In the above scenario, suppose the couch/mongo API is your own, and
that both fake and real implementations are yours. How will you test
that the code in your real implementation (sitting on top of 'http')
handles all the different kinds of errors and response codes you might
get from the database, so that your API behaves as expected? I
understand that your fake implementation lets you test the
higher-level code, but what about testing the lower-level code?
--
Martin Cooper
If you're testing an http client app, why not just make an http
server, and make requests to it? What's the difference between a
"fake" http module, and the real one, just listening on localhost?
There isn't built-in support, but Nock looks interesting for this kind of thing:
--
Martin Cooper
I prefer to mock the methods that do the http requests if possible though.
--
Mockery also enables this (using a different approach from that of
node-sandboxed-module).
https://github.com/mfncooper/mockery
--
Martin Cooper
You guys should really check nock. Really. :)
I'm currently using Mocha with Mockery for unit testing. I can't figure out a good way to deal with this situation:Test A requires a real module.Test B and mocks that same module.Running Test A in isolation passes.Running Test B in isolation passes.Running both at once: Test B fails every time.In the third scenario, Test B fails because the real module has already been loaded (even if Test A hasn't run yet).
This occurred with Gently as well, and with nodeunit instead of Mocha, so I don't think it's an issue with Mockery or Mocha. But I'm still guessing there's an easy solution to what is a trivial problem in other languages, and I'm just missing it. I'm fine with using a different mocking library if that's the solution, as long as it's simple to use and doesn't require me to pollute source files with weird statements like Gently does.
I have a repeatable set of source code and tests here: https://github.com/BryanDonovan/nodejs-mock-testAny advice would be greatly appreciated.
Thanks,Bryan
The bug has been fixed in Mockery 1.1.2, now available via npm. The
cause was a change in the Node module loader starting in Node v0.6.10.
The latest Mockery now works with both older and newer versions of
Node.
--
Martin Cooper