value in a mocking wsgi server?
Mark Hammond
The MoMo team is working on a couple of Python based projects and we
have been pondering how to best test these projects - both
unit/functional tests and stress/scale tests. I spent some time
looking for existing tools for this purpose and didn't find one thing to
meet the requirements. I'd like to outline an idea I have for such a
tool and hope this little list has both the experience and the
signal-to-noise ratio for good feedback - including negative feedback :)
Depending on the feedback I get here, I may also take it to the Web-SIG...
Consider our experimental 'linkdrop' application - it is a web service
built using pylons which itself uses external web services beyond our
control - services like twitter, facebook, URL shorteners, etc. The
testing scenarios we would like to support are:
* Regular unit/functional tests of the application. For example, we
want to unittest the 'send' functionality of linkdrop without actually
reaching out to twitter to post a status update. In this scenario we
could probably use in-process monkey patching - somewhat similar to
wsgi_intercept - to avoid an outgoing connection from being made.
* Stress testing/benchmarking the application. We'd like to create a
production-like environment and use tools like "ab" or similar to put
the server under heavy load to see how it reacts and scales. In this
scenario we do *not* want to 'stub' the calls to twitter etc - we want
our server to make a real outgoing connection to something that
*behaves* like twitter - including the random failwhales, raindom slow
responses, etc - but doesn't need to actually store the tweets, etc.
Ditto for facebook, url shorteners, etc.
While I didn't find an existing tool to meet these requirements, I did
find a collection of tools which could be cobbled together. I've been
experimenting with something that stitches together
pylons/paste/routes/etc (Hi Ian ;), wsgi_intercept (Hi Ian again ;) and
the 'mocker' library by Gustavo Niemeyer (http://labix.org/mocker)
In short, there is a server that starts without any special 'mocking'
characteristics. You then make requests to the server (using a helper
lib, or simply using curl etc) to setup requests you want mocked and how
specific request arguments should be handled, then you let your
application run as normal - but instead of talking to twitter.com etc,
it talks to the server and gets back the pre-arranged responses. Using
wsgi_intercept, the same server can be used in-process for regular unit
testing.
I'm pasting part of a "readme" and I'm looking for general feedback
before I blow any more time on it - in particular:
* Is there something I've missed which could be used instead or would be
better suited?
* Does this sound like it is meeting a real requirement others might
find interesting?
* Am I insane for thinking this is a reasonable idea?
The mock readme for the tool, which I've tentatively called 'parody', is
below (and I've a mock implementation which works as described below)
Thanks,
Mark
'Parody' is a mockable HTTP server. It is designed to help all facets
of testing an application (eg, a client application or a web service)
that itself depends on other web services which are out of your control.
For example, consider a web application which allows posting of status
updates to twitter - you can arrange for 'parody' to take the place of
twitter, returning whatever responses (including failure responses) you
need for the particular testing scenario.
It is built using 'best of breed' tools for the purpose:
* The 'mocker' library is used to to define and return mock responses
from remote web services.
* pylons and friends are used so parody can run as a real web server -
ie, your app will make socket connections, but to the parody server
instead of the real web server (eg, twitter). This is particularly
useful for stress or load testing of your application.
* wsgi_intercept is leveraged so parody can run in-process. In this
scenario, no socket connection is made to parody so it is particularly
useful for regular unit testing.
* The 'routes' package (also used by pylons etc) is used to allow very
flexible URL and argument matching for the server being mocked.
A helper library is provided so you can use the same steps to configure
the mocking behaviour whether parody is used in-process or as an
external server. Alternatively, tools like curl could be used to
configure the tool when it is being run as an external server.
An annotated example of mocking twitter:
>>> from parody.lib.intercept import add_intercept
>>> from parody.lib import consumer # some other setup code has been
snipped.
First, we tell parody we want to mock the twitter URL for posting status
updates.
>>> add_intercept("api.twitter.com", 80)
>>> mocker = consumer.create("/statuses/update.{format}")
Next, we tell parody how to mock this URL when it is called using JSON
format and with a particular status message:
>>> recorder = mocker.record("/statuses/update.json?status=hello")
>>> recorder.result('{"id": 1234}')
We can then use the twitter library to post a status update:
>>> from twitter.api import Twitter
>>> t = Twitter(...)
>>> t.statuses.update(status="hello")
{u'id': 1234}
In the example above, the twitter library connected to the parody server
and issued its normal request for posting a status update.
Another example to mock a similar request to facebook:
>>> add_intercept("graph.facebook.com", 80)
>>> mocker = consumer.create("/me/feed")
>>> recorder =
mocker.record("/me/feed?access_token=letmein&message=hello")
>>> recorder.result(value='{"id": 6666}')
In the example above, we configure parody to return a valid response
when a particular message and access token are supplied in the request.
Although not shown above, additional features of the 'mocker' library
allow for you to specify how many times and in what order calls must be
made, return different values for each request, simulate HTTP errors in
certain conditions, etc.
For more information on:
* The 'mocker' package: http://labix.org/mocker
* Why 'mocks' are different to 'stubs':
http://martinfowler.com/articles/mocksArentStubs.html
Ian Bicking
httplib is actually pretty easy to mock out. Surprisingly wsgi_intercept doesn't, and as a result is more complicated than it really needs to be IMHO, while also not mocking everything consistently. If you are using any of urllib, urllib2, or httplib2, they all go through httplib consistently (I had a chance to test this with my httplib monkeypatch in App Engine, which directs everything through App Engine's urlfetch function). If you are using curl or anything async, then patching httplib probably won't do anything, and maybe general mocking makes sense.
I personally find the general *concept* of wsgi_intercept useful, so if you aren't doing curl/async (and curl is just more mocking work) I'd still try to route things into a WSGI app. For instance:
from webob.dec import wsgify
from webob import Response
twitter_app = Response(json.dumps({some data}), content_type='text/json')
@wsgify.middleware
def delay(app, req, delay_time=1, prob=0.5):
if random.random() < prob:
time.sleep(delay_time)
return app
test_twitter_app = delay(twitter_app, delay_time=5, prob=0.1)
And you can throw in occasional overloads, retries, etc. Even throw socket errors from within the apps, they'll just bubble up largely indistinguishable from the real thing.
If possible I'd concentrate all the mocking in httplib and not use mock objects. The App Engine code would be a decent basis if you didn't use wsgi_intercept itself (http://googleappengine.googlecode.com/svn/trunk/python/google/appengine/dist/httplib.py).
Ian Bicking | http://blog.ianbicking.org
Mark Hammond
> I personally find the general *concept* of wsgi_intercept useful, so if
> you aren't doing curl/async (and curl is just more mocking work) I'd
> still try to route things into a WSGI app. For instance:
I'm not sure my rambling post was clear, and in particular, mentioning
wsgi_intercept made things even more confusing.
In the model I described, there *is* a WSGI app, and this WSGI app is
what hosts the mocking objects. The app under test has no mocking
objects or stubs at all. So instead of having explicit code in the test
server like:
> from webob.dec import wsgify
> from webob import Response
>
> twitter_app = Response(json.dumps({some data}), content_type='text/json')
>
> @wsgify.middleware
> def delay(app, req, delay_time=1, prob=0.5):
> if random.random() < prob:
> time.sleep(delay_time)
> return app
>
> test_twitter_app = delay(twitter_app, delay_time=5, prob=0.1)
You would instead make a few POST requests to the server to configure
its mocking behaviour to do what you describe above. IOW, when the WSGI
server is started it has no special knowledge about twitter responses or
random delays - that knowledge comes from the app under test based on
its requirements.
Assuming such a WSGI app exists, wsgi_intercept or similar could be used
to run the exact same server in-process for unit-testing - but that
detail is a distraction from the main idea.
> If possible I'd concentrate all the mocking in httplib and not use mock
> objects. The App Engine code would be a decent basis ...
I hope I've made things a little clearer - we don't really want to mock
httplib inside the app being tested. We want the app being tested to
use regular httplib to make a *real* socket connection out to a server
under our control, and that server is where the mocking of the twitter
API happens. Assuming the app under test uses a configurable host name
instead of 'twitter.com', no intercepts or mocks would happen in the app
at all.
You don't seem to be a big fan of mocking tools, which I can sympathize
with. While http://martinfowler.com/articles/mocksArentStubs.html makes
a compelling case that mocks are superior to stubs, the distinction is
very subtle in most cases. It would not be hard to convince me that
mocking objects are overkill here and that using explicit code like you
demonstrated above for twitter would be better suited - but I quite like
the idea that the app under test is capable of controlling the test
server responses, simply so that the test server itself doesn't need to
be changed for each new testing scenario.
Thanks for the reply and while I hope I've made things a little clearer,
it sounds like I've failed to make a case for a generic mocking WSGI
server :)
Cheer,
Mark
Ian Bicking
I might not entirely understand what you mean, but it sounds to me like you want to put up a fake twitter service on, e.g., localhost:8081, HTTP and everything. It doesn't sound particularly mock-like.
If that is the case, a couple tools that might be useful to you are wsgiproxy to start out with a fake service that is just a proxy to the real service, then using webtestrecorder to save a repeatable record (since performance tests should have a control against external issues), then some problem-inducing middleware like the delaying example I gave.
I'm still not entirely sure what you are looking for in this testing, but if you want http this at least might be the right direction?
--
Sent from a phone
On Sep 14, 2010 7:22 PM, "Mark Hammond" <mham...@skippinet.com.au> wrote:
Hi Ian,
> I personally find the general *concept* of wsgi_intercept useful, so if
> you aren't doing curl/...
I'm not sure my rambling post was clear, and in particular, mentioning wsgi_intercept made things even more confusing.
In the model I described, there *is* a WSGI app, and this WSGI app is what hosts the mocking objects. The app under test has no mocking objects or stubs at all. So instead of having explicit code in the test server like:
> from webob.dec import wsgify
> from webob import Response
>
> twitter_app = Response(json.dumps(...
Kumar McMillan
On Sep 13, 11:53 pm, Mark Hammond <mhamm...@skippinet.com.au> wrote:
> * Regular unit/functional tests of the application. For example, we
> want to unittest the 'send' functionality of linkdrop without actually
> reaching out to twitter to post a status update. In this scenario we
> could probably use in-process monkey patching - somewhat similar to
> wsgi_intercept - to avoid an outgoing connection from being made.
>
code is using a "trusted" API--i.e. calling the right Python methods
that you have already manually verified.
There are many tools for this but I really prefer writing declarations
of what I expect to happen, then executing code and letting it fail
when that declaration is not met. So I wrote a library called Fudge:
http://farmdev.com/projects/fudge/ I have actually used it for
exactly what you are talking about, to test Twitter API integration,
so there is an example of this in the docs:
http://farmdev.com/projects/fudge/using-fudge.html#fudging-a-web-service
Sometimes the declarative style works but other times it can make
things more complicated. In those cases I tend to use Mock which lets
you make assertions that methods were called after executing code.
http://www.voidspace.org.uk/python/mock/
I would recommend using some kind of library rather than just monkey
patching because otherwise you will be doing a lot of extra work in
setup/teardown.
-Kumar