how to test gen.engine based code ?

[oddskool]

Hi there,

I'm having trouble __testing__ a code based on `gen.engine` :(

The code works fine when manually tested through curl, but I must be
missing wthg in the way I write the unit test :

The handler code, file async.py:
"""
import sys
from tornado import ioloop , gen
from tornado.httpclient import AsyncHTTPClient, HTTPRequest
from tornado.web import asynchronous, RequestHandler, Application

class MainHandler(RequestHandler):
url = u'http://third-party.com/api'

@asynchronous
@gen.engine
def get(self):
http_client = AsyncHTTPClient()
request = HTTPRequest(self.url, headers={'Accept-
Language':'fr'})
print >>sys.stdout, request.url
try:
response = yield gen.Task(http_client.fetch, request)
self.write(repr(response))
except Exception as e:
print >>sys.stderr, "Woops:", e
finally:
print >>sys.stderr, "flushing"
self.finish()

@staticmethod
def create_app():
return Application([(r"/", MainHandler),], debug=True)

application = MainHandler.create_app()

if __name__ == "__main__":
application.listen(8888)
ioloop.IOLoop.instance().start()
"""

now the unit test, file async_test.py:
"""
import sys
from tornado.testing import AsyncHTTPTestCase
from async import MainHandler

# inspired by http://www.tornadoweb.org/documentation/testing.html
class MyHTTPTest(AsyncHTTPTestCase):
def get_app(self):
return MainHandler.create_app()
def test(self):
self.http_client.fetch(self.get_url('/'),self.stop())
response = self.wait()
print >>sys.stderr, response
self.assertTrue(response,"empty response")

#python -m tornado.testing async_test
"""

And it fails like this :
"""

None
F
======================================================================
FAIL: test (async_test2.MyHTTPTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "async_test2.py", line 12, in test
self.assertTrue(response,"empty response")
AssertionError: empty response

----------------------------------------------------------------------
Ran 1 test in 0.008s

FAILED (failures=1)
[E 120201 17:05:48 testing:374] FAIL
"""

Any ideas or best way to do this __greatly__ appreciated !

Thanks Folks !

oDDsKooL

[Alexey Kachayev]

Hi! Did you check examples of such testing in gen_tests.py
(https://github.com/facebook/tornado/blob/master/tornado/test/gen_test.py)?

2012/2/1 oddskool <odds...@gmail.com>:

--
Kind regards,
Alexey S. Kachayev, Senior Software Engineer
Cogniance Inc.
----------
http://codemehanika.org
Skype: kachayev
Tel: +380-996692092

[Ben Darnell]

Remove the parens after self.stop in the first line of the test - you
don't want to call stop() immediately, you want to pass the function
itself as a callback to self.fetch.

-Ben

[oddskool]

Ouch ! thanks for pointing this...

However, now it fails with a timeout :

"""
======================================================================
FAIL: test (async_test2.MyHTTPTest)
----------------------------------------------------------------------
Traceback (most recent call last):

File "async_test2.py", line 11, in test
response = self.wait()
File "/usr/lib/python2.6/site-packages/tornado/testing.py", line
169, in timeout_func
timeout)
AssertionError: Async operation timed out after 5 seconds

----------------------------------------------------------------------
Ran 1 test in 5.013s

FAILED (failures=1)
[E 120202 09:02:50 testing:374] FAIL

> > # inspired byhttp://www.tornadoweb.org/documentation/testing.html

[oddskool]

Yes, I read those, but maybe didn't take enough time to understand
them...

I'll give it another try today

On 1 fév, 17:42, Alexey Kachayev <kacha...@gmail.com> wrote:
> Hi! Did you check examples of such testing in gen_tests.py

> (https://github.com/facebook/tornado/blob/master/tornado/test/gen_test...
>
> 2012/2/1 oddskool <oddsk...@gmail.com>:

> > # inspired byhttp://www.tornadoweb.org/documentation/testing.html

> ----------http://codemehanika.org
> Skype: kachayev
> Tel: +380-996692092

[oddskool]

Actually gen_test.py tests something slightly different : a handler
that asynchronously calls another handler on the same IOLoop.

Here is the relevant subset:

"""
class GenSequenceHandler(RequestHandler):
@asynchronous
@gen.engine
def get(self):
self.io_loop = self.request.connection.stream.io_loop
self.io_loop.add_callback((yield gen.Callback("k1")))
yield gen.Wait("k1")
self.write("1")
self.io_loop.add_callback((yield gen.Callback("k2")))
yield gen.Wait("k2")
self.write("2")
# reuse an old key
self.io_loop.add_callback((yield gen.Callback("k1")))
yield gen.Wait("k1")
self.finish("3")

class GenTaskHandler(RequestHandler):
@asynchronous
@gen.engine
def get(self):
io_loop = self.request.connection.stream.io_loop
client = AsyncHTTPClient(io_loop=io_loop)
response = yield gen.Task(client.fetch,
self.get_argument('url'))
response.rethrow()
self.finish(b("got response: ") + response.body)

class GenWebTest(AsyncHTTPTestCase, LogTrapTestCase):
def get_app(self):
return Application([
('/sequence', GenSequenceHandler),
('/task', GenTaskHandler),
])

def test_task_handler(self):
response = self.fetch('/task?url=%s' %
url_escape(self.get_url('/sequence')))
self.assertEqual(response.body, b("got response: 123"))
"""

My case is testing a handler that asynchronously calls a third-party
ws.

Trying to reduce even more the problem, here is an executable snippet
that fails:

"""
from tornado import ioloop , gen
from tornado.httpclient import AsyncHTTPClient, HTTPRequest
from tornado.web import asynchronous, RequestHandler, Application

from tornado.testing import AsyncHTTPTestCase
import sys

class MainHandler(RequestHandler):
url = u'http://www.google.com'

@asynchronous
@gen.engine
def get(self):
http_client = AsyncHTTPClient()

request = HTTPRequest(self.url)

response = yield gen.Task(http_client.fetch, request)

response.rethrow()
self.write(repr(response))
self.finish()

application = Application([(r"/", MainHandler),], debug=True)

class Test(AsyncHTTPTestCase):
def get_app(self):
return application
def test_task_handler(self):
response = self.fetch('/')
print >>sys.stderr, response

if __name__ == "__main__":
application.listen(8888)
ioloop.IOLoop.instance().start()
"""

launch it with `python -m tornado.testing async` for unit-testing : it
fails

launch the server with `python async.py` and then `curl -gi 'http://
localhost:8888/'` : it succeeds...

[Ben Darnell]

The problem is that your server code (and the AsyncHTTPClient it
creates) use the default IOLoop.instance(), while the tests create
their own IOLoop so the default IOLoop is not started. You need to
either refactor things so you can pass the tests's self.io_loop from
get_app through to the creation of the AsyncHTTPClient, or configure
the tests to use the default IOLoop (by overriding get_new_ioloop to
return IOLoop.instance() instead of creating a new one). The former
is slightly better since it makes sure that nothing accidentally
sticks around from one test to the next, but the latter is generally
simpler.

-Ben

[oddskool]

Cool, thanks for this explanation Ben :)

Here is the simplest, working option for future reference (maybe this
sample could fit into Tornado's documentation in a "recipes"
section ?) :

"""
from tornado import ioloop , gen
from tornado.httpclient import AsyncHTTPClient, HTTPRequest
from tornado.web import asynchronous, RequestHandler, Application

from tornado.testing import AsyncHTTPTestCase
import sys

from tornado.ioloop import IOLoop
class MainHandler(RequestHandler):
url = u'http://www.google.com'

@asynchronous
@gen.engine
def get(self):
http_client = AsyncHTTPClient()

request = HTTPRequest(self.url)

response = yield gen.Task(http_client.fetch, request)

response.rethrow()
self.write(repr(response))
self.finish()
application = Application([(r"/", MainHandler),], debug=True)
class Test(AsyncHTTPTestCase):
def get_app(self):
return application
def test_task_handler(self):
response = self.fetch('/')
print >>sys.stderr, response

def get_new_ioloop(self):
return IOLoop.instance()

if __name__ == "__main__":
application.listen(8888)
ioloop.IOLoop.instance().start()
"""

Coming back to my original case, I suspect the cleanest strategy to
test a ws that uses gen.engine to access a third-part API is to
encapsulate the creation of the task and override ii the test case so
that the task switches from a real `http_fetch` to a mock'ed method
returning a (recorded) response.

cheers,

oDDsKooL

[Felinx Lee]

You can post it to  https://gist.github.com/ , then share it to http://tornadogists.org/ 

--
What can change the nature of a man?(Planescape Torment)
----------------------------------------------------------------------------------------
http://feilong.me            Felinx Lee's Blog (Chinese Only)
http://www.zhimaq.com IT Q&A (Chinese Only)
http://poweredsites.org  What powered your sites? PHP, Ruby or Python?

[oddskool]

done: http://tornadogists.org/1730451/