Basic problem with tornado.httpclient.AsyncHTTPClient()

[Don]

When my handler 'fetches' a URL from a remote server, the response
callback is called correctly (with a successful reply). But
self.finished() has already been called on the initial request so I
can't formulate a response. For example I can't call self.render() due
to an assertion error in web.py.

Any ideas why the original request has been 'finished' already?

This all works when I call the remote server with the non-async
tornado.httpclient.HTTPClient() (minus the callback function).

CODE SNIPPET:

class signup_handler(tornado.web.RequestHandler):
@tornado.web.asynchronous
def post(self):
...
http_client = tornado.httpclient.AsyncHTTPClient()
response = http_client.fetch("http://api-verify.recaptcha.net/
verify",
method="POST",
body=urllib.urlencode(post_args),
callback=self.async_callback
(self.on_captcha_response))
...
def on_captcha_response(self, response):
if self._finished: print "msg1" <- prints out msg1 to console
because original request has been 'finished' already
...

CONSOLE:

[I 091221 10:24:59 web:714] 200 POST /signup (1.2.3.4) 2.85ms
[I 091221 10:24:59 httpclient:395] POST http://api-verify.recaptcha.net/verify
msg1
[E 091221 10:25:00 web:664] Exception after headers written
Traceback (most recent call last):
File "/usr/lib/python2.5/site-packages/tornado/web.py", line
660, in wrapper
return callback(*args, **kwargs)
File "/root/my_app/my_app.py", line 168, in on_captcha_response
self.render("sentvalidation.html", email=email)
File "/usr/lib/python2.5/site-packages/tornado/web.py", line
366, in render
self.finish(html)
File "/usr/lib/python2.5/site-packages/tornado/web.py", line
437, in finish
assert not self._finished

[David P. Novakovic]

There is probably something in the "..." part that is doing something that triggers the finished condition? Can you show the rest of the code?

[Don]

> There is probably something in the "..." part that is doing something that
> triggers the finished condition? Can you show the rest of the code?

Sure can. Its still abridged but fuller - follows. Note that 'msgA' is
not printed but msg1 *is* printed. In other words by the first line of
the callback, the problem has already occurred (self._finished is true
which causes the assertion failure in web.py: assert not
self._finished). Thanks in advance for your attention.

def post(self):

# Get parameters
name = self.get_argument("name", default="");
...more get parameters

# Validate parameters
if len(name) == 0:
self.render("signup.html", msg="Please specify a name then retype
the password and two words", name=name, email=email, password="",
timezones=common_timezones, timezone=timezone)
return
... more validate parameters

# Ask the reCAPTCHA server if the recaptcha_response is correct
http_client = tornado.httpclient.AsyncHTTPClient()
try:
post_args = {"privatekey":
"some_key_excluded_from_code_for_security",
"remoteip": self.request.remote_ip,
"challenge": recaptcha_challenge,
"response": recaptcha_response}

response = http_client.fetch("http://api-verify.recaptcha.net/
verify",
method="POST",
body=urllib.urlencode(post_args),
callback=self.async_callback(self.on_captcha_response))

except tornado.httpclient.HTTPError, e:
self.render("signup.html", msg="Error: %s" % e, name=name,
email=email, password="", timezones=common_timezones,
timezone=timezone)
return

if self._finished: print "msgA"

def on_captcha_response(self, response):

if self._finished: print "msg1"
if response.error: raise tornado.web.HTTPError(500)

# Get parameters
name = self.get_argument("name", default="");
... more get parameters

# Test Captcha response
response_lines = response.body.splitlines()
if response_lines[0] != "true":
self.render("signup.html", msg="Error ["+response_lines[1]+"],
please retype the password and the two words", name=name, email=email$
return

...

[David P. Novakovic]

Try printing something in your except tornado.httpclient.HTTPError, e: block. If an exception is thrown there that doesn't cause the request to fail the self.render will run which will cause the response to 'finish'.

Basically be vary careful that .render is not being called. This is normally where i spot this particular error.  Remember.. once fetch is called the event is in IOLoop.. just because an error is thrown doesn't mean the response will never be called!

Anyway... just a bit of theorising there.. not sure how much of that is 100% fact. I'd try putting a print into the except block.

David

[Don]

David:

I follow your idea, good thinking. I put a print statement in the
exception block but it didn't get printed. Next I tried taking out the
whole try/except (i.e. not catching any exceptions and therefore not
potentially rendering anything) and still, the request is getting
prematurely finished.

I'm at about wits end. Thanks for trying. Any other ideas? If not I'll
leave it for a few days.

BTW, I could not find any example code that does an async 'POST', any
throughts on whether POST vs. GET would make any difference? I
inspected web.py and there's not much cost difference between the two
operations.

- Don

On Dec 21, 5:25 pm, "David P. Novakovic" <davidnovako...@gmail.com>
wrote:

> Try printing something in your except tornado.httpclient.HTTPError, e:
> block. If an exception is thrown there that doesn't cause the request to
> fail the self.render will run which will cause the response to 'finish'.
>
> Basically be vary careful that .render is not being called. This is normally
> where i spot this particular error.  Remember.. once fetch is called the
> event is in IOLoop.. just because an error is thrown doesn't mean the
> response will never be called!
>
> Anyway... just a bit of theorising there.. not sure how much of that is 100%
> fact. I'd try putting a print into the except block.
>
> David
>

[David P. Novakovic]

Don,

Just to confirm, you have the web.asyncronous decorator on the post method? I just noticed it was missing from your semi-full example.

I use async posts (a twitter client). They work fine.

David

[Don]

> Just to confirm, you have the web.asyncronous decorator on the post method?
> I just noticed it was missing from your semi-full example.

Uh, NO! I had it on the class (due to a lack of understanding of
decorators). I translated the example in the docs:

class MainHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous
def get(self):
...

into:

class MainHandler(tornado.web.RequestHandler):
@tornado.web.asynchronous
def get(self):
...
def post(def):
...

Changing that to:

class MainHandler(tornado.web.RequestHandler):
def get(self):
...
@tornado.web.asynchronous
def post(def):
...

... fixed it. Thanks so much for your help! Scoot me an email to
carbonsink [at] gmail.com, I'd like to send you a little thank-you.