does tornado AsyncHTTPClient support http keep-alive?

[Peng Dai]

I would like to write a server which would sometimes send a http-request to other server. I use no keep-alive first and saw it was really slow.

so I now use keep-alive, I test it with requests first, it seems the speed more fast. but I do not know how to use this in tornaod AsyncHTTPClient.

does it support this features?

[Yuan Wang]

AsyncHTTPClient dosn't handle or support keep-alive connection header. Every fetch will create a new iostream object.

--
You received this message because you are subscribed to the Google Groups "Tornado Web Server" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python-tornad...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

-----------------------------------------------------------------------------------------------------
王 远
QQ  ：807228688
 

[Ben Darnell]

simple_httpclient (which is used by default) cannot reuse connections, but curl_httpclient can. Simply install libcurl and pycurl and then run `AsyncHTTPClient.configure('tornado.curl_httpclient.CurlAsyncHTTPClient')` at the start of your program to use it.

-Ben

--

[Peng Dai]

hi, the curl_http_client do make the request much more quickly. But I got some confuse as below.

I write a script to test the speed of SimpleAsyncHTTPClient adn CurlAsyncHTTPClient,

  

import timeit

import time

from tornado import gen

from tornado.ioloop import IOLoop

from tornado.httpclient import AsyncHTTPClient

io = IOLoop.instance()

url = "http://www.easemob.com/hx/index.html"

@gen.coroutine

def simple_speed():

    client = AsyncHTTPClient()

    for i in range(5):

        yield client.fetch(url)

@gen.coroutine

def curl_speed():

    AsyncHTTPClient.configure('tornado.curl_httpclient.CurlAsyncHTTPClient')

    client = AsyncHTTPClient()

    for i in range(5):

        yield client.fetch(url)

# WARNING:  comment the three line, the next will finish quick

start = time.time()

print 'simple', simple_speed()

print 'end', time.time() - start

start = time.time()

print 'simple', curl_speed()

print 'end', time.time() - start

the both need 16 second to finish, but if I comment the first timer-code, the last timer-code will finish very quickly.

why just yield 5 future need 16 seconds?

why the last timer-code behavior so diffrently?

[Peng Dai]

and also, when I use 

io = IOLoop.instance()

io.run_sync(simpe_speed, timeout=10000000)

it will always give a timeout-error, even though I have set the timeout paramters to a very big number.

but the curl_speed works ok.

[Ben Darnell]

AsyncHTTPClient instances are cached; configure() must be called before the first instance is created.

If you want to compare the two classes side-by-side, create them directly instead of using AsyncHTTPClient.configure:

  simple_client = tornado.simple_httpclient.SimpleAsyncHTTPClient()

  curl_client = tornado.curl_httpclient.CurlAsyncHTTPClient().

Also, simply running a coroutine at the top level of your program will not wait for it to complete; you must run it on an IOLoop, perhaps with IOLoop.instance().run_sync(simple_speed).

-Ben

[Ben Darnell]

On Tue, Mar 3, 2015 at 12:42 AM, Peng Dai <daipen...@gmail.com> wrote:

and also, when I use 

io = IOLoop.instance()

io.run_sync(simpe_speed, timeout=10000000)

it will always give a timeout-error, even though I have set the timeout paramters to a very big number.

but the curl_speed works ok.

My guess is that that number overflows an int32 when converted to milliseconds; try a more reasonable value like 3600 (1 hour).

-Ben

[Peng Dai]

yes, you are right. set the timeout to 3600 work fine. thank you.

[Peng Dai]

first,  I know simple running a coroutine will not wait do the request, I just accidently saw yield 5 request with SimpleHTTPClient need 14 seconds, so I come to test for  it, not use io.run_sync() was not a mistake.

And, I have got this by checkout the tornado source-code. and the speed-improvment was really amazing.

import timeit

import tornado.simple_httpclient

import tornado.curl_httpclient

from tornado import gen

from tornado.ioloop import IOLoop

io = IOLoop.instance()

url = "http://www.easemob.com/hx/index.html"

@gen.coroutine

def simple_speed():

    client = tornado.simple_httpclient.SimpleAsyncHTTPClient()

    for i in range(5):

        yield client.fetch(url)

@gen.coroutine

def curl_speed():

    client = tornado.curl_httpclient.CurlAsyncHTTPClient()

    for i in range(5):

        yield client.fetch(url)

print "simple", timeit.repeat("io.run_sync(simple_speed, timeout=3600)",

                              setup="from __main__ import simple_speed, io",

                              number=3)

print "curl", timeit.repeat("io.run_sync(curl_speed, timeout=3600)",

                            setup="from __main__ import curl_speed, io",

                            number=3)

simple [59.33778500556946, 62.93559217453003, 71.27185702323914]

curl [11.058576107025146, 2.5515811443328857, 2.5450439453125]

At last, I was realy confuse, why the SimpleAsyncHTTPClient yield 5 request need 14 seconds. I check the source code, but not really understand it, it seems like the SimpleAsyncHTTClient will make connection but not simple yield a Request-future.