cherrypy & concurrency
Mike Nolet
>>> for t in range(1,11):
... test(t*10)
...
count=10: blocking: 142ms, parallel: 37ms
count=20: blocking: 298ms, parallel: 62ms
count=30: blocking: 448ms, parallel: 84ms
count=40: blocking: 581ms, parallel: 103ms
count=50: blocking: 717ms, parallel: 9254ms
count=60: blocking: 863ms, parallel: 9451ms
count=70: blocking: 1014ms, parallel: 9583ms
count=80: blocking: 1142ms, parallel: 9726ms
count=90: blocking: 1314ms, parallel: 9869ms
count=100: blocking: 9034ms, parallel: 10023ms
>>>
cherrypy.config.update({'server.socket_host': my_ip_address,
'server.socket_port': my_port,
'server.thread_pool': 100,
'server.socket_queue_size': 50,
})
class DummyAPIServer2():
exposed = True
def GET(self,id=None):
time.sleep(0.01)
return 'Done sleeping!'
if __name__ == '__main__':
cherrypy.tree.mount(
DummyAPIServer2(), '/dummy',
{'/':
{'request.dispatch': cherrypy.dispatch.MethodDispatcher()}
}
)
While test is running CPU on the server is low... (7308 PID is the client, 35347 the server)
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
[:~]$ top -d 0.5 | egrep '(python)'
7308 mnolet 20 0 3192m 46m 4708 S 2.0 1.2 0:21.47 python3.3
35347 mnolet 20 0 3289m 45m 5656 S 7.9 1.2 0:07.48 python3.3
7308 mnolet 20 0 3002m 39m 4708 S 4.0 1.0 0:21.49 python3.3
35347 mnolet 20 0 3289m 45m 5656 S 13.9 1.2 0:07.55 python3.3
7308 mnolet 20 0 2652m 31m 4708 S 6.0 0.8 0:21.52 python3.3
35347 mnolet 20 0 3289m 45m 5656 S 13.9 1.2 0:07.62 python3.3
7308 mnolet 20 0 2301m 30m 4708 S 5.9 0.8 0:21.55 python3.3
7308 mnolet 20 0 2231m 30m 4708 S 2.0 0.8 0:21.56 python3.3
35347 mnolet 20 0 3289m 45m 5656 S 2.0 1.2 0:07.63 python3.3
7308 mnolet 20 0 2231m 30m 4708 S 2.0 0.8 0:21.57 python3.3
35347 mnolet 20 0 3289m 45m 5656 S 2.0 1.2 0:07.64 python3.3
7308 mnolet 20 0 2231m 30m 4708 S 4.0 0.8 0:21.59 python3.3
35347 mnolet 20 0 3289m 45m 5656 S 2.0 1.2 0:07.65 python3.3
7308 mnolet 20 0 2231m 30m 4708 S 4.0 0.8 0:21.61 python3.3
7308 mnolet 20 0 2231m 30m 4708 S 4.0 0.8 0:21.63 python3.3
35347 mnolet 20 0 3289m 45m 5656 S 2.0 1.2 0:07.66 python3.3
7308 mnolet 20 0 2231m 30m 4708 S 4.0 0.8 0:21.65 python3.3
I've read plenty of the posts that say "put Apache in front of it" or "put nginx in front of it"... but feels like something is hanging inside cherrypy here no? Especially since this is consistently reproducible by just firing the a decent # of concurrent requests.
Curious to hear thoughts!
-Mike
PS: My URL classes for reference... please be gentle on my python, still learning =)
class URLRequest(object):
def __init__(self,url=None,type='GET',params=None,data=None,headers=None,timeout=2.5):
super().__init__()
self.type=type
self.type=type
self.url = url
self.params = params
self.data = data
self.headers = headers
def request(self):
if self.type == 'GET':
return requests.get(self.url,params=self.params,data=self.data,headers=self.headers)
elif self.type == 'PUT':
return requests.put(self.url,params=self.params,data=self.data,headers=self.headers)
elif self.type == 'POST':
return requests.post(self.url,params=self.params,data=self.data,headers=self.headers)
elif self.type == 'DELETE':
return requests.delete(self.url,params=self.params,data=self.data,headers=self.headers)
class bgURLRequest(URLRequest,Thread):
callback_obj = None
callback_func = None
callback_message = None
def __init__(self, **kwargs):
super().__init__(**kwargs)
def set_callback(self,obj,func,message=None):
self.callback_obj = obj
self.callback_func = func
self.callback_message = message
def run(self):
try:
self.response = self.request()
except Exception as e: # TODO -> Better error handling!!
print ("OOPS!")
self.response = e
f = getattr(self.callback_obj,self.callback_func)
if self.callback_message != None:
f(self.response,message=self.callback_message)
else:
f(self.response)
And test function:
def test(count=10):
r1=[]
start_time = time.time()
for i in range(count):
u = multiurl.URLRequest(url='http://localhost:8080/dummy/1')
d = Dummy(u.request())
r1.append(Dummy(d))
time_blocking = (time.time() - start_time)*1000
r2=[]
start_time=time.time()
for i in range(count):
u = multiurl.bgURLRequest(url='http://localhost:8080/dummy/1')
d = Dummy()
u.set_callback(d,'callback')
u.start()
r2.append(d)
while len([i for i in r2 if i.waiting_on_response]) > 0:
time.sleep(0.005)
time_parallel = (time.time() - start_time)*1000
print('count=%s: blocking: %2dms, parallel: %2dms' % (count,time_blocking,time_parallel))
David Bolen
> The behavior I've noticed is cherrypy seems to hit a pretty serious
> performance wall when it comes to even reasonable levels of concurrency.
going to create a thread pool with 10 worker threads, and it won't (I
don't believe) grow beyond that automatically. It also has a default
listen queue of 5, so essentially can backlog up to 50 requests before
they start being rejected. I suspect that explains most of the
results.
Given your request handler is guaranteed to take 10ms + "request
overhead" to run - that seems to match the ~14ms/req you see pretty
consistently in your blocking tests up to 90 - though I'm not sure I
understand why you got the jump at 100, but perhaps some other issue
was going on.
For parallel requests, with perfect parallelism (which won't happen
due the GIL) you'd be looking at ~14ms/10req. Given they have to be
processed at most 10 at a time due to the thread pool, from 10-40
requests you are landing in a range of 25-37ms per 10 requests (the
highest being at 10 itself which would also have the fewest requests
to amortize fixed overhead across) which seems fairly reasonable.
Now, at 50 you've got a huge leap upwards, but only for parallel
testing. But that's also right around the point where you'd expect to
start dropping requests due to the listen queue filling. So it sort
of feels like a retry mechanism may be involved somewhere. However,
you don't mention errors on the client side, nor am I positive what is
actually issuing the requests (the requests library?), so maybe
there's some internal retry logic built in that is hiding the initial
rejected connections?
So as written, you've got a server that can handle about 400 req/s (at
25ms/10req), but at higher loads than that it'll start rejecting new
connections.
Of course, you can also bump up the thread pool or listen queue size
(either initially or dynamically - there's a grow method you can use
though I don't think CherryPy ever does it itself), but there are
diminishing returns - unless it's to process requests that depend on
external resources (so don't run afoul of the GIL for pure Python
code), it's not necessarily going to help throughput all that much.
So this is where the recommendations to offload the static processing
to a front end server start to come in. For example, if you've got
nginx handling all the plain images, css, javascript, etc.. (which
it's extremely good at doing) and only letting through the dynamic
requests to CherryPy, you leverage the higher overhead of CherryPy's
processing (and threads) for what really needs it. And 400 dynamic
requests/s is a pretty decent load.
-- David
Eric Larson
David Bolen writes:
> Mike Nolet <mno...@gmail.com> writes:
>
>> The behavior I've noticed is cherrypy seems to hit a pretty serious
>> performance wall when it comes to even reasonable levels of concurrency.
>
> By default, CherryPy's internal web server (see wsgiserver2.py) is
> going to create a thread pool with 10 worker threads, and it won't (I
> don't believe) grow beyond that automatically. It also has a default
> listen queue of 5, so essentially can backlog up to 50 requests before
> they start being rejected. I suspect that explains most of the
> results.
>
listen queue was set to 50.
>> cherrypy.config.update({'server.socket_host': my_ip_address,
>> 'server.socket_port': my_port,
>> 'server.thread_pool': 100,
>> 'server.socket_queue_size': 50,
>> })
threadpool stops being productive after a while. Unfortunately, I don't
have any specific tests or examples to prove this.
With that said, in our production apps that are served behind a load
balancer, we typically will have at least one process per core and 30
threads per process. This number *was* found at one point to be optimal
for our uses, but again, I don't have the tests or scripts to prove
this.
It does make some sense that a higher threadpool doesn't automatically
amount to better performance. Each thread is still utilizing the same
CPU. At some point adding more threads really is just adding more
context switches. Essentially this means more time is spent switching
between threads than doing the actual work.
I'd try to experiment with some sort of load balancer, start a couple
cherrypy processes with lower thread counts and see if that helps. If
you still are having a problem, you can also try something like uWSGI
that can be configured with more than one process as well as threads. I
believe you'd lose the benefit of the cherrypy bus (cherrypy.engine) but
that might be reasonable for your use case.
All that said, there are tons of projects that can help manage processes
and load balancing. CherryPy is plenty fast and the lack of complexity
regarding multiple processes along side a really solid process handling
model (via the cherrypy.engine) makes the developing with cherrypy over
time a huge win.
I say this as we have apps that are getting close to 10 years old that
haven't had to go through any major upgrades due to cherrypy. Upgrading
python versions, databases, template systems, etc. have all been
reasonably easy thanks to CherryPy's unrelenting focus on allowing you
to write plain Python code. In all that time, CherryPy has never once
been the bottleneck for any performance problems.
Hope this helps and best of luck!
Eric
David Bolen
> In the original post it looks like the threadpool was set to 100 and the
> listen queue was set to 50.
It does seem odd that the stats appear to be suggestive of the default
values - I wonder if somehow that thread configuration isn't actually
taking effect.
Or maybe I was just finding an expected pattern in the data.
> With that said, I've done similar tests and found that bumping the
> threadpool stops being productive after a while. Unfortunately, I don't
> have any specific tests or examples to prove this.
least a little better parallelism in the test case if that many
threads existed, even if it wouldn't help as much in a real production
code base. Each request handler is sleeping, which should release the
GIL, so there should be able to be quite a few requests overlapped in
parallel.
> I say this as we have apps that are getting close to 10 years old that
> haven't had to go through any major upgrades due to cherrypy. Upgrading
> python versions, databases, template systems, etc. have all been
> reasonably easy thanks to CherryPy's unrelenting focus on allowing you
> to write plain Python code. In all that time, CherryPy has never once
> been the bottleneck for any performance problems.
app so far). I do admit, however, that outside of development testing, I
always offload the static stuff to nginx.
-- David
Sylvain Hellegouarch
That's been my experience to date as well (though only 5 years on the main
app so far). I do admit, however, that outside of development testing, I
always offload the static stuff to nginx.
- Sylvain
http://www.defuze.org
http://twitter.com/lawouach
Robert Brewer
I don't mean to excuse CherryPy here: it may actually be slower than what you need. But *never* trust a benchmark or load test that is:
a. run on the same host as the server,
b. run with multiple Python threads (instead of separate processes),
c. run over localhost, or
d. hand-written on the spur of the moment ;)
You have no idea whether the bottlenecks are in your code, the client request library, the GIL in the client, or memory/CPU pressure from the client skewing the performance of the server; nor whether/how much your loopback interface mocks real network latency. Your mileage WILL vary, greatly.
Robert Brewer
fuma...@aminus.org
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cherrypy-user...@googlegroups.com.
To post to this group, send email to cherryp...@googlegroups.com.
Visit this group at http://groups.google.com/group/cherrypy-users.
For more options, visit https://groups.google.com/groups/opt_out.
Mike Nolet
I don't mean to excuse CherryPy here: it may actually be slower than what you need. But *never* trust a benchmark or load test that is:
a. run on the same host as the server,
b. run with multiple Python threads (instead of separate processes),
c. run over localhost, or
d. hand-written on the spur of the moment ;)
On Thursday, February 27, 2014 10:29:42 AM UTC+13, David wrote:
By default, CherryPy's internal web server (see wsgiserver2.py) is
going to create a thread pool with 10 worker threads, and it won't (I
don't believe) grow beyond that automatically. It also has a default
listen queue of 5, so essentially can backlog up to 50 requests before
they start being rejected. I suspect that explains most of the
results.
Now, at 50 you've got a huge leap upwards, but only for parallel
testing. But that's also right around the point where you'd expect to
start dropping requests due to the listen queue filling. So it sort
of feels like a retry mechanism may be involved somewhere. However,
you don't mention errors on the client side, nor am I positive what is
actually issuing the requests (the requests library?), so maybe
there's some internal retry logic built in that is hiding the initial
rejected connections?
So as written, you've got a server that can handle about 400 req/s (at
25ms/10req), but at higher loads than that it'll start rejecting new
connections.
Of course, you can also bump up the thread pool or listen queue size
(either initially or dynamically - there's a grow method you can use
though I don't think CherryPy ever does it itself), but there are
diminishing returns - unless it's to process requests that depend on
external resources (so don't run afoul of the GIL for pure Python
code), it's not necessarily going to help throughput all that much.
So this is where the recommendations to offload the static processing
to a front end server start to come in. For example, if you've got
nginx handling all the plain images, css, javascript, etc.. (which
it's extremely good at doing) and only letting through the dynamic
requests to CherryPy, you leverage the higher overhead of CherryPy's
processing (and threads) for what really needs it. And 400 dynamic
requests/s is a pretty decent load.
Mike Nolet
>>> test(50)
count=50: blocking: 738ms, parallel: 143ms
>>> test(100)
count=100: blocking: 1477ms, parallel: 305ms
>>> test(100)
count=100: blocking: 1484ms, parallel: 291ms
>>> test(1000)
count=1000: blocking: 14619ms, parallel: 3158ms
Michiel Overtoom
On Mar 3, 2014, at 20:40, Mike Nolet wrote:
> "use nginx in front"?
Greetings,