Requests per second. How many is good?

[Stanislav]

We are developing a large site and we love Tornado, however we were
wondering how many requests per second is a good number for a single
Tornado instance? This instance is of course doing queries to SQL, and
rendering a template.

I would greatly appreciate a reply.

[keztoo]

There are 86,400 seconds in a day. If you handle 1 request per second
you are handling 86,400 requests per day. If you handle 10 requests
per second you are handling 864,000 requests per day (or nearly 1
million requests a day) and if you can handle 100 requests per second
you are handling 8,640,000 (over 8 million), so 100 requests per
second is usually a good starting point. Unfortunately, usage is not
evenly distributed like this and you tend to get bursts of requests
which can impact your server.

In your case, another factor to consider is how much data you deliver
(a 1k page, a 10k page, a 100k page, etc) and how long your sql
requests take. I tend to use a service oriented architecture where the
sql requests go over the wire to a farm of services which do the
actual sql access. I use async call backs on the front end and using
this approach I have noticed a communication overhead of about 2ms per
request. The actual sql takes about 20ms and misc overhead contributes
enough for us to say a typical request will take about 25ms.

Assuming a single threaded approach where requests happen serially
(which is not the case, but just for performance analysis) this means
you can get about 40 requests per second (1,000 ms per second divided
by 25ms per request) out of a single single threaded instance. Now in
the case of tornado you can get more than one request going at a time,
so you will get more than 40 requests per second out of a single
tornado instance.

Usually the limiting factor is the page payload and front-end
bandwidth. Assuming you deliver 100k per page and you have a 10mbs
link to your website, you can deliver about 10 pages a second
(remember 100k is in bytes, mbs is in bits; there is about a 10:1
ratio here) or put another way; you can support 10 simultaneous
users.

But lets forget about front-end bandwidth concerns (assume you are on
a cloud with unlimited bandwidth) and concentrate on server
performance. Now lets say each tornado instance as described above can
deliver roughly 40-100 requests per second (which is what we see on
our small cloud machine which is a 1Ghz Athelon with about 380Mb of
memory) then you can easily increase this by using nginx to load
balance several of the front-end (we call them MVCs or on-lines)
machines in parallel. To keep this post brief I will summarize what we
currently see ...

Using 1 Front-ends and 1 service instance we get about 25 rps.
Using 2 Front-ends and 2 service instances we get about 40 rps.
Using 4 Front-ends and 16 service instances we get about 80 rps.
Using 8 Front-ends and 32 service instances we get about 125 rps.

Not what we were expecting but we are investigating what is wrong as I
speak (for example, we also have couch db running on that box, a rails
app and some other stuff). Now keep in mind when I say "Using 8 Front-
ends and 32 service instances we get about 125 rps" what I really mean
is on that single 1Ghz machine (with about 380Mb of memory) we start
one instance of nginx (to load balance both the front and back ends)
32 instances of our back end service and 8 tornados (each on its on
port obviously) so we are probably overloading our poor 1Ghz cloud
machine but like I said, we are currently investigating. Still, we get
about 100rps for a single very small machine. I know that when I did
some tests at work (the small cloud machine is for a home start-up we
are working on) on dedicated hardware I was seeing closer to
500-700rps which is about what I expected and is not bad for a single
server.

So to summarize; if you want to support about a million requests a day
you need to be able to handle at least 10 requests per second
sustained. This should only require one small server.

Hope that helps a bit.

[Stanislav]

That is a wonderful response, I was just hoping for a target point to
aim for when benchmarking our site. I benchmarked various pages of our
site from pages that use 1 SQL query to more, and and one instance of
tornado goes from 25-30 to 70 RPS depending on page, so I am guessing
based on you reply that our RPS is in good standing.

[Stanislav]

While we are at it, out of curiosity how many open long polling
connections could one Tornado hold?

[keztoo]

On that same box I have 200 open at the same time but I am sure the
number is much higher as that was a simple test using a single python
app per request and we were running from swap almost immediately. I
would guess that 500-1000 (if they were not very busy) would be
possible.

[Stanislav]

I am having a weird thing going on while testing with AB.

A handler that according to tornado finishes in 10ms is getting half
the RPS as a handler that is generated in 30ms, and I am pretty
clueless atm.

[Stanislav]

Never mind, that was a massive brain fart didn't take into account my
download rate and response size =), RPS is now ranging from 150-300
and and thats running the test locally to the server.