Passenger max request queue size

[Sean Mikkelsen]

Hey guys, this is not strictly a Ruby question, but I'm hoping someone has an idea on what direction I should take to solve this problem.

So, long story short, I've got a rails app that is pretty basic. It takes a request, does a look up in the database, and renders a page. Not much to it.

The problem is that I am getting way too much simultaneous traffic, and something cannot handle the load. I've watched the server load, and it stays extremely low. Its almost as if nothing is happening on the server. but watching the passenger log, its taking requests so fast that I can't even read them as they go by. What I've noticed is that more than half of the requests are given an error from passenger because the queue is full.

example:

[ 2014-01-10 01:44:05.5626 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:05.6456 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:05.7382 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:06.3655 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:06.5241 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:07.0543 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:07.0946 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:07.6608 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:07.7320 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:07.9560 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:08.2632 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:08.6933 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:08.7497 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:08.8230 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:08.9167 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:09.1765 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:09.4440 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:09.6939 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:09.6972 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:09.7366 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:09.8361 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:10.1738 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

[ 2014-01-10 01:44:10.2735 6822/7fa08d717700 Pool2/Group.h:411 ]: Request queue is full. Returning an error

I've done some reading on passengers page here:

http://www.modrails.com/documentation/Users%20guide%20Apache.html#PassengerMaxRequestQueueSize

as well as on a few forums. I understand the problem in theory, but I don't understand what is causing it. The server clearly doesn't have a problem handling the load. its a 2 gb box on rackspace and this is the only app currently running on it. 

So there is an obvious thing I can do which is increase the queue size, but that will likely make the problem worse for users. Is there something at the application level that just can't handle the volume of requests? Or is this possibly a mysql problem? I'm using a dedicated database cloud solution so i'm not sure how I can check the health on that. Not sure how to see the load, etc.

Also, it may be worth noting that I increased the max connection pool in database.yml to 10 from the default of 5. I did this a while ago for other reasons, this could have an affect either positive, or negative.

Since my expertise do not lie in the server administration area, the best I can come up with is to set up a load balancer with smaller app servers to split up the traffic. I'm just hoping there is some simple config that I can do to fix it instead as it doesn't appear to be putting a dent on my server load.

Any ideas would be amazing!

[Peter Hamilton]

You need metrics. There are a dozen solutions out there too provide them. Establish a baseline of metrics, then find bottlenecks and try to improve them. Always compare to the baseline.

Once you have metrics, I suspect you'll see to many threads are blocked on IO. That's why CPU usage is low. Increase number of passenger workers. Add caching for frequent requests. Use HTTP caching headers.

--
http://utruby.org
http://groups.google.com/group/urug
 
please prefix the subject with [JOB] when regarding job opportunities
---
You received this message because you are subscribed to the Google Groups "Utah Ruby Users Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to urug+uns...@googlegroups.com.
To post to this group, send email to ur...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Benjamin Bytheway]

What is your PassengerMaxPoolSize set to?

--

[Sean Mikkelsen]

I literally just read about this for the first time right before you emailed :). Like I said... I'm not super good with the server end of things. I assume if I don't pass in this option when I start passenger, then it defaults to 1?

Also, its worth noting that i'm using passenger standalone. Not sure if that makes much of a difference either way. 

I am going to try setting max pool size to 8, and min instances to 8 also since the traffic is going to come in bursts. according to what I read, it can take some time for passenger to adjust to irregular traffic so this prevents that problem.

Not sure what a good number is for pool size until I play with it i guess. I agree that metrics would be useful Peter, i'll look into that as well.

[Benjamin Bytheway]

The default pool size is 6, and since ruby (MRI) has the global lock, it effectively is limited to processing one request per process, so you are basically limited to 6 (or your setting of 8) concurrent requests. If you get more concurrent requests than that, they'll queue up waiting for one of those processes to get free.

[Sean Mikkelsen]

Yes, i'm using MRI, I understood that part at least, but did not know about the multiple instances of passenger, so that helps a ton. Sounds like i'll need a lot more than 8 :). I'll keep playing with it and see what I come up with. That at least points me in the right direction cause I had no idea where to start.

Thanks again for the help!

[Michael Ries]

+1 to Peter's comments. without more detailed measurements it will be really hard for you to figure out where the bottleneck is.  It could be something in your app writing to disk, or a querying that is taking a long time under load, inserts to a table that has to keep a big index up-to-date, etc etc.

Rails has some default logging stuff built in so you can check your logs and see how much time rails spent in the database, rendering the views, etc which will give you a general direction to look, but getting more detailed measurements (like how long a given query is taking to run) will much more valuable. If you are new the process you might want use New Relic since it handles most of the instrumentation for you and tries to lead you in the right direction.