Understanding Passenger Load Balancing and Global Queuing

[segedunum]

Hi All,

I've got a site that has just experienced a pretty large spike in
requests and activity above normal. The CPU on the application and
database server is well within limits and I've increased the number of
Passenger processes as well as introducing global queuing to no avail
really. Global queuing is the only way I can see if there is anything
being bottlenecked:

passenger-status
----------- General information -----------
max = 18
count = 18
active = 18
inactive = 0
Waiting on global queue: 54

----------- Domains -----------
/var/www/rails/site/current:
PID: 22283 Sessions: 1 Processed: 1 Uptime: 10m 17s
PID: 22293 Sessions: 1 Processed: 1 Uptime: 9m 37s
PID: 22287 Sessions: 1 Processed: 2 Uptime: 10m 8s
PID: 22285 Sessions: 1 Processed: 2 Uptime: 10m 14s
PID: 22291 Sessions: 1 Processed: 3 Uptime: 9m 39s
PID: 22313 Sessions: 1 Processed: 0 Uptime: 8m 19s
PID: 22315 Sessions: 1 Processed: 0 Uptime: 8m 18s
PID: 22317 Sessions: 1 Processed: 0 Uptime: 8m 17s
PID: 22319 Sessions: 1 Processed: 0 Uptime: 8m 16s
PID: 22299 Sessions: 1 Processed: 2 Uptime: 8m 57s
PID: 22272 Sessions: 1 Processed: 14 Uptime: 11m 6s
PID: 22274 Sessions: 1 Processed: 8 Uptime: 10m 58s
PID: 22309 Sessions: 1 Processed: 2 Uptime: 8m 20s
PID: 22289 Sessions: 1 Processed: 3 Uptime: 9m 43s
PID: 22297 Sessions: 1 Processed: 2 Uptime: 9m 30s
PID: 22311 Sessions: 1 Processed: 1 Uptime: 8m 19s
PID: 22281 Sessions: 1 Processed: 6 Uptime: 10m 19s
PID: 22295 Sessions: 1 Processed: 3 Uptime: 9m 33s

The thing I'm confused about is that there are four processes there
that haven't processed anything but the global queue is 54? Am I
reading this right?

Thanks in advance,

[David Legg]

Interesting. The question is why have they been sat there for that long and what have they been doing? Unfortunately, we've rather 'inherited' this application so we can't really tell.

I'm wondering whether global queuing is hindering rather than helping here but it is the only way I can see whether things are really being held up. This is what I've got now after a restart and things ahve calmed down. We've got two processes that have done the bulk of the work:

----------- General information -----------
max = 14
count = 14
active = 13
inactive = 1
Waiting on global queue: 0

----------- Domains -----------
/var/www/rails/ccesd/site/current:
PID: 22775 Sessions: 0 Processed: 1289 Uptime: 2h 38m 7s
PID: 22777 Sessions: 1 Processed: 0 Uptime: 2h 38m 6s
PID: 22781 Sessions: 1 Processed: 0 Uptime: 2h 37m 44s
PID: 22787 Sessions: 1 Processed: 0 Uptime: 2h 37m 14s
PID: 22791 Sessions: 1 Processed: 0 Uptime: 2h 37m 5s
PID: 22793 Sessions: 1 Processed: 0 Uptime: 2h 37m 3s
PID: 22773 Sessions: 1 Processed: 1 Uptime: 2h 38m 17s
PID: 22785 Sessions: 1 Processed: 1 Uptime: 2h 37m 44s
PID: 22783 Sessions: 1 Processed: 2 Uptime: 2h 37m 44s
PID: 22755 Sessions: 1 Processed: 21 Uptime: 2h 44m 27s
PID: 22759 Sessions: 1 Processed: 14 Uptime: 2h 40m 36s
PID: 22757 Sessions: 1 Processed: 52 Uptime: 2h 43m 28s
PID: 22975 Sessions: 1 Processed: 0 Uptime: 1h 10m 25s
PID: 22779 Sessions: 1 Processed: 1299 Uptime: 2h 37m 55s

Would increasing pool size really help, which is what I've read when global queue is not showing zero?

[Hongli Lai]

On Wed, Jul 21, 2010 at 9:20 PM, David Legg <sege...@actuaria.co.uk> wrote:
> Interesting. The question is why have they been sat there for that long and what have they been doing? Unfortunately, we've rather 'inherited' this application so we can't really tell.

Try http://www.modrails.com/documentation/Users%20guide%20Apache.html#debugging_frozen

> Would increasing pool size really help, which is what I've read when global queue is not showing zero?

I don't think so. You should just figure out why your app processes are stuck.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)