passenger uses more than 1000% CPU

[Vince]

Hi,

We are running Passenger 3.0.7 and we have noticed that ruby process sometimes uses more than 1000% CPU. Sometimes a single rails process takes 2000% or even 3000% CPU (see below), which makes the entire system very slow and a lot of requests waiting on the global queue.

We are running 64-bit CentOS 6.0 on a four Intel E7- 4807 system, which has 48 virtual cores (4 CPU * 6 Core  * 2 Hyper Threading). Is there anyway we can find out the problem? We are running the same application on a dual Intel E5620 system with Passenger 3.0.7 + Centos 6.0 and everything is fine.

top - 00:15:40 up  9:07,  3 users,  load average: 27.93, 33.03, 43.04

Tasks: 693 total,  42 running, 651 sleeping,   0 stopped,   0 zombie

Cpu(s): 97.8%us,  0.7%sy,  0.0%ni,  1.2%id,  0.0%wa,  0.0%hi,  0.3%si,  0.0%st

Mem:  66088740k total, 12100144k used, 53988596k free,   150464k buffers

Swap: 68190200k total,        0k used, 68190200k free,  2271184k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND                                                                                                                                     

45665 webapps  20   0 1099m 340m 5820 R 2289.9  0.5  21:14.38 ruby                                                                                                                                       

45562 webapps  20   0 1000m 240m 5848 R 67.1  0.4  13:12.07 ruby                                                                                                                                         

49340 webapps  20   0  506m 211m 2968 R 49.4  0.3   0:06.03 ruby                                                                                                                                         

45599 webapps  20   0  996m 234m 6064 R 46.8  0.4  13:49.63 ruby                                                                                                                                         

48610 webapps  20   0  523m 225m 4064 R 46.8  0.3   2:29.85 ruby                                                                                                                                         

48160 webapps  20   0  533m 232m 3996 R 45.6  0.4   2:53.43 ruby                                                                                                                                         

49119 webapps  20   0  518m 223m 3540 R 45.6  0.3   1:23.97 ruby                                                                                                                                         

45575 webapps  20   0 1101m 340m 5832 R 43.0  0.5  20:54.49 ruby                                                                                                                                         

45662 webapps  20   0 1096m 334m 6084 R 43.0  0.5  21:01.22 ruby                                                                                                                                         

45686 webapps  20   0  542m 238m 4160 R 41.8  0.4   8:52.97 ruby                                                                                                                                         

45565 webapps  20   0 1005m 241m 5928 R 40.5  0.4  15:55.44 ruby                                                                                                                                         

45671 webapps  20   0 1101m 350m 6044 R 40.5  0.5  13:13.73 ruby                                                                                                                                         

45704 webapps  20   0  535m 235m 4092 R 40.5  0.4   7:56.79 ruby                                                                                                                                         

48736 webapps  20   0  528m 228m 4044 R 39.2  0.4   2:04.92 ruby                                                                                                                                         

45605 webapps  20   0 1102m 336m 6072 R 38.0  0.5  13:52.87 ruby 

Thanks,

Vince

[Hongli Lai]

You should check whether your application isn't stuck in some kind of
loop. Try debugging it by sending SIGHUP to the offending process, or
inspecting it's C backtrace with 'crash-watch --dump'
(https://github.com/FooBarWidget/crash-watch).

> --
> You received this message because you are subscribed to the Google Groups
> "Phusion Passenger Discussions" group.
> To post to this group, send email to phusion-...@googlegroups.com.
> To unsubscribe from this group, send email to
> phusion-passen...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/phusion-passenger?hl=en.

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Vince]

Hi Hongli,

 

I tried crash-watch but it dosn't seem to help because the 2000% CPU rails process doesn't crash. It stay at around 2000% CPU for only a few seconds and after that it becomes a normal process. And,"kill -s SIGHUP pid" doesn't help either as I cannot find any information regarding the SIGHUP process in production.log or exception.log.

 

Also we have the same application running and processing the same set of requests on another system via round-robin load balancing and it doesn't have this kind of problem.

 

any ideas?

 

thanks,

 

Vince

[Hongli Lai]

On Wed, Dec 28, 2011 at 7:22 PM, Vince <coolte...@gmail.com> wrote:
> Hi Hongli,
>
> I tried crash-watch but it dosn't seem to help because the 2000% CPU rails
> process doesn't crash.

That's not the only way to use crash-watch. Use --dump to immediately
dump the backtrace of a running process.

> It stay at around 2000% CPU for only a few seconds
> and after that it becomes a normal process. And,"kill -s SIGHUP pid" doesn't
> help either as I cannot find any information regarding the SIGHUP process in
> production.log or exception.log.

Look in the web server's global error log. SIGHUP causes the Ruby
backtrace to be written there if Ruby is able to respond to signals.