Stray 100% CPU Usage

1,652 views
Skip to first unread message

shmay

unread,
Nov 17, 2008, 7:26:21 PM11/17/08
to Phusion Passenger Discussions
I recently deployed an app with Passenger and MRI. And I've been
getting some seemingly random 100% CPU usages, which, looking through
some threads, has happened to others.

What should I do?

Thanks you.

Kennon Ballou

unread,
Nov 18, 2008, 4:03:29 AM11/18/08
to Phusion Passenger Discussions
This definitely happens to me and is the main reason why I have to
keep switching from REE back to normal Ruby (I really wish I could use
REE consistently, it's much faster and memory-efficient).

Does anyone know how to debug an already running passenger thread? It
happens so sporadically that I have no idea how to reproduce it. Is
there some way to attach to a running passenger process a la gdb?

Btw, Phusion do great work, passenger is amazing and once I can use
REE consistently, it will be amazing as well!

Hongli Lai

unread,
Nov 18, 2008, 7:42:50 AM11/18/08
to phusion-...@googlegroups.com

shmay, Kennon, what platforms are you running REE on?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

amos

unread,
Nov 18, 2008, 4:08:26 PM11/18/08
to Phusion Passenger Discussions
This happens to us a lot. Some of the things we've tried to do to fix
it:

- used 'conservative' spawning method
- used regular ruby
- wrote a bash script to find and kill these runaway processes (we've
now isolated the runaway processes to machines that handle media
uploads)

I tried once to use gdb to identify the issue. I got as far as seeing
an infinite loop in Ruby GC's finalize_list, but that was with REE on
Rails 1.2 (we've since upgraded). I haven't used REE on production
machines since. If you're looking for better performance with regular
ruby, try applying stefan kaes' GC patch.

Hongli Lai

unread,
Nov 18, 2008, 4:51:51 PM11/18/08
to phusion-...@googlegroups.com
amos wrote:
> This happens to us a lot. Some of the things we've tried to do to fix
> it:
>
> - used 'conservative' spawning method
> - used regular ruby
> - wrote a bash script to find and kill these runaway processes (we've
> now isolated the runaway processes to machines that handle media
> uploads)
>
> I tried once to use gdb to identify the issue. I got as far as seeing
> an infinite loop in Ruby GC's finalize_list, but that was with REE on
> Rails 1.2 (we've since upgraded). I haven't used REE on production
> machines since. If you're looking for better performance with regular
> ruby, try applying stefan kaes' GC patch.

Hi Amos.

Do you happen to have a spare staging server around on which you can
reproduce this problem? So far we haven't been able to find the cause of
this problem, or even seen the problem ourselves. If you had saved the
gdb backtrace somewhere then we'd be happy to take a look at it.

Also, finalize_list does not have any infinite loops. :) But I don't
exclude the possibility that something very weird might be going on
which causes finalize_list to freeze anyway.

Finally, what platform are your servers running on?

shmay

unread,
Nov 18, 2008, 6:36:05 PM11/18/08
to Phusion Passenger Discussions
I'm using regular MRI, on a Linode Ubuntu 360. Is that what you
meant?

I'm a noob, so it's probably me just doing something stupid, but it's
something I really want fixed. Is there some logs I could post, or
something?
> E-mail: i...@phusion.nl

Hongli Lai

unread,
Nov 18, 2008, 7:10:40 PM11/18/08
to phusion-...@googlegroups.com
shmay wrote:
> I'm using regular MRI, on a Linode Ubuntu 360. Is that what you
> meant?

Actually I was referring to Ruby Enterprise Edition.

But if you need to debug an application that has gone crazy, please try
to generate a backtrace for it and post it to this mailing list. You can
do it as follows:
1. Identify the PID of the process that has gone crazy.
2. Type:
sudo gdb
attach 12345 <--- replace "12345" with the actual PID
thread apply all bt

Please copy & paste the result.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

amos

unread,
Nov 18, 2008, 9:35:23 PM11/18/08
to Phusion Passenger Discussions
Here's where I originally posted the stack trace:

http://groups.google.com/group/emm-ruby/browse_thread/thread/c91b52e2c452d0c5/6bb0eaf6b51f3486?lnk=gst&q=finalize_list#6bb0eaf6b51f3486

We're using:

Centos 5 x86_64
Rails 2.1.0 (although the above trace was in Rails 1.2.3)

Some libraries:
five_runs memcache-client with our crc32 extension (http://github.com/
fiveruns/memcache-client/tree/master)
our version of data_fabric (which had some issues with passenger
reopening db connections in spawn_server since data_fabric does not
define a :production db connection)
rmagick
hpricot

I can't reproduce this problem on a staging server, only production
ones. I can possibly give you access to a live machine in a hanging
state if you contact me directly.

On Nov 18, 4:10 pm, Hongli Lai <hon...@phusion.nl> wrote:
> shmay wrote:
> > I'm using regular MRI, on a Linode Ubuntu 360.  Is that what you
> > meant?
>
> Actually I was referring to Ruby Enterprise Edition.
>
> But if you need to debug an application that has gone crazy, please try
> to generate a backtrace for it and post it to this mailing list. You can
> do it as follows:
> 1. Identify the PID of the process that has gone crazy.
> 2. Type:
> sudo gdb
> attach 12345     <--- replace "12345" with the actual PID
> thread apply all bt
>
> Please copy & paste the result.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl

Hongli Lai

unread,
Nov 19, 2008, 5:49:25 AM11/19/08
to phusion-...@googlegroups.com

Thanks. I seem to have missed that one.


> We're using:
>
> Centos 5 x86_64
> Rails 2.1.0 (although the above trace was in Rails 1.2.3)

Are other people who experience the same problem on 64-bit as well?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

Roderick van Domburg

unread,
Nov 19, 2008, 5:49:20 AM11/19/08
to phusion-...@googlegroups.com
We're running REE and Passenger on several 64-bit CentOS 5.2 boxes and have
not experienced this issue. That said, the applications on those boxes do
not use memcache.

Kind regards,

Roderick van Domburg
http://www.nedforce.com


-----Oorspronkelijk bericht-----
Van: phusion-...@googlegroups.com
[mailto:phusion-...@googlegroups.com] Namens Hongli Lai
Verzonden: woensdag 19 november 2008 11:49
Aan: phusion-...@googlegroups.com
Onderwerp: Re: Stray 100% CPU Usage

shmay

unread,
Nov 19, 2008, 6:55:44 PM11/19/08
to Phusion Passenger Discussions
How do I determine which process is causing the problem?

On Nov 18, 4:10 pm, Hongli Lai <hon...@phusion.nl> wrote:
> shmay wrote:
> > I'm using regular MRI, on a Linode Ubuntu 360.  Is that what you
> > meant?
>
> Actually I was referring to Ruby Enterprise Edition.
>
> But if you need to debug an application that has gone crazy, please try
> to generate a backtrace for it and post it to this mailing list. You can
> do it as follows:
> 1. Identify the PID of the process that has gone crazy.
> 2. Type:
> sudo gdb
> attach 12345     <--- replace "12345" with the actual PID
> thread apply all bt
>
> Please copy & paste the result.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl

amos

unread,
Nov 19, 2008, 7:54:04 PM11/19/08
to Phusion Passenger Discussions
I'm running a test with REE again. I'll try to give you another stack
trace if I see a runaway process.

On Nov 19, 2:49 am, Hongli Lai <hon...@phusion.nl> wrote:
> amos wrote:
> > Here's where I originally posted the stack trace:
>
> >http://groups.google.com/group/emm-ruby/browse_thread/thread/c91b52e2...
>
> Thanks. I seem to have missed that one.
>
> > We're using:
>
> > Centos 5 x86_64
> > Rails 2.1.0 (although the above trace was in Rails 1.2.3)
>
> Are other people who experience the same problem on 64-bit as well?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/
> E-mail: i...@phusion.nl

Hongli Lai

unread,
Nov 20, 2008, 4:21:49 AM11/20/08
to phusion-...@googlegroups.com

Thanks for the link. I seem to have missed that message.


> We're using:
>
> Centos 5 x86_64
> Rails 2.1.0 (although the above trace was in Rails 1.2.3)
>
> Some libraries:
> five_runs memcache-client with our crc32 extension (http://github.com/
> fiveruns/memcache-client/tree/master)
> our version of data_fabric (which had some issues with passenger
> reopening db connections in spawn_server since data_fabric does not
> define a :production db connection)
> rmagick
> hpricot

I've been spending some time stress testing a sample Rails app which
uses FiveRuns's memcache-client and RMagick with REE on 64-bit Ubuntu
8.10 server. So far I haven't been able to find any stability issues.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

Hongli Lai

unread,
Nov 20, 2008, 4:48:10 AM11/20/08
to phusion-...@googlegroups.com
amos wrote:
> I'm running a test with REE again. I'll try to give you another stack
> trace if I see a runaway process.

OK.

The stack trace you gave previously might not be accurate because of
compiler optimizations. Could you install REE without optimizations? You
can do that by setting the CFLAGS environment variable to an empty
string, then running the installer:

export CFLAGS=
./installer

And I'm also wondering whether your REE installation uses the system
Ruby's gems. By default, REE adds the system Ruby's gem path to its own
gem path, allowing it to use already-installed gems. But I've received a
few reports from users who say that this can cause crashes when REE
tries to load a native extension that's compiled for the system's Ruby.
So you are advised to reinstall all your gems for REE, e.g. with
'/path/to/ree/bin/gem install imagemagick'.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

amos

unread,
Nov 21, 2008, 1:49:04 AM11/21/08
to Phusion Passenger Discussions
unfortunately or fortunately i haven't been able to reproduce that old
infinite loop. i have a hunch it might when trying to allocate a bunch
of memory real quickly by loading in a ton of ar objects from a db
table. we've since optimized our code and removed a few memory leaks
by limiting the number of models we initialize at one time. i'll try
to do some debugging with the compilation options you gave me.

in the meantime, we still get the 100% usage on our upload servers.
i'm looking at one now. here is the output of strace:

lseek(8, -2, SEEK_CUR) = -1 ESPIPE (Illegal seek)
lseek(8, -2, SEEK_CUR) = -1 ESPIPE (Illegal seek)
lseek(8, -2, SEEK_CUR) = -1 ESPIPE (Illegal seek)
lseek(8, -2, SEEK_CUR) = -1 ESPIPE (Illegal seek)
....

gdb backtrace is:

#0 0x00000037f0ad1e5b in lseek64 () from /lib64/libc.so.6
#1 0x00000037f0a6b536 in _IO_new_do_write () from /lib64/libc.so.6
#2 0x00000037f0a6c9b2 in _IO_new_file_xsputn () from /lib64/libc.so.6
#3 0x00000037f0a61d8b in fwrite () from /lib64/libc.so.6
#4 0x00000037f1a50835 in rb_io_fptr_finalize () from /usr/lib64/
libruby.so.1.8
#5 0x00000037f1a5673b in rb_io_eof () from /usr/lib64/libruby.so.1.8
#6 0x00000037f1a31731 in rb_exc_jump () from /usr/lib64/libruby.so.
1.8
#7 0x00000037f1a31c38 in rb_exc_jump () from /usr/lib64/libruby.so.
1.8
#8 0x00000037f1a3bd86 in rb_apply () from /usr/lib64/libruby.so.1.8
#9 0x00000037f1a3d65d in rb_apply () from /usr/lib64/libruby.so.1.8
#10 0x00000037f1a31753 in rb_exc_jump () from /usr/lib64/libruby.so.
1.8
#11 0x00000037f1a31c38 in rb_exc_jump () from /usr/lib64/libruby.so.
1.8
#12 0x00000037f1a3bd86 in rb_apply () from /usr/lib64/libruby.so.1.8
#13 0x00000037f1a3fc0c in rb_apply () from /usr/lib64/libruby.so.1.8
#14 0x00000037f1a3c08b in rb_apply () from /usr/lib64/libruby.so.1.8
#15 0x00000037f1a3d65d in rb_apply () from /usr/lib64/libruby.so.1.8
#16 0x00000037f1a31753 in rb_exc_jump () from /usr/lib64/libruby.so.
1.8
#17 0x00000037f1a31c38 in rb_exc_jump () from /usr/lib64/libruby.so.
1.8
#18 0x00000037f1a3bd86 in rb_apply () from /usr/lib64/libruby.so.1.8
#19 0x00000037f1a3e00a in rb_apply () from /usr/lib64/libruby.so.1.8
....

this is with:
ruby 1.8.6 (2008-08-11 patchlevel 287) [x86_64-linux]

i don't know if it's something specific to our system or passenger,
but i don't think it ever happened with mongrel. any suggestions on
how to debug this one further?
> E-mail: i...@phusion.nl

Hongli Lai

unread,
Nov 24, 2008, 8:41:42 AM11/24/08
to phusion-...@googlegroups.com

I've been spending some time reading the Ruby source code and glibc
source code for clues. I haven't found any so far, but I suspect it
might have something to do with the way we handle Unix sockets.

Could you try the following:
1. Create a script, say '/usr/bin/ruby-wrapper-script', which sets
PASSENGER_NO_ABSTRACT_NAMESPACE_SOCKETS=1 and executes Ruby, like this:

#!/bin/bash
export PASSENGER_NO_ABSTRACT_NAMESPACE_SOCKETS=1
exec ruby "$@"

2. Set the PassengerRuby config option to your wrapper script.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

MarcoJ

unread,
Nov 24, 2008, 9:52:09 AM11/24/08
to Phusion Passenger Discussions
Over the last two week I have had the 100% cpu usage problem also on
my servers. Traffic has gone up during the last two weeks and also the
number of CPUs has been increased on these machines.

I found out that these cases always have been related to an illegal
seek in my production log. Seems similar to the previous mentioned
strace.

I found an interesting post from ep's blog who had the samen problem
and published a monkey patch that resolved it for him. He mentioned
that it is related to uploading a file: http://ep.blogware.com/blog/_archives/2008/10/14/3930392.html

I haven't tried the monkey patch myself yet. Oh, and I am using
FiveRuns as well, but I am not sure if that is related to the problem.

Hope this helps,

Marco

Stacktrace:

"Illegal seek"
Errno::ESPIPE

[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/
cgi_process.rb:215:in `flush'
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/
cgi_process.rb:215:in `out'
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/
dispatcher.rb:171:in `handle_request'
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/
dispatcher.rb:115:in `dispatch'
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/
dispatcher.rb:126:in `dispatch_cgi'
[RAILS_ROOT]/vendor/rails/actionpack/lib/action_controller/
dispatcher.rb:9:in `dispatch'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
request_handler.rb:38:in `process_request'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_request_handler.rb:163:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
application_spawner.rb:307:in `start_request_handler'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
application_spawner.rb:276:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/utils.rb:
165:in `safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/utils.rb:
163:in `fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/utils.rb:
163:in `safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
application_spawner.rb:274:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/utils.rb:
165:in `safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/utils.rb:
163:in `fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/utils.rb:
163:in `safe_fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
application_spawner.rb:273:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:317:in `__send__'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:317:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:168:in `start_synchronously'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:135:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:112:in `fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:112:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
application_spawner.rb:177:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
framework_spawner.rb:270:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
framework_spawner.rb:263:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
framework_spawner.rb:263:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:317:in `__send__'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:317:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:168:in `start_synchronously'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:135:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:112:in `fork'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:112:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/railz/
framework_spawner.rb:87:in `start'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
spawn_manager.rb:224:in `spawn_rails_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
spawn_manager.rb:219:in `synchronize'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
spawn_manager.rb:219:in `spawn_rails_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
spawn_manager.rb:122:in `spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
spawn_manager.rb:253:in `handle_spawn_application'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:317:in `__send__'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:317:in `main_loop'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/lib/passenger/
abstract_server.rb:168:in `start_synchronously'
/usr/lib/ruby/gems/1.8/gems/passenger-2.0.1/bin/passenger-spawn-server:
46
> E-mail: i...@phusion.nl

Hongli Lai

unread,
Nov 24, 2008, 11:20:42 AM11/24/08
to phusion-...@googlegroups.com
MarcoJ wrote:
> Over the last two week I have had the 100% cpu usage problem also on
> my servers. Traffic has gone up during the last two weeks and also the
> number of CPUs has been increased on these machines.
>
> I found out that these cases always have been related to an illegal
> seek in my production log. Seems similar to the previous mentioned
> strace.
>
> I found an interesting post from ep's blog who had the samen problem
> and published a monkey patch that resolved it for him. He mentioned
> that it is related to uploading a file: http://ep.blogware.com/blog/_archives/2008/10/14/3930392.html
>
> I haven't tried the monkey patch myself yet. Oh, and I am using
> FiveRuns as well, but I am not sure if that is related to the problem.
>
> Hope this helps,
>
> Marco

Very interesting. Can others confirm whether this money patch works and
whether the problem description is correct?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

amos

unread,
Nov 24, 2008, 6:16:52 PM11/24/08
to Phusion Passenger Discussions
I just implemented the patch on one of our production machines. I'll
respond back later if I feel like it's working.
> E-mail: i...@phusion.nl

amos

unread,
Nov 26, 2008, 2:23:35 PM11/26/08
to Phusion Passenger Discussions
Looks like the patch totally fixed the problem. Thank you, ep!

Hongli Lai

unread,
Nov 26, 2008, 2:27:01 PM11/26/08
to phusion-...@googlegroups.com
amos wrote:
> Looks like the patch totally fixed the problem. Thank you, ep!

Great. I've talked to the Rails core team about this problem, and they
think that fixing it in Rails is probably the best solution.

I still haven't figured out *why* this problem occurs so I'm going to
investigate it a bit more.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

Jonathan Wallace

unread,
Jan 7, 2009, 2:06:49 AM1/7/09
to Phusion Passenger Discussions
On Nov 26 2008, 2:27 pm, Hongli Lai <hon...@phusion.nl> wrote:
> I still haven't figured out *why* this problem occurs so I'm going to
> investigate it a bit more.

For what its worth, we're using:

ruby -v:
ruby 1.8.6 (2008-03-03 patchlevel 114) [i686-linux]

passenger 2.0.3

rails 2.1

uname -r:
2.6.18-xenU-ec2-v1.0

on gentoo

and have experienced this issue 3 times in the past two days. If it
continues to occur, we'll use ep's monkey patch.

Jonathan

Ramon Tayag

unread,
Nov 3, 2011, 2:11:34 PM11/3/11
to phusion-...@googlegroups.com
I found an ApplicationSpawner that was at 99% for over 5 minutes. I had to kill it, but before I did, I did what you asked: https://gist.github.com/1337203

Max Kaplan

unread,
Feb 8, 2012, 6:27:04 PM2/8/12
to phusion-...@googlegroups.com
I had the same issue with a rouge ruby process running at 99% CPU when I was running passenger 3.0.8.

I updated the gem to 3.0.11 and recompiled passenger and it seemed to fix the issue.

PS. I was running Apache 2 on CentOS

Jodi @ HomeStars

unread,
Jul 3, 2012, 9:28:41 AM7/3/12
to phusion-...@googlegroups.com
Good morning, could your problem be related to this ?



Jodi Showers | CTO | HomeStars
rating the renovators, retailers and repairmen
705-749-4955 | 416-488-0312www.HomeStars.com

On 2012-Jul-03, at 1:38 AM, Kenneth Thorman wrote:

Hi

I am also seeing this, and it does not seem to be intermittent, if I leave the server alone for a few minutes CPU will stay at close to 100%.
Any help would be appreciated.

Distro: Amazon Linux 64bit

uname -a
Linux 2.6.35.14-106.53.amzn1.x86_64 #1 SMP Fri Jan 6 16:20:10 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux


Apache:
Server version: Apache/2.2.22 (Unix)

#ruby -v
ruby 1.8.7 (2011-12-28 patchlevel 357) [x86_64-linux]

Rails 3.2.5

Passenger: 3.0.13

gbd output:
GNU gdb (GDB) Amazon Linux (7.2-50.11.amzn1)
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-amazon-linux-gnu".
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
(gdb) attach 18897
Attaching to process 18897
Reading symbols from /usr/bin/ruby1.8...(no debugging symbols found)...done.
Reading symbols from /usr/lib64/libruby.so.1.8...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/libruby.so.1.8
Reading symbols from /lib64/libpthread.so.0...(no debugging symbols found)...done.
[Thread debugging using libthread_db enabled]
[New Thread 0x7f91baa83700 (LWP 18935)]
Loaded symbols for /lib64/libpthread.so.0
Reading symbols from /lib64/librt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/librt.so.1
Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/libdl.so.2
Reading symbols from /lib64/libcrypt.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libcrypt.so.1
Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libm.so.6
Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
Loaded symbols for /lib64/libc.so.6
Reading symbols from /lib64/libgcc_s.so.1...(no debugging symbols found)...done.
Loaded symbols for /lib64/libgcc_s.so.1
Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols found)...done.
Loaded symbols for /lib64/ld-linux-x86-64.so.2
Reading symbols from /lib64/libfreebl3.so...(no debugging symbols found)...done.
Loaded symbols for /lib64/libfreebl3.so
Reading symbols from /usr/lib64/ruby/1.8/x86_64-linux/socket.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/ruby/1.8/x86_64-linux/socket.so
Reading symbols from /usr/lib/ruby/gems/1.8/gems/passenger-3.0.12/ext/ruby/ruby-1.8.7-x86_64-linux/passenger_native_support.so...done.
Loaded symbols for /usr/lib/ruby/gems/1.8/gems/passenger-3.0.12/ext/ruby/ruby-1.8.7-x86_64-linux/passenger_native_support.so
Reading symbols from /usr/lib64/ruby/1.8/x86_64-linux/thread.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/ruby/1.8/x86_64-linux/thread.so
Reading symbols from /usr/lib64/ruby/1.8/x86_64-linux/etc.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/ruby/1.8/x86_64-linux/etc.so
Reading symbols from /usr/lib64/ruby/1.8/x86_64-linux/fcntl.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/ruby/1.8/x86_64-linux/fcntl.so
Reading symbols from /usr/lib64/ruby/1.8/x86_64-linux/stringio.so...(no debugging symbols found)...done.
Loaded symbols for /usr/lib64/ruby/1.8/x86_64-linux/stringio.so
0x00007f91bbd328b3 in select () from /lib64/libc.so.6
Missing separate debuginfos, use: debuginfo-install ruby-1.8.7.357-1.18.amzn1.x86_64
(gdb) thread apply all bt

Thread 2 (Thread 0x7f91baa83700 (LWP 18935)):
#0  0x00007f91bc8b675b in pthread_cond_timedwait@@GLIBC_2.3.2 () from /lib64/libpthread.so.0
#1  0x00007f91bcaf9fac in ?? () from /usr/lib64/libruby.so.1.8
#2  0x00007f91bc8b27f1 in start_thread () from /lib64/libpthread.so.0
#3  0x00007f91bbd39ccd in clone () from /lib64/libc.so.6

Thread 1 (Thread 0x7f91bcfdc700 (LWP 18897)):
#0  0x00007f91bbd328b3 in select () from /lib64/libc.so.6
#1  0x00007f91bcb05f04 in rb_thread_schedule () from /usr/lib64/libruby.so.1.8
#2  0x00007f91bcb08149 in rb_thread_select () from /usr/lib64/libruby.so.1.8
#3  0x00007f91bcb220b4 in ?? () from /usr/lib64/libruby.so.1.8
#4  0x00007f91bcb02f1e in ?? () from /usr/lib64/libruby.so.1.8
#5  0x00007f91bcb030f7 in ?? () from /usr/lib64/libruby.so.1.8
#6  0x00007f91bcaffd54 in ?? () from /usr/lib64/libruby.so.1.8
#7  0x00007f91bcafd450 in ?? () from /usr/lib64/libruby.so.1.8
#8  0x00007f91bcafdd0e in ?? () from /usr/lib64/libruby.so.1.8
#9  0x00007f91bcafe6d5 in ?? () from /usr/lib64/libruby.so.1.8
#10 0x00007f91bcb02dd0 in ?? () from /usr/lib64/libruby.so.1.8
#11 0x00007f91bcb030f7 in ?? () from /usr/lib64/libruby.so.1.8
#12 0x00007f91bcaffd54 in ?? () from /usr/lib64/libruby.so.1.8
#13 0x00007f91bcb005be in ?? () from /usr/lib64/libruby.so.1.8
#14 0x00007f91bcb00471 in ?? () from /usr/lib64/libruby.so.1.8
#15 0x00007f91bcb005be in ?? () from /usr/lib64/libruby.so.1.8
#16 0x00007f91bcb02dd0 in ?? () from /usr/lib64/libruby.so.1.8
#17 0x00007f91bcb030f7 in ?? () from /usr/lib64/libruby.so.1.8
#18 0x00007f91bcaffe2d in ?? () from /usr/lib64/libruby.so.1.8
#19 0x00007f91bcb00471 in ?? () from /usr/lib64/libruby.so.1.8
#20 0x00007f91bcb0fb45 in ?? () from /usr/lib64/libruby.so.1.8
#21 0x00007f91bcb0fb85 in ruby_exec () from /usr/lib64/libruby.so.1.8
#22 0x00007f91bcb0fbb5 in ruby_run () from /usr/lib64/libruby.so.1.8
#23 0x0000000000400831 in main ()
(gdb)
--
You received this message because you are subscribed to the Google Groups "Phusion Passenger Discussions" group.
To view this discussion on the web visit https://groups.google.com/d/msg/phusion-passenger/-/r-pvlpq3vyYJ.
To post to this group, send email to phusion-...@googlegroups.com.
To unsubscribe from this group, send email to phusion-passen...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/phusion-passenger?hl=en.

Lupu Alexandru Emil

unread,
Jul 3, 2012, 11:24:46 AM7/3/12
to phusion-...@googlegroups.com
Hello!
I had the same issue on amazon in the weekend. I am running a nginx + unicorn setup and all my CPU appeared to be "stolen". 
A full reboot of the instance solved my issue. 

Alex
signature.asc

Craig Hansen

unread,
Jul 3, 2012, 6:07:00 PM7/3/12
to phusion-...@googlegroups.com
I had EC2 issues as well over the weekend -- their VA data center had power issues from the big storm and some services were affected.

- Craig

-- 
Craig Hansen
Platforms, LLC

Hongli Lai

unread,
Jul 4, 2012, 6:54:31 AM7/4/12
to phusion-...@googlegroups.com
On Tue, Jul 3, 2012 at 5:24 PM, Lupu Alexandru Emil
<gang....@gmail.com> wrote:
> Hello!
> I had the same issue on amazon in the weekend. I am running a nginx +
> unicorn setup and all my CPU appeared to be "stolen".
> A full reboot of the instance solved my issue.

That's the Linux kernel leap second bug. You can find info about this
all over the Internet.

--
Phusion | Ruby & Rails deployment, scaling and tuning solutions

Filipe Cifali

unread,
Jul 4, 2012, 8:35:40 AM7/4/12
to phusion-...@googlegroups.com
Here a link that may explain this issue about power:

http://www.datacenterknowledge.com/archives/2012/07/03/multiple-generator-failures-caused-amazon-outage/


2012/7/4 Hongli Lai <hon...@phusion.nl>
--
You received this message because you are subscribed to the Google Groups "Phusion Passenger Discussions" group.
To post to this group, send email to phusion-...@googlegroups.com.
To unsubscribe from this group, send email to phusion-passen...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/phusion-passenger?hl=en.




--
[]'s

Filipe Cifali Stangler

Reply all
Reply to author
Forward
0 new messages