mod_rails breaks after several hours

[joost]

Hi All,

We've recently moved to mod_rails. It is a nice and workable
deployment method .. but ..
It breaks after several hours.. there are numerous apache processes
and they take up the whole system..which starts swapping and stops
responding to requests..

It looks like the issue discussed here:
http://groups.google.com/group/phusion-passenger/browse_thread/thread/26f68ebfec300442/340125e420b246b8?lnk=gst&q=apache+processes#340125e420b246b8

So we've tried version 1.1.0 which is better.. but still breaks down
after some time.

Any suggestions would be great!

Joost

[joost]

After a while .. the system becomes idle again.. and you can kill all
httpd's and restart Apache. But not really a great production
environment :)

[Stanislav]

We're having exactly the same problem :( . Yesterday upgraded up to
1.1.0 (Fedora 9 , rails 2.0.2) however from time to time apache is
blocking the whole system with numerous processes.

How can we help for a solution of this problem or is there a current
solution?
10x in advance.

Stanislav

[Hongli Lai]

Stanislav wrote:
> We're having exactly the same problem :( . Yesterday upgraded up to
> 1.1.0 (Fedora 9 , rails 2.0.2) however from time to time apache is
> blocking the whole system with numerous processes.
>
> How can we help for a solution of this problem or is there a current
> solution?
> 10x in advance.
>
> Stanislav

Does the problem occur in 2.0 RC 1 as well?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl
Chamber of commerce no: 08173483 (The Netherlands)

[Stanislav]

Yes it happened again today.

May be, there're problems with it's installation. http processes still
take up to 260 MB VM.

[root@svejo passenger-1.9.0]# passenger-memory-stats
-------------- Apache processes ---------------
PID PPID Threads VMSize Private Name
-----------------------------------------------
350 29309 1 260.9 MB 0.9 MB /usr/sbin/httpd
353 29309 1 260.7 MB 0.8 MB /usr/sbin/httpd
417 29309 1 260.4 MB 0.5 MB /usr/sbin/httpd
..............
31761 29309 1 260.2 MB 0.3 MB /usr/sbin/httpd
32743 29309 1 338.6 MB 3.6 MB /usr/sbin/httpd
### Processes: 96
### Total private dirty RSS: 61.89 MB

--------- Passenger processes ----------
PID Threads VMSize Private Name
----------------------------------------
398 1 212.0 MB 62.7 MB Rails: /var/apps/svejo.net/releases/
20080609182340
435 1 221.9 MB 68.8 MB Rails: /var/apps/svejo.net/releases/
20080609182340
..........
32026 1 210.0 MB 60.9 MB Rails: /var/apps/svejo.net/releases/
20080609182340
32413 1 211.1 MB 62.2 MB Rails: /var/apps/svejo.net/releases/
20080609182340
32652 1 208.0 MB 61.2 MB Rails: /var/apps/svejo.net/releases/
20080609182340
32658 1 206.9 MB 59.8 MB Rails: /var/apps/svejo.net/releases/
20080609182340
32723 1 210.1 MB 61.6 MB Rails: /var/apps/svejo.net/releases/
20080609182340
### Processes: 23
### Total private dirty RSS: 1064.08 MB

I'll try the last version from the git repository (master).

The big problem with this server is that it has 2GB RAM and the swap
is on software RAID 1. So, it's really slow and CPU consumming to put
anything in the swap.

On Jun 11, 3:30 pm, Hongli Lai <hon...@phusion.nl> wrote:
> Stanislav wrote:
> > We're having exactly the same problem :( . Yesterday upgraded up to
> > 1.1.0 (Fedora 9 , rails 2.0.2) however from time to time apache is
> > blocking the whole system with numerous processes.
>
> > How can we help for a solution of this problem or is  there a current
> > solution?
> > 10x in advance.
>
> > Stanislav
>
> Does the problem occur in 2.0 RC 1 as well?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

Something doesn't seem right here. "ApplicationPoolServerExecutable"
should be showing up in the stats, but it isn't there.

There might be an installation problem. Could you post your Passenger
config options?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Hongli Lai]

And is your memory being hogged by Apache processes or by Rails processes?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Stanislav]

When having the memory problems I have more than 200 httpd instances.
The total sum of "private dirty RSS" is not a lot, however every
instance has ~260MB VM.

Quote from the blog:
"In Phusion Passenger 2.0, the VM size has been reduced by ten fold,
as shown by the following diagram:" - it doesn't go this way with my
installation. With 1.9.0 it seems to use the same VM.

On Jun 11, 7:43 pm, "Hongli Lai" <hon...@phusion.nl> wrote:
> And is your memory being hogged by Apache processes or by Rails processes?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

Stanislav wrote:
> When having the memory problems I have more than 200 httpd instances.
> The total sum of "private dirty RSS" is not a lot, however every
> instance has ~260MB VM.

Does the "ApplicationPoolServerExecutable" process appear in 'ps aux',
and if so, how does it look like?

Were you running on Mongrel before? If so, have you ever had similar
problems with Mongrel?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Stanislav]

Yes, the application was deployed on 8 mongrel servers behind a proxy
balancer. I can't remember similar problems.
The next time I get this error I'll check how
"ApplicationPoolServerExecutable" looks like ...

On Jun 11, 9:56 pm, Hongli Lai <hon...@phusion.nl> wrote:
> Stanislav wrote:
> > When having the memory problems I have more than 200 httpd instances.
> > The total sum of "private dirty RSS" is not a lot, however every
> > instance has ~260MB VM.
>
> Does the "ApplicationPoolServerExecutable" process appear in 'ps aux',
> and if so, how does it look like?
>
> Were you running on Mongrel before? If so, have you ever had similar
> problems with Mongrel?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[sandro d.]

We were having this problem here too. At least with the same symptoms.
It turned out it was a bug in Apache Pre-Fork, when listening in more
than one port, in our case 80 and 3333.

Removing the virtual-server in the port 3333 seems to have done the
trick for us.

Hope that helps.

Sandro Duarte

[Hongli Lai]

sandro d. wrote:
> We were having this problem here too. At least with the same symptoms.
> It turned out it was a bug in Apache Pre-Fork, when listening in more
> than one port, in our case 80 and 3333.
>
> Removing the virtual-server in the port 3333 seems to have done the
> trick for us.
>
> Hope that helps.
>
> Sandro Duarte

Interesting. Can anyone else confirm this?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Stanislav]

Well I diabled all other ports of Apache except 80.

I will share here in case of problems such as too many httpd
processes.

Stanislav

On Jun 13, 11:59 am, Hongli Lai <hon...@phusion.nl> wrote:
> sandro d. wrote:
> > We were having this problem here too. At least with the same symptoms.
> > It turned out it was a bug in Apache Pre-Fork, when listening in more
> > than one port, in our case 80 and 3333.
>
> > Removing the virtual-server in the port 3333 seems to have done the
> > trick for us.
>
> > Hope that helps.
>
> > Sandro Duarte
>
> Interesting. Can anyone else confirm this?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Stanislav]

OK, I just had the problem with too many httpd process. Also I have
stopped all other ports except 80.

netstat -tnlp | grep httpd
tcp 0 0 :::
80 :::* LISTEN 862/
httpd

Just tell me if any info from our server infrastructure whould be
helpful for debug . I'm scared of problems ...

[Hongli Lai]

Stanislav wrote:
> OK, I just had the problem with too many httpd process. Also I have
> stopped all other ports except 80.
>
> netstat -tnlp | grep httpd
> tcp 0 0 :::
> 80 :::* LISTEN 862/
> httpd
>
> Just tell me if any info from our server infrastructure whould be
> helpful for debug . I'm scared of problems ...

I'd like to help, but I'll need a lot of information to be properly able
to examine this problem. Could you post the following information?

- Your Apache MPM configuration.
- The number of virtual hosts in Apache.

Could you also post the following information the next time the problem
occurs?

- The full, verbatim output of 'passenger-memory-stats',
'passenger-status', 'ps aux' (feel free to censor sensitive processes)
and 'free -m'.
- Any data that you have on the website's traffic on that moment, e.g.
the number of people visiting your website.
- If you have any non-Rails virtual hosts, are they responsive?
- Are static assets (e.g. images and stylesheets) on your Rails virtual
host reachable?
- What happens if you perform a graceful restart?
- Are there any errors in the Apache log file or your Rails log file?

Finally, some tips:

- Try lowering the number of worker processes in your Apache MPM
configuration. It seems like it's currently set to an excessively high
value.
- Try switching to the worker MPM, it's more memory efficient,
especially if you set a low stack size for threads.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Stanislav]

It happened again :(

Here it is my setup:

I'm using apache with prefork currently. I tried running it with
worker MPM, however it crashed again. So my current MPM config is
this:
<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 10
ServerLimit 30
MaxClients 30
MaxRequestsPerChild 50000
</IfModule>

Number of VirtualHosts 4. All other hosts have activity almost equal
to 0.

----------- General information -----------
max = 15
count = 15
active = 15
inactive = 0

----------- Applications -----------
/var/apps/svejo.net/releases/20080609182340:
PID: 9005 Sessions: 2
PID: 17735 Sessions: 2
PID: 23792 Sessions: 2
PID: 6928 Sessions: 2
PID: 22242 Sessions: 2
PID: 22785 Sessions: 2
PID: 7704 Sessions: 2
PID: 1962 Sessions: 2
PID: 28965 Sessions: 2
PID: 5018 Sessions: 2
PID: 17441 Sessions: 2
PID: 5008 Sessions: 2
PID: 30963 Sessions: 2
PID: 24379 Sessions: 2
PID: 5064 Sessions: 2

-------------- Apache processes ---------------
PID PPID Threads VMSize Private Name
-----------------------------------------------

4854 20990 1 200.0 MB 0.7 MB /usr/sbin/httpd
4912 20990 1 200.1 MB 0.8 MB /usr/sbin/httpd
4931 20990 1 200.1 MB 0.8 MB /usr/sbin/httpd
4936 20990 1 200.1 MB 0.8 MB /usr/sbin/httpd
4938 20990 1 200.1 MB 0.8 MB /usr/sbin/httpd
4940 20990 1 200.0 MB 0.7 MB /usr/sbin/httpd
4946 20990 1 200.0 MB 0.7 MB /usr/sbin/httpd
4954 20990 1 200.0 MB 0.7 MB /usr/sbin/httpd
4956 20990 1 200.1 MB 0.8 MB /usr/sbin/httpd
4960 20990 1 200.1 MB 0.8 MB /usr/sbin/httpd
4962 20990 1 200.0 MB 0.7 MB /usr/sbin/httpd
4980 20990 1 200.0 MB 0.7 MB /usr/sbin/httpd
4983 20990 1 200.0 MB 0.7 MB /usr/sbin/httpd
5037 20990 1 200.1 MB 0.8 MB /usr/sbin/httpd
5054 20990 1 200.0 MB 0.7 MB /usr/sbin/httpd
5056 20990 1 200.1 MB 0.8 MB /usr/sbin/httpd
12861 20990 1 199.9 MB 0.0 MB /usr/sbin/httpd
13457 20990 1 199.8 MB 0.0 MB /usr/sbin/httpd
20990 1 1 199.6 MB 0.2 MB /usr/sbin/httpd
22516 20990 1 200.2 MB 0.6 MB /usr/sbin/httpd
22780 20990 1 199.9 MB 0.5 MB /usr/sbin/httpd
23623 20990 1 200.2 MB 0.8 MB /usr/sbin/httpd
23795 20990 1 199.9 MB 0.6 MB /usr/sbin/httpd
23807 20990 1 199.9 MB 0.6 MB /usr/sbin/httpd
23823 20990 1 199.9 MB 0.6 MB /usr/sbin/httpd
23827 20990 1 200.1 MB 0.7 MB /usr/sbin/httpd
23835 20990 1 200.2 MB 0.8 MB /usr/sbin/httpd
23850 20990 1 199.9 MB 0.6 MB /usr/sbin/httpd
23868 20990 1 199.9 MB 0.6 MB /usr/sbin/httpd
23875 20990 1 199.9 MB 0.5 MB /usr/sbin/httpd
23878 20990 1 199.8 MB 0.4 MB /usr/sbin/httpd
### Processes: 31
### Total private dirty RSS: 19.38 MB

---------- Passenger processes -----------

PID Threads VMSize Private Name

------------------------------------------
1962 1 214.2 MB 68.2 MB Rails: /var/apps/svejo.net/
releases/20080609182340
3721 1 114.7 MB 30.5 MB Passenger FrameworkSpawner: 2.0.2
5008 1 208.1 MB 59.7 MB Rails: /var/apps/svejo.net/
releases/20080609182340
5018 1 206.9 MB 60.4 MB Rails: /var/apps/svejo.net/
releases/20080609182340
5064 1 205.7 MB 61.3 MB Rails: /var/apps/svejo.net/
releases/20080609182340
6928 1 225.9 MB 69.0 MB Rails: /var/apps/svejo.net/
releases/20080609182340
7704 1 250.5 MB 93.7 MB Rails: /var/apps/svejo.net/
releases/20080609182340
9005 1 205.5 MB 3.2 MB Rails: /var/apps/svejo.net/
releases/20080609182340
17441 1 262.0 MB 107.3 MB Rails: /var/apps/svejo.net/
releases/20080609182340
17735 1 208.2 MB 40.9 MB Rails: /var/apps/svejo.net/
releases/20080609182340
22242 1 204.3 MB 54.4 MB Rails: /var/apps/svejo.net/
releases/20080609182340
22785 1 205.5 MB 56.2 MB Rails: /var/apps/svejo.net/
releases/20080609182340
23792 1 200.9 MB 53.3 MB Rails: /var/apps/svejo.net/
releases/20080609182340
24379 1 227.4 MB 76.9 MB Rails: /var/apps/svejo.net/
releases/20080609182340
28965 1 253.0 MB 104.6 MB Rails: /var/apps/svejo.net/
releases/20080609182340
30677 33 1849.8 MB 75.0 MB /usr/lib64/ruby/gems/1.8/gems/
passenger-1.9.0/ext/apache2/ApplicationPoolServerExecutable 0 /usr/
lib64/ruby/gems/1.8/gems/passenger-1.9.0/bin/passenger-spawn-server/
usr/bin/ruby /tmp/passenger_status.20990.fifo
30678 2 44.7 MB 10.8 MB Passenger spawn server
30963 1 219.8 MB 71.0 MB Rails: /var/apps/svejo.net/
releases/20080609182340
### Processes: 18
### Total private dirty RSS: 1096.42 MB

Static assets are on the same host.
There were ~ 200 online users. Last Friday it reached ~ 500 online
users and there were not such problems (The httpd serrver was with
default MPM configuration - ServerLimit 256, MaxRequestsPerChild
4000).

service httpd restart - fixed the problems wtih irresponsive apache/
rails. I didn't remember to check other hosts.

There are no errors in apache log, neither in the production.log of
Rails.

I really hope fixing those problem.

Stanislav

> E-mail: i...@phusion.nl

[Stanislav]

Passenger specific configs:

RailsMaxPoolSize 15
RailsPoolIdleTime 600

[Hongli Lai]

Stanislav wrote:
> It happened again :(
>
> Here it is my setup:
>
> I'm using apache with prefork currently. I tried running it with
> worker MPM, however it crashed again. So my current MPM config is
> this:

> [snip]

Your server might be swapping to death. Unfortunately you didn't include
the output of 'free -m' so I can't be sure what exactly is wrong.

- How much RAM does your server have?
- How many Mongrels were you running before?
- How many requests per second do you get at peak times?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Stanislav]

I think it wasn't swapping to death. In that cases the server load
increases up to 50 and more. Whith the above freeze I did had server
load 0.0 .

The server has 2G RAM from which 128MB (RES) for memcached and 128MB
for mysql.

Before that I was running with 12 Mongrels. Sorry, I really forgot to
get "free -m" info. I don't know how to give you information about
requests per second :( .

What does mean the number of Sessions in passenger_status.fifo?
Sometimes I've seen 7 processes with (Sessions: 0) and 1 process with
(Sessions: 30).

I changed the profork MPM configuration to this:

<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 10

ServerLimit 60
MaxClients 60
MaxRequestsPerChild 4000
</IfModule>

I hope that could help, becouse I want to stick to Passenger...

On Jun 16, 11:23 am, Hongli Lai <hon...@phusion.nl> wrote:
> Stanislav wrote:
> > It happened again :(
>
> > Here it is my setup:
>
> > I'm using apache with prefork currently. I tried running it with
> > worker MPM, however it crashed again. So my current MPM config is
> > this:
> > [snip]
>
> Your server might be swapping to death. Unfortunately you didn't include
> the output of 'free -m' so I can't be sure what exactly is wrong.
>
> - How much RAM does your server have?
> - How many Mongrels were you running before?
> - How many requests per second do you get at peak times?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

Stanislav wrote:
> I think it wasn't swapping to death. In that cases the server load
> increases up to 50 and more. Whith the above freeze I did had server
> load 0.0 .
>
> The server has 2G RAM from which 128MB (RES) for memcached and 128MB
> for mysql.
>
> Before that I was running with 12 Mongrels.

Then you should lower your RailsMaxPoolSize to 12.

> Sorry, I really forgot to
> get "free -m" info. I don't know how to give you information about
> requests per second :( .
>
> What does mean the number of Sessions in passenger_status.fifo?
> Sometimes I've seen 7 processes with (Sessions: 0) and 1 process with
> (Sessions: 30).

That's the number of requests currently in a Rails process's queue. More
information here:
http://www.modrails.com/documentation/Users%20guide%202.0.html#_inspecting_phusion_passenger_s_internal_status

Actually, now that I've re-read your emails, I'm not sure what the
problem is. Which of the following you mean by "crashing"?
1. Apache stops responding completely, even on static assets like
images, and on non-Rails virtual hosts.
2. The Rails processes stop responding, and Apache itself is unaffected.
That is, Apache can still access static assets like images, but not
Rails URIs.
3. One or more Apache processes exit with a segmentation fault.

Do you also see anything relevant in the Apache error log when the
problem occurs?

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[sandro d.]

Sorry,
I got the problem again too. I think we can discard the Apache bug.

In my case, what happens is:

1. Apache stops responding completely, even on static assets like
images, and on non-Rails virtual hosts.

Still trying to figure it out.

Sandro Duarte

On Jun 16, 6:49 am, Hongli Lai <hon...@phusion.nl> wrote:
> Stanislav wrote:
> > I think it wasn't swapping to death. In that cases the server load
> > increases up to 50 and more. Whith the above freeze I did had server
> > load 0.0 .
>
> > The server has 2G RAM from which 128MB (RES) for memcached and 128MB
> > for mysql.
>
> > Before that I was running with 12 Mongrels.
>
> Then you should lower your RailsMaxPoolSize to 12.
>
> > Sorry, I really forgot to
> > get "free -m" info. I don't know how to give you information about
> > requests per second :( .
>
> > What does mean the number of Sessions in passenger_status.fifo?
> > Sometimes I've seen 7 processes with (Sessions: 0) and 1 process with
> > (Sessions: 30).
>
> That's the number of requests currently in a Rails process's queue. More

> information here:http://www.modrails.com/documentation/Users%20guide%202.0.html#_inspe...

>
> Actually, now that I've re-read your emails, I'm not sure what the
> problem is. Which of the following you mean by "crashing"?
> 1. Apache stops responding completely, even on static assets like
> images, and on non-Rails virtual hosts.
> 2. The Rails processes stop responding, and Apache itself is unaffected.
> That is, Apache can still access static assets like images, but not
> Rails URIs.
> 3. One or more Apache processes exit with a segmentation fault.
>
> Do you also see anything relevant in the Apache error log when the
> problem occurs?
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

Stanislav wrote:
> I think it wasn't swapping to death. In that cases the server load
> increases up to 50 and more. Whith the above freeze I did had server
> load 0.0 .

> [snip]

By the way, please set PassengerLogLevel to 2, and post the last 300
lines of your Apache error log next time the problem occurs.

Thanks.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Stanislav]

Sorry for bothering so much with my problem, however it's a
production server and we're losing users.... :(

We had a downtime again. Only passenger/Rails freezes. I can access
static resources from the apache and the same application. It looks
that a lot of passenger instances are being spawned and that's the
problem. May be I should raise their number? I don't get any
exceptions in any of the logs.

Here they are some status info:
---------------------------------------------
free -m
total used free shared buffers
cached
Mem: 2013 1945 68 0 34
286
-/+ buffers/cache: 1624 389
Swap: 2000 231 1768

passenger-memory-status

-------------- Apache processes ---------------
PID PPID Threads VMSize Private Name
-----------------------------------------------

22700 31042 1 198.8 MB 0.6 MB /usr/sbin/httpd
23203 31042 1 198.7 MB 0.3 MB /usr/sbin/httpd
24945 31042 1 199.0 MB 0.7 MB /usr/sbin/httpd
........
25160 31042 1 198.8 MB 0.6 MB /usr/sbin/httpd
31042 1 1 198.5 MB 0.1 MB /usr/sbin/httpd
### Processes: 41
### Total private dirty RSS: 26.28 MB


---------- Passenger processes ----------

PID Threads VMSize Private Name
-----------------------------------------

14824 1 217.9 MB 65.3 MB Rails: /var/apps/svejo.net/
releases/20080609182340
14866 1 231.2 MB 74.2 MB Rails: /var/apps/svejo.net/
releases/20080609182340
15983 1 221.6 MB 74.8 MB Rails: /var/apps/svejo.net/
releases/20080609182340
17740 1 221.9 MB 68.4 MB Rails: /var/apps/svejo.net/
releases/20080609182340
17748 1 217.1 MB 64.3 MB Rails: /var/apps/svejo.net/
releases/20080609182340
19549 1 228.4 MB 75.1 MB Rails: /var/apps/svejo.net/
releases/20080609182340
19835 1 224.2 MB 75.6 MB Rails: /var/apps/svejo.net/
releases/20080609182340
21605 1 219.6 MB 64.8 MB Rails: /var/apps/svejo.net/
releases/20080609182340
21625 1 214.3 MB 63.4 MB Rails: /var/apps/svejo.net/
releases/20080609182340
21627 1 214.9 MB 63.9 MB Rails: /var/apps/svejo.net/
releases/20080609182340
22104 1 218.2 MB 71.9 MB Rails: /var/apps/svejo.net/
releases/20080609182340
22106 1 207.2 MB 60.9 MB Rails: /var/apps/svejo.net/
releases/20080609182340
24127 1 115.5 MB 30.6 MB Passenger FrameworkSpawner: 2.0.2
24144 1 205.6 MB 58.9 MB Rails: /var/apps/svejo.net/
releases/20080609182340
24150 1 205.5 MB 58.7 MB Rails: /var/apps/svejo.net/
releases/20080609182340
24799 1 205.1 MB 58.5 MB Rails: /var/apps/svejo.net/
releases/20080609182340
24801 1 208.1 MB 61.0 MB Rails: /var/apps/svejo.net/
releases/20080609182340
31044 43 1412.3 MB 88.5 MB /usr/lib64/ruby/gems/1.8/gems/

passenger-1.9.0/ext/apache2/ApplicationPoolServerExecutable 0 /usr/
lib64/ruby/gems/1.8/gems/passenger-1.9.0/bin/passenger-spawn-server /

usr/bin/ruby /tmp/passenger_status.31042.fifo
31045 2 44.5 MB 10.3 MB Passenger spawn server
### Processes: 19
### Total private dirty RSS: 1189.12 MB

cat /tmp/passenger_status.31042.fifo
----------- General information -----------
max = 16
count = 16
active = 16

inactive = 0

----------- Applications -----------
/var/apps/svejo.net/releases/20080609182340:

PID: 22106 Sessions: 3
PID: 14866 Sessions: 3
PID: 14824 Sessions: 3
PID: 24144 Sessions: 3
PID: 15983 Sessions: 3
PID: 17740 Sessions: 2
PID: 19549 Sessions: 2
PID: 21625 Sessions: 2
PID: 24799 Sessions: 2
PID: 17748 Sessions: 3
PID: 19835 Sessions: 2
PID: 21605 Sessions: 2
PID: 24801 Sessions: 2
PID: 22104 Sessions: 2
PID: 21627 Sessions: 3
PID: 24150 Sessions: 3



I think that I'm starting to repeat myself. I'll change the log level
for the next freeze.

On Jun 17, 9:24 am, Hongli Lai <hon...@phusion.nl> wrote:
> Stanislav wrote:
> > I think it wasn't swapping to death. In that cases the server load
> > increases up to 50 and more. Whith the above freeze I did had server
> > load 0.0 .
>
>  > [snip]
>
> By the way, please set PassengerLogLevel to 2, and post the last 300
> lines of your Apache error log next time the problem occurs.
>
> Thanks.
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[Hongli Lai]

Stanislav wrote:
> Sorry for bothering so much with my problem, however it's a
> production server and we're losing users.... :(

Sorry that you have to go through this. In this case I'd recommend you
to go back to Mongrel, and test Passenger on a staging server, or load
balance the thing so that Passenger won't take down the whole site.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Stanislav]

Thank you for your fast responses. I think to try the setup with load
balance with a few mongrels

Cheers

[Stanislav]

Hi again,

Just to mention. I continued using passenger and lowered the number of
instances down to 8. It was running almost 12 hours without problems.
Today it crashed again. I had this error in error logs -
[Wed Jun 18 11:28:46 2008] [error] server reached MaxClients setting,
consider raising the MaxClients setting

My setting for MaxClients was 64. I changed it now up to 128.
So my question is: Is it normal all passenger instances to freeze when
there's not enough httpd processes to spawn?

Passenger-status was this:
----------- General information -----------
max = 8
count = 8
active = 8

inactive = 0

----------- Applications -----------
/var/apps/svejo.net/releases/20080609182340:

PID: 11935 Sessions: 8
PID: 9904 Sessions: 8
PID: 13954 Sessions: 8
PID: 9579 Sessions: 8
PID: 12563 Sessions: 8
PID: 14621 Sessions: 8
PID: 14725 Sessions: 8
PID: 12020 Sessions: 8

I hope I'm getting closer to the problem.

[Hongli Lai]

From this it would seem like your Rails application is being
overloaded, i.e. there's more traffic than it can handle. The queues are
huge.

[Stanislav]

Yes it looks like this, but the server is not loaded. All passengers
are just frozen. I thought it should work very slowly , but not
freeze...

> E-mail: i...@phusion.nl

[Hongli Lai]

Stanislav wrote:
> Yes it looks like this, but the server is not loaded. All passengers
> are just frozen. I thought it should work very slowly , but not
> freeze...

If you kill one of the processes with SIGABRT, then it'll print its
stack trace to the Apache log file. That might show the cause of the
problem.

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Stanislav]

I have more info on the problem. I'm almost sure it's not a load
problem.

I raised MaxClients of apache prefork to 128. This time I had all 8
passenger processes frozen with every one having 16 sessions. It looks
that there's something that is locking those processes.

Wit PassengerLogLevel = 2, I have these errors in the apache error
log:

pid=9579 file=Hooks.cpp:532 time=06/18/08 18:56:46.682 ]:
Content-Length = 38904
[ pid=9898 file=Hooks.cpp:532 time=06/18/08 19:01:31.811 ]:
Content-Length = 82928
[ pid=11310 file=Hooks.cpp:532 time=06/18/08 19:30:51.268 ]:
Content-Length = 35481

[ pid=28781 file=Hooks.cpp:864 time=06/18/08 20:52:05.487 ]:
Shutting down Phusion Passenger...
[ pid=28781 file=ApplicationPoolServer.h:417 time=06/18/08
20:52:05.487 ]:
Waiting for existing ApplicationPoolServerExecutable (PID 7022) to
exit...
[ pid=28781 file=ApplicationPoolServer.h:435 time=06/18/08
20:52:05.488 ]:
ApplicationPoolServerExecutable exited.

[ pid=12174 file=Hooks.cpp:561 time=06/18/08 20:52:06.653 ]:
Initializing Phusion Passenger...
[ pid=12174 file=Hooks.cpp:864 time=06/18/08 20:52:06.731 ]:
Shutting down Phusion Passenger...
[ pid=12174 file=ApplicationPoolServer.h:417 time=06/18/08
20:52:06.732 ]:
Waiting for existing ApplicationPoolServerExecutable (PID 12175) to
exit...
[ pid=12175 file=ApplicationPoolServerExecutable.cpp:144 time=06/18/08
20:52:06.732 ]:
Shutting down server.
[ pid=12175 file=ApplicationPoolServerExecutable.cpp:112 time=06/18/08
20:52:06.733 ]:
Status report thread interrupted.
[ pid=12175 file=ApplicationPoolServerExecutable.cpp:167 time=06/18/08
20:52:06.736 ]:
Server shutdown complete.
[ pid=12175 file=SpawnManager.h:460 time=06/18/08 20:52:06.736 ]:
Shutting down spawn manager (PID 12176).
[ pid=12175 file=SpawnManager.h:463 time=06/18/08 20:52:06.738 ]:
Spawn manager exited.
[ pid=12174 file=ApplicationPoolServer.h:435 time=06/18/08
20:52:06.740 ]:
ApplicationPoolServerExecutable exited

I "solve" temporalily the problem with "service httpd reload". This
night I'll write a script that will monitor passenger-status and will
reload the apache. I know it's not the best solution, but it could be
a workaround.

Thanks again for your support.

Stanislav

On Jun 18, 1:50 pm, Hongli Lai <hon...@phusion.nl> wrote:
> Stanislav wrote:
> > Yes it looks like this, but the server is not loaded. All passengers
> > are just frozen. I thought it should work very slowly , but not
> > freeze...
>
> If you kill one of the processes with SIGABRT, then it'll print its
> stack trace to the Apache log file. That might show the cause of the
> problem.
>
> --
> Phusion | The Computer Science Company
>
> Web:http://www.phusion.nl/

> E-mail: i...@phusion.nl

[sandro d.]

Stanislav,

I've had the exact same problem. All my passenger processes eventually
froze and Apache stop responding.
Updated to 1.9.1.
No problems anymore, really! Two whole days. I think you should try
it.

Sandro

[Stanislav]

The last decribed problem is with exactly that version 1.9.1

sandro d. , please, share your configuration/setup . Thank you a lot.
If you want, you can do it in this thread:
http://groups.google.com/group/phusion-passenger/browse_thread/thread/71e70aea1ec220c9#
.

Cheers

[Stanislav]

I have more info on my problem. My output of passenger-status now:

----------- Applications -----------
PID: 30588 Sessions: 0
PID: 30582 Sessions: 0
PID: 30580 Sessions: 0
PID: 30586 Sessions: 0
PID: 30584 Sessions: 3
PID: 2446 Sessions: 1
PID: 30572 Sessions: 1
PID: 2444 Sessions: 1
PID: 30578 Sessions: 1

Process with pid 30584 is stale. Here it's output from strace:
write(4, "erver_sidebar_advert.js\"></scrip"..., 32128

So it looks that it have problems on the "write" function - returning
results. Any ideas how I can debug it further?

10x

On Jun 18, 10:58 pm, Stanislav <stanislav.bozh...@gmail.com> wrote:
> The last decribed problem is with exactly that version 1.9.1
>
> sandro d. , please, share your configuration/setup . Thank you a lot.

> If you want, you can do it in this thread:http://groups.google.com/group/phusion-passenger/browse_thread/thread...

[Hongli Lai]

Stanislav wrote:
> I have more info on my problem. My output of passenger-status now:
>
> ----------- Applications -----------
> PID: 30588 Sessions: 0
> PID: 30582 Sessions: 0
> PID: 30580 Sessions: 0
> PID: 30586 Sessions: 0
> PID: 30584 Sessions: 3
> PID: 2446 Sessions: 1
> PID: 30572 Sessions: 1
> PID: 2444 Sessions: 1
> PID: 30578 Sessions: 1
>
> Process with pid 30584 is stale. Here it's output from strace:
> write(4, "erver_sidebar_advert.js\"></scrip"..., 32128
>
> So it looks that it have problems on the "write" function - returning
> results. Any ideas how I can debug it further?

Please refer to the email that I sent to Sando D. ("Should one Passenger
process freeze the whole Apache?")

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Hongli Lai]

Hi Stanislav.

I think I've figured out why this happens and I believe to have found a
solution. I've prepared a version for you which enforces I/O timeouts in
a stricter manner. The source can be downloaded here:
http://github.com/FooBarWidget/passenger/tarball/experimental

Could you give this version a try?

Note that the I/O timeout is the same as the value given to Apache's
'Timeout' option. On Ubuntu, the default value is 15 (seconds).
Depending on your application, you may or may not want to tweak this
setting.

Regards,
Hongli Lai

--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[joost]

As Topic starter I wanted to let everybody know I found our problem.
It had nothing to do with Passenger/mod-rails. It was Google grabbing
the sitemap every couple of days. The sidemap code had a HUGE memory
leak (like 100Mb or so). This broke down the server.

So Passenger seems to do everything fine. However I still have the
issue with the sitemap grabbing about 10.000 Rails ActiveRecord
objects (yeah..a lot). And leaving a lot of memory ungarbage
collected. So if anybody knows why this happens (it also happened on a
clean Rails 2.1 install when grabbing 10.000 objects) please contact
me!

Also wanted to thank the Passenger guys. Nice presentation on
RubyEnRails! Cheers!

[Stanislav]

I'm running the experimental version for the last 12hours and it does
not have freezes yet. Thank U for the support.

> E-mail: i...@phusion.nl

[Stanislav]

It crashed again.

I think there's something specific with our app in order to crash so
often. However I switched back to the old stable mongrel setup.
Mongrel could be slower , however we really need stability.

I hope we'll have more stable passenger in the future. It's really
great, but I think it needs some more time and experience.

Stanislav

[nextpulse]

Was this ever resolved?
We moved to passenger recently - latest updated release. (running on
FC8).
Apache froze on us in under 24 hours - I was expecting some degrade
but not a complete lock out.
Only a restart of httpd will release the ruby processes.

Any suggestion?

(before we move back to mongrel)

thks.

[Hongli Lai]

nextpulse wrote:
> Was this ever resolved?
> We moved to passenger recently - latest updated release. (running on
> FC8).
> Apache froze on us in under 24 hours - I was expecting some degrade
> but not a complete lock out.
> Only a restart of httpd will release the ruby processes.
>
> Any suggestion?
>
> (before we move back to mongrel)
>
> thks.

I'm afraid I'm out of options until I can reproduce this problem myself.

Regards,
Hongli Lai
--
Phusion | The Computer Science Company

Web: http://www.phusion.nl/
E-mail: in...@phusion.nl

[Pooya Woodcock]

Hey guys :-)

I'm reviving this thread after trying the latest versions of rails,
ruby, and phusion passenger for the past few months. I have been
switching back and forth between mongrel, phusion, and nginx+thin, and
I would really love to get phusion working. I have a cluster of 6
servers sitting behind a hardware load balancer, running apache 2.2.9
and phusion 2.0.2. When the server crashes (roughly every 12 hours),
the system load on every machine goes to 0.00. The server doesn't swap
out of ram yet because I catch the machine before the # of httpd
processes gets too high.

Here are the technical details, using one server out of the cluster.

# cat status1.txt
----------- General information -----------
max = 60
count = 60
active = 57
inactive = 3

----------- Applications -----------
/var/home/wegadmin/multi_sites_members/releases/20080522145908:
PID: 12216 Sessions: 27
PID: 11979 Sessions: 28
PID: 26697 Sessions: 28
PID: 24594 Sessions: 28

/var/home/wegadmin/video_fly/releases/20080722180743:
PID: 21151 Sessions: 0
PID: 11170 Sessions: 0
PID: 7098 Sessions: 0
PID: 4335 Sessions: 2
PID: 2322 Sessions: 2
PID: 4333 Sessions: 2
PID: 14070 Sessions: 2
PID: 11187 Sessions: 2
PID: 21149 Sessions: 2
PID: 14072 Sessions: 2
PID: 14068 Sessions: 2
PID: 10728 Sessions: 3
PID: 10601 Sessions: 3
PID: 10686 Sessions: 3
PID: 10763 Sessions: 3
PID: 10194 Sessions: 3
PID: 10684 Sessions: 3
PID: 10761 Sessions: 3
PID: 10603 Sessions: 3
PID: 10672 Sessions: 3
PID: 10681 Sessions: 3
PID: 10192 Sessions: 3
PID: 10679 Sessions: 3
PID: 10688 Sessions: 3
PID: 10759 Sessions: 3
PID: 28653 Sessions: 3
PID: 10765 Sessions: 3
PID: 28657 Sessions: 3
PID: 10190 Sessions: 3
PID: 28655 Sessions: 3
PID: 10643 Sessions: 3
PID: 10757 Sessions: 3
PID: 5281 Sessions: 3
PID: 5277 Sessions: 3
PID: 11162 Sessions: 3
PID: 14154 Sessions: 3
PID: 5279 Sessions: 3
PID: 5283 Sessions: 3
PID: 10196 Sessions: 3
PID: 10767 Sessions: 3
PID: 8025 Sessions: 3
PID: 19347 Sessions: 3
PID: 19351 Sessions: 3
PID: 11722 Sessions: 3
PID: 11185 Sessions: 3
PID: 10206 Sessions: 3
PID: 19349 Sessions: 3
PID: 8027 Sessions: 3
PID: 19345 Sessions: 3
PID: 21153 Sessions: 3
PID: 19353 Sessions: 3
PID: 14152 Sessions: 3
PID: 7706 Sessions: 3
PID: 21147 Sessions: 3
PID: 19135 Sessions: 3
PID: 19993 Sessions: 1


# cat status2.xt
-------------- Apache processes --------------

PID PPID Threads VMSize Private Name
----------------------------------------------

668 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
986 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
1093 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
1213 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
1344 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
1731 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
1882 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
1933 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
2015 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
2299 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
2327 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
2558 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
2962 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
2989 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
3026 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
3052 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
3144 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
3454 10157 1 71.6 MB 0.4 MB /usr/local/apache_mongrel/bin/
httpd -k start
3532 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
3562 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
3565 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
3699 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
3705 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
4039 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
4093 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
4296 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
4486 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
4688 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
4689 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
4702 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
4706 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
4791 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5282 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5285 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5288 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5301 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5392 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5403 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5428 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5478 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5509 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5862 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5882 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
5885 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
6029 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
6081 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
6362 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
6423 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
6425 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
6454 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
6562 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7002 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7136 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7179 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7197 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
7307 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7428 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
7542 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7701 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7708 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7758 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7802 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
7824 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8165 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8210 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8250 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8283 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8290 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8457 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8602 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8782 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8806 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
8970 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9030 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9049 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
9160 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9344 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9405 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9419 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9577 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9581 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9644 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9776 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
9988 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10104 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10106 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10143 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10157 1 1 71.4 MB 0.4 MB /usr/local/apache_mongrel/bin/
httpd -k start
10199 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
10258 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
10259 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
10262 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
10264 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10546 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10655 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10754 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10786 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10788 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10847 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10958 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
10977 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11145 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11148 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11183 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11214 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11216 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11253 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11265 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11280 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11285 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11295 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11296 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11309 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11318 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11321 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11331 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11338 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11351 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11354 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
11379 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11387 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
11389 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
11399 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
11409 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11413 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
11417 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11424 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11431 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11703 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11707 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11709 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11717 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11731 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11735 10157 1 0.0 MB ? [httpd] <defunct>
11739 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11744 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11749 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11773 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11778 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11781 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11785 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11792 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11799 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11815 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11817 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11820 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11822 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11827 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11830 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11831 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11836 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11838 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11842 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11844 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11851 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11857 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
11859 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11863 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11865 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11867 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11873 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11875 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11881 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11883 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11885 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11921 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11954 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11956 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11963 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
11995 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
12093 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
12105 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
12107 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
12109 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
12110 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
12113 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
12118 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
12205 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
12737 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
12851 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
13118 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
13980 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
14757 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
14866 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
15126 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
15131 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
15192 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
15652 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
15672 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
15817 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
16116 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
16118 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
16123 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
16587 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
16623 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
16680 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
16802 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
17710 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
17736 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
17822 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
17857 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
17871 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
17886 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18195 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18255 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18281 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18332 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18373 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18391 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18395 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18812 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18860 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18875 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
18898 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
19213 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
19278 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
19300 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
19321 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
19391 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
19817 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
19877 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
19943 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
20258 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
20336 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
20463 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
20480 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
20486 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
20960 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
20964 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
20968 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
21024 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
21196 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
21387 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
21461 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
21598 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22037 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22111 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22469 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22496 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22540 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22593 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22595 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
22598 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
22614 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22950 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
22975 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
23465 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
23541 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24101 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24130 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24134 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24509 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24601 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24614 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24629 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24929 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
24975 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
25058 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
25098 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
25159 10157 1 71.6 MB 0.4 MB /usr/local/apache_mongrel/bin/
httpd -k start
25306 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
25494 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
25507 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
25511 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
25559 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
26638 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
26734 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27111 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27120 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27175 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27378 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27395 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
27400 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27463 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27483 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27824 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27842 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27861 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27863 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27865 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27894 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27898 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27923 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27963 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
27974 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28004 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
28005 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
28007 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
28008 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
28013 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28020 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
28022 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
28400 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28405 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28467 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28488 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28499 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28501 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28525 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28546 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
28581 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
29007 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
29063 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
29065 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
29113 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
29120 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
29209 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
29483 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
29550 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
29633 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
30029 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
30198 10157 1 71.6 MB 0.4 MB /usr/local/apache_mongrel/bin/
httpd -k start
30269 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
30648 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
30678 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
30733 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
30925 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
31240 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
31285 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
31301 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
31320 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
31322 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
31330 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
31347 10157 1 71.5 MB 0.2 MB /usr/local/apache_mongrel/bin/
httpd -k start
31371 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
31810 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
32030 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
32117 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
32400 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
32429 10157 1 71.5 MB 0.3 MB /usr/local/apache_mongrel/bin/
httpd -k start
### Processes: 330
### Total private dirty RSS: 92.21 MB (?)

---------- Passenger processes ----------
PID Threads VMSize Private Name
-----------------------------------------

2322 1 188.6 MB 101.9 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
4333 1 196.8 MB 109.7 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
4335 1 180.2 MB 94.5 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
5277 1 150.2 MB 64.4 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
5279 1 223.8 MB 136.6 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
5281 1 144.9 MB 59.1 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
5283 1 205.3 MB 118.1 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
7098 1 284.2 MB 197.6 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
7706 1 232.5 MB 145.8 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
8025 1 189.7 MB 103.9 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
8027 1 261.2 MB 174.0 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10158 331 123.5 MB 5.2 MB /usr/local/lib/ruby/gems/1.8/gems/
passenger-2.0.2/ext/apache2/ApplicationPoolServerExecutable 0 /usr/
local/lib/ruby/gems/1.8/gems/passenger-2.0.2/bin/passenger-spawn-
server /usr/local/bin/ruby /tmp/passenger_status.10157.fifo
10161 1 37.2 MB 9.4 MB Passenger spawn server
10190 1 228.2 MB 142.2 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10192 1 175.9 MB 92.0 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10194 1 185.7 MB 98.4 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10196 1 237.0 MB 151.2 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10206 1 265.5 MB 178.4 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10601 1 142.2 MB 56.3 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10603 1 142.7 MB 56.8 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10643 1 220.1 MB 132.9 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10672 1 154.0 MB 68.2 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10679 1 193.6 MB 106.5 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10681 1 184.2 MB 98.2 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10684 1 135.2 MB 51.2 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10686 1 144.5 MB 58.6 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10688 1 206.2 MB 118.9 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10728 1 143.3 MB 57.4 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10757 1 200.1 MB 114.2 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10759 1 146.9 MB 60.9 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10761 1 159.5 MB 73.6 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10763 1 181.3 MB 95.4 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10765 1 226.8 MB 140.0 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
10767 1 299.2 MB 213.3 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
11162 1 146.1 MB 60.3 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
11170 1 323.9 MB 236.8 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
11185 1 141.9 MB 58.0 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
11187 1 233.8 MB 146.8 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
11722 1 230.5 MB 143.5 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
11979 1 136.3 MB 54.2 MB Rails: /var/home/wegadmin/
multi_sites_members/releases/20080522145908
12216 1 141.5 MB 57.3 MB Rails: /var/home/wegadmin/
multi_sites_members/releases/20080522145908
14068 1 203.7 MB 116.6 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
14070 1 138.0 MB 54.2 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
14072 1 149.1 MB 63.2 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
14152 1 261.5 MB 175.8 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
14154 1 147.8 MB 62.1 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
19135 1 183.8 MB 97.9 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
19345 1 238.2 MB 151.0 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
19347 1 194.1 MB 107.0 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
19349 1 193.7 MB 106.9 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
19351 1 206.7 MB 120.9 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
19353 1 245.7 MB 159.8 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
19993 1 309.5 MB 223.6 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
21147 1 190.6 MB 103.5 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
21149 1 230.9 MB 145.1 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
21151 1 299.6 MB 212.6 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
21153 1 155.2 MB 69.3 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
24594 1 142.2 MB 58.0 MB Rails: /var/home/wegadmin/
multi_sites_members/releases/20080522145908
26697 1 137.2 MB 55.1 MB Rails: /var/home/wegadmin/
multi_sites_members/releases/20080522145908
28653 1 162.5 MB 76.7 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
28655 1 198.0 MB 111.0 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
28657 1 158.4 MB 72.5 MB Rails: /var/home/wegadmin/
video_fly/releases/20080722180743
### Processes: 62
### Total private dirty RSS: 6584.71 MB



My settings are:
# mod rails
LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/
passenger-2.0.2/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/lib/ruby/gems/1.8/gems/passenger-2.0.2
PassengerRuby /usr/local/bin/ruby

RailsMaxPoolSize 60
RailsPoolIdleTime 120
PassengerPoolIdleTime 120
# end mod rails

[Pooya Woodcock]

On Jul 23, 7:05 am, Pooya Woodcock <pooya.woodc...@gmail.com> wrote:
> My settings are:
> # mod rails
>    LoadModule passenger_module /usr/local/lib/ruby/gems/1.8/gems/
> passenger-2.0.2/ext/apache2/mod_passenger.so
>    PassengerRoot /usr/local/lib/ruby/gems/1.8/gems/passenger-2.0.2
>    PassengerRuby /usr/local/bin/ruby
>
>    RailsMaxPoolSize 60
>    RailsPoolIdleTime 120
>    PassengerPoolIdleTime 120
> # end mod rails

I'm going to increase the PassengerLogLevel to 2 then report back.

Thanks,
Pooya

[Hongli Lai]

The load doesn't seem to be fairly distributed between your virtual
hosts. You might be interested in the 'PassengerMaxInstancesPerApp' option.

[Pooya Woodcock]

On Jul 23, 8:39 am, Hongli Lai <hon...@phusion.nl> wrote:

> The load doesn't seem to be fairly distributed between your virtual
> hosts. You might be interested in the 'PassengerMaxInstancesPerApp' option.
>

Hongli,

Thanks for the suggestion. I removed the first project from my config
to rule that out and let it run for some hours. Here's the result with
more verbose logging:

[ pid=32035 file=Hooks.cpp:625 time=07/24/08 02:54:19.639 ]:
Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe
(32) (this warning is harmless if you're currently restarting or
shutting down Apache)

[Thu Jul 24 02:54:19 2008] [notice] child pid 32032 exit signal
Aborted (6)
[Thu Jul 24 02:54:19 2008] [notice] child pid 32033 exit signal
Aborted (6)
[Thu Jul 24 02:54:19 2008] [notice] child pid 32034 exit signal
Aborted (6)
[Thu Jul 24 02:54:19 2008] [notice] child pid 32035 exit signal
Aborted (6)
[Thu Jul 24 02:54:20 2008] [error] [client 82.1.23.9] *** Unexpected
error in Passenger: The ApplicationPool server ex
ited unexpectedly., referer:
http://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&gfns=1&q=the_search_string
[ pid=32036 file=Hooks.cpp:625 time=07/24/08 02:54:20.641 ]:
Cannot initialize Passenger in an Apache child process: Could not
connect to the ApplicationPool server: Broken pipe
(32) (this warning is harmless if you're currently restarting or
shutting down Apache)


I think we may be getting somewhere now!

[Pooya Woodcock]

Here is some more info from the error_log when I scrolled up, perhaps
I ran into the ulimit on open files?

[ pid=31752 file=ApplicationPoolServerExecutable.cpp:144 time=07/24/08
02:54:17.662 ]:
Shutting down server.
[ pid=31752 file=ApplicationPoolServerExecutable.cpp:112 time=07/24/08
02:54:17.662 ]:
Status report thread interrupted.
[ pid=31752 file=ApplicationPoolServerExecutable.cpp:346 time=07/24/08
02:54:17.663 ]:
Client thread 0x2aaaac0cb680 interrupted.
<snip>
[ pid=31752 file=ApplicationPoolServerExecutable.cpp:346 time=07/24/08
02:54:17.759 ]:
Client thread 0x1ad6cec0 interrupted.
[ pid=31752 file=ApplicationPoolServerExecutable.cpp:167 time=07/24/08
02:54:17.772 ]:
Server shutdown complete.
[ pid=31752 file=SpawnManager.h:460 time=07/24/08 02:54:17.772 ]:
Shutting down spawn manager (PID 31755).
[ pid=31752 file=SpawnManager.h:463 time=07/24/08 02:54:17.776 ]:
Spawn manager exited.
[ pid=31752 file=ApplicationPoolServerExecutable.cpp:471 time=07/24/08
02:54:17.776 ]:
Cannot create an anonymous Unix socket: Too many open files (24)

[Hongli Lai]

Pooya Woodcock wrote:
>
> Here is some more info from the error_log when I scrolled up, perhaps
> I ran into the ulimit on open files?

That's possible. What is your ulimit on open files?

Also, could you try the git version? It's a little more robust than
2.0.x. http://github.com/FooBarWidget/passenger/tree/master

[Pooya Woodcock]

On Jul 24, 3:31 am, Hongli Lai <hon...@phusion.nl> wrote:
> Pooya Woodcock wrote:
>
> > Here is some more info from the error_log when I scrolled up, perhaps
> > I ran into the ulimit on open files?
>
> That's possible. What is your ulimit on open files?
>
> Also, could you try the git version? It's a little more robust than
> 2.0.x.http://github.com/FooBarWidget/passenger/tree/master

Thanks, I'll try the github version first then try increasing to 2048
(the limit was 1024).
-Pooya

[Pooya Woodcock]

The GitHub version is showing the leaky memory problem that TonyLa had
earlier.

--------- Passenger processes ----------
PID Threads VMSize Private Name
----------------------------------------

5428 92 304.8 MB 22.7 MB /var/src/FooBarWidget-
passenger-1b49a8d7bd7c270447a14032ffa7139d48e65837/ext/apache2/
ApplicationPoolServerExecutable 2 /var/src/FooBarWidget-
passenger-1b49a8d7bd7c270447a14032ffa7139d48e65837/bin/passenger-spawn-
server /usr/local/bin/ruby /tmp/passenger_status.5427.fifo
5434 1 36.3 MB 8.5 MB Passenger spawn server
<snip>
### Processes: 25
### Total private dirty RSS: 1548.92 MB

2.0.2 seems to be doing better than what's in
http://github.com/FooBarWidget/passenger/tree/master ... did I pull
the right version? I guess I should be pulling experimental?

I have one of my other load balanced servers running 2.0.2 and the
memory status is:

14017 96 91.2 MB 1.6 MB /usr/local/lib/ruby/gems/1.8/gems/
passenger-2.0.2/ext/apache2/ApplicationPoolServerExecutable 0 /usr/
local/lib/ruby/gems/1.8/gems/passenger-2.0.2/bin/passenger-spawn-
server /usr/local/bin/ruby /tmp/passenger_status.14016.fifo
14018 1 36.0 MB 8.2 MB Passenger spawn server
14031 1 121.2 MB 43.6 MB Passenger FrameworkSpawner: 2.1.0
<snip>
### Processes: 18
### Total private dirty RSS: 902.46 MB

[Pooya Woodcock]

> The GitHub version is showing the leaky memory problem that TonyLa had
> earlier.
>
> --------- Passenger processes ----------
> PID   Threads  VMSize    Private   Name
> ----------------------------------------
> 5428  92       304.8 MB  22.7 MB   /var/src/FooBarWidget-
> passenger-1b49a8d7bd7c270447a14032ffa7139d48e65837/ext/apache2/
> ApplicationPoolServerExecutable 2 /var/src/FooBarWidget-
> passenger-1b49a8d7bd7c270447a14032ffa7139d48e65837/bin/passenger-spawn-
> server  /usr/local/bin/ruby  /tmp/passenger_status.5427.fifo
> 5434  1        36.3 MB   8.5 MB    Passenger spawn server

That same server is now at:
5428 92 446.5 MB 36.6 MB /var/src/FooBarWidget-

passenger-1b49a8d7bd7c270447a14032ffa7139d48e65837/ext/apache2/
ApplicationPoolServerExecutable 2 /var/src/FooBarWidget-
passenger-1b49a8d7bd7c270447a14032ffa7139d48e65837/bin/passenger-spawn-
server /usr/local/bin/ruby /tmp/passenger_status.5427.fifo
5434 1 36.3 MB 8.5 MB Passenger spawn server

I downloaded from github again... this time I pulled
2de5e1c93be18d167b69d0a1b95ca76e4fccece5 ... trying this one out

[Hongli Lai]

The 'master' branch is the correct one.

Could you run the ApplicationPoolServerExecutable in Valgrind, and
report its output? Instructions can be found at
http://groups.google.com/group/phusion-passenger/browse_thread/thread/d2e0306cb78b9ca1/f70cf76fd3682c88?hl=en&lnk=gst&q=valgrind#b5789a4acf8ab3bd

Thanks.

[Pooya Woodcock]

> The 'master' branch is the correct one.
>
> Could you run the ApplicationPoolServerExecutable in Valgrind, and

> report its output? Instructions can be found athttp://groups.google.com/group/phusion-passenger/browse_thread/thread...
>

I must have been running with the wrong version from git, because this
one I just downloaded is not leaking anywhere near like the previous.
In any case, I compiled in Valgrind support and here's the output
after about 40 mins: http://dpaste.com/hold/67212/ . Also I tried
mongrel again and that keeps leaking as well. I'm going to debug our
application and upgrade to 1.8.6-p120 from p111 as well and update
later.

Thanks,

[Hongli Lai]

Pooya Woodcock wrote:
> I must have been running with the wrong version from git, because this
> one I just downloaded is not leaking anywhere near like the previous.
> In any case, I compiled in Valgrind support and here's the output
> after about 40 mins: http://dpaste.com/hold/67212/ . Also I tried
> mongrel again and that keeps leaking as well. I'm going to debug our
> application and upgrade to 1.8.6-p120 from p111 as well and update
> later.
>
> Thanks,

According to Valgrind, there are no memory leaks in Passenger itself.
Are you still experiencing memory leak problems now that you compiled
the right version?