Performance of Passenger vs. Mongrel
Mark Plaksin
How does Passenger perform compared to Mongrel? For us Passenger looks
worse. We have two puppetmasters; one does file serving, the other does
everything else. We just started running 0.25 beta2 on both. The
file-server is running RubyEE + Passenger and the non-file-server is
running Mongrel. When our heavy-hitter schedule runs the load average
on our file server spikes much higher than it did when it was running
0.25 beta1 with Mongrel.
We will try 0.25 beta2 + Mongrel on our file server to see if that makes
a difference but I'm curious what others have seen and what we should
expect.
Thanks!
Scott Smith
> Howdy:
>
> How does Passenger perform compared to Mongrel? For us Passenger looks
> worse. We have two puppetmasters; one does file serving, the other does
> everything else. We just started running 0.25 beta2 on both. The
> file-server is running RubyEE + Passenger and the non-file-server is
> running Mongrel. When our heavy-hitter schedule runs the load average
> on our file server spikes much higher than it did when it was running
> 0.25 beta1 with Mongrel.
>
Are you running the same # of mongrels as Passenger workers? What is
your Passenger config? What are the specs of your app server?
-scott
Mark Plaksin
> Mark Plaksin wrote:
>
>> Howdy:
>>
>> How does Passenger perform compared to Mongrel? For us Passenger looks
>> worse. We have two puppetmasters; one does file serving, the other does
>> everything else. We just started running 0.25 beta2 on both. The
>> file-server is running RubyEE + Passenger and the non-file-server is
>> running Mongrel. When our heavy-hitter schedule runs the load average
>> on our file server spikes much higher than it did when it was running
>> 0.25 beta1 with Mongrel.
>
> Are you running the same # of mongrels as Passenger workers?
Sort of :) We run 12 masters in mongrel. Passenger doesn't seem have a
"run at least this many puppetmasters" setting. We set the max to 12 to
match mongrel but we never saw more than 6 masters running. We bumped
max to 24 late yesterday and I now see 11 masters running. We'll see
how it performs during today's big schedule run.
> What is your Passenger config?
We're using what the docs suggest except MaxPoolSize is now 24:
http://github.com/reductivelabs/puppet/tree/607b01e82ea294068fdd554e59bc8e5fe3f9761a/ext/rack/files
> What are the specs of your app server?
All of our puppetmasters are running on the same size hardware. The
machines have 8 cores and 12G of RAM.
Do you expect Passenger to perform better?
Nigel Kersten
>
> Scott Smith <sc...@ohlol.net> writes:
>
>> Mark Plaksin wrote:
>>
>>> Howdy:
>>>
>>> How does Passenger perform compared to Mongrel? For us Passenger looks
>>> worse. We have two puppetmasters; one does file serving, the other does
>>> everything else. We just started running 0.25 beta2 on both. The
>>> file-server is running RubyEE + Passenger and the non-file-server is
>>> running Mongrel. When our heavy-hitter schedule runs the load average
>>> on our file server spikes much higher than it did when it was running
>>> 0.25 beta1 with Mongrel.
>>
>> Are you running the same # of mongrels as Passenger workers?
>
> Sort of :) We run 12 masters in mongrel. Passenger doesn't seem have a
> "run at least this many puppetmasters" setting. We set the max to 12 to
> match mongrel but we never saw more than 6 masters running. We bumped
> max to 24 late yesterday and I now see 11 masters running. We'll see
> how it performs during today's big schedule run.
connections.
>
>> What is your Passenger config?
>
> We're using what the docs suggest except MaxPoolSize is now 24:
> http://github.com/reductivelabs/puppet/tree/607b01e82ea294068fdd554e59bc8e5fe3f9761a/ext/rack/files
>
>> What are the specs of your app server?
>
> All of our puppetmasters are running on the same size hardware. The
> machines have 8 cores and 12G of RAM.
>
> Do you expect Passenger to perform better?
>
>
> >
>
Nigel Kersten
nig...@google.com
System Administrator
Google, Inc.
Scott Smith
> Sort of :) We run 12 masters in mongrel. Passenger doesn't seem have a
> "run at least this many puppetmasters" setting. We set the max to 12 to
> match mongrel but we never saw more than 6 masters running. We bumped
> max to 24 late yesterday and I now see 11 masters running. We'll see
> how it performs during today's big schedule run.
>
Yeah, PassengerMaxPoolSize is a bit confusing - they say it's for tuning
memory usage, but then say it is the max # of worker Rails processes
it'll spawn. I don't care if you've got 12TB of ram, if you're only
running a single core you don't want to set it very high.
> All of our puppetmasters are running on the same size hardware. The
> machines have 8 cores and 12G of RAM.
>
> Do you expect Passenger to perform better?
>
Not necessarily. But load average isn't exactly a great performance
metric. BTW, what's the load average getting up to? You've got 8 cores,
so something between 10-16 is probably fine, no?
-scott
Mark Plaksin
>> All of our puppetmasters are running on the same size hardware. The
>> machines have 8 cores and 12G of RAM.
>>
>> Do you expect Passenger to perform better?
>
> Not necessarily. But load average isn't exactly a great performance
> metric. BTW, what's the load average getting up to? You've got 8 cores,
> so something between 10-16 is probably fine, no?
Load average is peaking around 10. That's not so good compared to
running under mongrel where the load peaked at less than 5 (both with
0.25b1 and 0.25b2).
Christian Hofstaedtler
> Scott Smith <sc...@ohlol.net> writes:
> > Mark Plaksin wrote:
> > Are you running the same # of mongrels as Passenger workers?
>
> Sort of :) We run 12 masters in mongrel. Passenger doesn't seem have a
> "run at least this many puppetmasters" setting. We set the max to 12 to
> match mongrel but we never saw more than 6 masters running. We bumped
> max to 24 late yesterday and I now see 11 masters running. We'll see
> how it performs during today's big schedule run.
set PassengerPoolIdleTime much larger, say 172800. You probably would
only see an effect on the next day, though.
Christian
Mark Plaksin
Our servers don't seem to get idle enough to get killed. Once they
start they hang around until the next day (or until
PassengerMaxRequests, I guess).
Scott Smith
> Load average is peaking around 10. That's not so good compared to
> running under mongrel where the load peaked at less than 5 (both with
> 0.25b1 and 0.25b2).
>
Were the puppetmasters verifiably slower? If so, how did you measure it?
Mark Plaksin
> Mark Plaksin wrote:
>
>> Load average is peaking around 10. That's not so good compared to
>> running under mongrel where the load peaked at less than 5 (both with
>> 0.25b1 and 0.25b2).
>
> Were the puppetmasters verifiably slower? If so, how did you measure it?
I didn't measure but load average has been a good indicator for us. Oh,
and our stats system has stats based on Puppet reports. The "how long
did the file part of all manifests on all machines take to run" graph
says that our file server is slower with Passenger than it is with
Mongrel. The difference is about a factor of 2--just like the
difference in the peak load averages.
Christian Hofstaedtler
your load is twice as high. So you're actually seeing a 4-time worse
performance, which is /very/ bad.
Which processes do you see running? puppetmasterds and Apaches or only
one kind of them?
How long does file serving take for you? Are there very large files?
If so, you might want to test with PassengerUseGlobalQueue on.
Which versions are your clients? 0.25beta too?
Christian
Mark Plaksin
> On Jun 26, 9:56 pm, Mark Plaksin <ha...@usg.edu> wrote:
>> Scott Smith <sc...@ohlol.net> writes:
>> > Mark Plaksin wrote:
>>
>> >> Load average is peaking around 10. That's not so good compared to
>> >> running under mongrel where the load peaked at less than 5 (both with
>> >> 0.25b1 and 0.25b2).
>>
>> > Were the puppetmasters verifiably slower? If so, how did you measure it?
>>
>> I didn't measure but load average has been a good indicator for us. Oh,
>> and our stats system has stats based on Puppet reports. The "how long
>> did the file part of all manifests on all machines take to run" graph
>> says that our file server is slower with Passenger than it is with
>> Mongrel. The difference is about a factor of 2--just like the
>> difference in the peak load averages.
>
> What worries me is, that you are saying it takes twice as long and
> your load is twice as high. So you're actually seeing a 4-time worse
> performance, which is /very/ bad.
Heh. Good point.
> Which processes do you see running? puppetmasterds and Apaches or only
> one kind of them?
By "running" do you mean "busy"? I'm not positive but I think
puppetmasterds were at the top of top when we were running Passenger.
We're running Mongrel now so I can't double-check.
> How long does file serving take for you? Are there very large files?
The vast majority are under a meg. The top 15 are between .9M and 7M
and only get distributed to one machine.
> If so, you might want to test with PassengerUseGlobalQueue on.
>
> Which versions are your clients? 0.25beta too?
The clients are nearly all 0.24.7. Once the file ignore bug (#2358) is
fixed we'll try 0.25 on more clients.
Scott Smith
>
> What worries me is, that you are saying it takes twice as long and
> your load is twice as high. So you're actually seeing a 4-time worse
> performance, which is /very/ bad.
> Which processes do you see running? puppetmasterds and Apaches or only
> one kind of them?
>
Even then, I don't think a load average of 10 on his system is something
to worry about. A load average of 5 on his 8 core system simply means
that he is not CPU bound. The CPU(s) are underutilized. Even at 10 I
would wager that he still is not.
We need more data. What is the iowait? etc.
Mark Plaksin
The machine almost never has iowait. Spikes in user CPU match the load
average spikes (this is true with both Passenger and mongrel). The
machine is doing nothing but Puppet file serving. Its 12G of RAM is
never all used--it usually has about 8G completely free. Buffer cache
is uses around 3G.
Christian Hofstaedtler
On Jun 26, 11:19 pm, Scott Smith <sc...@ohlol.net> wrote:
> Christian Hofstaedtler wrote:
>
> > What worries me is, that you are saying it takes twice as long and
> > your load is twice as high. So you're actually seeing a 4-time worse
> > performance, which is /very/ bad.
> > Which processes do you see running? puppetmasterds and Apaches or only
> > one kind of them?
>
> Even then, I don't think a load average of 10 on his system is something
> to worry about. A load average of 5 on his 8 core system simply means
> that he is not CPU bound. The CPU(s) are underutilized. Even at 10 I
> would wager that he still is not.
clients per server when scaling up.
It'd be interesting to compare this setup with 0.24.8, as the
passenger-related integration code got completely rewritten for 0.25.
Christian
Larry Ludwig
The example Apache config file with Puppet is not correct. I have
submitted a patch so the example configuration file is correct.
http://projects.reductivelabs.com/issues/2430
In communication with USG, once the options were properly set, the
load is now much lower and similar with Mongrel but the Puppet compile
times are much lower.
-L
--
Larry Ludwig
Reductive Labs