High CPU load / queued passenger requests

[Luke Baker]

After restarting our puppetmaster host, we are seeing high CPU load and a ton of queued passenger requests. I would suspect that our puppetmaster is falling victim to the thundering herd of puppet agents, but we have deployed most of our agents using a randomized cron job. The others are using 'splay = true' if they are running puppet agent.

Some info:
Puppetmaster is RHEL6 x86_64 (VM)
4 vCPU 8GB memory
Apache 2.2
Passenger 3.0.7

The vhost for our puppetmaster includes these passenger options:
PassengerHighPerformance on
PassengerUseGlobalQueue on
PassengerPoolIdleTime 1500
PassengerMaxPoolSize 24
PassengerMaxRequests 4000
PassengerStatThrottleRate 120
PassengerSpawnMethod smart
RackAutoDetect Off
RailsAutoDetect Off


Look at all of these queued requests!

Passenger-status:
----------- General information -----------
max      = 24
count    = 24
active   = 22
inactive = 2
Waiting on global queue: 234

----------- Application groups -----------
/etc/puppet/rack:
  App root: /etc/puppet/rack
  * PID: 24840   Sessions: 1    Processed: 528     Uptime: 9h 41m 19s
  * PID: 24703   Sessions: 1    Processed: 513     Uptime: 9h 42m 17s
  * PID: 24642   Sessions: 1    Processed: 567     Uptime: 9h 42m 29s
  * PID: 24827   Sessions: 1    Processed: 572     Uptime: 9h 41m 28s
  * PID: 24869   Sessions: 1    Processed: 530     Uptime: 9h 41m 4s
  * PID: 24586   Sessions: 1    Processed: 572     Uptime: 9h 42m 32s
  * PID: 24715   Sessions: 1    Processed: 539     Uptime: 9h 42m 17s
  * PID: 29046   Sessions: 1    Processed: 487     Uptime: 8h 52m 35s
  * PID: 24609   Sessions: 1    Processed: 575     Uptime: 9h 42m 30s
  * PID: 24605   Sessions: 1    Processed: 488     Uptime: 9h 42m 30s
  * PID: 24831   Sessions: 1    Processed: 555     Uptime: 9h 41m 27s
  * PID: 29036   Sessions: 1    Processed: 549     Uptime: 8h 52m 35s
  * PID: 24583   Sessions: 1    Processed: 565     Uptime: 9h 42m 32s
  * PID: 24768   Sessions: 1    Processed: 521     Uptime: 9h 41m 56s
  * PID: 24850   Sessions: 1    Processed: 579     Uptime: 9h 41m 16s
  * PID: 29050   Sessions: 1    Processed: 497     Uptime: 8h 52m 35s
  * PID: 24567   Sessions: 1    Processed: 536     Uptime: 9h 42m 37s
  * PID: 24761   Sessions: 1    Processed: 570     Uptime: 9h 42m 5s
  * PID: 24709   Sessions: 1    Processed: 575     Uptime: 9h 42m 17s
  * PID: 29042   Sessions: 1    Processed: 488     Uptime: 8h 52m 35s
  * PID: 24865   Sessions: 1    Processed: 559     Uptime: 9h 41m 5s
  * PID: 29039   Sessions: 1    Processed: 506     Uptime: 8h 52m 35s

/usr/share/foreman:
  App root: /usr/share/foreman
  * PID: 24724   Sessions: 0    Processed: 5743    Uptime: 9h 42m 16s
  * PID: 24717   Sessions: 0    Processed: 5099    Uptime: 9h 42m 17s

[Luke Baker]

I ended up restart the mysql database on the storeconfigs server which helped this problem dramatically. But, I'm still open for more tuning options if they're out there : )

[Ramin K]

On 7/31/2012 2:17 PM, Luke Baker wrote:
> I ended up restart the mysql database on the storeconfigs server which
> helped this problem dramatically. But, I'm still open for more tuning
> options if they're out there : )
>

> Some info:
> Puppetmaster is RHEL6 x86_64 (VM)
> 4 vCPU 8GB memory
> Apache 2.2
> Passenger 3.0.7
>

PassengerHighPerformance on
PassengerMaxPoolSize 8
PassengerMinInstances 1
PassengerMaxRequests 10000
PassengerStatThrottleRate 120
PassengerPreStart https://puppet.mydomain.com:8140/

In Passenger 3.0.x GlobalQueue is on by default so you can drop that.

MinInstances is new in 3.0.x. Using it obviates the need for timeout
settings. Setting MinInstances beyond 1 unless done at the vhost level
for specific reasons is a bad idea because Passenger can balance
application instances better than you. Keep in mind that MinInstances
does not start applications when your http daemon starts for the first
time.

PassengerPreStart does start applications when the server restarts and
takes a URL as an argument which it uses in the host headers when it
connects to localhost. This should also be applied at the vhost level.

MaxRequests can be reasonably high with a mature app like Puppet.

MaxPoolSize is a little harder to size. A Puppet master is usually CPU
bound and I default to (cores * 2), but it depends on your apps. I'd aim
for 80% of CPU with all Rack processes in use and add or drop from the
poolsize to achieve it. Keep in mind that a Puppet Rack process will
weigh in around 150-200MB and you should have enough RAM to support the
max pool size.

Rails/Rack autodetect shouldn't be needed with a normal vhost for Puppet.

I would not touch the Passenger spawn method at all unless there is a
specific reason to modify it. There is not a reason to modify it with
Puppet in my opinion.
http://www.modrails.com/documentation/Users%20guide%20Apache.html#spawning_methods_explained

Ramin

[Jake - USPS]

If that seemed to help maybe you need to tune your mysql server(s)?  I've tuned mine quite a bit from default OS distro settings using  http://mysqltuner.pl/mysqltuner.pl as I myself am not a mysql expert.

I've also separated mysql to a different server then what my puppetmasters run on.

Regards,

Jake

[ad]

On Thursday, August 2, 2012 8:34:55 AM UTC-5, Jake - USPS wrote:

If that seemed to help maybe you need to tune your mysql server(s)?  I've tuned mine quite a bit from default OS distro settings using  http://mysqltuner.pl/mysqltuner.pl as I myself am not a mysql expert.

Check out Sheeri K. Cabral's fork here:

https://launchpad.net/mysqltuner 

Regards,

Adam