Configuring max_children correctly

[Chris]

I'm really hoping I can get some advice here as there seems to be a lot of misinformation on the internet regarding how to set the max_children setting.

Occasionally my site will get a small spike of traffic and reach the max_children setting (37) that I have configured.

My server spec is: 2GB RAM / 2 CPUs (DigitalOcean)

My stack is: Ubuntu 14.04, Apache 2.4, php5-fpm, MariaDB (MySQL), Memcached

I'm using the Laravel framework and am caching all of my database queries with Memcached (I've also checked that the queries themselves are optimal).

I have configured my max_children based off available memory divided by average process size, which I have read on a lot of php-fpm "setup guides".

The numbers for me are 1500MB for php and 40MB process size = 37.5.

listen = /var/run/php5-fpm.sock

pm = dynamic

pm.max_children = 37

pm.start_servers = 2

pm.min_spare_servers = 1
pm.max_spare_servers = 3

pm.max_requests = 500

There was a spike today of around 200 requests per minute over 5 minutes:

[05-Mar-2016 12:23:14] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children

[05-Mar-2016 12:29:27] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 8 total children

[05-Mar-2016 12:29:28] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 9 total children

[05-Mar-2016 12:29:29] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 10 total children

[05-Mar-2016 12:53:15] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children

[05-Mar-2016 12:53:16] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 7 total children

[05-Mar-2016 12:53:17] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 8 total children

[05-Mar-2016 12:53:18] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 9 total children

[05-Mar-2016 12:53:19] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 10 total children

[05-Mar-2016 12:53:20] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 11 total children

[05-Mar-2016 12:53:21] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 12 total children

[05-Mar-2016 12:53:22] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 13 total children

[05-Mar-2016 12:53:23] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 14 total children

[05-Mar-2016 12:53:24] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 15 total children

[05-Mar-2016 12:53:25] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 16 total children

[05-Mar-2016 12:53:26] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 17 total children

[05-Mar-2016 12:53:27] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 18 total children

[05-Mar-2016 12:53:28] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 19 total children

[05-Mar-2016 12:53:29] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 20 total children

[05-Mar-2016 12:53:30] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 21 total children

[05-Mar-2016 12:53:31] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 22 total children

[05-Mar-2016 12:53:32] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 23 total children

[05-Mar-2016 12:53:33] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 24 total children

[05-Mar-2016 12:53:34] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 25 total children

[05-Mar-2016 12:53:35] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 26 total children

[05-Mar-2016 12:53:36] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 27 total children

[05-Mar-2016 12:53:37] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 28 total children

[05-Mar-2016 12:53:38] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 29 total children

[05-Mar-2016 12:53:39] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 30 total children

[05-Mar-2016 12:53:40] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 31 total children

[05-Mar-2016 12:53:41] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 32 total children

[05-Mar-2016 12:53:42] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 33 total children

[05-Mar-2016 12:53:43] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 34 total children

[05-Mar-2016 12:53:44] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 35 total children

[05-Mar-2016 12:53:45] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 36 total children

[05-Mar-2016 12:53:46] WARNING: [pool www] server reached pm.max_children setting (37), consider raising it

I'm using New Relic to monitor my app response times and my server stats. During the time span in the log file above, New Relic reports that php5-fpm was only using 3 processes, and around 100MB memory in total. I'm not sure how that can be true.

I'm not sure which direction to go from here. I thought that a 2GB VPS would be able to handle that amount of requests, or is that very optimistic? Do I need more RAM? Do I need to increase the setting as per the logs? Also, what happens after that warning message? Do new requests have to wait because all the children are busy? I'm led to believe that setting the max_children too high would cause fpm to start swapping to disk, resulting in very slow response times which I'd like to avoid.

I've tried to give as much info as possible, but a couple of things might have slipped my mind. Any help would be greatly appreciated.

Chris

[Jérôme Loyet]

Hi Chris,

if your server is dedicated to PHP-FPM, you should use pm = static. No need for PHP-FPM to fork and kill processes for free. Then set pm.max.children to a value that your server is used at its best (it means, mainly, optimal RAM use without swapping). If, then, you still reach max_children then it just means you should either upgrade your server, load balance with another server, or optimize your application.

But RAM is not the only issue. You have also CPU consumption. If you have too many requests that use a lot of CPU, requests will take longer, so RAM will be used longer and at the end you'll be able to handle less requests at the same time.

Same problem with disk usage btw.

Setting max_children is not an easy task if you want to optimize your setup without changing hardware. Many doesn't bother optimizing, they just upgrade hardware without trying to understand what is going on' (and they still sell their product as "green")

Hope it helps

++ jerome

--

---
You received this message because you are subscribed to the Google Groups "highload-php-en" group.
To unsubscribe from this group and stop receiving emails from it, send an email to highload-php-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chris]

Hi Jerome,

I will switch from dynamic to static as the response time is affected a little when spawning new children.

I had another traffic spike early this morning. According to New Relic it was serving 500 transactions per minute over 20 minutes, the peak was 700. During this the site became slow. Around 2,000 users in Google Analytics.

But New Relic reports that php-fpm memory usage was 100MB (3 procs) and the CPU usage 30%. Very weird. This would mean that I *can* increase pm.max_children. I'm not sure if New Relic is reporting correctly!

The slow down was around 5:50 AM but apparently FPM wasn't using all children:

[06-Mar-2016 05:11:55] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 7 total children

[06-Mar-2016 05:11:56] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 8 total children

[06-Mar-2016 05:28:34] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children

[06-Mar-2016 05:28:35] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 16 children, there are 0 idle, and 7 total children

[06-Mar-2016 05:28:36] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 8 total children

[06-Mar-2016 05:28:37] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 9 total children

[06-Mar-2016 05:28:38] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 10 total children

[06-Mar-2016 05:28:39] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 11 total children

[06-Mar-2016 05:28:40] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 12 total children

[06-Mar-2016 05:28:41] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 13 total children

[06-Mar-2016 05:28:42] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 14 total children

[06-Mar-2016 05:28:43] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 32 children, there are 0 idle, and 15 total children

[06-Mar-2016 06:05:09] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children

[06-Mar-2016 06:07:03] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children

[06-Mar-2016 06:11:56] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 7 total children

[06-Mar-2016 06:30:54] WARNING: [pool www] seems busy (you may need to increase pm.start_servers, or pm.min/max_spare_servers), spawning 8 children, there are 0 idle, and 6 total children

Disk reads were very low and writes peaked about 50 KB/s.

Then set pm.max.children to a value that your server is used at its best (it means, mainly, optimal RAM use without swapping)

I have just set pm to static and restarted fpm. It's memory usage is now ~900MB with 37 processes. Maybe I should be more generous and allow 50-60MB per child. Reducing pm.max_children a bit.

If I get "server reached pm.max_children" warning now, this would mean I need to upgrade the server to more RAM right?

[Chris]

After changing pm to static, the app response times have gone up :/

I've upgraded to 4GB RAM and set max_children to 70 (70 * 50MB = 3500) but the response times were sometimes 1,000ms

I've now set max_children to 10 based off this answer - http://serverfault.com/a/513398/316086 and response times are ~300ms for just 85 reqs per minute for example. I'm getting tired of this. :(