PHP-FPM and Persistent Connections

[tres]

I've recently been doing load testing on an app that is using PHP and
MongoDB. Here is some background:

- 20000 connection limit (ulimit -n 1000000)
- Nginx + PHP-FPM + 5.3.6 + APC
- Persistent Connections
- No replication at the moment
- 8 Cores
- 16gb of ram
- linux
- MongoDB 1.8.1
- PHP driver off master branch at Github
- Server load during both tests stayed around 0.3.

The first load test we ran handled 2400 concurrent users with flying
colors. No errors.

The second load test was done after modifying some settings in the
Nginx and PHP-FPM config. MongoCursorExceptions started being thrown
at about 2270 concurrent users. Server was not stressed at all.

Nginx before:
worker_processes 4;
events {
worker_connections 1024;
}

PHP-FPM before:
pm = static
pm.max_children = 4096
pm.max_requests = 1000

Nginx after:
worker_processes 6;
events {
worker_connections 1024;
}

PHP-FPM after:
pm = static
pm.max_children = 6144
pm.max_requests = 1000

During the first test, the number of MongoDB connections were around
16000 at 2400 concurrency. The second maxed and began to fail after
hitting 20000 at 2270. I understand that the concurrency may call for
more connections to be made, however this raises 2 questions:

1. The concurrency shouldn't be maxing out my connection limit since
after 6144 the server would start to back up processes and eventually
the load would become to large for it to handle. At that point, the
number of connections that the PHP driver should be pooling would be
6144, no?

2. Given the "static" setting of the PHP-FPM process, this spawns all
of the specified children at once. Since the master process will keep
each of the 6144 of the children around for 1000 requests before
respawning them, is there a chance that this would cause the number of
MongoDB connections to rise really high and never come down?

Keep in mind that the connections never get released. And since I can
only ever handle a maximum of 6144 requests at once, shouldn't the
number of connections to Mongo never get above that? Even so,
shouldn't these connections be reused and cleaned up after a given
amount of time of inactivity?

Thanks.

--
Trey

[Nat]

Tres,

- What is the error message you see?

- Can you check how many active/inactive sockets you have both on mongod server and php/webserver?

- What db.serverStatus() look like when it starts failing?

[tres]

1. In the mongo logs, just that it cannot connect because it has
exhausted its connection limit.
2. Not anymore. Netstat showed 20000 connections when I grepped it on
the database server, though.
3. I could not check because there were no connections for me to use
to check. But I'm sure it would say 20000 used, 0 available.

Some more background:

- 2 Webservers.
- Both contain the PHP app and are routed to via a load balancer.
- One of those servers is also running the mongod process.

[tres]

Kristina, you handle the PHP driver. Can you help shed any light on
this? I believe our support ticket Jira was updated to include a link
to this post.

Another bit of info is that the 2270 failure with the "after" settings
is benchmarked as being faster than the 2400+ benchmark with the
"before" settings it's just that Mongo's connection limit is exceeded
which should not be the case as each request is completed in around
300 to 500 milliseconds max.

[k]

Maybe I'm misreading your second test, but aren't you creating 6
worker processes with 6144 requests, i.e., 36000 connections at once?
MongoDB can't handle more than 20000 connections.