PHP FPM socket failure

[inte...@googlemail.com]

Hi,
we get some php5-fpm sock failed messages and I'm looking for a way to get a clue what exactly is going on.

Our setup:

- Mongo Cluster with three shards a three mongodb nodes. Version 3.0.6
- Four nginx servers with php-fpm connected via socket. PHP Version 5.5.9 on
- In front of our nginx there is a haproxy as loadbalancer
- zabbix clients on each server
- between 100 and 300 session per second, measured at the haproxy
- according mongo cloud manager there are between 30 and 50 inserts per second

Every day at 10 there is a peak and nearly every day at the same time there are php fpm socket timeouts happen. (11: Resource temporarily unavailable).
I have added an TTL index yesterday evening on a collection with 8 mio documents and the same happens. A lot of socket timeouts happen for about 30 seconds.
I have checked CPU iowait time, processor load, page faults at the cloud manager .... everything looks quite normal. No peak somewhere which could point me into a direction.

Most of the requests are handled via a simple php script which just saves one document, no query, no update. The write concern for this save is set to unacknowledged. So this script may actually never fail if e.g. the mongodb is busy. But on the other hand the socket timout happens if there is somehting going on at the mongo database (migration chunks / creating index)

Maybe someone has a hint or an idea where to search or which performance indicator to look for.

Regards Marcus