PHP FPM processes opening a ton of files and leaving them open

[Joe Auty]

I'm finding that my PHP FPM processes are opening a ton of files, so many that I'm bumping up against OS limits causing my FPM processes to die. I'm assuming that this is caused by memory leaky PHP apps that are crashing and these connections not severing? What sort of config options would be best for me to focus on in turning over new connections and closing files so that I'm dealing with this more gracefully and don't have to use the emergency threshold limit or rely on Monit to automatically reload PHP FPM?

[Antony Dovgal]

On 12/15/2011 07:13 AM, Joe Auty wrote:
> I'm finding that my PHP FPM processes are opening a ton of files, so many that I'm bumping up against OS limits causing my FPM processes to die.
>I'm assuming that this is caused by memory leaky PHP apps that are crashing and these connections not severing?

If process dies, all the descriptors are closed by the system.
And 'memory leaky PHP apps' have nothing do with that, too.

> What sort of config options would be best for me to focus on in turning over new connections and
>closing files so that I'm dealing with this more gracefully and don't have to use the emergency
>threshold limit or rely on Monit to automatically reload PHP FPM?

You didn't provide any information except that you think you have some problems with file descriptors.
How about starting from the very beginning?
OS? PHP version? Are you using some prepackaged version?
What *exactly* is your problem and why do you think that file descriptors are leaking?

--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime profiling for PHP

[Joe Auty]

Antony Dovgal

December 15, 2011 12:12 AM

Okay...

Debian Squeeze, latest dotdeb packages (PHP 5.3.8), FastCGI 2.4.7~0910052141.

The primary problem I've been experiencing is the number of open files ballooning to crazy numbers and reaching OS limits (after increasing the amount from its default) - around four times more so than normal, causing "too many files open" errors being logged. At one point this may have caused PHP-FPM to quit entirely, but that hasn't happened since. Doing an lsof I see a lot of  "can't identify protocol" in the "NAME" column belonging to php5-fpm.

I don't know if the FDs are leaking, but this crossed my mind given how the number of open files seems to randomly balloon for no explicable reason.

Joe Auty

December 14, 2011 10:13 PM

I'm finding that my PHP FPM processes are opening a ton of files, so many that I'm bumping up against OS limits causing my FPM processes to die. I'm assuming that this is caused by memory leaky PHP apps that are crashing and these connections not severing? What sort of config options would be best for me to focus on in turning over new connections and closing files so that I'm dealing with this more gracefully and don't have to use the emergency threshold limit or rely on Monit to automatically reload PHP FPM?

---

--
Joe Auty, NetMusician
NetMusician helps musicians, bands and artists create beautiful, professional, custom designed, career-essential websites that are easy to maintain and to integrate with popular social networks.
www.netmusician.org
j...@netmusician.org

[Antony Dovgal]

On 12/15/2011 10:20 AM, Joe Auty wrote:
> Debian Squeeze, latest dotdeb packages (PHP 5.3.8), FastCGI 2.4.7~0910052141.

Did you try to report this problem to Debian package maintainers?
I would also recommend to try with clean sources from php.net, since Debian adds a lot of their stuff to the sources.

> The primary problem I've been experiencing is the number of open files ballooning to crazy
>numbers and reaching OS limits (after increasing the amount from its default) -
>around four times more so than normal, causing "too many files open" errors being logged.
>At one point this may have caused PHP-FPM to quit entirely, but that hasn't happened since.
>Doing an lsof I see a lot of "can't identify protocol" in the "NAME" column belonging to php5-fpm.

> I don't know if the FDs are leaking, but this crossed my mind given how the number of open files seems to randomly balloon for no explicable reason.

Do you see anything suspicious in the FPM logs?

[Joe Auty]

Antony Dovgal

December 15, 2011 2:36 AM

On 12/15/2011 10:20 AM, Joe Auty wrote:

Debian Squeeze, latest dotdeb packages (PHP 5.3.8), FastCGI 2.4.7~0910052141.

Did you try to report this problem to Debian package maintainers?
I would also recommend to try with clean sources from php.net, since Debian adds a lot of their stuff to the sources.

I could do this, but I figured that it would be best to learn more about what is going on so that I know where to focus my attention, be it the packages or elsewhere, hence this thread :)

 

The primary problem I've been experiencing is the number of open files ballooning to crazy
numbers and reaching OS limits (after increasing the amount from its default) -
around four times more so than normal, causing "too many files open" errors being logged.
At one point this may have caused PHP-FPM to quit entirely, but that hasn't happened since.
Doing an lsof I see a lot of "can't identify protocol" in the "NAME" column belonging to php5-fpm.

I don't know if the FDs are leaking, but this crossed my mind given how the number of open files seems to randomly balloon for no explicable reason.

Do you see anything suspicious in the FPM logs?

Yeah, lots of stuff like the following:

[pool www] child 27620 exited with code 0 after 9987.151221 seconds from start

Joe Auty

December 15, 2011 1:20 AM

Antony Dovgal

December 15, 2011 12:12 AM

On 12/15/2011 07:13 AM, Joe Auty wrote:

I'm finding that my PHP FPM processes are opening a ton of files, so many that I'm bumping up against OS limits causing my FPM processes to die.
I'm assuming that this is caused by memory leaky PHP apps that are crashing and these connections not severing?

If process dies, all the descriptors are closed by the system.
And 'memory leaky PHP apps' have nothing do with that, too.

What sort of config options would be best for me to focus on in turning over new connections and
closing files so that I'm dealing with this more gracefully and don't have to use the emergency
threshold limit or rely on Monit to automatically reload PHP FPM?

You didn't provide any information except that you think you have some problems with file descriptors.
How about starting from the very beginning?
OS? PHP version? Are you using some prepackaged version?
What *exactly* is your problem and why do you think that file descriptors are leaking?

Okay...

Debian Squeeze, latest dotdeb packages (PHP 5.3.8), FastCGI 2.4.7~0910052141.

The primary problem I've been experiencing is the number of open files ballooning to crazy numbers and reaching OS limits (after increasing the amount from its default) - around four times more so than normal, causing "too many files open" errors being logged. At one point this may have caused PHP-FPM to quit entirely, but that hasn't happened since. Doing an lsof I see a lot of  "can't identify protocol" in the "NAME" column belonging to php5-fpm.

I don't know if the FDs are leaking, but this crossed my mind given how the number of open files seems to randomly balloon for no explicable reason.

Joe Auty

December 14, 2011 10:13 PM

I'm finding that my PHP FPM processes are opening a ton of files, so many that I'm bumping up against OS limits causing my FPM processes to die. I'm assuming that this is caused by memory leaky PHP apps that are crashing and these connections not severing? What sort of config options would be best for me to focus on in turning over new connections and closing files so that I'm dealing with this more gracefully and don't have to use the emergency threshold limit or rely on Monit to automatically reload PHP FPM?

---

Antony Dovgal

December 15, 2011 12:12 AM

On 12/15/2011 07:13 AM, Joe Auty wrote:

I'm finding that my PHP FPM processes are opening a ton of files, so many that I'm bumping up against OS limits causing my FPM processes to die.
I'm assuming that this is caused by memory leaky PHP apps that are crashing and these connections not severing?

If process dies, all the descriptors are closed by the system.
And 'memory leaky PHP apps' have nothing do with that, too.

What sort of config options would be best for me to focus on in turning over new connections and
closing files so that I'm dealing with this more gracefully and don't have to use the emergency
threshold limit or rely on Monit to automatically reload PHP FPM?

You didn't provide any information except that you think you have some problems with file descriptors.
How about starting from the very beginning?
OS? PHP version? Are you using some prepackaged version?
What *exactly* is your problem and why do you think that file descriptors are leaking?

Joe Auty

December 14, 2011 10:13 PM

I'm finding that my PHP FPM processes are opening a ton of files, so many that I'm bumping up against OS limits causing my FPM processes to die. I'm assuming that this is caused by memory leaky PHP apps that are crashing and these connections not severing? What sort of config options would be best for me to focus on in turning over new connections and closing files so that I'm dealing with this more gracefully and don't have to use the emergency threshold limit or rely on Monit to automatically reload PHP FPM?

---

[Antony Dovgal]

On 12/15/2011 12:20 PM, Joe Auty wrote:
>> Do you see anything suspicious in the FPM logs?
>
> Yeah, lots of stuff like the following:
>
> [pool www] child 27620 exited with code 0 after 9987.151221 seconds from start

Well, that's completely normal - all childs have to respect max_requests limit and exit after they've reached it.

[Joe Auty]

Okay,

I'm still having this problem. I'm not seeing anything suspicious in my PHP error log, and I'm starting to think that the problem has more to do with my underlying ZFS RAID and/or my NFS mounts. I've tried switching back to using the Apache PHP module and while I may not have given it enough testing time, it didn't seem to help keep the number of open files down.

That being said, since restarting PHP FPM does restore the number of open files to sane levels, aside from a disruptive cronjob that runs every hour or whatever that restarts FPM (a /etc/init.d/php5-fpm reload command does not help), can you think of any other hackjob sort of general Linux techniques for keeping this number down? I realize that this is outside the scope of this forum, but I'd appreciate a little creativity and flexibility here :)

[Maciej Lisiewski]

On 31 grudzień 2011 22:06:05, Joe Auty wrote:
> Okay,
>
> I'm still having this problem. I'm not seeing anything suspicious in
> my PHP error log, and I'm starting to think that the problem has more
> to do with my underlying ZFS RAID and/or my NFS mounts. I've tried
> switching back to using the Apache PHP module and while I may not have
> given it enough testing time, it didn't seem to help keep the number
> of open files down.
>

Just going on a hunch here:
Do you have max_requests set to some high value? 100k+ for example?
If so try setting it to a lower value - try 10k or 20k and see what
happens.

--
Maciej Lisiewski

[Joe Auty]

I have this setting set to 0 - endless spawning. I've just this to the suggested non-default option of 500 to see whether this helps... I'll report back :)

Thanks!

Maciej Lisiewski

January 1, 2012 3:11 AM

Just going on a hunch here: Do you have max_requests set to some high value? 100k+ for example? If so try setting it to a lower value - try 10k or 20k and see what happens.

Joe Auty

December 31, 2011 4:06 PM

Okay,

I'm still having this problem. I'm not seeing anything suspicious in my PHP error log, and I'm starting to think that the problem has more to do with my underlying ZFS RAID and/or my NFS mounts. I've tried switching back to using the Apache PHP module and while I may not have given it enough testing time, it didn't seem to help keep the number of open files down.

That being said, since restarting PHP FPM does restore the number of open files to sane levels, aside from a disruptive cronjob that runs every hour or whatever that restarts FPM (a /etc/init.d/php5-fpm reload command does not help), can you think of any other hackjob sort of general Linux techniques for keeping this number down? I realize that this is outside the scope of this forum, but I'd appreciate a little creativity and flexibility here :)

Joe Auty

December 14, 2011 10:13 PM

I'm finding that my PHP FPM processes are opening a ton of files, so many that I'm bumping up against OS limits causing my FPM processes to die. I'm assuming that this is caused by memory leaky PHP apps that are crashing and these connections not severing? What sort of config options would be best for me to focus on in turning over new connections and closing files so that I'm dealing with this more gracefully and don't have to use the emergency threshold limit or rely on Monit to automatically reload PHP FPM?

[Joe Auty]

Awesome, so far this setting change seems to have solved my problems!

I'm probably just a moron for not thinking of adjusting this setting, but wouldn't it be smart to set this to some non-zero value by default with the project config file? Everything else seems to be set conservatively except for this setting. With PHP memory leaks being so common and the config file making multiple references to this sort, wouldn't it be best to play it safe here? Is there a huge performance upside to less respawning?

Joe Auty

January 1, 2012 3:49 AM