php-fpm randomly becomes unresponsive, segfaults

[c2h5oh]

I'm running PHP 5.2.12 with PHP-FPM patch, nginx accesses it via
socket. Couple of times a day php-fpm become unresponsive, resulting
in 502 and 504. There is no load spike (CPU usage prior to the
incident is ~50%, and once it starts drops close to 0), there is
plenty of ram available, io isn't the reason either (actually watched
it happen in iotop - small .5mb/s io load dropped close to 0). Oh, and
it's Debian Lenny.

How is it unresponsive? First there is a noticeable slowdown (my
guess: less php-fpm processes available - longer queue), then full
halt with 502/504. Sometimes (rarely) starts working in 30-50 minutes,
but usually I have to restart php-fpm.

There is nothing suspicious in php-fpm log, even if I set logging
level to debug, so no, it's not rlimit etc.

In nginx error:
connect() to unix:/var/run/php-fpm/php.sock failed (11: Resource
temporarily unavailable) while connecting to upstream

in dmesq:
[81503.175657] php-cgi[28972]: segfault at bf7ff4a0 ip 0831c300 sp
bf7ff4a0 error 6 in php-cgi[8048000+5ef000]
[81508.170733] php-cgi[28965]: segfault at bf7ff4a0 ip 0831c300 sp
bf7ff4a0 error 6 in php-cgi[8048000+5ef000]
[180609.535546] php-cgi[26208]: segfault at bf7fee30 ip 0831c300 sp
bf7fee30 error 6 in php-cgi[8048000+5ef000]
[180662.879631] php-cgi[26206]: segfault at bf7fee30 ip 0831c300 sp
bf7fee30 error 6 in php-cgi[8048000+5ef000]

but I have only checked dmesg after I have restarted php-fpm, so I am
not sure if segfault message was there prior to me restarting or it
appeared at restart.


I'd really appreciate any help.


--
Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

[Daniel Carrillo]

2010/4/16 c2h5oh <maciej.l...@gmail.com>

Hi.

I have seen a similar behaviour on the past. I was able to fix it, tricking with: 

<value name="backlog">4096</value>

It seems that php-fpm not takes backlog value from sysctl when -1 is set.

Hope this helps.

Kind Regards.

[c2h5oh]

Thans, trying it right now.

[c2h5oh]

Nope, setting backlog made no difference.

[Frieder Kundel]

+1
i tried 5.3.2 with fpm from php svn - and i get random segfaults
within seconds.. 5.2.10 with fpm is running fine but i haven't checked
if it is one of my extensions (apc,memcache,gearman) so i didn't mail
yesterday.

regards,
frieder

2010/4/16 c2h5oh <maciej.l...@gmail.com>:

[Jérôme Loyet]

2010/4/16 Frieder Kundel <frieder...@gmail.com>:

> +1
> i tried 5.3.2 with fpm from php svn - and i get random segfaults
> within seconds.. 5.2.10 with fpm is running fine but i haven't checked
> if it is one of my extensions (apc,memcache,gearman) so i didn't mail
> yesterday.
>

can you post more about your segfault problems please ?

I never had such problems. Having a log file, a conf and a gdb stack
trace would be helpful.

thx
++ Jerome

[c2h5oh]

It's most likely 2 separate problems:
1. Mine: stability problems, possible segfault after long use and with
5.2.x
2. Frieder Kundel's one: immediate segfault on start and with 5.3.x

It seems they need separate threads..

Going back to 1st one:

setting

<value name="backlog">4096</value>

vs default
<value name="backlog">-1</value>
made no difference

No matter what logging level I use in php-fpm/nginx I just can't find
any extra info. Segfault listed in dmesg seems to have nothing to do
with the problem - it didn't appear last 2 times it happened.
I have tried stressing the system, but it seems even saturating
100mbit pipe with php-generated html using ab -n50000 -c64 (with all
the modules I normally use in production: xcache, memcached) haven't
triggered the problem.

I would appreciate if someone pointed me in the right direction in
terms of how to debug this problem and pinpoint the reason.

[Jérôme Loyet]

2010/4/17 c2h5oh <maciej.l...@gmail.com>:

Can you use the last SVN version with php 5.3.2 ?

for debuggin, you can set log_level to debug, daemonize to no and
rlimit_core to "unlimited". FPM will run in foreground and error logs
will print on stderr. when it crashed you should have a coredump. gdb
it with

gdb /path/to/fpm_binary /path/to/core_file
and show the stack trace with "bt"

++ Jerome

[c2h5oh]

> Can you use the last SVN version with php 5.3.2 ?

No, the php code I'm running does not work with 5.3.x and I don't even
have ETA on update.

> for debuggin, you can set log_level to debug, daemonize to no and
> rlimit_core to "unlimited". FPM will run in foreground and error logs
> will print on stderr. when it crashed you should have a coredump. gdb
> it with
>
> gdb /path/to/fpm_binary /path/to/core_file
> and show the stack trace with "bt"

logging is already set to debug - nothing out of ordinary in log. I'm
setting rlimit_core and disabling daemonization now - I will get back
to you.

[Jérôme Loyet]

2010/4/17 c2h5oh <maciej.l...@gmail.com>:

>
>> Can you use the last SVN version with php 5.3.2 ?
>
> No, the php code I'm running does not work with 5.3.x and I don't even
> have ETA on update.

OK I understand. But I'll do my best to help you but I'm not very
familiar with FPM version prior to PHP svn inclusion.

>
>
>> for debuggin, you can set log_level to debug, daemonize to no and
>> rlimit_core to "unlimited". FPM will run in foreground and error logs
>> will print on stderr. when it crashed you should have a coredump. gdb
>> it with
>>
>> gdb /path/to/fpm_binary /path/to/core_file
>> and show the stack trace with "bt"
>
> logging is already set to debug - nothing out of ordinary in log. I'm
> setting rlimit_core and disabling daemonization now - I will get back
> to you.

I'm waiting to hear from you

[c2h5oh]

> I'm waiting to hear from you

All I got is:

Apr 17 17:21:09.731348 [WARNING] fpm_request_check_timed_out(), line
146: child 25467, script '[edited]' (pool default) executing too slow
(30.078848 sec), logging
Apr 17 17:21:09.731409 [NOTICE] fpm_got_signal(), line 48: received
SIGCHLD
Apr 17 17:21:09.731427 [NOTICE] fpm_children_bury(), line 194: child
25467 stopped for tracing
Apr 17 17:21:09.731441 [NOTICE] fpm_php_trace(), line 139: about to
trace 25467
Apr 17 17:21:09.771622 [NOTICE] fpm_php_trace(), line 167: finished
trace of 25467

Just restarted without execution time limit and waiting for new output.

[Jérôme Loyet]

2010/4/17 c2h5oh <maciej.l...@gmail.com>:

>
>> I'm waiting to hear from you
>
> All I got is:
>
> Apr 17 17:21:09.731348 [WARNING] fpm_request_check_timed_out(), line
> 146: child 25467, script '[edited]' (pool default) executing too slow
> (30.078848 sec), logging
> Apr 17 17:21:09.731409 [NOTICE] fpm_got_signal(), line 48: received
> SIGCHLD
> Apr 17 17:21:09.731427 [NOTICE] fpm_children_bury(), line 194: child
> 25467 stopped for tracing
> Apr 17 17:21:09.731441 [NOTICE] fpm_php_trace(), line 139: about to
> trace 25467
> Apr 17 17:21:09.771622 [NOTICE] fpm_php_trace(), line 167: finished
> trace of 25467
>
> Just restarted without execution time limit and waiting for new output.
>

yes set request_slowlog_timeout to 0 and request_terminate_timeout to 0.

++ jerome

[c2h5oh]

Without execution time limit nothing is logged/output to stderr - just
the basic notices, child start, child termination on requests limit
etc.

[Jérôme Loyet]

2010/4/17 c2h5oh <maciej.l...@gmail.com>:

> Without execution time limit nothing is logged/output to stderr - just
> the basic notices, child start, child termination on requests limit
> etc.

and did the problem appear ?

[Usu]

Finally somebody that has my very same problem (except for the
segfaults, they stopped a few weeks ago, probably due to some upgrade,
I don't really remember), but I don't think this is strictly php-fpm
related, here's why:

- Tried many different versions of php (5.2 and 5.3 branch)
- Tried using 2 different spawn-cgi script instead of php-fpm
- Tried using another webserver instead of nginx (lighttpd)
- Tried with lighttpd 1.5 + php-fpm and with lighttpd 1.4.x + the
built-in spawning system

All the above resulted in having the same error you described (502
after minutes/hours/days and nothing in the logs that could explain
it).
Ok, here's the interesting part: last wednesday I set a very high
number of max_children in php-fpm.conf (I'm using the apache-like
configuration so the number of child increases or decreases
dinamically) and since then I haven't had any 502! I should wait
another week before saying the error it's gone, but I'm pretty
confident it is.

My PHP version atm: 5.3.99-dev
My OS: Debian 5.0.4
My nginx version: 0.8.35

[Michael Shadle]

Unless Jerome disagrees since he put it in place I would still call
the apache-like or dynamic process manager experimental. Use static
and adjust children appropriately. Note that dynamic is only in svn,
apache-like is from launchpad and neither are in the patch (5.2)

For 5.3 use svn and try that. If that doesn't work, downgrade to 5.2
and use the patch.

[Antony Dovgal]

On 18.04.2010 00:00, Michael Shadle wrote:
> Unless Jerome disagrees since he put it in place I would still call
> the apache-like or dynamic process manager experimental. Use static
> and adjust children appropriately. Note that dynamic is only in svn,
> apache-like is from launchpad and neither are in the patch (5.2)
>
> For 5.3 use svn and try that. If that doesn't work, downgrade to 5.2
> and use the patch.

If that doesn't work, provide a GDB backtrace at last!
If you don't know how to do it, read this: http://bugs.php.net/bugs-generating-backtrace.php

Please, always start with the backtrace whenever you have any segfaults, it saves a lot of time and traffic.

--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime statistics for PHP

[Andrea Usuelli]

Yeah, I meant dynamic from svn, I've been using it for a few weeks
without problems (except for the 502 thing but it's unreleated since
it happens also using static and old php-fpm versions).

Using static with an appropriate number of children leads to the 502
error, maybe the dynamic thing is just a workaround for this error
because the processes are stopped and started a lot but this is just a
guess... in fact this error seems to be disappeared after setting a
very high number of max_children in php-fpm.conf (using dynamic
process) in order to never reach the limit

On 17 Apr, 22:00, Michael Shadle <mike...@gmail.com> wrote:
> Unless Jerome disagrees since he put it in place I would still call  
> the apache-like or dynamic process manager experimental. Use static  
> and adjust children appropriately. Note that dynamic is only in svn,  
> apache-like is from launchpad and neither are in the patch (5.2)
>
> For 5.3 use svn and try that. If that doesn't work, downgrade to 5.2  
> and use the patch.
>

[Michael Shadle]

I'd like to add a section on the site about how to report a bug. That
way it is a one-stop shop on explaining the best things to try.
Jerome/Antony/anyone please send me your ideas on the steps, I'll get
a basic version up and we can tweak (feel free to take it off-list or
on-list, whichever you feel is appropriate)

I suppose it also depends on what the error is too... sometimes
increasing or decreasing the children works, things like that.

Something like this...

"
Common Stuff:
- Be sure to include the following information: OS, distribution, PHP
configure string, relevant snippets from the PHP-FPM log, and any
relevant snippets from nginx or the webserver logs.
- If you are still having issues with segfaults, provide a GDB
backtrace. If you don't know how to do it, read this:
http://bugs.php.net/bugs-generating-backtrace.php


For SVN (5.3+):
- ??
- If you are still having issues with segfaults, provide a GDB
backtrace. If you don't know how to do it, read this:
http://bugs.php.net/bugs-generating-backtrace.php
- Note: at some point soon the bugs.php.net bug tracker should have a
category for this

For the patch:
- ??
- Make sure you are using the 'static' process manager type.
'apache-like' does not actually work.
- Make sure you have enough max_children defined, but not too many.

For the Launchpad version: no longer supported, either change to using
5.3 from SVN, or downgrade to 5.2 and use the patch."

On Sat, Apr 17, 2010 at 1:19 PM, Antony Dovgal <to...@daylessday.org> wrote:

> If that doesn't work, provide a GDB backtrace at last!
> If you don't know how to do it, read this: http://bugs.php.net/bugs-generating-backtrace.php
>
> Please, always start with the backtrace whenever you have any segfaults, it saves a lot of time and traffic.


--

Subscription settings: http://groups.google.com/group/highload-php-en/subscribe?hl=en

[Antony Dovgal]

On 18.04.2010 00:30, Michael Shadle wrote:
> I'd like to add a section on the site about how to report a bug.

We already have that, no need for a duplicate:
http://bugs.php.net/how-to-report.php

A category for FPM bugs has been added recently, too.

--
Wbr,
Antony Dovgal
---
http://pinba.org - realtime statistics for PHP

[Michael Shadle]

Alright. I'm going to make a page, but for SVN/5.3+ I'm going to link
to that page, and put down something about the patch is not supported
by the PHP team, don't report a bug there. (A little bit of a triage)

[Michael Shadle]

http://php-fpm.org/bugs/

Let me know if you think anything is insufficient.

On Sat, Apr 17, 2010 at 1:44 PM, Michael Shadle <mik...@gmail.com> wrote:
> Alright. I'm going to make a page, but for SVN/5.3+ I'm going to link
> to that page, and put down something about the patch is not supported
> by the PHP team, don't report a bug there. (A little bit of a triage)

[c2h5oh]

> > Without execution time limit nothing is logged/output to stderr - just
> > the basic notices, child start, child termination on requests limit
> > etc.
>
> and did the problem appear ?

Yes, it did.

[Jérôme Loyet]

2010/4/17 Michael Shadle <mik...@gmail.com>:

> http://php-fpm.org/bugs/
>
> Let me know if you think anything is insufficient.
>

Only the SVN Version is fully supported !

[Jérôme Loyet]

2010/4/17 Michael Shadle <mik...@gmail.com>:

> Unless Jerome disagrees since he put it in place I would still call the
> apache-like or dynamic process manager experimental. Use static and adjust
> children appropriately.

yes it is experimental ! which doesn't mean it doesn't work or is buggy :)

[Jérôme Loyet]

2010/4/17 c2h5oh <maciej.l...@gmail.com>:

>
>> > Without execution time limit nothing is logged/output to stderr - just
>> > the basic notices, child start, child termination on requests limit
>> > etc.
>>
>> and did the problem appear ?
>
> Yes, it did.

I'll say the same as tony:

If that doesn't work, provide a GDB backtrace at last!
If you don't know how to do it, read this:
http://bugs.php.net/bugs-generating-backtrace.php

Please, always start with the backtrace whenever you have any
segfaults, it saves a lot of time and traffic.

[Michael Shadle]

Yeah but it also means it's brand new and not necessarily 100% :)

I wish I had 5.3 code I supported so I could try it out myself!

[Michael Shadle]

we seem to do a "best we can do" help on the patch still typically.
And we should - 5.2 is very prevalent still

[c2h5oh]

> If that doesn't work, provide a GDB backtrace at last!
> If you don't know how to do it, read this:http://bugs.php.net/bugs-generating-backtrace.php
>
> Please, always start with the backtrace whenever you have any
> segfaults, it saves a lot of time and traffic.

As I wrote earlier I wasn't sure if segfaults were related to the
issue and as the time goes I am more and more sure they aren't - last
5 incidents happened without a segfault being logged. There is nothing
to backtrace.
So to follow the bug reporting guide:

1. What happens:
From time to time (2-4 times a day) PHP-FPM processes pool becomes
less and less responsive to finally grind to full halt with 502/504
errors thrown by nginx. This is not related to cpu/io/memory spike in
any way - each and every time I have seen it happen there was plenty
of free RAM, almost no IO load, and CPU load under 50% - and as the
responsiveness gets worse server load gets lower.
With PHP-FPM logging level set to debug NOTHING out of ordinary is
logged in PHP-FPM log. If PHP-FPM is run not daemonized I get
[WARNING] fpm_request_check_timed_out() if request_slowlog_timeout /
request_terminate_timeout is set or absolutely nothing out of ordinary
if left at 0.
Nginx logs connect() to unix:/var/run/php-fpm/php.sock failed (11:

Resource temporarily unavailable) while connecting to upstream

Sometimes, rarely, PHP-FPM pool regains responsiveness over time
(30-60min+), but mostly it has to be restarted to work again.

2. Latest version:
- I can't use 5.3.x, because some of the PHP code running on my server
is not 5.3.x ready yet - so I'm stuck with 5.2.x for now
- I have first noticed the problem with PHP 5.2.12 + PHP-FPM patch
- I have updated to PHP 5.2.13 + PHP-FPM patch ( php-5.2.13-
fpm-0.5.13 ) - made no difference.

3. System info:
- Debian 5.0 Lenny
- PHP 5.2.13, compiled with '--enable-bcmath' '--with-gd' '--with-
jpeg-dir=/usr/lib' '--with-zlib' '--with-mysql' '--enable-gd-native-
ttf' '--with-png-dir' '--with-ttf' '--with-pear' '--with-openssl' '--
with-curl' '--with-imap-ssl' '--enable-fastcgi' '--with-mcrypt' '--
with-mhash' '--enable-zip' '--enable-ftp' '--enable-magic-quotes' '--
enable-mbstring' '--enable-exif' '--with-bz2' '--enable-force-cgi-
redirect' '--with-config-file-path=/etc/php5' '--with-freetype-dir' '--
with-gettext' '--enable-pdo' '--with-pdo-mysql' '--with-pdo-pgsql' '--
with-pgsql' '--enable-discard-path' '--enable-fpm' '--with-imap' '--
with-kerberos' '--enable-calendar' '--enable-cli'

[Jérôme Loyet]

2010/4/19 c2h5oh <maciej.l...@gmail.com>:

it seems that your php scripts are running out of time and are
consuming the max_children processes. When you have bad response time,
you can count the number of fpm processes and see if it's the same as
max_children + 1 (1= the master process).

++ Jerome

[dangerousdave]

Maybe start it with one child and start or attach strace to it. It may
give you some ideas as to slow downs when they occur. Bed return
values from system calls etc. This approach always yields some useful
information for me in these situations.

Dave.

On Apr 19, 9:42 am, Jérôme Loyet <m...@fatbsd.com> wrote:
> 2010/4/19 c2h5oh <maciej.lisiew...@gmail.com>:

[c2h5oh]

> it seems that your php scripts are running out of time and are
> consuming the max_children processes. When you have bad response time,
> you can count the number of fpm processes and see if it's the same as
> max_children + 1 (1= the master process).

yes, it is.

[c2h5oh]

On 19 Kwi, 02:07, dangerousdave <dkier...@gmail.com> wrote:
> Maybe start it with one child and start or attach strace to it. It may
> give you some ideas as to slow downs when they occur. Bed return
> values from system calls etc.  This approach always yields some useful
> information for me in these situations.

Waiting for next incident to do just that - will get back to you.

[Jérôme Loyet]

2010/4/19 c2h5oh <maciej.l...@gmail.com>:

>
>
> On 19 Kwi, 02:07, dangerousdave <dkier...@gmail.com> wrote:
>> Maybe start it with one child and start or attach strace to it. It may
>> give you some ideas as to slow downs when they occur. Bed return
>> values from system calls etc.  This approach always yields some useful
>> information for me in these situations.
>
> Waiting for next incident to do just that - will get back to you.

You can set
- request_slowlog_timeout to 5s
- slowlog to /tmp/slowlog.log

and if a request takes more than 5s, a ptrace of the process will be
written into /tmp/slowlog:

Apr 17 18:20:31.713958 [pool test] pid 25163
script_filename = /home/www/test/docs/wait10.php
[0x8a6c4020] sleep() /home/www/test/docs/wait10.php:4

it could help you debugging.

btw, for me it's not a FPM problem :)

[c2h5oh]

On 19 Kwi, 02:07, dangerousdave <dkier...@gmail.com> wrote:

> Maybe start it with one child and start or attach strace to it. It may
> give you some ideas as to slow downs when they occur. Bed return
> values from system calls etc.  This approach always yields some useful
> information for me in these situations.

I've got

restart_syscall(<... resuming interrupted call ...>

hanging for each and every child process

[Jérôme Loyet]

2010/4/19 c2h5oh <maciej.l...@gmail.com>:

>
>
> On 19 Kwi, 02:07, dangerousdave <dkier...@gmail.com> wrote:
>> Maybe start it with one child and start or attach strace to it. It may
>> give you some ideas as to slow downs when they occur. Bed return
>> values from system calls etc.  This approach always yields some useful
>> information for me in these situations.
>
> I've got
>
> restart_syscall(<... resuming interrupted call ...>
>
> hanging for each and every child process

can you try to have a stack trace please ?

gdb /path/to/php-fpm_binary pid_of_child
bt

++ jerome

[c2h5oh]

> can you try to have a stack trace please ?
>
> gdb /path/to/php-fpm_binary pid_of_child
> bt
>

Sorry about the delay, heres gdb output:

Reading symbols from /lib/i686/cmov/libcrypt.so.1...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libcrypt-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libcrypt.so.1
Reading symbols from /usr/lib/libc-client.so.2002edebian...done.
Loaded symbols for /usr/lib/libc-client.so.2002edebian
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libpq.so.5...done.
Loaded symbols for /usr/lib/libpq.so.5
Reading symbols from /lib/i686/cmov/librt.so.1...Reading symbols from /
usr/lib/debug/lib/i686/cmov/librt-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/librt.so.1
Reading symbols from /usr/lib/libmhash.so.2...done.
Loaded symbols for /usr/lib/libmhash.so.2
Reading symbols from /usr/lib/libmcrypt.so.4...done.
Loaded symbols for /usr/lib/libmcrypt.so.4
Reading symbols from /usr/lib/libltdl.so.3...done.
Loaded symbols for /usr/lib/libltdl.so.3
Reading symbols from /lib/libpam.so.0...done.
Loaded symbols for /lib/libpam.so.0
Reading symbols from /usr/lib/libfreetype.so.6...done.
Loaded symbols for /usr/lib/libfreetype.so.6
Reading symbols from /usr/lib/libpng12.so.0...done.
Loaded symbols for /usr/lib/libpng12.so.0
Reading symbols from /usr/lib/libjpeg.so.62...done.
Loaded symbols for /usr/lib/libjpeg.so.62
Reading symbols from /lib/libbz2.so.1.0...done.
Loaded symbols for /lib/libbz2.so.1.0
Reading symbols from /lib/i686/cmov/libresolv.so.2...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libresolv-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libresolv.so.2
Reading symbols from /lib/i686/cmov/libdl.so.2...Reading symbols from /
usr/lib/debug/lib/i686/cmov/libdl-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libdl.so.2
Reading symbols from /lib/i686/cmov/libnsl.so.1...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libnsl-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libnsl.so.1
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /usr/lib/i686/cmov/libssl.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libssl.so.0.9.8
Reading symbols from /usr/lib/i686/cmov/libcrypto.so.0.9.8...done.
Loaded symbols for /usr/lib/i686/cmov/libcrypto.so.0.9.8
Reading symbols from /usr/lib/libcurl.so.4...Reading symbols from /usr/
lib/debug/usr/lib/libcurl.so.4.1.0...done.
done.
Loaded symbols for /usr/lib/libcurl.so.4
Reading symbols from /usr/lib/libmysqlclient.so.15...done.
Loaded symbols for /usr/lib/libmysqlclient.so.15
Reading symbols from /lib/i686/cmov/libpthread.so.0...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libpthread-2.7.so...done.
[Thread debugging using libthread_db enabled]
[New Thread 0xb735f6d0 (LWP 14326)]
done.
Loaded symbols for /lib/i686/cmov/libpthread.so.0
Reading symbols from /lib/i686/cmov/libm.so.6...Reading symbols from /
usr/lib/debug/lib/i686/cmov/libm-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libm.so.6
Reading symbols from /usr/lib/libxml2.so.2...done.
Loaded symbols for /usr/lib/libxml2.so.2
Reading symbols from /lib/i686/cmov/libc.so.6...Reading symbols from /
usr/lib/debug/lib/i686/cmov/libc-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libc.so.6
Reading symbols from /usr/lib/libldap_r-2.4.so.2...done.
Loaded symbols for /usr/lib/libldap_r-2.4.so.2
Reading symbols from /lib/ld-linux.so.2...Reading symbols from /usr/
lib/debug/lib/ld-2.7.so...done.
done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /usr/lib/libidn.so.11...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /usr/lib/libssh2.so.1...done.
Loaded symbols for /usr/lib/libssh2.so.1
Reading symbols from /usr/lib/liblber-2.4.so.2...done.
Loaded symbols for /usr/lib/liblber-2.4.so.2
Reading symbols from /usr/lib/libsasl2.so.2...done.
Loaded symbols for /usr/lib/libsasl2.so.2
Reading symbols from /usr/lib/libgnutls.so.26...done.
Loaded symbols for /usr/lib/libgnutls.so.26
Reading symbols from /usr/lib/libgcrypt.so.11...done.
Loaded symbols for /usr/lib/libgcrypt.so.11
Reading symbols from /usr/lib/libgpg-error.so.0...done.
Loaded symbols for /usr/lib/libgpg-error.so.0
Reading symbols from /usr/lib/libtasn1.so.3...done.
Loaded symbols for /usr/lib/libtasn1.so.3
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/xcache.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/xcache.so
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/memcache.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/memcache.so
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/memcached.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/memcached.so
Reading symbols from /usr/local/lib/libmemcached.so.2...done.
Loaded symbols for /usr/local/lib/libmemcached.so.2
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/imagick.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/imagick.so
Reading symbols from /usr/lib/libWand.so.10...done.
Loaded symbols for /usr/lib/libWand.so.10
Reading symbols from /usr/lib/libMagick.so.10...done.
Loaded symbols for /usr/lib/libMagick.so.10
Reading symbols from /usr/lib/liblcms.so.1...done.
Loaded symbols for /usr/lib/liblcms.so.1
Reading symbols from /usr/lib/libtiff.so.4...done.
Loaded symbols for /usr/lib/libtiff.so.4
Reading symbols from /usr/lib/libfontconfig.so.1...done.
Loaded symbols for /usr/lib/libfontconfig.so.1
Reading symbols from /usr/lib/libXext.so.6...done.
Loaded symbols for /usr/lib/libXext.so.6
Reading symbols from /usr/lib/libXt.so.6...done.
Loaded symbols for /usr/lib/libXt.so.6
Reading symbols from /usr/lib/libSM.so.6...done.
Loaded symbols for /usr/lib/libSM.so.6
Reading symbols from /usr/lib/libICE.so.6...done.
Loaded symbols for /usr/lib/libICE.so.6
Reading symbols from /usr/lib/libX11.so.6...done.
Loaded symbols for /usr/lib/libX11.so.6
Reading symbols from /usr/lib/libexpat.so.1...done.
Loaded symbols for /usr/lib/libexpat.so.1
Reading symbols from /usr/lib/libXau.so.6...done.
Loaded symbols for /usr/lib/libXau.so.6
Reading symbols from /usr/lib/libxcb-xlib.so.0...done.
Loaded symbols for /usr/lib/libxcb-xlib.so.0
Reading symbols from /usr/lib/libxcb.so.1...done.
Loaded symbols for /usr/lib/libxcb.so.1
Reading symbols from /usr/lib/libXdmcp.so.6...done.
Loaded symbols for /usr/lib/libXdmcp.so.6
Reading symbols from /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/mailparse.so...done.
Loaded symbols for /usr/local/lib/php/extensions/no-debug-non-
zts-20060613/mailparse.so
Reading symbols from /lib/i686/cmov/libnss_files.so.2...Reading
symbols from /usr/lib/debug/lib/i686/cmov/libnss_files-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libnss_files.so.2
Reading symbols from /lib/i686/cmov/libnss_compat.so.2...Reading
symbols from /usr/lib/debug/lib/i686/cmov/libnss_compat-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libnss_compat.so.2
Reading symbols from /lib/i686/cmov/libnss_nis.so.2...Reading symbols
from /usr/lib/debug/lib/i686/cmov/libnss_nis-2.7.so...done.
done.
Loaded symbols for /lib/i686/cmov/libnss_nis.so.2
0xb7f7f424 in __kernel_vsyscall ()
(gdb) bt
#0 0xb7f7f424 in __kernel_vsyscall ()
#1 0xb7809083 in __read_nocancel () from /lib/i686/cmov/libpthread.so.
0
#2 0xb78899d8 in vio_read () from /usr/lib/libmysqlclient.so.15
#3 0xb7889a2b in vio_read_buff () from /usr/lib/libmysqlclient.so.15
#4 0xb788ad19 in ?? () from /usr/lib/libmysqlclient.so.15
#5 0x08831288 in ?? ()
#6 0x08835a20 in ?? ()
#7 0x00000004 in ?? ()
#8 0x0884a918 in ?? ()
#9 0x00000000 in ?? ()

[Jérôme Loyet]

2010/4/26 c2h5oh <maciej.l...@gmail.com>:

>> can you try to have a stack trace please ?
>>
>> gdb /path/to/php-fpm_binary pid_of_child
>> bt
>>
>
> Sorry about the delay, heres gdb output:

can you remove all extensions and try again please ?

[c2h5oh]

> can you remove all extensions and try again please ?

Wish I could - I am unable to force this error to happen (as I wrote
before I have even tried hammering it with AB for hours), therefore I
am down to running this on my production servers with 15s interval
monitoring. Since it's production imagick and memcached have to stay.
I can replace xcache with APC.

Sorry for making it extra difficult to pinpoint the error - If you can
offer some suggestions how to trigger the error on purpose I'm willing
to setup another box for just testing purpose.

[Jérôme Loyet]

2010/4/26 c2h5oh <maciej.l...@gmail.com>:

>
>> can you remove all extensions and try again please ?
>
> Wish I could - I am unable to force this error to happen (as I wrote
> before I have even tried hammering it with AB for hours), therefore I
> am down to running this on my production servers with 15s interval
> monitoring. Since it's production imagick and memcached have to stay.
> I can replace xcache with APC.
>
> Sorry for making it extra difficult to pinpoint the error - If you can
> offer some suggestions how to trigger the error on purpose I'm willing
> to setup another box for just testing purpose.

Your stack trace shows a problem in the mysql library. But there is
not more informations about the previous calls. Maybe you can try to
compile PHP with --enable-debug to have a better stack trace. Or try
to use another mysql library ... I don't really know how to help you

[Juan Fco. Giordana]

On 04/26/2010 05:38 AM, Jérôme Loyet wrote:
> Your stack trace shows a problem in the mysql library. But there is
> not more informations about the previous calls. Maybe you can try to
> compile PHP with --enable-debug to have a better stack trace. Or try
> to use another mysql library ... I don't really know how to help you

Yeah,

Try to compile PHP with the mysql native driver instead:

--with-mysql=mysqlnd \
--with-mysql-sock=/srv/var/run/mysqld/mysql.sock \
--with-mysqli=mysqlnd \
--enable-pdo \
--with-pdo-mysql=mysqlnd \

[Teri]

I have similar problem. Sometimes php-fpm start to use so many
resources.. and the vps is blocked. I can't login to it.
Restart of php-fpm resolves the problem but I can't see why it becomes.

Posted at Nginx Forum: http://forum.nginx.org/read.php?3,75656,128103#msg-128103

[Teri]

I found solution of this. I don't need to start many servers (childs),
because they eat the memory. So for VPS with 640 MB, 7 childs are
enough. If I start more, they start to eat the memory and the descibed
situation occurs.

Posted at Nginx Forum: http://forum.nginx.org/read.php?3,75656,129680#msg-129680

[grigori]

On 12 сен, 11:02, "Teri" <nginx-fo...@nginx.us> wrote:
> I found solution of this. I don't need to start many servers (childs),
> because they eat the memory. So for VPS with 640 MB, 7 childs are
> enough. If I start more, they start to eat the memory and the descibed
> situation occurs.

I run 20 child processes and a MySQL with hash tables on a VPS with
256 MB RAM, serving 400k requests daily.
Neither FPM nor PHP uses "so many resources" - it's your scripts only.

[jordi]

@c2h5oh did you ever solved the problem?

Jordi

Posted at Nginx Forum: http://forum.nginx.org/read.php?3,75656,139903#msg-139903

[Maciej Lisiewski]

> @c2h5oh did you ever solved the problem?
>
> Jordi
>
> Posted at Nginx Forum: http://forum.nginx.org/read.php?3,75656,139903#msg-139903
>

Yeah, actually it was MySQL connection limit.
For some reason php-fpm became unresponsive if it was unable to connect
to MySQL and failed to report that error.
My guess is that connection timeout was longer than php-fpm process
timeout and that's why error was never logged, but as for the reason why
child process that encountered this problem became unresponsive I have
no clue - I was trying to fix a problem on a live site so I was more
concentrated on fixing or working around the issue than pinpointing the
exact php-fpm problem.

--
Maciej Lisiewski

[jordi]

I have a master/slave schema

My master shows this:

mysql> show variables like 'max_connections';
+-----------------+-------+
| max_connections | 2248 |
+-----------------+-------+

mysql> SHOW STATUS LIKE 'Max_used_connections';
+----------------------+-------+
| Max_used_connections | 274 |
+----------------------+-------+

slaves show this:

show variables like 'max_connections';
+-----------------+-------+
| max_connections | 200 |
+-----------------+-------+

SHOW STATUS LIKE 'Max_used_connections';
+----------------------+-------+
| Max_used_connections | 62 |
+----------------------+-------+

Any other guess?

Posted at Nginx Forum: http://forum.nginx.org/read.php?3,75656,139908#msg-139908