Socket or TCP Limit while using Fastcgi?

[Mr_Vista]

Hello,

on my machine are running mysql, nginx and fastcgi via spawn-fcgi.
with a lot of visitors there are some problems.

when i use fastcgi via socket (/tmp/fastcgi.socket), it serves php
very
quickly but 10 to 20 percent are 500 error - depending on amount of
visitors.
when i use fastcgi via ip (127.0.0.1:1026) there are no 500 errors,
but
therefore it serves php very slow.

static html pages are served very quickly at any time. mysql database
is
also very fast... there are no great table locks or something that
would
slow up php.
anyway.. php slows down without mysql connections.

cpu idle is about 60% when php slows down... so there is enough power.

so i had the idea that there could be a problem with the amount of
connections on tcp and/or the socket. is there a limit for socket or
tcp
connections?
how can i figure out if i reach this limits and how could i solve
this
problem?

thanks a lot
greets, alex

[mike]

honestly, i've heard a variety of issues and oddities when using
fastcgi via socket.

i use php over localhost TCP and it works great - and fast - not sure
why it's slow to you...

perhaps ensure its "127.0.0.1" and not "localhost" ... i don't know.

[Mr_Vista]

thanks for your answer mike!

i also used / use tcp... i tried to figure out why its so slow and so
i tried using socket for fastcgi and it was fast... but with lots of
500 error.

how much requests per second do you have? on our site its about 2000 -
3000 request per second... mostly php... so i think somewhere is a
limit
of tcp connections that can be handled or so. :-/

On 2 Jul., 20:47, mike <mike...@gmail.com> wrote:
> honestly, i've heard a variety of issues and oddities when using
> fastcgi via socket.
>
> i use php over localhost TCP and it works great - and fast - not sure
> why it's slow to you...
>
> perhaps ensure its "127.0.0.1" and not "localhost" ... i don't know.
>

[Andrei Nigmatulin]

On Wednesday 02 July 2008 22:37, Mr_Vista wrote:
> Hello,
>
> on my machine are running mysql, nginx and fastcgi via spawn-fcgi.
> with a lot of visitors there are some problems.
>
> when i use fastcgi via socket (/tmp/fastcgi.socket), it serves php
> very
> quickly but 10 to 20 percent are 500 error - depending on amount of
> visitors.
> when i use fastcgi via ip (127.0.0.1:1026) there are no 500 errors,
> but
> therefore it serves php very slow.

It seems there are two problems that make this behavour possible, one is in
linux kernel and second is in nginx. When the backlog for a listening unix
socket is full (when all workers are busy for some reason) the kernel returns
EAGAIN on next connect() attempt (not EINPROGRESS as you would expect for tcp
sockets). It seems that nginx cannot handle this situation normally and just
drop connection to a backend returning error to the client:

http://www.ruby-forum.com/topic/132314

Can you check nginx' error_log file for exact message when it issues an
error ?

> static html pages are served very quickly at any time. mysql database
> is
> also very fast... there are no great table locks or something that
> would
> slow up php.
> anyway.. php slows down without mysql connections.
>
> cpu idle is about 60% when php slows down... so there is enough power.

It might be shared file locks or something like that. You may try to attach to
php worker process with strace and see what happening inside single php
request - this might give you an idea where to dig next.

> so i had the idea that there could be a problem with the amount of
> connections on tcp and/or the socket. is there a limit for socket or
> tcp
> connections?
> how can i figure out if i reach this limits and how could i solve
> this
> problem?
>
> thanks a lot
> greets, alex

--
Andrei Nigmatulin
GPG PUB KEY 6449830D

Now I lay me down to sleep(3)
Pray the OS my core to keep
If I die before I wake
Pray the Disk my core to take

[Mr_Vista]

thanks Andrei,

well the error message in error.log is "Transport endpoint is not
connected" when i use fastcgi via socket.
when is use fastcgi via tcp its only very slooow without this error.

so did i understand you correct that the socket cant wait, but sends
an error?

i read the thread you posted. could you tell me anything about that?:
"Try to increase backlog in the fcgi spawn utility."

thanks for help!



On 2 Jul., 21:09, Andrei Nigmatulin <andrei.nigmatu...@gmail.com>
wrote:

[mike]

perhaps you just need more fastcgi engines listening on the tcp socket?

that could be why - it's waiting for the engines to process so it can
process... have you tried setting more engines up?

this would be where the statistics in php-fpm would be a good help
you'd know how many engines you have being called on average and know
if you're hitting your limits

On 7/2/08, Mr_Vista <style...@googlemail.com> wrote:
>

[Mr_Vista]

hmm shame on me but im still using spawn-fcgi from the lightly
project...

but of course i tried all possibilitys ... 1 parent 200 children, 1
parent 20 children, 2 parents each 50 children and so on.
nothing helps. best "feeled" results i got with 1 parent and 30
childs.

On 2 Jul., 21:34, mike <mike...@gmail.com> wrote:
> perhaps you just need more fastcgi engines listening on the tcp socket?
>
> that could be why - it's waiting for the engines to process so it can
> process... have you tried setting more engines up?
>
> this would be where the statistics in php-fpm would be a good help
> you'd know how many engines you have being called on average and know
> if you're hitting your limits
>

[Andrei Nigmatulin]

On Wednesday 02 July 2008 23:33, Mr_Vista wrote:
> thanks Andrei,
>
> well the error message in error.log is "Transport endpoint is not
> connected" when i use fastcgi via socket.
> when is use fastcgi via tcp its only very slooow without this error.
>
> so did i understand you correct that the socket cant wait, but sends
> an error?

Yes, exactly. TCP sockes behave differently at this point.

> i read the thread you posted. could you tell me anything about that?:
> "Try to increase backlog in the fcgi spawn utility."

To set a backlog to higher value:
$ echo "4096" > /proc/sys/net/core/somaxconn
and restart php-fpm or spawn-fcgi.

But this will not solve your problem with hanging php. This should only reduce
a probability of error occurrence.

To solve problem completely you have to found the reason why php executes too
slow on your server while cpu is not fully loaded.

[Mr_Vista]

thank you so much Andrei

tried it out and till now (5 minutes) it runs without 500 error and
using sockets.
the peak hour is over so i will report tomorrow how it works.

mike, thank you too!


On 2 Jul., 21:58, Andrei Nigmatulin <andrei.nigmatu...@gmail.com>

[Mr_Vista]

so here my report...

fastcgi runs via socket without 500er error now... BUT it slows down
on peak hour like fastcgi via tcp... (any idea what it could be when
all runs great but php?)
so tcp wasnt the problem i guess... the bottleneck has to be something
different in my case.
of course your solution with the backlog helped me to figure out this
fact! thanks!

now i use nginx upstream to distribute the php processes on different
server... works fine for now.