504 gateway timeouts

426 views
Skip to first unread message

Anoop Alias

unread,
Sep 18, 2008, 4:30:11 AM9/18/08
to highloa...@googlegroups.com
Hi,

i have a NGINX-PHP-FPM  - MySQL  system doing virtual hosting

It works fine most of the time but recently i installed a moodle learning management system in it. and problems started

Once i take moodle page ..the entire server hangs and all other sites hosted on the server gives "504 gateway timeout error"

I thought it was a problem with moodle and shutdown moodle..but the same error occured in another website built with symfony project :(

The system load and memory usage are well below normal.The only problem i could fins was a lot of connections to php-fpm in TIME_WAIT

=============================================
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN     25601/master
tcp        0      0 127.0.0.1:9000          0.0.0.0:*               LISTEN     13321/php-cgi
tcp     1097      0 127.0.0.1:9000          127.0.0.1:44335         CLOSE_WAIT -
tcp      977      0 127.0.0.1:9000          127.0.0.1:59439         CLOSE_WAIT -
tcp     1249      0 127.0.0.1:9000          127.0.0.1:59693         CLOSE_WAIT -
tcp     1113      0 127.0.0.1:9000          127.0.0.1:59433         CLOSE_WAIT -
tcp     1201      0 127.0.0.1:9000          127.0.0.1:59943         CLOSE_WAIT -
tcp     1089      0 127.0.0.1:9000          127.0.0.1:59429         CLOSE_WAIT -
tcp     1193      0 127.0.0.1:9000          127.0.0.1:59685         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:40226         CLOSE_WAIT 13337/php-cgi
tcp     1201      0 127.0.0.1:9000          127.0.0.1:59683         CLOSE_WAIT -
tcp     1465      0 127.0.0.1:9000          127.0.0.1:55840         CLOSE_WAIT -
tcp     1425      0 127.0.0.1:9000          127.0.0.1:59424         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:40246         CLOSE_WAIT 13323/php-cgi
tcp      873      0 127.0.0.1:9000          127.0.0.1:60470         CLOSE_WAIT -
tcp      713      0 127.0.0.1:9000          127.0.0.1:59959         CLOSE_WAIT -
tcp     1065      0 127.0.0.1:9000          127.0.0.1:59700         CLOSE_WAIT -
tcp      977      0 127.0.0.1:9000          127.0.0.1:59184         CLOSE_WAIT -
tcp     1089      0 127.0.0.1:9000          127.0.0.1:59150         CLOSE_WAIT -
tcp     1089      0 127.0.0.1:9000          127.0.0.1:59146         CLOSE_WAIT -
tcp     2289      0 127.0.0.1:9000          127.0.0.1:59144         CLOSE_WAIT -
tcp     1113      0 127.0.0.1:9000          127.0.0.1:59394         CLOSE_WAIT -
tcp        0      0 127.0.0.1:9000          127.0.0.1:45342         ESTABLISHED-
tcp        9      0 127.0.0.1:9000          127.0.0.1:39967         CLOSE_WAIT 13334/php-cgi
tcp     2265      0 127.0.0.1:9000          127.0.0.1:59167         CLOSE_WAIT -
tcp     1025      0 127.0.0.1:9000          127.0.0.1:59164         CLOSE_WAIT -
tcp     1161      0 127.0.0.1:9000          127.0.0.1:59677         CLOSE_WAIT -
tcp     1113      0 127.0.0.1:9000          127.0.0.1:59161         CLOSE_WAIT -
tcp     1017      0 127.0.0.1:9000          127.0.0.1:59411         CLOSE_WAIT -
tcp      905      0 127.0.0.1:9000          127.0.0.1:59665         CLOSE_WAIT -
tcp      985      0 127.0.0.1:9000          127.0.0.1:44399         CLOSE_WAIT -
tcp      873      0 127.0.0.1:9000          127.0.0.1:59754         CLOSE_WAIT -
tcp     1465      0 127.0.0.1:9000          127.0.0.1:59498         CLOSE_WAIT -
tcp     1113      0 127.0.0.1:9000          127.0.0.1:59242         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:58982         CLOSE_WAIT 13328/php-cgi
tcp     1337      0 127.0.0.1:9000          127.0.0.1:59239         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:40551         CLOSE_WAIT 13327/php-cgi
tcp     1345      0 127.0.0.1:9000          127.0.0.1:59748         CLOSE_WAIT -
tcp      873      0 127.0.0.1:9000          127.0.0.1:60002         CLOSE_WAIT -
tcp      897      0 127.0.0.1:9000          127.0.0.1:59518         CLOSE_WAIT -
tcp     1025      0 127.0.0.1:9000          127.0.0.1:59519         CLOSE_WAIT -
tcp      969      0 127.0.0.1:9000          127.0.0.1:60027         CLOSE_WAIT -
tcp      833      0 127.0.0.1:9000          127.0.0.1:59768         CLOSE_WAIT -
tcp      969      0 127.0.0.1:9000          127.0.0.1:59766         CLOSE_WAIT -
tcp      897      0 127.0.0.1:9000          127.0.0.1:59253         CLOSE_WAIT -
tcp     1169      0 127.0.0.1:9000          127.0.0.1:59506         CLOSE_WAIT -
tcp     1049      0 127.0.0.1:9000          127.0.0.1:44403         CLOSE_WAIT -
tcp     1305      0 127.0.0.1:9000          127.0.0.1:59248         CLOSE_WAIT -
tcp     1113      0 127.0.0.1:9000          127.0.0.1:60016         CLOSE_WAIT -
tcp     1313      0 127.0.0.1:9000          127.0.0.1:59249         CLOSE_WAIT -
tcp      985      0 127.0.0.1:9000          127.0.0.1:59761         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:58958         CLOSE_WAIT 13324/php-cgi
tcp     1113      0 127.0.0.1:9000          127.0.0.1:44367         CLOSE_WAIT -
tcp     1089      0 127.0.0.1:9000          127.0.0.1:59468         CLOSE_WAIT -
tcp     1001      0 127.0.0.1:9000          127.0.0.1:59465         CLOSE_WAIT -
tcp      857      0 127.0.0.1:9000          127.0.0.1:59719         CLOSE_WAIT -
tcp     1049      0 127.0.0.1:9000          127.0.0.1:59714         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:40514         CLOSE_WAIT 13336/php-cgi
tcp     1089      0 127.0.0.1:9000          127.0.0.1:59228         CLOSE_WAIT -
tcp     1113      0 127.0.0.1:9000          127.0.0.1:59737         CLOSE_WAIT -
tcp     1089      0 127.0.0.1:9000          127.0.0.1:59225         CLOSE_WAIT -
tcp     1209      0 127.0.0.1:9000          127.0.0.1:59474         CLOSE_WAIT -
tcp      873      0 127.0.0.1:9000          127.0.0.1:59219         CLOSE_WAIT -
tcp     1065      0 127.0.0.1:9000          127.0.0.1:59987         CLOSE_WAIT -
tcp      985      0 127.0.0.1:9000          127.0.0.1:59216         CLOSE_WAIT -
tcp     1009      0 127.0.0.1:9000          127.0.0.1:45648         CLOSE_WAIT -
tcp     1121      0 127.0.0.1:9000          127.0.0.1:44462         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:40617         CLOSE_WAIT 13335/php-cgi
tcp     1137      0 127.0.0.1:9000          127.0.0.1:59303         CLOSE_WAIT -
tcp     1001      0 127.0.0.1:9000          127.0.0.1:59815         CLOSE_WAIT -
tcp      801      0 127.0.0.1:9000          127.0.0.1:59300         CLOSE_WAIT -
tcp      961      0 127.0.0.1:9000          127.0.0.1:59301         CLOSE_WAIT -
tcp     1065      0 127.0.0.1:9000          127.0.0.1:59299         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:59041         CLOSE_WAIT 13344/php-cgi
tcp        9      0 127.0.0.1:9000          127.0.0.1:40127         CLOSE_WAIT 13322/php-cgi
tcp        9      0 127.0.0.1:9000          127.0.0.1:39868         CLOSE_WAIT 13339/php-cgi
tcp      921      0 127.0.0.1:9000          127.0.0.1:59579         CLOSE_WAIT -
tcp     1265      0 127.0.0.1:9000          127.0.0.1:59577         CLOSE_WAIT -
tcp      945      0 127.0.0.1:9000          127.0.0.1:59319         CLOSE_WAIT -
tcp      657      0 127.0.0.1:9000          127.0.0.1:45748         CLOSE_WAIT -
tcp      977      0 127.0.0.1:9000          127.0.0.1:59317         CLOSE_WAIT -
tcp        9      0 127.0.0.1:9000          127.0.0.1:40625         CLOSE_WAIT 13338/php-cgi
tcp      793      0 127.0.0.1:9000          127.0.0.1:45708         CLOSE_WAIT -
tcp     1025      0 127.0.0.1:9000          127.0.0.1:45453         CLOSE_WAIT -
tcp      921      0 127.0.0.1:9000          127.0.0.1:59529         CLOSE_WAIT -
tcp     1097      0 127.0.0.1:9000          127.0.0.1:59783         CLOSE_WAIT -
tcp     1113      0 127.0.0.1:9000          127.0.0.1:59781         CLOSE_WAIT -
tcp        0      0 127.0.0.1:9000          127.0.0.1:45443         ESTABLISHED-
tcp     1033      0 127.0.0.1:9000          127.0.0.1:59779         CLOSE_WAIT -
tcp     1233      0 127.0.0.1:9000          127.0.0.1:59806         CLOSE_WAIT -
tcp      905      0 127.0.0.1:9000          127.0.0.1:59550         CLOSE_WAIT -
tcp      969      0 127.0.0.1:9000          127.0.0.1:59292         CLOSE_WAIT -
tcp      697      0 127.0.0.1:9000          127.0.0.1:60056         CLOSE_WAIT -
tcp     1057      0 127.0.0.1:9000          127.0.0.1:59798         CLOSE_WAIT -
tcp     1249      0 127.0.0.1:9000          127.0.0.1:59287         CLOSE_WAIT -
tcp     1329      0 127.0.0.1:
=================================================================

As a side note i use a virtuosso VPS and an external MySQL server connected over TCP/IP




My php-fpm.conf
==============================================================
Time limit on waiting execution of single request
                        Should be used when 'max_execution_time' ini option does not terminate execution for some reason
                        <value name="request_execution_timeout">31s</value>

                        Set open file desc rlimit
                        <value name="rlimit_files">1024</value>

                        Set max core size rlimit
                        <value name="rlimit_core">0</value>

                        Chroot to this directory at the start
                        <value name="chroot"></value>

                        Chdir to this directory at the start
                        <value name="chdir"></value>

                        Redirect workers' stdout and stderr into main error log.
                        If not set, they will be redirected to /dev/null, according to FastCGI specs
                        <value name="catch_workers_output">yes</value>

                        How much requests each process should execute before respawn.
                        Useful to work around memory leaks in 3rd party libraries.
                        For endless request processing please specify 0
                        Equivalent to PHP_FCGI_MAX_REQUESTS
                        <value name="max_requests">500</value>

                        Comma separated list of ipv4 addresses of FastCGI clients that allowed to connect.
                        Equivalent to FCGI_WEB_SERVER_ADDRS environment in original php.fcgi (5.2.2+)
                        Makes sense only with AF_INET listening socket.
                        <value name="allowed_clients">127.0.0.1</value>

---------------------
--------------------------------

Process manager settings
                        <value name="pm">

                                Sets style of controling worker process count.
                                Valid values are 'static' and 'apache-like'
                                <value name="style">apache-like</value>

                                Sets the limit on the number of simultaneous requests that will be served.
                                Equivalent to Apache MaxClients directive.
                                Equivalent to PHP_FCGI_CHILDREN environment in original php.fcgi
                                Used with any pm_style.
                                <value name="max_children">15</value>

                                Settings group for 'apache-like' pm style
                                <value name="apache_like">

                                        Sets the number of server processes created on startup.
                                        Used only when 'apache-like' pm_style is selected
                                        <value name="StartServers">20</value>

                                        Sets the desired minimum number of idle server processes.
                                        Used only when 'apache-like' pm_style is selected
                                        <value name="MinSpareServers">5</value>

                                        Sets the desired maximum number of idle server processes.
                                        Used only when 'apache-like' pm_style is selected
                                        <value name="MaxSpareServers">35</value>

                                </value>
=================================================================



NGINX config file

===========================================
server {
listen       80;
server_name domain.com www.domain.com *.domain.com;
location / {
root   /home/jail/home/domain.com/httpdocs/alpha/web;
index  index.php index.html;
if (!-e $request_filename) {
rewrite  ^(.*)$  /index.php?q=$1  last;
break;
}
location ~ .php$ {
include /usr/local/nginx/conf/fastcgi_params;
fastcgi_pass 127.0.0.1:9000;
fastcgi_index index.php;
}
}
}
==============================================

Thanks in advance in helping me out

--
Anoop P Alias (PGP Key ID : 0x014F9953)
GNU system administrator
http://GnuSys.net

jackbillow

unread,
Sep 18, 2008, 6:36:25 AM9/18/08
to highloa...@googlegroups.com
you can set max_execution_time = 80 in php.ini file.
and set "fastcgi_buffers 8 128k send_timeout 60" in nginx.conf

Andrei Nigmatulin

unread,
Sep 18, 2008, 6:56:14 AM9/18/08
to highloa...@googlegroups.com
On Thursday 18 September 2008 12:30, Anoop Alias wrote:
> Hi,
>
> i have a NGINX-PHP-FPM - MySQL system doing virtual hosting
>
> It works fine most of the time but recently i installed a moodle learning
> management system in it. and problems started
>
> Once i take moodle page ..the entire server hangs and all other sites
> hosted on the server gives "504 gateway timeout error"
>
> I thought it was a problem with moodle and shutdown moodle..but the same
> error occured in another website built with symfony project :(

It seems like all workers are busy at the moment you get 504 error. Can you
make a backtrace of any php worker to make sure ?

--
Andrei Nigmatulin
GPG PUB KEY 6449830D

Now I lay me down to sleep(3)
Pray the OS my core to keep
If I die before I wake
Pray the Disk my core to take

Anoop Alias

unread,
Sep 20, 2008, 11:56:08 AM9/20/08
to highloa...@googlegroups.com
Hi all,

Guess what , i dumped the Dedicated database server and moved the database to the same vps running nginx and php-fpm and everything seems to be smooth now

Previously i had two vps of 300 MB each and everything was in chaos and now everything is squeezed into a single vps of 300 MB RAM .

I reached this conclusion after seeing many connections to the mysql port also in TIME_WAIT . (the latency between the server was ~ 1 ms) ;so i guess the php-fpm process were timing out  because of some problems with getting data from the database (most of the sites were database intensive)

Just thought the above info would be valuable for anyone who just thought they can scale the system by keeping a separate  dedicated database server
Reply all
Reply to author
Forward
0 new messages