Exception occurred processing WSGI script, IOError: client connection closed
138 views
Skip to first unread message
atomekk
Nov 27, 2009, 3:09:23 AM11/27/09
to modwsgi
Hi
I'm having trouble with site i'm working with. Website is build using
Pylons 0.9.7 and setup is on Apache with mod_wsgi 2.6. Generally
everything is working fine but sometimes after a week or so the wsgi
process get 'killed' with that trace in Apache logs:
mod_wsgi (pid=20008): Exception occurred processing WSGI script '/var/
www/tpn-cms/cgi/wsgi.py'.
[Fri Nov 27 07:32:07 2009] [error] [client 79.77.134.29] IOError:
client connection closed
mod_wsgi conf in apache:
WSGIDaemonProcess tpnssl user=cms group=cms threads=5 maximum-
requests=1000 inactivity-timeout=60 display-name=%{GROUP}
WSGIProcessGroup tpnssl
WSGIScriptAlias /cms /var/www/tpn-cms/cgi/wsgi.py
<Directory /var/www/tpn-cms>
Order deny,allow
Allow from all
WSGIRestrictProcess tpnssl
SSLOptions +StdEnvVars
</Directory>
I'm considering maybe to setup some watchdog to monitor wsgi apache
process and restart apache when its got killed/crashed.
Thomas
I'm having trouble with site i'm working with. Website is build using
Pylons 0.9.7 and setup is on Apache with mod_wsgi 2.6. Generally
everything is working fine but sometimes after a week or so the wsgi
process get 'killed' with that trace in Apache logs:
mod_wsgi (pid=20008): Exception occurred processing WSGI script '/var/
www/tpn-cms/cgi/wsgi.py'.
[Fri Nov 27 07:32:07 2009] [error] [client 79.77.134.29] IOError:
client connection closed
mod_wsgi conf in apache:
WSGIDaemonProcess tpnssl user=cms group=cms threads=5 maximum-
requests=1000 inactivity-timeout=60 display-name=%{GROUP}
WSGIProcessGroup tpnssl
WSGIScriptAlias /cms /var/www/tpn-cms/cgi/wsgi.py
<Directory /var/www/tpn-cms>
Order deny,allow
Allow from all
WSGIRestrictProcess tpnssl
SSLOptions +StdEnvVars
</Directory>
I'm considering maybe to setup some watchdog to monitor wsgi apache
process and restart apache when its got killed/crashed.
Thomas
Graham Dumpleton
Nov 27, 2009, 4:07:42 AM11/27/09
to mod...@googlegroups.com
2009/11/27 atomekk <ato...@gmail.com>:
From the FAQ at:
http://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions
this is described as:
"""
Q: Why do I get the error 'IOError: client connection closed'
appearing in the error logs?
A: This occurs when the HTTP client making the request closes the
connection before the complete response for a request has been
written.
This can occur where a user force reloads a web page before it had
been completely displayed. It can also occur when using benchmarking
tools such as 'ab' as they will over commit on the number of requests
they make when doing concurrent requests, killing off any extra
requests once the required number has been reached.
In general this error message can be ignored. For more information see
mod_wsgi issue #29.
"""
Where issue #29 is:
http://code.google.com/p/modwsgi/issues/detail?id=29
As described in that issue, mod_wsgi 3.0 was changed to not be so
noisy about that condition occurring and will only be logged in most
cases if Apache LogLevel set to debug.
So, if that is all you are seeing, it is nothing to worry about.
If you are seeing other information in the logs to suggest process is
truly crashing then post that information.
Do note that since you have specified both inactivity timeout and
maximum requests option for daemon process directive, the daemon
process is going to be shutdown and restart on a periodic basis
anyway.
Graham
> mod_wsgi conf in apache:
>
> WSGIDaemonProcess tpnssl user=cms group=cms threads=5 maximum-
> requests=1000 inactivity-timeout=60 display-name=%{GROUP}
> WSGIProcessGroup tpnssl
> WSGIScriptAlias /cms /var/www/tpn-cms/cgi/wsgi.py
> <Directory /var/www/tpn-cms>
> Order deny,allow
> Allow from all
> WSGIRestrictProcess tpnssl
> SSLOptions +StdEnvVars
> </Directory>
>
> I'm considering maybe to setup some watchdog to monitor wsgi apache
> process and restart apache when its got killed/crashed.
>
> Thomas
>
> --
>
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>
>
> Hi
>
> I'm having trouble with site i'm working with. Website is build using
> Pylons 0.9.7 and setup is on Apache with mod_wsgi 2.6. Generally
> everything is working fine but sometimes after a week or so the wsgi
> process get 'killed' with that trace in Apache logs:
>
> mod_wsgi (pid=20008): Exception occurred processing WSGI script '/var/
> www/tpn-cms/cgi/wsgi.py'.
> [Fri Nov 27 07:32:07 2009] [error] [client 79.77.134.29] IOError:
> client connection closed
That message in itself is not an indicator that the process is getting killed.
>
> I'm having trouble with site i'm working with. Website is build using
> Pylons 0.9.7 and setup is on Apache with mod_wsgi 2.6. Generally
> everything is working fine but sometimes after a week or so the wsgi
> process get 'killed' with that trace in Apache logs:
>
> mod_wsgi (pid=20008): Exception occurred processing WSGI script '/var/
> www/tpn-cms/cgi/wsgi.py'.
> [Fri Nov 27 07:32:07 2009] [error] [client 79.77.134.29] IOError:
> client connection closed
From the FAQ at:
http://code.google.com/p/modwsgi/wiki/FrequentlyAskedQuestions
this is described as:
"""
Q: Why do I get the error 'IOError: client connection closed'
appearing in the error logs?
A: This occurs when the HTTP client making the request closes the
connection before the complete response for a request has been
written.
This can occur where a user force reloads a web page before it had
been completely displayed. It can also occur when using benchmarking
tools such as 'ab' as they will over commit on the number of requests
they make when doing concurrent requests, killing off any extra
requests once the required number has been reached.
In general this error message can be ignored. For more information see
mod_wsgi issue #29.
"""
Where issue #29 is:
http://code.google.com/p/modwsgi/issues/detail?id=29
As described in that issue, mod_wsgi 3.0 was changed to not be so
noisy about that condition occurring and will only be logged in most
cases if Apache LogLevel set to debug.
So, if that is all you are seeing, it is nothing to worry about.
If you are seeing other information in the logs to suggest process is
truly crashing then post that information.
Do note that since you have specified both inactivity timeout and
maximum requests option for daemon process directive, the daemon
process is going to be shutdown and restart on a periodic basis
anyway.
Graham
> mod_wsgi conf in apache:
>
> WSGIDaemonProcess tpnssl user=cms group=cms threads=5 maximum-
> requests=1000 inactivity-timeout=60 display-name=%{GROUP}
> WSGIProcessGroup tpnssl
> WSGIScriptAlias /cms /var/www/tpn-cms/cgi/wsgi.py
> <Directory /var/www/tpn-cms>
> Order deny,allow
> Allow from all
> WSGIRestrictProcess tpnssl
> SSLOptions +StdEnvVars
> </Directory>
>
> I'm considering maybe to setup some watchdog to monitor wsgi apache
> process and restart apache when its got killed/crashed.
>
> Thomas
>
>
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>
>
atomekk
Nov 27, 2009, 5:00:29 AM11/27/09
to modwsgi
On Nov 27, 10:07 am, Graham Dumpleton <graham.dumple...@gmail.com>
wrote:
> 2009/11/27 atomekk <atom...@gmail.com>:
I dont see any other error in the logs apache or pylons one.
Our Pylons app/website is served in Daemon mode for two domains/
VirtualHosts, when everything is ok i can see in console:
top -u cms
5873 cms 19 0 104m 13m 3904 S 0.0 0.4 0:00.00 httpd
9855 cms 20 0 104m 13m 3904 S 0.0 0.4 0:00.00 httpd
Two processes are running for each VirtualHost, so today when i got
this IOError in top -u cms there was only one process.
Thomas
Graham Dumpleton
Nov 27, 2009, 5:11:02 AM11/27/09
to mod...@googlegroups.com
2009/11/27 atomekk <ato...@gmail.com>:
Read:
http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
In there is describes how Apache server child processes can come and
go in number. In other words you probably aren't even looking at
mod_wsgi daemon processes there, especially if you version of top
shows the name as overridden by mod_wsgi due to you using display-name
option.
Suggest you use instead:
ps auxwww | grep wsgi
The 'wsgi' bit comes from fact that the value of %{GROUP} for
display-name option means that process will get renamed as
(wsgi:tpnssl) instead of httpd.
Also, when using WSGIDaemonProcess, the number of process in mod_wsgi
daemon process group is fixed. In your case, since no process option,
is defaulting to one process. As such, can never be more than one
process in that daemon process group.
So, believe you are just seeing the normal Apache server child
processes, which as I saw can come and go and change in number anyway.
Graham
http://code.google.com/p/modwsgi/wiki/ProcessesAndThreading
In there is describes how Apache server child processes can come and
go in number. In other words you probably aren't even looking at
mod_wsgi daemon processes there, especially if you version of top
shows the name as overridden by mod_wsgi due to you using display-name
option.
Suggest you use instead:
ps auxwww | grep wsgi
The 'wsgi' bit comes from fact that the value of %{GROUP} for
display-name option means that process will get renamed as
(wsgi:tpnssl) instead of httpd.
Also, when using WSGIDaemonProcess, the number of process in mod_wsgi
daemon process group is fixed. In your case, since no process option,
is defaulting to one process. As such, can never be more than one
process in that daemon process group.
So, believe you are just seeing the normal Apache server child
processes, which as I saw can come and go and change in number anyway.
Graham
atomekk
Nov 27, 2009, 5:30:31 AM11/27/09
to modwsgi
> So, believe you are just seeing the normal Apache server child
> processes, which as I saw can come and go and change in number anyway.
>
> Graham
As i'm running wsgi daemon under specified user 'cms' and i got two
> processes, which as I saw can come and go and change in number anyway.
>
> Graham
instances of it so i guess top -u cms shows correct processes:
top -u cms
11195 cms 19 0 104m 13m 3904 S 0.0 0.4 0:00.00 httpd
12777 cms 19 0 124m 37m 12m S 0.0 1.0 0:01.33 httpd
and
ps auxwww | grep wsgi
cms 11195 0.0 0.3 106560 14272 ? Sl 10:05 0:00
(wsgi:tpnssl)
cms 12777 1.9 0.9 127232 38656 ? Sl 10:18 0:01
(wsgi:gleaxssl)
top and ps auxwww shows the same PID numbers. Maybe i'm understand it
wrong :/.
Anyway after today error in Apache logs i mentioned in first post,
website was down till i restarted apache again.
Maybe i should remove that inactivity-timeout directive ?
Thomas
Graham Dumpleton
Nov 27, 2009, 5:49:05 AM11/27/09
to mod...@googlegroups.com
2009/11/27 atomekk <ato...@gmail.com>:
So, one process per daemon process group.
> top and ps auxwww shows the same PID numbers. Maybe i'm understand it
> wrong :/.
No you aren't confused, top doesn't show the overridden names. The
htop on some systems may, but top doesn't. And BSD derived system (or
'ps' program to be correct), which show the overridden name.
They are the same processes.
> Anyway after today error in Apache logs i mentioned in first post,
> website was down till i restarted apache again.
If a daemon process dies, then one will automatically be restarted in its place.
> Maybe i should remove that inactivity-timeout directive ?
The inactivity timeout doesn't make the process go away, it just means
that it will shutdown and restart. It is purely to throw stuff out of
memory and reduce process back to starting memory size. Actual
application wouldn't be reloaded until next request for that
application.
What you should do is ensure that in Apache configuration you have:
LogLevel info
rather than default of:
LogLevel warn
This will cause mod_wsgi to output lot of logging about daemon
processes and when they are started, stopped, restarted and
unexpectedly exited.
You should be able to get a better idea of what is going on from that.
Details of setting LogLevel can be found in:
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques
Graham
>> So, believe you are just seeing the normal Apache server child
>> processes, which as I saw can come and go and change in number anyway.
>>
>> Graham
>
> As i'm running wsgi daemon under specified user 'cms' and i got two
> instances of it so i guess top -u cms shows correct processes:
>
> top -u cms
> 11195 cms 19 0 104m 13m 3904 S 0.0 0.4 0:00.00 httpd
> 12777 cms 19 0 124m 37m 12m S 0.0 1.0 0:01.33 httpd
>
> and
>
> ps auxwww | grep wsgi
> cms 11195 0.0 0.3 106560 14272 ? Sl 10:05 0:00
> (wsgi:tpnssl)
> cms 12777 1.9 0.9 127232 38656 ? Sl 10:18 0:01
> (wsgi:gleaxssl)
Which shows you have another daemon process group called 'gleaxssl'.
>> processes, which as I saw can come and go and change in number anyway.
>>
>> Graham
>
> As i'm running wsgi daemon under specified user 'cms' and i got two
> instances of it so i guess top -u cms shows correct processes:
>
> top -u cms
> 11195 cms 19 0 104m 13m 3904 S 0.0 0.4 0:00.00 httpd
> 12777 cms 19 0 124m 37m 12m S 0.0 1.0 0:01.33 httpd
>
> and
>
> ps auxwww | grep wsgi
> cms 11195 0.0 0.3 106560 14272 ? Sl 10:05 0:00
> (wsgi:tpnssl)
> cms 12777 1.9 0.9 127232 38656 ? Sl 10:18 0:01
> (wsgi:gleaxssl)
So, one process per daemon process group.
> top and ps auxwww shows the same PID numbers. Maybe i'm understand it
> wrong :/.
htop on some systems may, but top doesn't. And BSD derived system (or
'ps' program to be correct), which show the overridden name.
They are the same processes.
> Anyway after today error in Apache logs i mentioned in first post,
> website was down till i restarted apache again.
> Maybe i should remove that inactivity-timeout directive ?
that it will shutdown and restart. It is purely to throw stuff out of
memory and reduce process back to starting memory size. Actual
application wouldn't be reloaded until next request for that
application.
What you should do is ensure that in Apache configuration you have:
LogLevel info
rather than default of:
LogLevel warn
This will cause mod_wsgi to output lot of logging about daemon
processes and when they are started, stopped, restarted and
unexpectedly exited.
You should be able to get a better idea of what is going on from that.
Details of setting LogLevel can be found in:
http://code.google.com/p/modwsgi/wiki/DebuggingTechniques
Graham
Reply all
Reply to author
Forward
0 new messages