WSGI script timeout

[colorpyen]

I have been using wsgi3.0c in daemon for almost a year and recently
upgraded to 3.3. Strange is now the server is not responding?

Any help is appreciated.!!


// server version
Apache/2.2.14 (Ubuntu) mod_wsgi/3.3 Python/2.6.5

[Fri Aug 06 03:02:15 2010] [error] [client 220.181.94.226] Script
timed out before returning headers: application.wsgi

Thanks a lot.

-peter

[colorpyen]

some interesting logs

[info] mod_wsgi (pid=12985): Daemon process deadlock timer expired,
stopping process 'koistore'.
[Fri Aug 06 02:46:12 2010] [info] mod_wsgi (pid=12985): Shutdown
requested 'koistore'.
[Fri Aug 06 02:46:17 2010] [info] mod_wsgi (pid=12985): Aborting
process 'koistore'.
[Fri Aug 06 02:46:17 2010] [error] [client 114.32.30.80] Premature end
of script headers: application.wsgi, referer: http://apps.facebook
[Fri Aug 06 02:46:17 2010] [info] mod_wsgi (pid=13088): Attach
interpreter ''.
[Fri Aug 06 02:46:17 2010] [info] mod_wsgi (pid=13088): Adding '/usr/
local/pythonenv/DJANGO/lib/python2.6/site-packages' to path.
[Fri Aug 06 02:46:17 2010] [info] mod_wsgi (pid=13088): Create
interpreter 'www.evermai.com|'.
[Fri Aug 06 02:46:17 2010] [info] mod_wsgi (pid=13088): Adding '/usr/
local/pythonenv/DJANGO/lib/python2.6/site-packages' to path.
[Fri Aug 06 02:46:17 2010] [info] [client 220.181.94.226] (32)Broken
pipe: core_output_filter: writing data to the network
[Fri Aug 06 02:46:17 2010] [info] [client 220.181.94.226] mod_wsgi
(pid=13088, process='koistore', application='www.evermai.com|'): Loadi
[Fri Aug 06 02:46:18 2010] [info] [client 220.181.94.226] (32)Broken
pipe: core_output_filter: writing data to the network
[Fri Aug 06 02:46:18 2010] [info] [client 220.181.94.226] (32)Broken
pipe: core_output_filter: writing data to the network
[Fri Aug 06 02:46:18 2010] [info] [client 220.181.94.226] (32)Broken
pipe: core_output_filter: writing data to the network
[Fri Aug 06 02:46:18 2010] [info] [client 114.45.139.105] (32)Broken
pipe: core_output_filter: writing data to the network
[Fri Aug 06 02:46:18 2010] [error] [client 220.181.94.226] Script

timed out before returning headers: application.wsgi

[Graham Dumpleton]

On 6 August 2010 05:15, colorpyen <colo...@gmail.com> wrote:
> some interesting logs
>
> [info] mod_wsgi (pid=12985): Daemon process deadlock timer expired,
> stopping process 'koistore'.

What are the WSGIDaemonProcess directive options you are using?

That message above implies that some code in the application, normally
a C extension module, acquired the Python GIL and then did a long
operation which blocks, without releasing the GIL.

The default deadlock timeout is 5 minutes however, so that the daemon
process wasn't processing other requests should have been noticeable.

Have you overridden the default deadlock timeout?

Graham

> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To post to this group, send email to mod...@googlegroups.com.
> To unsubscribe from this group, send email to modwsgi+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/modwsgi?hl=en.
>
>

[Graham Dumpleton]

Please use reply-all and keep followups on mailing list.

On 6 August 2010 13:05, colorpyen <colo...@gmail.com> wrote:
> Hi Graham,
>
> Thanks for your quick response. My DaemonProcess directive is listed
> below. I didn't try to override the deadlock timeout as it didn't
> cause any problem in wsgi3.0c. I tried to downgrade wsgi to 2.8 and
> now it works okay. But I am still curious about why the deadlock
> problem may cause. Any C extension you can think of as it may
> introduce the problem. I use PIL 1.7 and Lxml.

The lxml package has had known problems working in Python sub
interpreters in the past.

Try with mod_wsgi 3.3, but add:

WSGIApplicationGroup %{GLOBAL}

to force use of main Python interpreter.

If you are hosting multiple WSGI applications, you will want to
restrict that directive to context of that specific application, or
ensure that each application is run in its own daemon process group to
ensure separation in case framework being used cant handle multiple
instances running in same interpreter.

Forcing use of main interpreter will avoid potential for issues where
packages not compatible with sub interpreters.

> Is overriding deadlock timeout introducing any performance degradation
> if set?
>
>   WSGIDaemonProcess koistore processes=1

Don't set 'processes=1', let it default to '1'. Use of the 'processes'
option, no matter the value, flags to WSGI application that it is
running across multiple processes. Triggering that prevents things
which expect to run in a single process, and check that value, from
working.

> threads=5 display-name=%
> {GROUP} \
>                      home=/var/www/koistore \
>                      python-path=/usr/local/pythonenv/DJANGO/lib/
> python2.6/site-packages \
>                      python-eggs=/tmp/python-eggs

It is generally a bad idea to put Python eggs cache in /tmp, at least
on systems where server used by different users.

Graham

>    # process group that serves this WSGI application in the virtual
> host
>    WSGIProcessGroup koistore
>
> Thanks a lot
>
> -peter
>
> On Aug 6, 7:45 am, Graham Dumpleton <graham.dumple...@gmail.com>
> wrote:

[Peter Yen]

Hi Graham,

Thanks for your response and feedback. That really makes me plan to re-learn the wiki and pay more attention to implication on directives.

Set to WSGIApplicationGroup %{GLOBAL}, makes mod_wsgi 3.3 works. and I've changed the eggs to a secure place and disable the processes flag.

Is there any concern if using apache worker MPM with WSGI daemon mode with threaded properties?

Thanks for all the help.


-peter

--
mobile: 0963499558
Pinkoi.com

[Graham Dumpleton]

On 6 August 2010 14:00, Peter Yen <colo...@gmail.com> wrote:
> Hi Graham,
>

> Thanks for your response and feedback. That really makes me plan to re-learn
> the wiki and pay more attention to implication on directives.
>
> Set to WSGIApplicationGroup %{GLOBAL}, makes mod_wsgi 3.3 works. and I've
> changed the eggs to a secure place and disable the processes flag.
>
> Is there any concern if using apache worker MPM with WSGI daemon mode with
> threaded properties?

Not sure I understand the question.

Multiple threads in daemon mode work fine if using either prefork or
worker MPM in Apache.

The only requirement is that when using prefork MPM, the underlying
APR libraries must have thread support compiled into them.

Also, your code should also be thread safe.

Obviously how many threads to use is a topic involving much hand
waving, as well as lots of smoke and mirrors as it depends on so many
factors.

Anyway, as I said, not sure what exactly you are wanting to know.

Graham

[Peter Yen]

Hi Graham,

Yeah, I just wanan know if multiple threads in daemon mode works okay with either prefork or worker MPM.
Thanks!

-peter

--
mobile: 0963499558
Pinkoi.com

[Graham Dumpleton]

On 21 March 2013 10:43, Chris <ch...@spitfireinnovation.com> wrote:

On Thursday, August 5, 2010 11:12:51 PM UTC-4, Graham Dumpleton wrote:

Try with mod_wsgi 3.3, but add:

 WSGIApplicationGroup %{GLOBAL}

to force use of main Python interpreter.

Graham, thanks so much for this advice.  This solved a persistent problem for me.

Are there downsides to doing this -- should I be motivated to find & solve the root cause?  Or is the WSGIApplicationGroup approach something I can use in the long term and not worry about?

If you are hosting only the one WSGI application there is no downside and forcing the use of the main interpreter is actually recommended as it will save a little memory.

If you are hosting more than one WSGI application, using this directive will cause issues if you are using a framework such as Django where it is not possible to run more than one Django site instance in the same interpreter context.

If running multiple WSGI applications and using separate daemon processes for each application, then it is again safe to use.

 

Is there a way to identify the offending module that's causing the GIL problem?

It can be possible to use gdb to extract C level stack traces when a process is in a dead locked state. See:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB

Graham

 

Thank you

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

To post to this group, send email to mod...@googlegroups.com.

Visit this group at http://groups.google.com/group/modwsgi?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.