Script timed out before returning headers, my production site give time out after one day.

[Ariel Isaac]

I have a django web application that's becoming unresponsive every one day approximately in a production server, when it happens, it never
recovers, I can leave it there for hours and it will not server any more requests, then I restart the apache and everything work fine for more one day.
I am using apache, wsgi, django, mysql.

The apache logs error says this:

[Fri Feb 08 12:20:12 2013] [error] [client "ipnumber1" ] Script timed out before returning headers: django.wsgi, referer: http://www.sitedomain.com/url
[Fri Feb 08 12:14:16 2013] [error] [client "ipnumber2" ] Script timed out before returning headers: django.wsgi, referer: http://www.sitedomain.com/url1
[Fri Feb 08 12:20:59 2013] [error] [client "ipnumber3" ] Script timed out before returning headers: django.wsgi, referer: http://www.sitedomain.com/url2
[Fri Feb 08 12:21:03 2013] [error] [client "ipnumber1" ] Script timed out before returning headers: django.wsgi, referer: http://www.sitedomain.com/url3
[Fri Feb 08 12:26:29 2013] [error] [client "ipnumber5" ] Script timed out before returning headers: django.wsgi, referer: http://www.sitedomain.com/url4
[Fri Feb 08 12:26:42 2013] [error] [client "ipnumber6" ] Script timed out before returning headers: django.wsgi, referer: http://www.sitedomain.com/url5


Apache config:

WSGIDaemonProcess sitename threads=15 user=www-sitename group=www-sitename maximum-requests=1024 display-name=%{GROUP}
WSGIProcessGroup sitename
WSGIScriptAlias / /somepath-sitename/django/django.wsgi



Could somebody help me to find out what's happening ???

Regards,

Ariel

[Graham Dumpleton]

Try adding:

WSGIApplicationGroup %{GLOBAL}

See:

http://code.google.com/p/modwsgi/wiki/ApplicationIssues#Python_Simplified_GIL_State_API

Your code is likely using a C extension for Python which doesn't work in sub interpreters properly and is causing a deadlock on the Python GIL and stopping everything.

BTW, maximum-requests is not recommended in production systems unless you have absolutely no over choice because your application is leaking resources and you cant do anything about it or cant find the problem.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Ariel Isaac]

Hi Graham, I have already add WSGIApplicationGroup %{GLOBAL} I have
the configuration like this:

WSGIApplicationGroup %{GLOBAL}

but the problem it is still there, why is this happening ?

I hope you can help me.

Regards,
Ariel

[Graham Dumpleton]

Validate that you are:

http://code.google.com/p/modwsgi/wiki/CheckingYourInstallation#Sub_Interpreter_Being_Used

Also try:

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces

Your code is either deadlocking due to thread locks, or blocking and never returning on calls to back end services.

Graham

[Ariel Isaac]

Hi, thanks again for your answer Graham.

I thought I had solved the problem when I changed the configuration I wrote in previous email by this:

WSGIDaemonProcess sitename processes=3 threads=5 user=www-sitename group=www-sitename display-name=%{GROUP}
WSGIApplicationGroup %{GLOBAL}

WSGIProcessGroup sitename
WSGIScriptAlias / /somepath-sitename/django/django.wsgi

Notice that I removed the 'maximum-request=1024' directive and I added the 'processes=3' directive and I updated the 'threads=5' directive

But after 2 weeks the timeout error raised again and I need to restart he apache each time this happen, How could I solve the problem ??? what is the better configuration for this ??? Should I increase the number of processes to 5 ??? Should I increase the number of threads ???

I'll really appreciate any help.

Kind regards,
Thanks in advance.
Ariel

[Graham Dumpleton]

If the daemon processes are still in existence at that point, try and work out where you code may be blocking.

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Extracting_Python_Stack_Traces

http://code.google.com/p/modwsgi/wiki/DebuggingTechniques#Debugging_Crashes_With_GDB

In the latter using gdb, you aren't debugging a crash, simply trying to get some stack traces out for all threads (ie., thread apply all bt) to work out where things are stuck.

So long as your site sees a constant flow of traffic and you don't have legitimate long running requests, you could set on WSGIDaemonProcess the option:

  inactivity-timeout=120

This will cause the daemon process to be restarted if it was receiving requests and then didn't receive any for 2 minutes. 

More important to you though, is that if a specific request blocks and there is HTTP request content read, or or HTTP response content returned for 2 minutes, it will also try restarting the process.

How the one option handles both is not ideal. In mod_wsgi 4.0 I was playing with a new blocked-timeout option which would specifically look for just stuck requests, but no idea when that may be done and not even sure where I was up to with it since has been so long since was doing it. 

Graham

[Ariel Isaac]

Hi Graham, thanks again for your answer.

So, you mean that if I add the 'inactivity-timeout=120' to the WSGIDaemonProcess I can solve the problem of timeout while I debug the thread that is causing the lock ?

Otherwise, I have read the #Extracting_Python_Stack_Traces and #Sub_Interpreter_Being_Used but I have not idea how to debug it and where to put the code they explain there, by instance in #Sub_Interpreter_Being_Used they say I must use this code:

<code>
import sys

def application(environ, start_response):
    status = '200 OK'
    output = 'mod_wsgi.application_group = %s' % repr(environ['mod_wsgi.application_group'])

    response_headers = [('Content-type', 'text/plain'),
                        ('Content-Length', str(len(output)))]
    start_response(status, response_headers)

    return [output]
</code>

Where must I put that piece of code ? in django.wsgi ??? or in another file ?
Excuse me because I am not an apache-wsgi expert.

I would really appreciate your help,
Thanks in advance,
Ariel