Error: "Timeout when reading response headers from daemon process 'localhost:80'..."

[Gregory Gundersen]

Hi,

I have a Flask web application using Apache server, mod_wsgi, and Docker. Here is my version information:

Debian

Apache 2.2.22
mod_wsgi 4.4.11
Docker 1.5.0
Python 2.7.3

The application is responsible for processing a variable amount of gene expression data. When the data sets are small, say less than a few MB, the application runs smoothly. But when the application must process a lot of data—my current failing test is around 18 MB—I get an timeout error. I doubt it's the size of the file so much as the time required to process the data. Here is my error:

Timeout when reading response headers from daemon process 'localhost:80': /etc/g2e/htdocs/g2e, referer: http://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE40585

The request from the NCBI website is coming from a Chrome extension, so that's my front-end. It seems to happen at exactly 60 seconds.

For what it's worth, I used mod-wsgi-express to build my Apache/mod_wsgi config files, so I'm pretty sure I'm using just the defaults in terms of timeouts.

Any help is appreciated.

Thanks,

Greg

[Graham Dumpleton]

Sorry for the delay in responding, was travelling and still catching up.

If left to run, how long would the request actually take to run?

You are going to run up against two possible timeouts designed to protect against ensuring the web application is still alive and hasn't got blocked.

The first of these is:

    optparse.make_option('--socket-timeout', type='int', default=60,

            metavar='SECONDS', help='Maximum number of seconds allowed '

            'to pass before timing out on a read or write operation on '

            'a socket and aborting the request. Defaults to 60 seconds.'),

That is, an Apache child worker process will wait at most 60 seconds by default when using mod_wsgi-express, for any data to be received related to a response from the mod_wsgi daemon process. This also applies where an Apache child worker process is reading or writing data with a HTTP client.

If you have very long running requests, you would need to set this higher, but increasing it carries dangers and needing to set it higher generally indicates what you are trying to do is probably a bad way of doing it.

In general, what you are better off doing is not doing such long running work inside of the actual web request. You should offload the processing to a backed task system queue. The web request would therefore return immediately once queued and the task system would pick up the job. The web UI can then poll or use some other mechanism to determine when the task is complete and the data available.

Now although you can set this higher, the next timeout that will cause a problem is:

    optparse.make_option('--request-timeout', type='int', default=60,

            metavar='SECONDS', help='Maximum number of seconds allowed '

            'to pass before the worker process is forcibly shutdown and '

            'restarted when a request does not complete in the expected '

            'time. In a multi threaded worker, the request time is '

            'calculated as an average across all request threads. Defaults '

            'to 60 seconds.'),

This timeout is designed to detect blocked requests or requests which are taking too long in the actual mod_wsgi daemon process. When this timeout is triggered the mod_wsgi daemon process itself will be restarted to try and recover the process. For a single threaded process this will be at 60 seconds by default. For a multithread process the timeout is actually more dynamic and depends on what other concurrent requests are being handled by the process, allowing more time where there might be parallel requests that would be interrupted if done strictly at 60 seconds.

So if increasing --socket-timeout you may also have to increase this as well, or definitely will if a single thread process. May still be okay for a multi threaded process.

Thinking about it, for a multithreaded daemon process, the interaction between this timeout and the socket timeout one isn't ideal. This is because also request timeout will allow the request to run longer in a multithread process, the socket timeout on the client will already have expired and the client would have got a gateway timeout error already. I will need to think about that one, but there isn't necessarily a simple answer as to what to do. Simply increasing the default for the socket timeout to 90 from 60, only delays the inevitable.

Graham

[Gregory Gundersen]

Thanks for the help. I was able to use the --socket-timeout option just as described. Setting the --request-timeout did not seem necessary.

increasing it carries dangers and needing to set it higher generally indicates what you are trying to do is probably a bad way of doing it.

Okay, I think you're right here. After benchmarking my application, I found several areas where I either had bugs or could make significant improvements in processing time, although I think your point was that if processing has to take longer than a minute, I should consider a different architecture. I'm a bit past that point right now, but it is good to know for future projects.

 

--
You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/cuZlSO9vN18/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Michal Seidl]

Hello, but where to write these configuration? I search through http://modwsgi.readthedocs.org and no matches? It should be used as ./configure parameter at compilation time of mod_wsgi?

Thanks Michal

[Graham Dumpleton]

The discussion was about when using mod_wsgi-express. That is, the pip installable mod_wsgi version on PyPi.

https://pypi.python.org/pypi/mod_wsgi

The equivalents for when configuring mod_wsgi directly are ‘socket-timeout’ and ‘request-timeout’ options to WSGIDaemonProcess directive.

The ReadTheDocs site search function is a pain in that it will not pick up search terms with ‘-‘ in them.

http://modwsgi.readthedocs.org/en/develop/release-notes/version-4.1.0.html?highlight=request-timeout

http://modwsgi.readthedocs.org/en/develop/release-notes/version-4.1.0.html?highlight=socket-timeout

You need to be using a recent mod_wsgi version and not ancient versions that some Linux distributions ship.

Graham

You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

To post to this group, send email to mod...@googlegroups.com.

Visit this group at https://groups.google.com/group/modwsgi.