How to get a stack trace upon server timeout?

[Daniel Haude]

Hello,

about once a day I'm getting "Timeout when reading response headers from daemon process" from my WSGI/Flask application, of course never when I test/use the app myself. I'd like to find out at which point the code hangs. The MySQL "long query" or error logs doesn't showe anything, but I'm also attaching to other DBs where I don't have access to the logs.

In the mod_wsgi logs I found this paragraph: "When stack traces were being dumped upon request timeout expiring, the line numbers of the definition of each function in the stack trace was being displayed, instead of the actual line number within the body of the function that was executing at the time." , so it must be possible to get stack traces, only there is no hint on how to activate them.

Thanks!

[Graham Dumpleton]

You need to set the request-timeout option on WSGIDaemonProcess.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/b4204300-ca26-4d26-8251-219a06846837o%40googlegroups.com.

[Daniel Haude]

Hello Graham,

thanks for your tip. I set  the request-timeout to 30 seconds (well shorter than Apache's patience), and lo and behold, the timeout errors have stopped appearing in the logs altogether.

Now I'm wondering: Since the request-timeout parameter makes mod_wsgi restart itself before Apache kills it from outside, is it possible that my code gets stuck as before but the timeout isn't logged at all? I was hoping that hitting mod_wsgi's request-timeout limit would trigger a meaningful stack trace into the Apache logs. Or possibly my script just happened to not hang since a couple of days ago (it did it about once per day before).

Best regards: **D

On Monday, June 8, 2020 at 12:46:36 PM UTC+2, Graham Dumpleton wrote:

You need to set the request-timeout option on WSGIDaemonProcess.

On 8 Jun 2020, at 6:01 pm, Daniel Haude <danie...@gmail.com> wrote:

Hello,

about once a day I'm getting "Timeout when reading response headers from daemon process" from my WSGI/Flask application, of course never when I test/use the app myself. I'd like to find out at which point the code hangs. The MySQL "long query" or error logs doesn't showe anything, but I'm also attaching to other DBs where I don't have access to the logs.

In the mod_wsgi logs I found this paragraph: "When stack traces were being dumped upon request timeout expiring, the line numbers of the definition of each function in the stack trace was being displayed, instead of the actual line number within the body of the function that was executing at the time." , so it must be possible to get stack traces, only there is no hint on how to activate them.

Thanks!

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

[Graham Dumpleton]

Ensure you are using LogLevel of info for Apache and not err or warn. You will only see stack traces in info level output.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/de0b5052-647f-43d7-89f5-07af5cfa1f94o%40googlegroups.com.

[Daniel Haude]

On Thursday, June 11, 2020 at 9:07:15 AM UTC+2, Graham Dumpleton wrote:

Ensure you are using LogLevel of info for Apache and not err or warn. You will only see stack traces in info level output.

Thanks for the tip, I did that. So far no timeouts, but loads and loads of stuff like below. Is that normal? Hundreds of hundreds of interpreter sgutdown / restarts. Were do all those processe s come from? "ps" only shows two WSGI processes with un-changing PIDs.

[Fri Jun 12 16:39:08.000146 2020] [wsgi:info] [pid 8459] mod_wsgi (pid=8459): Destroying interpreters.
[Fri Jun 12 16:39:08.000182 2020] [wsgi:info] [pid 8459] mod_wsgi (pid=8459): Cleanup interpreter ''.
[Fri Jun 12 16:39:08.008493 2020] [wsgi:info] [pid 8459] mod_wsgi (pid=8459): Terminating Python.
[Fri Jun 12 16:39:08.014036 2020] [wsgi:info] [pid 8459] mod_wsgi (pid=8459): Python has shutdown.
[Fri Jun 12 16:39:10.006916 2020] [wsgi:info] [pid 5055] mod_wsgi (pid=5055): Destroying interpreters.
[Fri Jun 12 16:39:10.006942 2020] [wsgi:info] [pid 5055] mod_wsgi (pid=5055): Cleanup interpreter ''.
[Fri Jun 12 16:39:10.014463 2020] [wsgi:info] [pid 5055] mod_wsgi (pid=5055): Terminating Python.
[Fri Jun 12 16:39:10.019861 2020] [wsgi:info] [pid 5055] mod_wsgi (pid=5055): Python has shutdown.
[Fri Jun 12 16:39:52.085182 2020] [wsgi:info] [pid 8759] mod_wsgi (pid=8759): Initializing Python.
[Fri Jun 12 16:39:52.104435 2020] [wsgi:info] [pid 8759] mod_wsgi (pid=8759): Attach interpreter ''.
[Fri Jun 12 16:39:52.119136 2020] [wsgi:info] [pid 8759] mod_wsgi (pid=8759): Imported 'mod_wsgi'.
[Fri Jun 12 16:40:10.136108 2020] [wsgi:info] [pid 8083] mod_wsgi (pid=8083): Destroying interpreters.

[Graham Dumpleton]

If you are using daemon mode, at global scope in the Apache configuration set:

    WSGIRestrictEmbedded On

This will turn off initialisation of Python interpreter in Apache child processes.

You don't need them since using daemon mode. It will save on some memory and startup time for the worker processes, and should get rid of these messages for when the Apache child worker processes get restarted due to server scaling events. Daemon processes are not scaled dynamically and so should stay running unless restarted due to events like request timeouts, or whole server restart due to log file rotation etc.

After you have set that, if still see the messages, then would need to investigate why daemon processes are restarting.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/42429c42-9d13-417f-a86a-7dd68393e008o%40googlegroups.com.

[Daniel Haude]

On Saturday, June 13, 2020 at 12:39:35 AM UTC+2, Graham Dumpleton wrote:

If you are using daemon mode, at global scope in the Apache configuration set:

    WSGIRestrictEmbedded On

Hi Graham,

I'll try that. First let me say thank you very much for your patience. I must admit that after reading your answers I looked into the mod_wsgi docs and found the same information there. However, I still don't think I could've found it without your help because, as excellent as the mod_wsgi docs are, the whole Apache / Python interaction seems to be a complex one. I consider myself to be a decent Python programmer, but if you (like me) don't have much experience with multi-process / multi-threaded applications or if you (like me) don't know the difference between an interpreter and a sub-interpreter much of it is white noise.

Regards! ++D

[Daniel Haude]

Update: Tried it, works like a charm. Source of timeout is identified. Best regards, and thanks!