timeouts

[Bob Bobsled]

Hi,

I'm trying to troubleshoot a poorly performing django site.  This started acting up when I upgraded on Fedora to python 3.10.

I've seen this post:

https://groups.google.com/g/modwsgi/c/_i6MGs6fh6w?pli=1

and

ttps://modwsgi.readthedocs.io/en/develop/user-guides/debugging-techniques.html#tracking-request-and-response

But I'm not sure how to debug with the wsgi middleware for error logging.

I've inspected the error_log, at /var/log/httpd, and see the problem reported there, but not sure where to try to fix it.

I did try increasing KeepAliveTimeout in my httpd.conf but the problem in general is very weird...it'll work fine for a few clicks and then hang up again with a timeout.  Restarting apache fixes it momentarily and then it's buggered again. 

Doesn't seem to be port conflicts since nip.io returns various ports, and nip.io is working fine when applied to my other sites on the same server.  And other sites on the same server not using modwsgi work fine too.

Any ideas or comments of where to look next would be appreciated.

error_log

[Thu Mar 10 13:32:13.737319 2022] [wsgi:error] [pid 1239:tid 1393] (70007)The timeout specified has expired: [client xxx.xxx.xxx.xxx:51768] mod_wsgi (pid=1239): Failed to proxy response from daemon.
[Thu Mar 10 13:34:08.063648 2022] [wsgi:error] [pid 3409:tid 3439] [client xxx.xxx.xxx.xxx:51770] Timeout when reading response headers from daemon process 'myarchive': /var/www/html/myarchive/myarchive/wsgi.py, referer: http://myarchive.xxx.xxx.xxx.xxx.nip.io/ephemera/

Thank you,

Bob

[Graham Dumpleton]

I presume you are manually configuring Apache. If yes, what is the mod_wsgi configuration? I need to see the VirtualHost, Directory/Location blocks and how mod_wsgi directives sit in respect of those and what arguments are given to them.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CALWZDaN8o4L8YMNz-PvUOwNTM8FQNQCg63p84LacHjmtHB%3D5cA%40mail.gmail.com.

[Bob Bobsled]

Hi Graham,

Thank you for your reply.  Here is my mod_wsgi.conf from fedora /etc/httpd/conf.d

Is this the info you needed?

My mod_wsgi is 4.9.0 was installed in a virtual env (/opt/myenv) with pip.

Regards,

Bob

<VirtualHost *:80>
ServerName localhost
ServerAlias localhost
DocumentRoot /var/www/html
   <Directory "/var/www/html/">
   Options +Indexes +Includes +FollowSymLinks +MultiViews
   AllowOverride All
   Require all granted
   </Directory>
</VirtualHost>

<VirtualHost *:80>
ServerName myarchive.xxx.xxx.xxx.xxx.nip.io
ServerAlias myarchive.xxx.xxx.xxx.xxx.nip.io
DocumentRoot /var/www/html
WSGIScriptAlias / "/var/www/html/myarchive/myarchive/wsgi.py"
WSGIDaemonProcess myarchive python-home=/opt/myenv python-path=/var/www/html/myarchive
WSGIProcessGroup myarchive

<Directory "/var/www/html/myarchive/myarchive">
   <Files wsgi.py>
   Require all granted
   </Files>
</Directory>

<Directory /var/www/html/myarchive>
   AllowOverride all
   Require all granted
   Options FollowSymlinks
 </Directory>
 
 Alias /static/ /var/www/html/myarchive/static/
 
 <Directory /var/www/html/myarchive/static>
  Require all granted
 </Directory>

 Alias /media/ /var/www/html/myarchive/media/

 <Directory /var/www/html/myarchive/media/>
  Order deny,allow
  Allow from all
 </Directory>

</VirtualHost>

WSGIPythonHome "/opt/myenv"

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/8ACF71E3-1C97-4126-BF5C-7341E0AEBF4A%40gmail.com.

[Graham Dumpleton]

After:

WSGIProcessGroup myarchive

also add:

WSGIApplicationGroup %{GLOBAL}

You are likely using a Python package that doesn't work properly in a sub interpreter and it is deadlock or hanging requests, meaning you get a timeout in the Apache child worker process when communicating with the daemon process.

If you only have one WSGI application per daemon process, you should always use that extra directive to force the use of the main Python interpreter context to avoid issues with poorly designed C extension packages.

https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api

So try that, and if still no joy then we can look at add something like request-timeout=60 to WSGIDaemonProcess to force recovery of the daemon process when requests get stuck. A side effect of that will be that tracebacks are logged telling you where it gets stuck.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CALWZDaPPgt0mu17QQhSoQTYQ1h2SOFhE6Nc%2BsQaZS3SrgfHZBA%40mail.gmail.com.

[Bob Bobsled]

Hi Graham,

re: timeouts

Adding WSGIApplicationGroup %{GLOBAL}  to the mod_wsgi.conf  seems to solve the problem.  I don't think I would have been able to come up with that as a solution on my own.  Thanks so much for your help.

When fedora upgraded to python 3.10 I created the virtual env again, and there's not much in there, but something is not playing as nicely as it was in 3.9.

(myenv) [refstudent@localhost ~]$ pip freeze
asgiref==3.4.1
Django==4.0
django-filter==21.1
django-modeladmin-reorder==0.3.1
django-paginator-rawqueryset==0.1.1
djangorestframework==3.12.4
mod-wsgi==4.9.0
pytz==2021.3
sqlparse==0.4.2

Best Regards,

Bob

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/7995D5A8-649B-49CF-8E54-76E410C595CA%40gmail.com.

[Graham Dumpleton]

Yeah that's odd. None of the usual Python packages which can trigger this sort of issue are being used. All of the packages look like they are pure Python packages, which shouldn't have an issue since the problem arises in C extensions with it usually being those which use a specific limited API in C Python rather than using a more complete API that works in sub interpreters.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/CALWZDaNAkMxk2pBOBcEXyLK9nNYDqPcy-5AUbhtuBkR%3DomE0Ug%40mail.gmail.com.