Timeout when reading response header from daemon process.

97 views
Skip to first unread message

A

unread,
Jan 23, 2023, 2:15:33 PM1/23/23
to modwsgi
It keeps timing out and I've been trying to solve it to no avail.
Here my modwsgi.conf

<VirtualHost *:80>
      ServerName localhost
      ServerAlias ----------.com *.----------.com
     
      Define project_name     ----------
      Define user             -------------
     
      Define project_path     /srv/http/fosware
      Define wsgi_path        /srv/http/fosware/fosware
      Define environment_path /srv/http/fosware/venv
     
      WSGIDaemonProcess ${user}-${project_name} user=${user} group=${user} processes=1 threads=1 python-eggs=/tmp/python-eggs/ python-path=${project_path}:${environment_path}/lib/python2.7/site-packages
      WSGIProcessGroup ${user}-${project_name}

      WSGIScriptAlias / ${wsgi_path}/wsgi.py

      <Directory ${project_path}>
        <IfVersion < 2.3 >
        Order allow,deny
        Allow from all
        </IfVersion>
        <IfVersion >= 2.3>
        Require all granted
        </IfVersion>
      </Directory>        
     

      Alias /static ${project_path}/static
      <Directory ${project_path}/static>
             Require all granted
             SetHandler None
             FileETag none
             Options FollowSymLinks
      </Directory>

      Alias /media ${project_path}/media
      <Directory ${project_path}/media>
             Require all granted
             SetHandler None
             FileETag none
             Options FollowSymLinks
             ErrorDocument 404 /error404
      </Directory>

      ErrorLog /var/log/httpd/${user}-${project_name}-error.log
      LogLevel info
      CustomLog /var/log/httpd/${user}-${project_name}-access.log combined
</VirtualHost>

Graham Dumpleton

unread,
Jan 23, 2023, 2:21:44 PM1/23/23
to mod...@googlegroups.com
Your configuration means only a single request can be handled at a time by the daemon process, so if you have a very long running request, any other requests would block waiting.

Even if not a long running request, a problem may be that you are using a Python package that isn't designed to work in a Python sub interpreter. This could cause that package to hang and so everything blocks up (same as long running request at that point) for that request.


Try the solution described in the docs of forcing the use of the main Python interpreter by adding:

    WSGIApplicationGroup %{GLOBAL}

If that doesn't help you will need to work out where your code is blocking. For that try adding:

    request-timeout=30

option to WSGIDaemonProcess directive.

This will cause daemon process in your case to restart after 30 seconds when it blocks and in doing that will attempt to log to error log file the stack traces of where your code was blocked.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/02fb2171-ca59-4053-9be3-8ff75e9cf9edn%40googlegroups.com.

A

unread,
Jan 24, 2023, 4:39:40 AM1/24/23
to modwsgi
Dear Graham,

It was a matter of adding that line and everything fell into place!


Thanks a lot,
A

Carsten Fuchs

unread,
Feb 5, 2023, 11:23:29 AM2/5/23
to modwsgi
Dear Graham,

I experienced the same timeout errors after having upgraded a server from Ubuntu 20.04 LTS to Ubuntu 22.04 LTS, thereby also upgrading from Python 3.8 to Python 3.10. The application is a relatively simple Django project that used to work well until the upgrade. After the upgrade, I deleted the old virtualenv and built a new one, using `pip install --no-cache-dir -r requirements.txt` to install it. However, I experience the same problem.

Adding `WSGIApplicationGroup %{GLOBAL}` solved the problem, but I am still concerned because the site worked well with the older Ubuntu 20.04 LTS and I would prefer to not mask a potential problem and rather find its root.

Therefore, I added the `request-timeout=30` option to `WSGIDaemonProcess` (and temporarily commented `WSGIApplicationGroup` out again) in order to get a stack trace, however it doesn't seem to have any effect: Requests time out only much later than 30 seconds.

Can you please advise what may have caused the problem when upgrading from Ubuntu 20.04 LTS to Ubuntu 22.04 LTS and why `request-timeout=30` may not have any effect?

Best regards,
Carsten

Carsten Fuchs

unread,
Feb 5, 2023, 11:35:45 AM2/5/23
to modwsgi
Adding to my recent post:

It seems that the `request-timeout=30` is effective, after all: The error messages have changed, but there is no stack trace.
For completeness, here is the site config and error log excerpt:

Site config:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html

    # Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
    # error, crit, alert, emerg.
    # It is also possible to configure the loglevel for particular
    # modules, e.g.
    #LogLevel info ssl:warn
    LogLevel info

    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    # Siehe https://bz.apache.org/bugzilla/show_bug.cgi?id=45023
    # Hierfür muss mod_headers aktiv sein: sudo a2enmod headers
    RequestHeader edit "If-None-Match" '^"((.*)-gzip)"$' '"$1", "$2"'

    Alias /static/ /var/www/HallCam-static/
    <Directory /var/www/HallCam-static>
        Require all granted
    </Directory>

    Alias /media/ /var/www/HallCam-media/
    <Directory /var/www/HallCam-media>
        Require all granted
    </Directory>

    WSGIDaemonProcess cf-hallcam-site request-timeout=30 user=carsten group=carsten processes=2 display-name=%{GROUP} python-home=/home/carsten/.virtualenvs/HallCam-web python-path=/home/carsten/HallCam/web
    WSGIProcessGroup cf-hallcam-site
    # WSGIApplicationGroup %{GLOBAL}

    WSGIScriptAlias / /home/carsten/HallCam/web/HallCam/wsgi.py
    <Directory /home/carsten/HallCam/web/HallCam>
        <Files wsgi.py>
            Require all granted
        </Files>
    </Directory>
</VirtualHost>


From `/var/log/apache2/error.log`:

[Sun Feb 05 17:18:55.470532 2023] [mpm_event:notice] [pid 635:tid 140277991982976] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Sun Feb 05 17:18:55.690430 2023] [core:notice] [pid 635:tid 140277991982976] AH00094: Command line: '/usr/sbin/apache2'
[Sun Feb 05 17:18:56.338443 2023] [wsgi:info] [pid 638:tid 140277991982976] mod_wsgi (pid=638): Attach interpreter ''.
[Sun Feb 05 17:18:56.342866 2023] [wsgi:info] [pid 636:tid 140277991982976] mod_wsgi (pid=636): Attach interpreter ''.
[Sun Feb 05 17:18:56.366079 2023] [wsgi:info] [pid 636:tid 140277991982976] mod_wsgi (pid=636): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:18:56.371777 2023] [wsgi:info] [pid 638:tid 140277991982976] mod_wsgi (pid=638): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:19:24.823860 2023] [wsgi:info] [pid 636:tid 140277959865920] mod_wsgi (pid=636): Create interpreter '192.168.1.222:32228|'.
[Sun Feb 05 17:19:24.837012 2023] [wsgi:info] [pid 636:tid 140277959865920] mod_wsgi (pid=636): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:19:24.837752 2023] [wsgi:info] [pid 636:tid 140277959865920] [remote 88.75.25.178:50558] mod_wsgi (pid=636, process='cf-hallcam-site', application='192.168.1.222:32228|'): Loading Python script file '/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Sun Feb 05 17:19:43.356895 2023] [wsgi:info] [pid 638:tid 140277867546176] mod_wsgi (pid=638): Create interpreter '192.168.1.222:32228|'.
[Sun Feb 05 17:19:43.369787 2023] [wsgi:info] [pid 638:tid 140277867546176] mod_wsgi (pid=638): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:19:43.370532 2023] [wsgi:info] [pid 638:tid 140277867546176] [remote 88.75.25.178:48210] mod_wsgi (pid=638, process='cf-hallcam-site', application='192.168.1.222:32228|'): Loading Python script file '/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Sun Feb 05 17:23:21.631286 2023] [wsgi:info] [pid 636:tid 140277976651328] mod_wsgi (pid=636): Daemon process request time limit exceeded, stopping process 'cf-hallcam-site'.
[Sun Feb 05 17:23:21.631405 2023] [wsgi:info] [pid 636:tid 140277991982976] mod_wsgi (pid=636): Shutdown requested 'cf-hallcam-site'.
[Sun Feb 05 17:23:26.631650 2023] [wsgi:info] [pid 636:tid 140277591307840] mod_wsgi (pid=636): Aborting process 'cf-hallcam-site'.
[Sun Feb 05 17:23:26.631699 2023] [wsgi:info] [pid 636:tid 140277591307840] mod_wsgi (pid=636): Exiting process 'cf-hallcam-site'.
[Sun Feb 05 17:23:26.714616 2023] [wsgi:error] [pid 639:tid 140277725460032] [client 88.75.25.178:48224] Truncated or oversized response headers received from daemon process 'cf-hallcam-site': /home/carsten/HallCam/web/HallCam/wsgi.py, referer: http://vdzuggmrroo5k7e9.myfritz.net:32228/upload/
[Sun Feb 05 17:23:26.714952 2023] [wsgi:error] [pid 639:tid 140277959865920] (104)Connection reset by peer: [client 88.75.25.178:50558] mod_wsgi (pid=639): Failed to proxy response from daemon., referer: http://vdzuggmrroo5k7e9.myfritz.net:32228/upload/
[Sun Feb 05 17:23:26.983714 2023] [wsgi:info] [pid 885:tid 140277991982976] mod_wsgi (pid=885): Attach interpreter ''.
[Sun Feb 05 17:23:26.994980 2023] [wsgi:info] [pid 885:tid 140277991982976] mod_wsgi (pid=885): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:23:37.641476 2023] [wsgi:info] [pid 638:tid 140277976651328] mod_wsgi (pid=638): Daemon process request time limit exceeded, stopping process 'cf-hallcam-site'.
[Sun Feb 05 17:23:37.641609 2023] [wsgi:info] [pid 638:tid 140277991982976] mod_wsgi (pid=638): Shutdown requested 'cf-hallcam-site'.
[Sun Feb 05 17:23:42.641863 2023] [wsgi:info] [pid 638:tid 140277591307840] mod_wsgi (pid=638): Aborting process 'cf-hallcam-site'.
[Sun Feb 05 17:23:42.641910 2023] [wsgi:info] [pid 638:tid 140277591307840] mod_wsgi (pid=638): Exiting process 'cf-hallcam-site'.
[Sun Feb 05 17:23:42.648248 2023] [wsgi:error] [pid 639:tid 140277717067328] [client 37.81.109.237:49336] Truncated or oversized response headers received from daemon process 'cf-hallcam-site': /home/carsten/HallCam/web/HallCam/wsgi.py
[Sun Feb 05 17:23:42.648624 2023] [wsgi:error] [pid 639:tid 140277733852736] (104)Connection reset by peer: [client 88.75.25.178:48210] mod_wsgi (pid=639): Failed to proxy response from daemon., referer: http://vdzuggmrroo5k7e9.myfritz.net:32228/
[Sun Feb 05 17:23:42.997522 2023] [wsgi:info] [pid 906:tid 140277991982976] mod_wsgi (pid=906): Attach interpreter ''.
[Sun Feb 05 17:23:43.039478 2023] [wsgi:info] [pid 906:tid 140277991982976] mod_wsgi (pid=906): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:25:01.761551 2023] [wsgi:info] [pid 885:tid 140277959865920] mod_wsgi (pid=885): Create interpreter '192.168.1.222:32228|'.
[Sun Feb 05 17:25:01.774488 2023] [wsgi:info] [pid 885:tid 140277959865920] mod_wsgi (pid=885): Adding '/home/carsten/HallCam/web' to path.
[Sun Feb 05 17:25:01.775225 2023] [wsgi:info] [pid 885:tid 140277959865920] [remote 37.81.109.237:49338] mod_wsgi (pid=885, process='cf-hallcam-site', application='192.168.1.222:32228|'): Loading Python script file '/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Sun Feb 05 17:25:03.155261 2023] [wsgi:error] [pid 885:tid 140277959865920] [remote 37.81.109.237:49338] Picture saved to /var/www/HallCam-media/pictures/camera-1/pic_20230205_172500_4.jpg (image/jpeg, 896262 bytes, camera camera-1)
[Sun Feb 05 17:30:03.323110 2023] [wsgi:info] [pid 885:tid 140277976651328] mod_wsgi (pid=885): Daemon process deadlock timer expired, stopping process 'cf-hallcam-site'.
[Sun Feb 05 17:30:03.323253 2023] [wsgi:info] [pid 885:tid 140277991982976] mod_wsgi (pid=885): Shutdown requested 'cf-hallcam-site'.

Graham Dumpleton

unread,
Feb 5, 2023, 5:32:29 PM2/5/23
to mod...@googlegroups.com
Two points here to clarify.

In your case the final error is:

  Daemon process deadlock timer expired

This means that the full Python interpreter locked up. In this case the request timeout may not apply and the feature where by stack traces can be dumped might not happen. Depends on what lead up to the issue.

By default the deadlock timeout is 300 seconds.

deadlock-timeout=sss

Defines the maximum number of seconds allowed to pass before the daemon process is shutdown and restarted after a potential deadlock on the Python GIL has been detected. The default is 300 seconds.

This option exists to combat the problem of a daemon process freezing as the result of a rogue Python C extension module which doesn’t properly release the Python GIL when entering into a blocking or long running operation.

This can occur when you are using third party Python packages which aren't designed to work in Python sub interpreters. More details in:
The solution as you found is to set:

  WSGIApplicationGroup %{GLOBAL}

So likely when you updated Python versions, some third party package you are using has shifted to a newer version which breaks in this respect, or which runs code different when using newer Python version.

As to request timeout if it were coming into play, the timeout is 30 seconds when you have daemon processes which are single threaded.

For multithreaded daemon process the timeout can trigger after 30 seconds as uses an average across all active request handler threads.

request-timeout=sss

Defines the maximum number of seconds that a request is allowed to run before the daemon process is restarted. This can be used to recover from a scenario where a request blocks indefinitely, and where if all request threads were consumed in this way, would result in the whole WSGI application process being blocked.

How this option is seen to behave is different depending on whether a daemon process uses only one thread, or more than one thread for handling requests, as set by the threads option.

If there is only a single thread, and so the process can only handle one request at a time, as soon as the timeout has passed, a restart of the process will be initiated.

If there is more than one thread, the request timeout is applied to the average running time for any requests, across all threads. This means that a request can run longer than the request timeout. This is done to reduce the possibility of interupting other running requests, and causing a user to see a failure. So where there is still capacity to handle more requests, restarting of the process will be delayed if possible.


In your case though because hitting a full lock up of the Python interpreter, even though the request timeout may still have triggered, it couldn't dump stack traces as doing so requires getting the global Python interpreter lock, which wasn't being released and thus threads to dump stack traces got stuck.

To understand the cause further and identify which third party package may be the problem, I would need to see what extra Python packages you are installing. Note that numpy can cause this, so if using anything which relies on that, that could be the reason.

Graham

Carsten Fuchs

unread,
Feb 6, 2023, 3:14:08 PM2/6/23
to mod...@googlegroups.com

Hello Graham,

many thanks for your detailed reply!

I must add that my Django project is a simple webcam application: A webcam sends POST requests to upload the images, the views show them. A single webcam is connected over a very slow internet connection (32 kBit/s), so a single POST can take several minutes. However, that never was and still doesn't seem to be a problem: Uploads still succeed, it's only the GET requests that fail. The GET requests in turn do more than just sending some HTML with the image URLs: especially they use the Pillow library to lazily produce thumbnails, put timestamps into the images, etc.

Therefore I thought that Pillow (9.4.0) might be the culprit, so I removed the related code, made a new virtuelenv without Pillow from scratch and restarted Apache. Unfortunately, it didn't help – please see the error log below. The timeout occurs only after the browser has received a large portion, possibly all, of the view's response.

`pip list` shows

Package              Version
-------------------- -------
asgiref              3.6.0
Django               4.1.5
django-improved-user 2.0a2
django-widget-tweaks 1.4.12
pip                  22.0.2
python-dateutil      2.8.2
setuptools           59.6.0
six                  1.16.0
sqlparse             0.4.3

Is any of these know for causing problems like numpy?

Best regards,
Carsten

The above mentioned error log:

[Mon Feb 06 20:52:20.329545 2023] [mpm_event:notice] [pid 25844:tid 140714434877312] AH00489: Apache/2.4.52 (Ubuntu) mod_wsgi/4.9.0 Python/3.10 configured -- resuming normal operations
[Mon Feb 06 20:52:20.330207 2023] [core:notice] [pid 25844:tid 140714434877312] AH00094: Command line: '/usr/sbin/apache2'
[Mon Feb 06 20:52:20.380741 2023] [wsgi:info] [pid 25845:tid 140714434877312] mod_wsgi (pid=25845): Attach interpreter ''.
[Mon Feb 06 20:52:20.395024 2023] [wsgi:info] [pid 25846:tid 140714434877312] mod_wsgi (pid=25846): Attach interpreter ''.
[Mon Feb 06 20:52:20.395771 2023] [wsgi:info] [pid 25845:tid 140714434877312] mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 20:52:20.411711 2023] [wsgi:info] [pid 25846:tid 140714434877312] mod_wsgi (pid=25846): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 20:52:48.621284 2023] [wsgi:info] [pid 25845:tid 140714369189440] mod_wsgi (pid=25845): Create interpreter '192.168.1.222:32228|'.
[Mon Feb 06 20:52:48.634464 2023] [wsgi:info] [pid 25845:tid 140714369189440] mod_wsgi (pid=25845): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 20:52:48.635248 2023] [wsgi:info] [pid 25845:tid 140714369189440] [remote 88.75.25.178:42162] mod_wsgi (pid=25845, process='cf-hallcam-site', application='192.168.1.222:32228|'): Loading Python script file '/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Mon Feb 06 20:57:40.661043 2023] [wsgi:info] [pid 25845:tid 140714419545664] mod_wsgi (pid=25845): Daemon process request time limit exceeded, stopping process 'cf-hallcam-site'.
[Mon Feb 06 20:57:40.661184 2023] [wsgi:info] [pid 25845:tid 140714434877312] mod_wsgi (pid=25845): Shutdown requested 'cf-hallcam-site'.
[Mon Feb 06 20:57:45.661421 2023] [wsgi:info] [pid 25845:tid 140714177934912] mod_wsgi (pid=25845): Aborting process 'cf-hallcam-site'.
[Mon Feb 06 20:57:45.661464 2023] [wsgi:info] [pid 25845:tid 140714177934912] mod_wsgi (pid=25845): Exiting process 'cf-hallcam-site'.
[Mon Feb 06 20:57:45.666987 2023] [wsgi:error] [pid 25847:tid 140714226689600] [client 37.81.109.237:49940] Truncated or oversized response headers received from daemon process 'cf-hallcam-site': /home/carsten/HallCam/web/HallCam/wsgi.py
[Mon Feb 06 20:57:45.667005 2023] [wsgi:error] [pid 25848:tid 140714335794752] (104)Connection reset by peer: [client 88.75.25.178:42162] mod_wsgi (pid=25848): Failed to proxy response from daemon., referer: http://vdzuggmrroo5k7e9.myfritz.net:32228/
[Mon Feb 06 20:57:46.621804 2023] [wsgi:info] [pid 25977:tid 140714434877312] mod_wsgi (pid=25977): Attach interpreter ''.
[Mon Feb 06 20:57:46.633564 2023] [wsgi:info] [pid 25977:tid 140714434877312] mod_wsgi (pid=25977): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 21:00:01.811479 2023] [wsgi:info] [pid 25846:tid 140714360796736] mod_wsgi (pid=25846): Create interpreter '192.168.1.222:32228|'.
[Mon Feb 06 21:00:01.824795 2023] [wsgi:info] [pid 25846:tid 140714360796736] mod_wsgi (pid=25846): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 21:00:01.825535 2023] [wsgi:info] [pid 25846:tid 140714360796736] [remote 37.81.109.237:49942] mod_wsgi (pid=25846, process='cf-hallcam-site', application='192.168.1.222:32228|'): Loading Python script file '/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Mon Feb 06 21:00:02.506063 2023] [wsgi:info] [pid 25977:tid 140714402760256] mod_wsgi (pid=25977): Create interpreter '192.168.1.222:32228|'.
[Mon Feb 06 21:00:02.519453 2023] [wsgi:info] [pid 25977:tid 140714402760256] mod_wsgi (pid=25977): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 21:00:02.520239 2023] [wsgi:info] [pid 25977:tid 140714402760256] [remote 88.75.25.178:43650] mod_wsgi (pid=25977, process='cf-hallcam-site', application='192.168.1.222:32228|'): Loading Python script file '/home/carsten/HallCam/web/HallCam/wsgi.py'.
[Mon Feb 06 21:00:03.299821 2023] [core:info] [pid 25847:tid 140714151155264] [client 88.75.25.178:43652] AH00128: File does not exist: /var/www/HallCam-media/thumbs/camera-2/pic_20230206_205532_4.jpg, referer: http://vdzuggmrroo5k7e9.myfritz.net:32228/
[Mon Feb 06 21:05:02.996169 2023] [wsgi:info] [pid 25977:tid 140714419545664] mod_wsgi (pid=25977): Daemon process deadlock timer expired, stopping process 'cf-hallcam-site'.
[Mon Feb 06 21:05:02.996308 2023] [wsgi:info] [pid 25977:tid 140714434877312] mod_wsgi (pid=25977): Shutdown requested 'cf-hallcam-site'.
[Mon Feb 06 21:05:07.996544 2023] [wsgi:info] [pid 25977:tid 140714177934912] mod_wsgi (pid=25977): Aborting process 'cf-hallcam-site'.
[Mon Feb 06 21:05:07.996587 2023] [wsgi:info] [pid 25977:tid 140714177934912] mod_wsgi (pid=25977): Exiting process 'cf-hallcam-site'.
[Mon Feb 06 21:05:08.002106 2023] [wsgi:error] [pid 25848:tid 140713647920704] [client 37.81.109.237:49944] Truncated or oversized response headers received from daemon process 'cf-hallcam-site': /home/carsten/HallCam/web/HallCam/wsgi.py
[Mon Feb 06 21:05:08.002137 2023] [wsgi:error] [pid 25848:tid 140714025395776] (104)Connection reset by peer: [client 88.75.25.178:43650] mod_wsgi (pid=25848): Failed to proxy response from daemon., referer: http://vdzuggmrroo5k7e9.myfritz.net:32228/
[Mon Feb 06 21:05:09.015715 2023] [wsgi:info] [pid 26028:tid 140714434877312] mod_wsgi (pid=26028): Attach interpreter ''.
[Mon Feb 06 21:05:09.027591 2023] [wsgi:info] [pid 26028:tid 140714434877312] mod_wsgi (pid=26028): Adding '/home/carsten/HallCam/web' to path.
[Mon Feb 06 21:05:29.082057 2023] [wsgi:info] [pid 25846:tid 140714419545664] mod_wsgi (pid=25846): Daemon process deadlock timer expired, stopping process 'cf-hallcam-site'.
[Mon Feb 06 21:05:29.082193 2023] [wsgi:info] [pid 25846:tid 140714434877312] mod_wsgi (pid=25846): Shutdown requested 'cf-hallcam-site'.
[Mon Feb 06 21:05:34.082447 2023] [wsgi:info] [pid 25846:tid 140714177934912] mod_wsgi (pid=25846): Aborting process 'cf-hallcam-site'.
[Mon Feb 06 21:05:34.082493 2023] [wsgi:info] [pid 25846:tid 140714177934912] mod_wsgi (pid=25846): Exiting process 'cf-hallcam-site'.
[Mon Feb 06 21:05:35.038614 2023] [wsgi:info] [pid 26047:tid 140714434877312] mod_wsgi (pid=26047): Attach interpreter ''.
[Mon Feb 06 21:05:35.050453 2023] [wsgi:info] [pid 26047:tid 140714434877312] mod_wsgi (pid=26047): Adding '/home/carsten/HallCam/web' to path.



Am 05.02.23 um 23:32 schrieb Graham Dumpleton:

Graham Dumpleton

unread,
Feb 6, 2023, 6:55:22 PM2/6/23
to mod...@googlegroups.com
Go into the lib/python3.10/site-packages directory of your Python virtual environment. What .so files exist in that directory? These are the packages with C extensions. It is only code in C extensions that should normally be able to trigger a Python deadlock.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

Carsten Fuchs

unread,
Feb 7, 2023, 10:55:51 AM2/7/23
to mod...@googlegroups.com

Hello,

hmmm. There are no .so files there:

$ pwd
/home/carsten/.virtualenvs/HallCam-web/lib/python3.10/site-packages

$ dir
insgesamt 120
drwxrwxr-x  3 carsten carsten  4096 Feb  6 20:41 asgiref
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 asgiref-3.6.0.dist-info
drwxrwxr-x  6 carsten carsten  4096 Feb  6 20:41 dateutil
drwxrwxr-x  3 carsten carsten  4096 Feb  6 20:40 _distutils_hack
drwxrwxr-x 18 carsten carsten  4096 Feb  6 20:41 django
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 Django-4.1.5.dist-info
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 django_improved_user-2.0a2.dist-info
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 django_widget_tweaks-1.4.12.dist-info
drwxrwxr-x  4 carsten carsten  4096 Feb  6 20:41 improved_user
drwxrwxr-x  5 carsten carsten  4096 Feb  6 20:40 pip
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:40 pip-22.0.2.dist-info
drwxrwxr-x  6 carsten carsten  4096 Feb  6 20:40 pkg_resources
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 __pycache__
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 python_dateutil-2.8.2.dist-info
drwxrwxr-x  7 carsten carsten  4096 Feb  6 20:40 setuptools
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:40 setuptools-59.6.0.dist-info
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 six-1.16.0.dist-info
drwxrwxr-x  5 carsten carsten  4096 Feb  6 20:41 sqlparse
drwxrwxr-x  2 carsten carsten  4096 Feb  6 20:41 sqlparse-0.4.3.dist-info
drwxrwxr-x  4 carsten carsten  4096 Feb  6 20:41 widget_tweaks
-rw-rw-r--  1 carsten carsten   152 Feb  6 20:40 distutils-precedence.pth
-rw-rw-r--  1 carsten carsten 34549 Feb  6 20:41 six.py

$ find -iname "*.so"

The last command returned no output, but I additionally looked into the directories also in many other ways to make sure that I didn't run a wrong search. Hmmm.

Best regards,
Carsten

Am 07.02.23 um 00:55 schrieb Graham Dumpleton:

Graham Dumpleton

unread,
Feb 7, 2023, 4:42:35 PM2/7/23
to mod...@googlegroups.com
Is any of your code using ctypes module for Apache, or any other module which is used to call into C code by writing Python code only?

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

Graham Dumpleton

unread,
Feb 7, 2023, 4:45:01 PM2/7/23
to mod...@googlegroups.com
Only other thing can suggest if you know point at which it is hanging, is to test and extract C stack trace by attaching gdb to the running process. This may or may not work depending on your system setup and what tools are installed.


That may give some clues.

Graham

Carsten Fuchs

unread,
Feb 8, 2023, 3:02:10 PM2/8/23
to mod...@googlegroups.com
Ok, I got a stack trace. My understanding of this is limited, but the thread seems to be stuck at polling something without timeout? (gdb reports that file ../sysdeps/unix/sysv/linux/poll.c cannot be found)

The full session follows. (Using plaintext email today in the hope that this helps with readibility. Please let me know if I should send differently instead.)
Between the two "thread apply all bt" commands I waited about two minutes:


$ sudo gdb /usr/sbin/apache2 60501
GNU gdb (Ubuntu 12.1-0ubuntu1~22.04) 12.1
Copyright (C) 2022 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Type "show copying" and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<https://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.

For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/sbin/apache2...
(No debugging symbols found in /usr/sbin/apache2)
Attaching to program: /usr/sbin/apache2, process 60501
[New LWP 60532]
[New LWP 60533]
[New LWP 60534]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
29 ../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht gefunden.
(gdb) thread apply all bt

Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"):
#0 __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=-1359973880, abstime=0x7f68ad6f7220, clockid=-1359689157, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f68ad6f7220, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68ad6f7220, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68ad6f7220) at ./nptl/pthread_cond_wait.c:652
#5 0x00007f68aefe5df5 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#6 0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#7 0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#8 0x00007f68ac30d6e3 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#9 0x00007f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0
#10 0x00007f68a750b2d6 in ?? () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0
#11 0x00007f68ac3087d6 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#12 0x00007f68aef118ba in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#13 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#14 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#15 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#16 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#17 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#18 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#19 0x00007f68aee9c9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#20 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#21 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#22 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#23 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#24 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#25 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#26 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#27 0x00007f68aef08118 in PyVectorcall_Call () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#28 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#29 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#30 0x00007f68aee9962e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#31 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#32 0x00007f68aef08618 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#33 0x00007f68aee99458 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#34 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#35 0x00007f68aef086ac in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#36 0x00007f68af41680f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#37 0x00007f68af41c9c8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#38 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#39 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f68adefa640 (LWP 60533) "apache2"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f68adef9d80, op=137, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7f68adef9d80, clockid=0, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f68adef9d80, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68adef9d80, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68adef9d80) at ./nptl/pthread_cond_wait.c:652
#5 0x00007f68aefe5df5 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#6 0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#7 0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#8 0x00007f68af41cf30 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#9 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f68ae6fb640 (LWP 60532) "apache2"):
#0 0x00007f68af72e7ed in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7f68ae6fad60) at ../sysdeps/unix/sysv/linux/select.c:69
#1 0x00007f68af865c09 in apr_sleep () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#2 0x00007f68af41d11c in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#3 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f68af59a780 (LWP 60501) "apache2"):
#0 0x00007f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f68af85e0fb in apr_poll () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f68af4222d8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#3 0x00007f68af423f1f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#4 0x000055c9771cc73d in ap_run_pre_mpm ()
#5 0x00007f68af478455 in ?? () from target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x000055c9771c30e8 in ap_run_mpm ()
#7 0x000055c9771c2609 in main ()
(gdb)
(gdb)
(gdb)
(gdb)
(gdb) thread apply all bt

Thread 4 (Thread 0x7f68ad6f9640 (LWP 60534) "apache2"):
#0 __futex_abstimed_wait_common64 (private=-1359973880, cancel=true, abstime=0x7f68ad6f7220, op=137, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=-1359973880, abstime=0x7f68ad6f7220, clockid=-1359689157, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f68ad6f7220, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68ad6f7220, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68ad6f7220) at ./nptl/pthread_cond_wait.c:652
#5 0x00007f68aefe5df5 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#6 0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#7 0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#8 0x00007f68ac30d6e3 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#9 0x00007f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0
#10 0x00007f68a750b2d6 in ?? () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0
#11 0x00007f68ac3087d6 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#12 0x00007f68aef118ba in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#13 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#14 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#15 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#16 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#17 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#18 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#19 0x00007f68aee9c9b8 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#20 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#21 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#22 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#23 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#24 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#25 0x00007f68aee9aeee in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#26 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#27 0x00007f68aef08118 in PyVectorcall_Call () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#28 0x00007f68aee97766 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#29 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#30 0x00007f68aee9962e in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#31 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#32 0x00007f68aef08618 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#33 0x00007f68aee99458 in _PyEval_EvalFrameDefault () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#34 0x00007f68aefe781f in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#35 0x00007f68aef086ac in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#36 0x00007f68af41680f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#37 0x00007f68af41c9c8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#38 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#39 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 3 (Thread 0x7f68adefa640 (LWP 60533) "apache2"):
#0 __futex_abstimed_wait_common64 (private=0, cancel=true, abstime=0x7f68adef9d80, op=137, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:57
#1 __futex_abstimed_wait_common (cancel=true, private=0, abstime=0x7f68adef9d80, clockid=0, expected=0, futex_word=0x7f68af3b49a8 <_PyRuntime+424>) at ./nptl/futex-internal.c:87
#2 __GI___futex_abstimed_wait_cancelable64 (futex_word=futex_word@entry=0x7f68af3b49a8 <_PyRuntime+424>, expected=expected@entry=0, clockid=clockid@entry=1, abstime=abstime@entry=0x7f68adef9d80, private=private@entry=0) at ./nptl/futex-internal.c:139
#3 0x00007f68af6a6f1b in __pthread_cond_wait_common (abstime=0x7f68adef9d80, clockid=1, mutex=0x7f68af3b49b0 <_PyRuntime+432>, cond=0x7f68af3b4980 <_PyRuntime+384>) at ./nptl/pthread_cond_wait.c:503
#4 ___pthread_cond_timedwait64 (cond=0x7f68af3b4980 <_PyRuntime+384>, mutex=0x7f68af3b49b0 <_PyRuntime+432>, abstime=0x7f68adef9d80) at ./nptl/pthread_cond_wait.c:652
#5 0x00007f68aefe5df5 in ?? () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#6 0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#7 0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#8 0x00007f68af41cf30 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#9 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#10 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 2 (Thread 0x7f68ae6fb640 (LWP 60532) "apache2"):
#0 0x00007f68af72e7ed in __GI___select (nfds=0, readfds=0x0, writefds=0x0, exceptfds=0x0, timeout=0x7f68ae6fad60) at ../sysdeps/unix/sysv/linux/select.c:69
#1 0x00007f68af865c09 in apr_sleep () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
--Type <RET> for more, q to quit, c to continue without paging--
#2 0x00007f68af41d11c in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#3 0x00007f68af6a7b43 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:442
#4 0x00007f68af739a00 in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81

Thread 1 (Thread 0x7f68af59a780 (LWP 60501) "apache2"):
#0 0x00007f68af72bd7f in __GI___poll (fds=0x7ffe9f0b9e90, nfds=1, timeout=-1) at ../sysdeps/unix/sysv/linux/poll.c:29
#1 0x00007f68af85e0fb in apr_poll () from target:/lib/x86_64-linux-gnu/libapr-1.so.0
#2 0x00007f68af4222d8 in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#3 0x00007f68af423f1f in ?? () from target:/usr/lib/apache2/modules/mod_wsgi.so
#4 0x000055c9771cc73d in ap_run_pre_mpm ()
#5 0x00007f68af478455 in ?? () from target:/usr/lib/apache2/modules/mod_mpm_event.so
#6 0x000055c9771c30e8 in ap_run_mpm ()
#7 0x000055c9771c2609 in main ()
(gdb)


Best regards,
Carsten


Am 07.02.23 um 22:44 schrieb Graham Dumpleton:
> Only other thing can suggest if you know point at which it is hanging, is to test and extract C stack trace by attaching gdb to the running process. This may or may not work depending on your system setup and what tools are installed.
>
> Debugging Techniques — mod_wsgi 4.9.4 documentation <https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb>
> modwsgi.readthedocs.io <https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb>
> favicon.ico <https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb>
>
> <https://modwsgi.readthedocs.io/en/master/user-guides/debugging-techniques.html#debugging-crashes-with-gdb>

Graham Dumpleton

unread,
Feb 8, 2023, 3:09:44 PM2/8/23
to mod...@googlegroups.com
The key bit of information in that is the call to sqlite3LeaveMutexAndCloseZombie(). It looks like sqlite module for Python is doing something really strange that isn't working with sub interpreters. This will need some research as it may be a bug they have introduced in Python 3.10 for the sqlite module.

Graham
> --
> You received this message because you are subscribed to the Google Groups "modwsgi" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/a09f8380-eddd-e859-911b-5919d44ea610%40cafu.de.

Graham Dumpleton

unread,
Feb 8, 2023, 3:23:23 PM2/8/23
to mod...@googlegroups.com
More specifically, if you look at the stack trace you see:

#6  0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#7  0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#8  0x00007f68ac30d6e3 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#9  0x00007f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0

Note how that function ends up calling PyGILState_Ensure(). That is a big no no when working with Python sub interpreters.

In other words, it uses the simplified GIL state API for some callback into Python from C code of SQLite and using that  when in a sub interpreter can cause deadlocks.

favicon.ico

So need to find the code for that function in sqlite module and work out what it does.

Graham

Graham Dumpleton

unread,
Feb 8, 2023, 3:29:19 PM2/8/23
to mod...@googlegroups.com
Possibly related issues:


There are possibly some known issues with sub interpreters in code used by sqlite module there.

What exact version of Python 3.10 as you using? Can you update to latest patch revision of it?

Graham

On 9 Feb 2023, at 7:22 am, Graham Dumpleton <Graham.D...@gmail.com> wrote:

More specifically, if you look at the stack trace you see:

#6  0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#7  0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
#8  0x00007f68ac30d6e3 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#9  0x00007f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0

Note how that function ends up calling PyGILState_Ensure(). That is a big no no when working with Python sub interpreters.

In other words, it uses the simplified GIL state API for some callback into Python from C code of SQLite and using that  when in a sub interpreter can cause deadlocks.

Carsten Fuchs

unread,
Feb 8, 2023, 3:53:34 PM2/8/23
to mod...@googlegroups.com
Many thanks for your help and the detailed information!
My Python version is 3.10.6.
I'll try to switch to the latest point release next, but it will a couple of days before I can do it. Will let you know the results as soon as I got it done.

Best regards,
Carsten



Am 08.02.23 um 21:28 schrieb Graham Dumpleton:
> Possibly related issues:
>
> https://bugs.archlinux.org/task/72979 <https://bugs.archlinux.org/task/72979> 
> https://bugs.python.org/issue46006 <https://bugs.python.org/issue46006>
>
> There are possibly some known issues with sub interpreters in code used by sqlite module there.
>
> What exact version of Python 3.10 as you using? Can you update to latest patch revision of it?
>
> Graham
>
>> On 9 Feb 2023, at 7:22 am, Graham Dumpleton <Graham.D...@gmail.com> wrote:
>>
>> More specifically, if you look at the stack trace you see:
>>
>> #6  0x00007f68aefe6292 in PyEval_RestoreThread () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>> #7  0x00007f68af02f1dc in PyGILState_Ensure () from target:/lib/x86_64-linux-gnu/libpython3.10.so.1.0
>> #8  0x00007f68ac30d6e3 in ?? () from target:/usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
>> #9  0x00007f68a750b01a in sqlite3LeaveMutexAndCloseZombie () from target:/lib/x86_64-linux-gnu/libsqlite3.so.0
>>
>> Note how that function ends up calling PyGILState_Ensure(). That is a big no no when working with Python sub interpreters.
>>
>> In other words, it uses the simplified GIL state API for some callback into Python from C code of SQLite and using that  when in a sub interpreter can cause deadlocks.
>>
>> Application Issues — mod_wsgi 4.9.4 documentation <https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api>
>> modwsgi.readthedocs.io <https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api>
>> <favicon.ico> <https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api>
>>
>> <https://modwsgi.readthedocs.io/en/master/user-guides/application-issues.html#python-simplified-gil-state-api>
>>>> 29../sysdeps/unix/sysv/linux/poll.c: Datei oder Verzeichnis nicht gefunden.
> You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/bntPYwPnsU8/unsubscribe <https://groups.google.com/d/topic/modwsgi/bntPYwPnsU8/unsubscribe>.
> To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com <mailto:modwsgi+u...@googlegroups.com>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/7715AEF5-6F1F-407A-B899-EB4871707856%40gmail.com <https://groups.google.com/d/msgid/modwsgi/7715AEF5-6F1F-407A-B899-EB4871707856%40gmail.com?utm_medium=email&utm_source=footer>.

--
Dipl.-Inf. Carsten Fuchs
Industriegebiet 3 ℅ Rofu
55768 Hoppstädten-Weiersbach
https://www.cafu.de

Carsten Fuchs

unread,
Feb 9, 2023, 1:53:33 AM2/9/23
to mod...@googlegroups.com
Argh, sorry, I think I may have spoken too soon: The deadsnakes PPA does not contain Python 3.10 for Ubuntu 22.04 and I'm not really prepared to compile from source.

Would it help you, mod_wsgi or someone else if I pursued installing the latest Python 3.10 point release for further testing?

Otherwise, now that the reason for the problem is clear, I consider to just use the `WSGIApplicationGroup %{GLOBAL}` or switch to another database in order to evade the issue.

Best regards,
Carsten


Am 08.02.23 um 21:53 schrieb Carsten Fuchs:

Graham Dumpleton

unread,
Feb 9, 2023, 7:08:12 PM2/9/23
to mod...@googlegroups.com
-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

Carsten Fuchs

unread,
Feb 13, 2023, 3:45:54 AM2/13/23
to mod...@googlegroups.com
Hello Graham,

many thanks for your kind and detailed help!

Best regards,
Carsten


Am 10.02.23 um 01:07 schrieb Graham Dumpleton:
> Use WSGIApplicationGroup to force the use of the main Python sub interpreter. That is only safe way to deal with it.
>
> I have created issue against CPython itself to separately highlight the problem.
>
> 101767.png
> sqlite callback functions can deadlock in Python subinterpreters · Issue #101767 · python/cpython <https://github.com/python/cpython/issues/101767>
> github.com <https://github.com/python/cpython/issues/101767>
>
> <https://github.com/python/cpython/issues/101767>
>
> You might also look at:
>
> 59956.png
> [subinterpreters] Make the PyGILState API compatible with subinterpreters · Issue #59956 · python/cpython <https://github.com/python/cpython/issues/59956>
> github.com <https://github.com/python/cpython/issues/59956>
>
> <https://github.com/python/cpython/issues/59956>
Reply all
Reply to author
Forward
0 new messages