Daemon Process deadlock timer expired inspite of using WSGIApplicationGroup %{GLOBAL}

48 views
Skip to first unread message

Shakim Mohammed

unread,
Jan 22, 2019, 9:37:58 AM1/22/19
to modwsgi
Hello,

I have a Flask + Sqlalchemy application(connecting to a Sybase database) that I am trying to run in Apache + mod_wsgi setup

Few pointers on my wsgi setup:
  • Using WSGIDaemonProcess <nameOfApp> processes=x threads=y
  • Using WSGIApplicationGroup %{GLOBAL}
No matter what combination of values of x and y I use, after processing some request, my apache/mod_wsgi hangs up for sometime showing logs like :
  • Daemon process deadlock timer expired, stopping process <>
  • Shutdown process <nameOfApp>
  • Aborting process <nameOfApp>
  • Exiting process <nameOfApp>
I believe it hangs for 300 seconds(which is default deadlock-timeout)
If I set deadlock-timeout = 60, apache/mod_wsgi recovers in 60sec

Few other logs that I see are :
  • Truncated or oversized response headers received from daemon process <nameOfApp>
  • server seems busy, (you may need to increase StartServers, or Min/MaxServers, spawning 16 children, there are three idle
What could be the reason of this deadlock and other logs shown above?

Could this be because of a rogue C extension as pointed out by Graham at various places?

There are bunch of third party modules that I am using which has C extension like -> http://python-sybase.sourceforge.net/download.html

Is there anyway to fix this or make it more stable(not going into deadlock too frequently and giving 500 to clients)?

FYI, I am using prefork instead of worker at this moment. I am not loading unnecessary modules like mod_php.
The only modules that I am loading in my apache conf are "mod_authz_host.so", "mod_log_config.so" and "mod_wsgi.so"
I believe the issue I am trying to highlight is totally independent of whether it is a prefork or worker setup

Graham Dumpleton

unread,
Jan 22, 2019, 9:41:18 AM1/22/19
to mod...@googlegroups.com
Can you show the actual parts of the Apache configuration for mod_wsgi and your WSGI application. The clues you have given aren't really enough. Need to see the actual config to know if you have it correct. The actual log messages also are important as they confirm in what context the WSGI application runs. Ensure though you have LogLevel set to info to get most useful information.

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

Shakim Mohammed

unread,
Jan 22, 2019, 12:33:12 PM1/22/19
to modwsgi
Thank you very much for a quick reply:

I cannot copy the logs/configs from my firm but I will try my best to re-create(manual copy paste) it here

Apache Configuration:

LoadModule log_config_module <...../mod_log_config.so>
LoadModule wsgi_module <...../mod_wsgi.so>
LoadModule authz_host_module <..../mod_authz_host.so>

Listen 8080

PidFile <...path to pid file>

LogLevel Info
ErroLog /local/scratch/apache.err
TransferLog /local/scratch/apache.out

WSGIApplicationGroup %{GLOBAL}

WSGIRestrictEmbedded On

WSGISocketPrefix /local/scratch/run/wsgi

<VirtualHost *:8080>

    ServerName xyz.com

    WSGIDaemonProcess myapp processes=5 threads=5 display-name=%{GROUP} deadlock-timeout=60
    WSGIApplicationGroup %{GLOBAL}
    WSGIProcessGroup myapp
    WSGIScriptAlias / /local/scratch/....../bin/webapp-1.0.wsgi

    <Directory /local/scratch/....../bin>
        Order allow, deny
        Allow from all
    </Directory>

</VirtualHost>

Logs:

[info] mod_wsgi (pid=5038): Starting process 'myapp' with threads=5
[info] mod_wsgi (pid=5038): Initializing Python.
[notice] Apache/2.2.24 (Unix) mod_wsgi/4.3.2 Python/2.7.1 configured -- resuming normal operations
[info] Server built: Aug 2 2013 13.51.41
[info] mod_wsgi (pid=5040): Starting process 'myapp' with threads=5
[info] mod_wsgi (pid=5041): Starting process 'myapp' with threads=5
[info] mod_wsgi (pid=5040): Initializing Python.
[info] mod_wsgi (pid=5039): Starting process 'myapp' with threads=5
[info] mod_wsgi (pid=5039): Initializing Python.
[info] mod_wsgi (pid=5042): Starting process 'myapp' with threads=5
[info] mod_wsgi (pid=5041): Initializing Python.
[info] mod_wsgi (pid=5042): Initializing Python.
[info] mod_wsgi (pid=5038): Attaching interpreter ''.
[info] mod_wsgi (pid=5042): Attaching interpreter ''.
[info] mod_wsgi (pid=5041): Attaching interpreter ''.
[info] mod_wsgi (pid=5040): Attaching interpreter ''.
[info] mod_wsgi (pid=5039): Attaching interpreter ''.

[info] [client 10.79.29.2] mod_wsgi (pid=5038, process='myapp', application=''): Loading WSGI script '/local/scratch/....../bin/webapp-1.0.wsgi'

[info] [client 10.79.29.2] mod_wsgi (pid=5042, process='myapp', application=''): Loading WSGI script '/local/scratch/....../bin/webapp-1.0.wsgi'

[info] [client 10.79.29.2] mod_wsgi (pid=5040, process='myapp', application=''): Loading WSGI script '/local/scratch/....../bin/webapp-1.0.wsgi'

[info] [client 10.79.29.2] mod_wsgi (pid=5039, process='myapp', application=''): Loading WSGI script '/local/scratch/....../bin/webapp-1.0.wsgi'

[info] [client 10.79.29.2] mod_wsgi (pid=5041, process='myapp', application=''): Loading WSGI script '/local/scratch/....../bin/webapp-1.0.wsgi'

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................

[info] server seems busy, (you many need to increase StartServers, or Min/MaxSpareServers), spawning 8 children, there are 0 idle, and 12 total children

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................

[info] server seems busy, (you many need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 0 idle, and 20 total children

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................

[info] server seems busy, (you many need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 36 total children

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................

[info] server seems busy, (you many need to increase StartServers, or Min/MaxSpareServers), spawning 32 children, there are 0 idle, and 68 total children

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................

[info] mod_wsgi (pid=5038): Daemon process deadlock timer expired, stopping process 'myapp'
[info] mod_wsgi (pid=5038): Shutdown requested 'myapp'
[info] mod_wsgi (pid=5041): Daemon process deadlock timer expired, stopping process 'myapp'
[info] mod_wsgi (pid=5041): Shutdown requested 'myapp'
[info] mod_wsgi (pid=5042): Daemon process deadlock timer expired, stopping process 'myapp'
[info] mod_wsgi (pid=5042): Shutdown requested 'myapp'
[info] mod_wsgi (pid=5038): Aborting process 'myapp'
[info] mod_wsgi (pid=5038): Exiting process 'myapp'

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................

[info] mod_wsgi (pid=5038): Process 'myapp' had died, deregister and restart it
[info] mod_wsgi (pid=5038): Process 'myapp' has been deregistered and will no longer be monitored
[info] mod_wsgi (pid=5405): Starting process 'myapp' with threads=5
[info] mod_wsgi (pid=5405): Initializing Python.
[info] mod_wsgi (pid=5405): Attaching interpreter ''.

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................

[info] mod_wsgi (pid=5041): Aborting process 'myapp'
[info] mod_wsgi (pid=5041): Exiting process 'myapp'


[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................


[info] [client 10.79.29.2] mod_wsgi (pid=5405, process='myapp', application=''): Loading WSGI script '/local/scratch/....../bin/webapp-1.0.wsgi'


[info] mod_wsgi (pid=5042): Aborting process 'myapp'
[info] mod_wsgi (pid=5042): Exiting process 'myapp'

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................


[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

[error] [client 10.79.29.2] Truncated or oversized response headers received from daemon process 'myapp': /local/scratch/....../bin/webapp-1.0.wsgi

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................


[info] mod_wsgi (pid=5041): Process 'myapp' had died, deregister and restart it
[info] mod_wsgi (pid=5041): Process 'myapp' has been deregistered and will no longer be monitored
[info] mod_wsgi (pid=5042): Process 'myapp' had died, deregister and restart it
[info] mod_wsgi (pid=5042): Process 'myapp' has been deregistered and will no longer be monitored

................
................
[error] <All the print statements from python app is getting logged here as an error>
................
................

Please let me know if there is any more info required

bo li

unread,
Jun 30, 2020, 4:15:07 AM6/30/20
to modwsgi
I have the same problem

i also Using WSGIApplicationGroup %{GLOBAL}, but it appear " Daemon process deadlock timer expired, stopping process"

if u have solved the problem , can you reply to me ?

Thanks

在 2019年1月22日星期二 UTC+8下午10:37:58，Shakim Mohammed写道：

Graham Dumpleton

unread,
Jun 30, 2020, 4:16:31 AM6/30/20
to mod...@googlegroups.com
Provide the mod_wsgi configuration you are using so can verify you are in fact setting it up correctly.

Also check:


and indicate what you get.

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.

bo li

unread,
Jun 30, 2020, 5:18:05 AM6/30/20
to modwsgi
'''

LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

<VirtualHost *:8777>
    LogLevel info
    WSGIDaemonProcess ceilometer-api processes=4 threads=8 user=ceilometer group=ceilometer display-name=%{GROUP} python-path=/usr/lib/python2.7/site-packages
    WSGIProcessGroup ceilometer-api

    WSGIScriptReloading On
    WSGIScriptAlias / /usr/lib/python2.7/site-packages/ceilometer/api/app.wsgi

    WSGIApplicationGroup %{GLOBAL}

    <Directory "/usr/lib/python2.7/site-packages/ceilometer/api">
        <IfVersion >= 2.4>
            Require all granted
        </IfVersion>
        <IfVersion < 2.4>
            Order allow,deny
            Allow from all
        </IfVersion>
    </Directory>
    ErrorLog /dev/stdout
    CustomLog /dev/stdout combined
</VirtualHost>
'''

this is my wsgi configuraton.

is it correctly?

Thanks

在 2020年6月30日星期二 UTC+8下午4:16:31，Graham Dumpleton写道：
To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.

Graham Dumpleton

unread,
Jun 30, 2020, 7:11:52 PM6/30/20
to mod...@googlegroups.com
What instructions are telling you to set it up in this way?

That is, setting the WSGI script file to be a path inside of the installed Python installation. This is highly unconventional and wouldn't usually be done.

Also concerning that setting python-path to the site-packages of the Python installation. This is also wrong. If this is necessary it suggests that you mod_wsgi version may not even be compiled for Python 2.7, which may be part of the problem, as you would be mixing Python packages with extensions compiled for Python 2.7, with a mod_wsgi compiled for a different Python version.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/6e0c27b4-6bb1-4805-949e-b86ce05e4369o%40googlegroups.com.

bo li

unread,
Jul 1, 2020, 1:50:29 AM7/1/20
to modwsgi
Thank you very much for a quick reply.

so, how to fix it.

Do i need to make sure that my mod_wsgi version is compiled for python 2.7?

Can you give me some advice?

Thanks


在 2020年7月1日星期三 UTC+8上午7:11:52，Graham Dumpleton写道：
To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/6e0c27b4-6bb1-4805-949e-b86ce05e4369o%40googlegroups.com.

bo li

unread,
Jul 1, 2020, 2:24:05 AM7/1/20
to modwsgi
Excuse me.

In addition, the project now runs fine and deadlocks are occasionally generated. 

So, can you give us some solutions？

在 2020年7月1日星期三 UTC+8上午7:11:52，Graham Dumpleton写道：
What instructions are telling you to set it up in this way?
To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/6e0c27b4-6bb1-4805-949e-b86ce05e4369o%40googlegroups.com.

Graham Dumpleton

unread,
Jul 1, 2020, 4:42:54 AM7/1/20
to mod...@googlegroups.com
Use this test to see what Python version mod_wsgi is compiled for:


Show the results of the test.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/d9abac5a-c298-4186-889f-b6711f7e6d1do%40googlegroups.com.

bo li

unread,
Jul 1, 2020, 8:20:51 AM7/1/20
to modwsgi
Thanks for your reply;

I exec the command "ldd mod_wsgi.so":

()[root@ceilometer-api-8656fbc7cf-vd2pz /]# ldd /usr/lib64/httpd/modules/mod_wsgi.so
linux-vdso.so.1 =>  (0x00007ffc52373000)
libpython2.7.so.1.0 => /lib64/libpython2.7.so.1.0 (0x00007f528939b000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f528917f000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f5288f7b000)
libutil.so.1 => /lib64/libutil.so.1 (0x00007f5288d78000)
libm.so.6 => /lib64/libm.so.6 (0x00007f5288a76000)
libc.so.6 => /lib64/libc.so.6 (0x00007f52886b3000)
/lib64/ld-linux-x86-64.so.2 (0x00007f5289991000)

and my python version is 2.7.5:

()[root@ceilometer-api-8656fbc7cf-vd2pz /]# python --version
Python 2.7.5

this result matches each other. This suspicion does not seem to hold

Are there any other reasons for this bug？

Thanks a lot.

在 2020年7月1日星期三 UTC+8下午4:42:54，Graham Dumpleton写道：
To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/d9abac5a-c298-4186-889f-b6711f7e6d1do%40googlegroups.com.

Graham Dumpleton

unread,
Jul 1, 2020, 8:27:31 AM7/1/20
to mod...@googlegroups.com
If you did the test in:


like I already mentioned and the application group was an empty string, then it could just be an issue with ceilometer.

I would suggest asking the ceilometer developers.

To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/093db3b7-06cc-4fd0-989b-262c118e3fabo%40googlegroups.com.

bo li

unread,
Jul 1, 2020, 8:34:46 AM7/1/20
to modwsgi
thanks you very much for suggestion.

best wishes.

在 2020年7月1日星期三 UTC+8下午8:27:31，Graham Dumpleton写道：
To unsubscribe from this group and stop receiving emails from it, send an email to mod...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/093db3b7-06cc-4fd0-989b-262c118e3fabo%40googlegroups.com.

Reply all
Reply to author
Forward
0 new messages