end of script output before headers: wsgi.py

[Clay Templeton]

Hello, hoping very dearly that someone might solve this mystery for me. This is a new issue that crept into a working codebase while I wasn't looking.  Issue: Apache hangs for several minutes then sends 'Internal Server Error' message. Error log shows  'End of script output before headers: wsgi.py'. Details follow:

Error Message sent to browser looks like this: 

Internal Server Error

The server encountered an internal error or misconfiguration and was unable to complete your request.

Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.

More information about this error may be available in the server error log.

Main Apache error log looks like this: 

[Tue Nov 29 23:44:17.038082 2016] [core:notice] [pid 8164] SELinux policy enabled; httpd running as context system_u:system_r:httpd_t:s0

[Tue Nov 29 23:44:17.038701 2016] [suexec:notice] [pid 8164] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)

AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using localhost.localdomain. Set the 'ServerName' directive globally to suppress this message

[Tue Nov 29 23:44:17.050318 2016] [auth_digest:notice] [pid 8164] AH01757: generating secret for digest authentication ...

[Tue Nov 29 23:44:17.051277 2016] [lbmethod_heartbeat:notice] [pid 8164] AH02282: No slotmem from mod_heartmonitor

[Tue Nov 29 23:44:17.054149 2016] [mpm_prefork:notice] [pid 8164] AH00163: Apache/2.4.6 (CentOS) mod_wsgi/3.4 Python/2.7.5 configured -- resuming normal operations

[Tue Nov 29 23:44:17.054181 2016] [core:notice] [pid 8164] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'

Virtual host error log looks like this: 

[Tue Nov 29 23:49:52.583315 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Attach interpreter ''.

[Tue Nov 29 23:49:52.583471 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Adding '/var/www/ims_web' to path.

[Tue Nov 29 23:49:52.583626 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Adding '/var/www/anaconda2/envs/ims_web/lib/python2.7/site-packages' to path.

[Tue Nov 29 23:50:13.060200 2016] [authz_core:debug] [pid 8227] mod_authz_core.c(809): [client 10.0.2.2:36474] AH01626: authorization result of Require all granted: granted

[Tue Nov 29 23:50:13.060236 2016] [authz_core:debug] [pid 8227] mod_authz_core.c(809): [client 10.0.2.2:36474] AH01626: authorization result of <RequireAny>: granted

[Tue Nov 29 23:50:13.060451 2016] [authz_core:debug] [pid 8227] mod_authz_core.c(809): [client 10.0.2.2:36474] AH01626: authorization result of Require all granted: granted

[Tue Nov 29 23:50:13.060465 2016] [authz_core:debug] [pid 8227] mod_authz_core.c(809): [client 10.0.2.2:36474] AH01626: authorization result of <RequireAny>: granted

[Tue Nov 29 23:50:13.071420 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Create interpreter 'localhost.localdomain:8080|/services'.

[Tue Nov 29 23:50:13.072845 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Adding '/var/www/ims_web' to path.

[Tue Nov 29 23:50:13.072970 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Adding '/var/www/anaconda2/envs/ims_web/lib/python2.7/site-packages' to path.

[Tue Nov 29 23:50:13.073484 2016] [:info] [pid 8224] [remote 10.0.2.2:168] mod_wsgi (pid=8224, process='main', application='localhost.localdomain:8080|/services'): Loading WSGI script '/var/www/ims_web/ims_web/wsgi.py'.

[Tue Nov 29 23:55:13.648316 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Daemon process deadlock timer expired, stopping process 'main'.

[Tue Nov 29 23:55:13.648461 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Shutdown requested 'main'.

[Tue Nov 29 23:55:18.648793 2016] [:info] [pid 8224] mod_wsgi (pid=8224): Aborting process 'main'.

[Tue Nov 29 23:55:18.656494 2016] [core:error] [pid 8227] [client 10.0.2.2:36474] End of script output before headers: wsgi.py

[Tue Nov 29 23:55:19.296001 2016] [:info] [pid 8264] mod_wsgi (pid=8264): Attach interpreter ''.

[Tue Nov 29 23:55:19.296134 2016] [:info] [pid 8264] mod_wsgi (pid=8264): Adding '/var/www/ims_web' to path.

[Tue Nov 29 23:55:19.296370 2016] [:info] [pid 8264] mod_wsgi (pid=8264): Adding '/var/www/anaconda2/envs/ims_web/lib/python2.7/site-packages' to path.

Main httpd.conf file is as out-of-the-box, except LogLevel has been changed to 'debug'. Custom .conf file looks like this: 

<VirtualHost *:80>

    Header set Access-Control-Allow-Origin "*"

    DocumentRoot /var/www/html/

    Alias /media /var/www/data

    Alias /static /var/www/static

    WSGIDaemonProcess main python-path=/var/www/ims_web:/var/www/anaconda2/envs/ims_web/lib/python2.7/site-packages

    WSGIProcessGroup main

    WSGIScriptAlias /services /var/www/ims_web/ims_web/wsgi.py

    ErrorLog /var/www/log/error_log 

    LogLevel debug

    <Directory /var/www/ims_web/static>

        Require all granted

    </Directory>

    <Directory /var/www/ims_web/ims_web>

        <Files wsgi.py>

            Require all granted

        </Files>

    </Directory>

</VirtualHost>

wsgi.py looks like this: 

import os

from django.core.wsgi import get_wsgi_application

os.environ.setdefault("DJANGO_SETTINGS_MODULE", "ims_web.settings.production")

application = get_wsgi_application()

Many Thanks

Clay

[Graham Dumpleton]

Try adding the directive:

WSGIApplicationGroup %{GLOBAL}

after the WSGIProcessGroup line.

There are known issues with various third party Python packages with C extension components that mean they will not work with Python sub interpreters properly. That directive forces the application to run in the main interpreter context. Such modules are then happy.

Symptoms of this being the problem are deadlocking code, or processes crashing. Docs mention it in:

    http://modwsgi.readthedocs.io/en/develop/user-guides/application-issues.html#python-simplified-gil-state-api

Also be aware that mod_wsgi version 3.4 is ridiculously out of date. Something like 5+ years old and over 30 versions behind. :-)

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Clay Templeton]

Thanks :-)