Truncated or oversized response headers received from daemon process

[Erik Curiel]

Hello, I've read all the previous posts on this issue and tried all the suggested fixes, but nothing has worked, so I'm posting my info here to see whether you can help me solve the problem.  (By the way, thanks for all your great work!)  I'm running Django through Apache.

First, I'll say that before I upgraded a few things, I never had this problem.  The system set up for which there was no problems:

Mac OS X 10.11.6

Apache: 2.4.18

mod_wsgi: 4.4.21

Django: 1.9.7

python: 2.7.13

The only change I made was to upgrade Django to 1.10.4 (and simultaneously upgrade djangorestframework from 3.3.3 to 3.5.3).  That is when the problem started.  I then upgraded mod_wsgi to 4.5.11, but the problem continued. 

Here is my httpd config:

    WSGIApplicationGroup %{GLOBAL}

    WSGIDaemonProcess trunk.delphicloud.io user=erik group=staff processes=1 display-name=%{GROUP}

    WSGIProcessGroup trunk.delphicloud.io

    WSGIScriptAlias / /Users/erik/web/delphicloud.io/dc-django-app/trunk/delphi_cloud/delphicloud/wsgi.py

Here is the output of the apache error_log (with LogLevel = info):

[Thu Dec 29 22:27:29.937943 2016] [wsgi:info] [pid 72223:tid 140735160061952] mod_wsgi (pid=72223): Starting process 'trunk.delphicloud.io' with uid=502, gid=20 and threads=15.

[Thu Dec 29 22:27:29.938345 2016] [wsgi:info] [pid 72224:tid 140735160061952] mod_wsgi (pid=72224): Starting process 'prod-branch.delphicloud.io' with uid=502, gid=20 and threads=15.

[Thu Dec 29 22:27:29.938665 2016] [mpm_event:notice] [pid 69988:tid 140735160061

952] AH00489: Apache/2.4.18 (Unix) mod_wsgi/4.5.11 Python/2.7 configured -- resuming normal operations

[Thu Dec 29 22:27:29.938692 2016] [mpm_event:info] [pid 69988:tid 140735160061952] AH00490: Server built: Aug  9 2016 00:09:14

[Thu Dec 29 22:27:29.938716 2016] [core:notice] [pid 69988:tid 140735160061952] AH00094: Command line: '/usr/local/apache2/bin/httpd -f /Users/erik/web/apache2/httpd.conf'

[Thu Dec 29 22:27:29.939548 2016] [wsgi:info] [pid 72223:tid 140735160061952] mod_wsgi (pid=72223): Initializing Python.

[Thu Dec 29 22:27:29.939875 2016] [wsgi:info] [pid 72224:tid 140735160061952] mod_wsgi (pid=72224): Initializing Python.

[Thu Dec 29 22:27:29.940361 2016] [wsgi:info] [pid 72225:tid 140735160061952] mod_wsgi (pid=72225): Initializing Python.

[Thu Dec 29 22:27:29.990508 2016] [wsgi:info] [pid 72223:tid 140735160061952] mod_wsgi (pid=72223): Attach interpreter ''.

[Thu Dec 29 22:27:29.990567 2016] [wsgi:info] [pid 72224:tid 140735160061952] mod_wsgi (pid=72224): Attach interpreter ''.

[Thu Dec 29 22:27:29.990572 2016] [wsgi:info] [pid 72225:tid 140735160061952] mod_wsgi (pid=72225): Attach interpreter ''.

[Thu Dec 29 22:28:08.373654 2016] [wsgi:info] [pid 72223:tid 123145303912448] [remote 127.0.0.1:60088] mod_wsgi (pid=72223, process='trunk.delphicloud.io', application=''): Loading WSGI script '/Users/erik/web/delphicloud.io/dc-django-app/trunk/delphi_cloud/delphicloud/wsgi.py'.

[Thu Dec 29 22:28:08.975756 2016] [wsgi:info] [pid 72230:tid 140735160061952] mod_wsgi (pid=72230): Initializing Python.

[Thu Dec 29 22:28:09.007544 2016] [wsgi:info] [pid 72230:tid 140735160061952] mod_wsgi (pid=72230): Attach interpreter ''.

[Thu Dec 29 22:28:17.599267 2016] [wsgi:error] [pid 72230:tid 123145312497664] [client 127.0.0.1:60094] Truncated or oversized response headers received from daemon process 'trunk.delphicloud.io': /Users/erik/web/delphicloud.io/dc-django-app/trunk/delphi_cloud/delphicloud/wsgi.py, referer: http://aias.delphicloud/question/1

[Thu Dec 29 22:28:17.977361 2016] [core:notice] [pid 69988:tid 140735160061952] AH00052: child pid 72223 exit signal Illegal instruction (4)

[Thu Dec 29 22:28:17.977527 2016] [wsgi:info] [pid 69988:tid 140735160061952] mod_wsgi (pid=72223): Process 'trunk.delphicloud.io' has died, deregister and restart it.

[Thu Dec 29 22:28:17.977540 2016] [wsgi:info] [pid 69988:tid 140735160061952] mod_wsgi (pid=72223): Process 'trunk.delphicloud.io' terminated by signal 4

[Thu Dec 29 22:28:17.977548 2016] [wsgi:info] [pid 69988:tid 140735160061952] mod_wsgi (pid=72223): Process 'trunk.delphicloud.io' has been deregistered and will no longer be monitored.

[Thu Dec 29 22:28:17.978275 2016] [wsgi:info] [pid 72266:tid 140735160061952] mod_wsgi (pid=72266): Starting process 'trunk.delphicloud.io' with uid=502, gid=20 and threads=15.

[Thu Dec 29 22:28:17.979449 2016] [wsgi:info] [pid 72266:tid 140735160061952] mod_wsgi (pid=72266): Initializing Python. 

[Thu Dec 29 22:28:18.011622 2016] [wsgi:info] [pid 72266:tid 140735160061952] mod_wsgi (pid=72266): Attach interpreter ''. 

 Any help you can give me would be greatly appreciated.  Thank you!

[Graham Dumpleton]

You aren’t using system Python but some other Python installation.

What Python installation are you using? Python Software Foundation or Home Brew?

Has Python been upgraded at some point as well?

Were you using and activating a Python virtual environment from your WSGI script file?

Where is the mod_wsgi.so module file located and what configuration have you added to Apache to load it?

What do you get if you run ‘otool -L mod_wsgi.so’ from the directory the mod_wsgi module is in?

Graham

-- 
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at https://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Erik Curiel]

Thank you for your fast reply.  I'm using macport's python27, which was upgraded on 18. Dec 2017 to 2.7.13.  That is the system-wide python installation, as the box I'm using is dedicated solely to development of this one piece of software, so there is no virtualenv needed, so in particular nothing like that is being used or activated in the wsgi script file.  This is the entirety of the wsgi script:

import os, sys

# make path to the project root the first entry in sys.path, so we can

# import params.py

sys.path.insert(0, os.path.dirname(

    os.path.dirname(os.path.abspath(__file__))))

# this adds everything else to sys.path defined by the current

# profile, sets up os.environ

import params

from django.core.wsgi import get_wsgi_application

application = get_wsgi_application()

 That hasn't changed in any way in months.  The mod_wsgi.so file is located at /usr/local/apache2/modules/mod_wsgi.so.   /usr/local/apache2 is the local apache home, where the apache I compiled is installed.  I compiled mod_wsgi from source using the following configuration:

./configure --with-apxs=/usr/local/apache2/bin/apxs --with-python=/opt/local/bin/python

/opt/local/bin/python is a soft link to /opt/local/Library/Frameworks/Python.framework/Versions/2.7/bin/python2.7, the macport global python installation binary.  The apache configuration to load mod_wsgi.so is:

LoadModule wsgi_module /usr/local/apache2/modules/mod_wsgi.so

The only further configuration happens under the <VirtualHost> directive, which is:

    WSGIApplicationGroup %{GLOBAL}

    # WSGIDaemonProcess trunk.delphicloud.io user=erik group=staff processes=1 display-name=%{GROUP} header-buffer-size=65536

    WSGIDaemonProcess trunk.delphicloud.io user=erik group=staff processes=1 display-name=%{GROUP}

    WSGIProcessGroup trunk.delphicloud.io

    WSGIScriptAlias / /Users/erik/web/delphicloud.io/dc-django-app/trunk/delphi_cloud/delphicloud/wsgi.py

The commented out line was my attempt to try manually increasing the header-buffer-size parameter to see whether that would fix the problem.  It didn't.

When I run otool, I get this:

$ otool -L mod_wsgi.so

mod_wsgi.so:

/opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python (compatibility version 2.7.0, current version 2.7.0)

/System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1259.0.0)

/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1226.10.1)

/opt/local/lib/libgcc/libgcc_s.1.dylib (compatibility version 1.0.0, current version 1.0.0)

Please let me know if I can give you any other information to help.  Thanks.

Erik

[Graham Dumpleton]

That all looks reasonable.

I can only suggest that this point if you have a working ‘gdb’, is to following instructions in:

* http://modwsgi.readthedocs.io/en/develop/user-guides/debugging-techniques.html#debugging-crashes-with-gdb

to try and attach gdb to the daemon process and then fire the request and get a stack trace of all the threads when the process crashes.

Graham