Truncated or oversized response headers received from daemon process when configured apache + mod_wsgi + django in centOS 6.9

[Suyash Jain]

Hi,

I am facing error Truncated or oversized response headers received from daemon process when configured apache + mod_wsgi + django

System python is 2.6.6 and I needed Python 3.7.5

So i installed python using below cmds (with enable shared option)

wget https://www.python.org/ftp/python/3.7.5/Python-3.7.5.tgz

tar xzf  Python-3.7.5.tgz

cd Python-3.7.5 >

./configure --enable-optimizations --enable-shared

Then complied mpd_wsgi with virtualenv created from installed python 3.7.5 ie test1_env using pyenv virtualenv

yum install httpd-devel   (apxs)

git clone https://github.com/GrahamDumpleton/mod_wsgi.git

./configure --with-apxs=/usr/sbin/apxs --with-python=/home/abc/.pyenv/versions/test1_env/bin/python3

make

System details :

CentOS release 6.9 (Final)

Server version: Apache/2.2.15 (Unix)

Architecture:   64-bit

Server MPM:     Prefork

threaded:     no

forked:     yes (variable process count)

Insallted Package details inside test1_env virtual env

python - 3.7.5

Django  3.1.2

mod_wsgi 4.7.1

pyenv virtualenv used ie test1_env

Error logs: 

Truncated or oversized response headers received from daemon process 'appproject': /home/abc/workspace/project/core/wsgi.py

[info] mod_wsgi (pid=124460): Attach interpreter ''.

[info] mod_wsgi (pid=124460): Adding '/home/abc/workspace' to path.

[info] mod_wsgi (pid=124460): Adding '/home/abc/workspace/project' to path.

[info] mod_wsgi (pid=124460): Adding '/home/abc/.pyenv/versions/test1_env/lib/python3.7/site-packages' to path.

[info] mod_wsgi (pid=124460): Imported 'mod_wsgi'.

The WSGI portion of the configuration for the vhost in Apache looks like this

        ServerAdmin xx...@yyy.com

        ServerName example.com

        DocumentRoot /home/abc/workspace/project

        Alias /static /home/abc/workspace/project/static/

        <Directory "/home/abc/workspace/project/static/">

                Allow from all

        </Directory>

    

        LogLevel info

        ErrorLog logs/project_error.log

        CustomLog logs/project_access.log combined

       

        WSGIDaemonProcess appproject user=xx group=yy  processes=2 threads=15 python-path=/home/abc/.pyenv/versions/test1_env/lib/python3.7/site-packages:/home/abc/workspace:/home/abc/workspace/project python-home=/home/abc/.pyenv/versions/test1_env display-name='appproject'

        WSGIProcessGroup appproject

        WSGIApplicationGroup %{GLOBAL}

        WSGIScriptAlias / /home/abc/workspace/project/core/wsgi.py 

        <Directory /home/abc/workspace/project/core>

               <Files wsgi.py>

                        Order allow,deny

                        Allow from all

                </Files>

        </Directory>

</VirtualHost>

Please provide solution for above issue.

Thanks

Suyash

[Graham Dumpleton]

Your are responding with a set of HTTP headers which exceed the typical maximum Apache header size. You can override the default when using daemon mode of mod_wsgi by using the 'header-buffer-size' option to WSGIDaemonProcess. See:

* https://modwsgi.readthedocs.io/en/master/configuration-directives/WSGIDaemonProcess.html

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/7eca331a-d0d1-4582-a5dd-e8bb74f9d9f8n%40googlegroups.com.

[Graham Dumpleton]

Sorry, my memory is bad here and it is likely nothing to do with that. It is more likely because the application process crashes. You didn't provide sufficient log messages from before the error message to confirm that, so please provide log messages form before and after. You can also look in the main Apache error log message and see if there was a "Segmentation fault" message.

[Graham Dumpleton]

BTW, the options you have used to compile your custom Python may not be sufficient. Ensure you read:

    http://blog.dscpl.com.au/2015/06/installing-custom-python-version-into.html

Ignore that talks about docker, most of it should still be relevant. In other words, look to how system Python is compiled to work out what options you use.

Also, you don't show that you ran 'make install' or how you told Apache to load your custom mod_wsgi. So another cause of the crash could be that you are actually using system package for mod_wsgi compiled with system Python, with your virtual environment for your Python. You need to make sure you have uninstalled system package for mod_wsgi and that Apache is using correct mod_wsgi.

Graham

[Suyash Jain]

logs from

[Wed Feb 03 04:19:37 2021] [info] [client 116.xx.xxx.xxx] Connection to child 1 established (server example.com:443)

[Wed Feb 03 04:19:37 2021] [info] Seeding PRNG with 144 bytes of entropy

[Wed Feb 03 04:19:37 2021] [debug] ssl_engine_kernel.c(1876): OpenSSL: Handshake: start

[Wed Feb 03 04:19:37 2021] [debug] ssl_engine_kernel.c(1884): OpenSSL: Loop: before/accept initialization.

.

.

ssl_engine_io and OPENSSL calls

.

.

[Wed Feb 03 04:19:37 2021] [debug] ssl_engine_kernel.c(1884): OpenSSL: Loop: SSLv3 read finished A

[Wed Feb 03 04:19:37 2021] [debug] ssl_engine_kernel.c(1884): OpenSSL: Loop: SSLv3 write session ticket A

[Wed Feb 03 04:19:37 2021] [debug] ssl_engine_kernel.c(1884): OpenSSL: Loop: SSLv3 write change cipher spec A

[Wed Feb 03 04:19:37 2021] [debug] ssl_engine_kernel.c(1884): OpenSSL: Loop: SSLv3 write finished A

[Wed Feb 03 04:19:37 2021] [debug] ssl_engine_kernel.c(1884): OpenSSL: Loop: SSLv3 flush data

[Wed Feb 03 04:19:37 2021] [debug] ssl_engine_kernel.c(1880): OpenSSL: Handshake: done

[Wed Feb 03 04:19:37 2021] [info] Connection: Client IP: 116.xx.xxx.xxx, Protocol: TLSv1.2, Cipher: xxxxxx (128/128 bits)

[Wed Feb 03 04:19:39 2021] [error] [client 116.xx.xxx.xxx] Truncated or oversized response headers received from daemon process 'appproject': /home/abc/workspace/project/core/wsgi.py

[Wed Feb 03 04:19:39 2021] [debug] ssl_engine_kernel.c(1894): OpenSSL: Write: SSL negotiation finished successfully

[Wed Feb 03 04:19:39 2021] [info] [client 116.xx.xxx.xxx] Connection closed to child 1 with standard shutdown (server example.com:443)

[Wed Feb 03 04:19:39 2021] [info] mod_wsgi (pid=114381): Attach interpreter ''.

[Wed Feb 03 04:19:39 2021] [info] mod_wsgi (pid=114381): Adding '/home/abc/workspace' to path.

[Wed Feb 03 04:19:39 2021] [info] mod_wsgi (pid=114381): Adding '/home/abc/workspace/project' to path.

[Wed Feb 03 04:19:39 2021] [info] mod_wsgi (pid=114381): Adding '/home/abc/.pyenv/versions/test1_env/lib/python3.7/site-packages' to path.

[Wed Feb 03 04:19:39 2021] [info] mod_wsgi (pid=114381): Imported 'mod_wsgi'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 0 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 1 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 3 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 2 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 5 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 4 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 6 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 7 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 9 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 8 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 10 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 13 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 11 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 12 in daemon process 'appproject'.

[Wed Feb 03 04:19:39 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=114381): Started thread 14 in daemon process 'appproject'.

[Wed Feb 03 04:19:55 2021] [debug] ssl_engine_io.c(1930): OpenSSL: read 5/5 bytes from BIO#xxxxxxx [mem: xxxxxxx] (BIO dump follows)

[Suyash Jain]

First i ran this cmd to get mod_wsgi module  for apache:

>mod_wsgi-express module-config

LoadModule wsgi_module "/home/abc/.pyenv/versions/test1_env/lib/python3.7/site-packages/mod_wsgi/server/mod_wsgi-py37.cpython-37m-x86_64-linux-gnu.so"

WSGIPythonHome "/home/abc/.pyenv/versions/test1_env"

then Check module using ldd cmd

under dir /etc/httpd/modules

>ldd mod_wsgi.so 

linux-vdso.so.1 =>  (xxxxxxxxxxxxx)

libpython3.7m.so.1.0 => /usr/local/lib/libpython3.7m.so.1.0 (xxxxxxxxxxx)

libpthread.so.0 => /lib64/libpthread.so.0 (xxxxxxxxxxx)

And under httpd.conf have mentioned

LoadModule wsgi_module modules/mod_wsgi.so

Is that all right and sufficient??

[Graham Dumpleton]

At a guess the problem is that when you are compiling your custom Python then that Python isn't compiled with the same OpenSSL libraries that Apache uses. The version compatibility can cause it to crash. Can you disable mod_ssl in Apache and see if the issue goes away. If it does, that will confirm that is the issue.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/ef2f6054-b50c-41f3-b9f7-9c853e29c608n%40googlegroups.com.

[Suyash Jain]

Hi Graham

I checked with your mention points from prev mail:

Check1

You need to make sure you have uninstalled system package for mod_wsgi and that Apache is using correct mod_wsgi.

>  there is no system package for mod_wsgi installed and Apache is using correct mod_wsgi

Check2

Can you disable mod_ssl in Apache and see if the issue goes away:

> apache uses mod_ssl for SSL cert and version is OpenSSL 1.0.2r  26 Feb 2019

Python which i complied has openssl version of OpenSSL 1.0.2r  26 Feb 2019' 

Since they are same, but still i will check once with mod_ssl disable.

Check3

I check apache error logs too,   it mentions:

> [notice] child pid 116566 exit signal Segmentation fault (11)

Also i have few question , it will be great you can answer them:

Q1. If we configure mod_wsgi daemon process inside Virtual host file like i have mention in first mail. Do we need to start explicitly the django server as like :

python manage.py runserver  , along with apache running the wsgi configured vhost file.

Q2. Does centOS 6.9 has some compatible issue which doesnt allow mod_wsgi = 4.7,1 and python (custom build) = 3.7.5  to configure with apache.

Should we use centOS 7/8 versions for the same?

Q3. Also do we need to mention WSGIRestrictEmbedded On  outside of VirtualHost in main apache conf file ?

Thanks 

Suyash

[Graham Dumpleton]

On 12 Feb 2021, at 8:36 pm, Suyash Jain <sj...@tenplustwo.com> wrote:

Hi Graham

I checked with your mention points from prev mail:

Check1

You need to make sure you have uninstalled system package for mod_wsgi and that Apache is using correct mod_wsgi.

>  there is no system package for mod_wsgi installed and Apache is using correct mod_wsgi

Check2

Can you disable mod_ssl in Apache and see if the issue goes away:

> apache uses mod_ssl for SSL cert and version is OpenSSL 1.0.2r  26 Feb 2019

Python which i complied has openssl version of OpenSSL 1.0.2r  26 Feb 2019' 

Since they are same, but still i will check once with mod_ssl disable.

Check3

I check apache error logs too,   it mentions:

> [notice] child pid 116566 exit signal Segmentation fault (11)

Also i have few question , it will be great you can answer them:

Q1. If we configure mod_wsgi daemon process inside Virtual host file like i have mention in first mail. Do we need to start explicitly the django server as like :

python manage.py runserver  , along with apache running the wsgi configured vhost file.

No you don't. Using 'python manage.py runserver' starts a complete separate WSGI server only really usable for local development. It is not needed when using mod_wsgi.

Q2. Does centOS 6.9 has some compatible issue which doesnt allow mod_wsgi = 4.7,1 and python (custom build) = 3.7.5  to configure with apache.

Should we use centOS 7/8 versions for the same?

Q3. Also do we need to mention WSGIRestrictEmbedded On  outside of VirtualHost in main apache conf file ?

Since you are using daemon mode, it is recommended to add that directive.

As to the crash, can you try with mod_wsgi source code from develop branch of GitHub. If using pip use:

    pip install -U --no-cache-dir https://github.com/GrahamDumpleton/mod_wsgi/archive/develop.zip

More details of using pip based version in:

    https://pypi.org/project/mod-wsgi/

It has a change to address an issue which has been causing process crashes when Python deprecation warnings have been turned on for everything.

Graham

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/bb95febc-8aa9-4e44-98a9-5c66f8bc1a56n%40googlegroups.com.

[Suyash Jain]

Hi Graham, 

i followed as you suggested, compiled openssl from source and compiled python 3.7.5 with option (-./configure --with-openssl=/usr/src/openssl-1.0.2o --enable-optimizations --enable-shared).

Then make -> make install.

then Install mod_wsgi by activating virtualenv usinf (pip install mod_wsgi) and copy module to apache/modules directory.

I change the LogLevel in apache conf to info.

When i hit the URL  example.com :

i see below error in error_log of apache :

 [notice] child pid 74569 exit signal Segmentation fault (11)

 [info] mod_wsgi (pid=74569): Process 'app' has died, deregister and restart it.

 [info] mod_wsgi (pid=74569): Process 'app' terminated by signal 11

 [info] mod_wsgi (pid=74569): Process 'app' has been deregistered and will no longer be monitored.

 [info] mod_wsgi (pid=74625): Starting process 'app' with uid=630, gid=48 and threads=15.

 [info] mod_wsgi (pid=74625): Python home /home/abc/workspace/test_env.

 [info] mod_wsgi (pid=74625): Initializing Python.

 [notice] child pid 74625 exit signal Segmentation fault (11)

 [info] mod_wsgi (pid=74625): Process 'app' has died, deregister and restart it.

 [info] mod_wsgi (pid=74625): Process 'app' terminated by signal 11

 [info] mod_wsgi (pid=74625): Process 'app' has been deregistered and will no longer be monitored.

 [info] mod_wsgi (pid=74654): Starting process 'app' with uid=630, gid=48 and threads=15.

 [info] mod_wsgi (pid=74654): Python home /home/abc/workspace/test_env.

 [info] mod_wsgi (pid=74654): Initializing Python.

project_error.log configured in virtual host file:

[Fri Feb 26 01:16:41 2021] [info] Initial (No.1) HTTPS request received for child 68 (server example.com:443)

[Fri Feb 26 01:16:41 2021] [error] SYS PATH -  ['/home/abc/workspace', '/home/abc/workspace/project', '/usr/local/lib/python37.zip', '/usr/local/lib/python3.7', '/usr/local/lib/python3.7/lib-dynload', '/home/abc/workspace/test_env/lib/python3.7/site-packages']

[Fri Feb 26 01:16:42 2021] [error] [client 116.xx.xxx.xxx] Truncated or oversized response headers received from daemon process 'app': /home/abc/workspace/project/core/wsgi.py, referer: https://example.com

[Fri Feb 26 01:16:42 2021] [debug] ssl_engine_kernel.c(1894): OpenSSL: Write: SSL negotiation finished successfully

[Fri Feb 26 01:16:42 2021] [info] [client 116.73.52.163] Connection closed to child 68 with standard shutdown (server example.com:443)

[Fri Feb 26 01:16:42 2021] [info] mod_wsgi (pid=75016): Attach interpreter ''.

[Fri Feb 26 01:16:42 2021] [info] mod_wsgi (pid=75016): Adding '/home/abc/workspace' to path.

[Fri Feb 26 01:16:42 2021] [info] mod_wsgi (pid=75016): Adding '/home/abc/workspace/project' to path.

[Fri Feb 26 01:16:42 2021] [info] mod_wsgi (pid=75016): Imported 'mod_wsgi'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 0 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 2 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 1 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 4 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 3 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 6 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 5 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 7 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 8 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 10 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 9 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 13 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 11 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 12 in daemon process 'app'.

[Fri Feb 26 01:16:42 2021] [debug] src/server/mod_wsgi.c(9115): mod_wsgi (pid=75016): Started thread 14 in daemon process 'app'.

Same repeat for another pid = 74989

As you can see in both error_log and project_error, in both same error appear twice , even when i have mentioned only one process to be run inside WSGIDaemonProcess 

WSGIDaemonProcess app user=xx group=yy processes=1 threads=15 python-path=.. python-home=..

Please help here , can not understand whats the error here.

Thanks

Suyash

[Suyash Jain]

Just to add also. when i disabled the ssl in Apache, mod_wsgi was working fine then.

As mentioned above i installed open ssl from source , referred this url - https://stackoverflow.com/questions/53543477/building-python-3-7-1-ssl-module-failed

But still getting error mentioned in above mail.

If you can suggest some doc how we can compile openssl with proper instructions.

Thanks 

Suyash

[Graham Dumpleton]

If you use a WSGI hello world application (not your real application), with mod_ssl enabled, does it still fail? Don't use a virtual environment when doing the test.

If that works, add to the WSGI hello world an import of the ssl module and see if it still works.

This will help to eliminate whether it is an issue with anything in your virtual environment or third party Python packages you are using.

To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/c761c6f1-9d72-4e6c-bb4e-9ee1a420f255n%40googlegroups.com.