Truncated or oversized response headers received from daemon process

[Sergey]

Hello,

We need to move ubuntu 16.04/Apache + mod_wsgi + django 1.8/Nginx server to new hardware.

I setup everything the same (apache + python + django app) as on old ubuntu 16.04 server using docker container with ubuntu 16.04. nginx on host is used as proxy.

New website is working, but some requests are failing with "Truncated or oversized response headers received from daemon process" record in log. As I understand wsgi processes are crashing and relaunched by apache (PIDs of processes are changing after this error). If launch Django app in debug mode without wsgi - everything works well. On old server everything is also working well. WSGIApplicationGroup %{GLOBAL} is in config. Django app is using modules which load C++ .so libraries.

Server version: Apache/2.4.41 (Ubuntu)
Server built:   2020-04-13T17:19:17

---------- apache config for wsgi app:

LoadModule wsgi_module /usr/local/lib/python2.7/dist-packages/mod_wsgi/server/mod_wsgi-py27.so

Listen 127.0.0.1:8040

<VirtualHost 127.0.0.1:8040>
    WSGIDaemonProcess nm4-live processes=24 threads=1 home=/app/app/ python-path=/usr/local/lib/python2.7/dist-packages user=nm group=nm display-name='wsgi:nm4-live'
    WSGIProcessGroup nm4-live
    WSGIApplicationGroup %{GLOBAL}

    WSGIScriptAlias / /app/app/nm4/wsgi_live.py

    LogLevel  debug
    CustomLog /app/log/apache/access.log "%h (%{X-Forwarded-For}i) %t \"%r\" %>s (\"%{X-Original-Request}i\") %O %{sessionid}C"
    ErrorLog /app/log/apache/error.log

    # There is streaming URL. Apache won't stream output if gzip is enabled.
    # Also not much sense to gzip traffic between apache and nginx.
    # So just disable it for whole virtual host
    SetEnv no-gzip
</VirtualHost>

# Allow Apache to read wsgi.py files, otherwise it'll return "forbidden"
<Directory /app/app/nm4>
    Order deny,allow
    Require all granted
</Directory>
----------------

pip freeze:
cachetools==3.1.1
certifi==2020.12.5
cffi==1.14.5
chardet==3.0.4
click==7.1.2
click-plugins==1.1.1
cligj==0.7.1
colour==0.1.5
contextlib2==0.6.0.post1
cryptography==3.3.2
Django==1.8.19
django-common-helpers==0.9.2
django-cron==0.5.1
django-jsonfield==0.9.15
django-recaptcha==1.3.1
djorm-pgarray==1.2
docopt==0.4.0
enum34==1.1.10
Fiona==1.6.3
google-api-python-client==1.7.3
google-auth==1.30.0
google-auth-httplib2==0.1.0
httplib2==0.19.1
idna==2.7
ipaddress==1.0.23
mailchimp==2.0.9
mandrill==1.0.57
markdown2==2.3.5
mod-wsgi==4.7.1
numexpr==2.7.3
numpy==1.10.1
oauth2client==4.1.2
oauthlib==3.1.0
pandas==0.17.0
Pillow==5.2.0
psycopg2-binary==2.8.6
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.20
PyJWT==1.7.1
pymongo==2.8
pyOpenSSL==20.0.1
pyparsing==2.4.7
Pyste==0.9.10
python-dateutil==2.7.3
python-memcached==1.59
python-openid==2.2.5
python-social-auth==0.2.21
pytz==2018.5
raven==6.9.0
requests==2.19.1
requests-oauthlib==1.3.0
rsa==3.4.2
Shapely==1.5.13
six==1.16.0
sorl-thumbnail==12.4.1
stripe==1.35.0
tables==3.4.4
uritemplate==3.0.1
urllib3==1.23
uWSGI==2.0.19.1
---

python --version
Python 2.7.12
---

error in apache log:

[Thu May 13 16:06:02.110200 2021] [authz_core:debug] [pid 1712:tid 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: authorization result of Require all granted: granted, referer: ...
[Thu May 13 16:06:02.110238 2021] [authz_core:debug] [pid 1712:tid 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: authorization result of <RequireAny>: granted, referer: ...
[Thu May 13 16:06:02.110298 2021] [authz_core:debug] [pid 1712:tid 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: authorization result of Require all granted: granted, referer: ...
[Thu May 13 16:06:02.110307 2021] [authz_core:debug] [pid 1712:tid 139677168236288] mod_authz_core.c(809): [client 127.0.0.1:38898] AH01626: authorization result of <RequireAny>: granted, referer: ...
[Thu May 13 16:06:36.186653 2021] [wsgi:error] [pid 1712:tid 139677168236288] [client 127.0.0.1:38898] Truncated or oversized response headers received from daemon process 'nm4-live': /app/app/nm4/wsgi_live.py, referer: ...

--------

As last resort I installed uwsgi for testing this situation and launched it instead of apache:

---

uwsgi --chdir=/app/app/ \
    --module=nm4.wsgi_live:application \
    --master --pidfile=/tmp/nm4-live-uwsgi-master.pid \
    --http-socket=127.0.0.1:8040 \
    --processes=25 \
    --uid=1000 --gid=1000 \
    --harakiri=20 \
    --max-requests=5000 \
    --vacuum \
    --stats :1717 \
    --daemonize=/app/log/uwsgi.log

---

Now there is no such a mass of errors as with apache.

But if send a lot of requests uwsgi does not response for several of them and this error is returned by nginx: 502 Bad Gateway.

In this case in uwsgi log I see this:

----

Thu May 13 19:08:01 2021 - HARAKIRI !!! end of worker 17 status !!!
DAMN ! worker 17 (pid: 10800) died, killed by signal 9 :( trying respawn ...
Respawned uWSGI worker 17 (new pid: 10836)

----

New server has less CPU frequency. May be these harakiry is because django processes request longer than 20 secs on high load (it makes postgreSQL/postgis requests).

Any help/hints will be appreciated.

Regards,

Sergey.

[Graham Dumpleton]

Since you have LogLevel debug, mod_wsgi should be outputting messages about the daemon process restarts and this can include from memory whether was killed by certain signal. The information may be in the main Apache error log rather than virtual host log. See if you can find them.

Based on the message from uwsgi, I would speculate the reason is the same for Apache. The message in uwsgi logs is:

DAMN ! worker 17 (pid: 10800) died, killed by signal 9 :( trying respawn ...

As it is signal 9, this would tend to indicate that the Linux OOM reaper felt the OS was running out of memory and decided for kill off the process running your web application due to consuming a large amount of memory.

Graham

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/384288f2-e5a6-4490-9b19-38091687908fn%40googlegroups.com.

[Sergey]

My main apache log level was set to warn, so I didn't see error messages which are having info level.

After I changed it to debug, I found  a lot of such lines in /var/log/apache2/error.log:

[Fri May 14 19:36:45.659418 2021] [wsgi:info] [pid 11806:tid 140271581906816] mod_wsgi (pid=11823): Process 'nm4-live' has died, d
eregister and restart it.
[Fri May 14 19:36:45.659990 2021] [wsgi:info] [pid 11806:tid 140271581906816] mod_wsgi (pid=11823): Process 'nm4-live' terminated
by signal 9
[Fri May 14 19:36:45.660009 2021] [wsgi:info] [pid 11806:tid 140271581906816] mod_wsgi (pid=11823): Process 'nm4-live' has been de
registered and will no longer be monitored.

I found out that almost all memory of 32GB server RAM is resumed when apache processing requests.

The old server had 256GB of RAM. New server has 32GB.

I decreased the number of processes from 24 to 6 and the error disappeared.

Thank you very much! I spent a week trying to find out the reason of the problem and lost any hope of solving it.

Regards,

Sergey

пт, 14 мая 2021 г. в 03:55, Graham Dumpleton <graham.d...@gmail.com>:

You received this message because you are subscribed to a topic in the Google Groups "modwsgi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/modwsgi/qOJF48JEuks/unsubscribe.
To unsubscribe from this group and all its topics, send an email to modwsgi+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/modwsgi/6BC57CE5-F43D-4208-BD60-2FF55319334E%40gmail.com.