Serious problem with Truncated or oversize response headers / Segfault

[Jason Garber]

Hi Graham,

We were upgraded to python33-mod_wsgi-4.4.0-1.ius.el6.x86_64 this morning and started getting strange errors on a major production site.

The message in the log was about Truncated or oversized response headers.  But upon further looking - it looks like a Segmentation Fault is being emitted to the main apache error log.

Here was the upgrade:

Dec 18 03:27:39 Updated: python33-mod_wsgi-4.4.0-1.ius.el6.x86_64

Here was the downgrade:

Dec 18 10:46:56 Installed: python33-mod_wsgi-4.3.2-1.ius.el6.x86_64

After the downgrade, the segfaults stopped happening. 

I traced it down to a `global` statement:

def parse_form_data(self):

  global Log  #SEGFAULT CAUSED HERE

This would only happen every few requests.  

So the question comes down to:

1. Is mod_wsgi built against the current python version?  (Asking IUS that...)

2. Why would `global Log` ever cause a segfault?

3. Why would it cause a segfault in 4.4.0 but not in 4.3.2

Thanks for your help Graham,

Jason

--------------------------------

Apache Log Example:

[Thu Dec 18 12:08:47 2014] [error] [client 192.168.50.235] ExtendedRequest:Start:/User/API/ShockBox/

[Thu Dec 18 12:08:47 2014] [notice] child pid 27348 exit signal Segmentation fault (11)

[Thu Dec 18 12:08:47 2014] [error] [client 192.168.50.235] Truncated or oversized response headers received from daemon process 'Port60011': /home/jason/DevLevel.2/AMSTMT/Web/Dashboard/User/index.wsgi, referer: https://dashboard--technologymarketingtoolkit--com--jason.step.appcove.net/User/ShockBox/

Note: no ExtendedRequest:End log message here as would be expected, just before start_response() is called and an iterator is returned.  So the request is being terminated

Software Stack:

Red Hat Enterprise Linux Server release 6.6 (Santiago)

httpd-2.2.15-39.0.1.el6.x86_64

python33-mod_wsgi-4.4.0-1.ius.el6.x86_64

python33-3.3.6-1.ius.el6.x86_64

HTTP Config

<IfModule !python_module>

    <IfModule !wsgi_module>

        LoadModule wsgi_module modules/python33-mod_wsgi.so

        WSGISocketPrefix run/wsgi

        WSGIApplicationGroup %{GLOBAL}

    </IfModule>

</IfModule>

WSGIDaemonProcess Port60011 processes=2 threads=2 python-path=/home/jason/DevLevel.2/AMSTMT/Python socket-timeout=600

Listen 127.0.0.1:60011

NameVirtualHost 127.0.0.1:60011

# dashboard--technologymarketingtoolkit--com--jason.step.appcove.net

<VirtualHost 127.0.0.1:60011>

  ServerName _default_;

  DocumentRoot /home/jason/DevLevel.2/AMSTMT/Web/Dashboard

  AddDefaultCharset UTF-8

  RewriteEngine on

  RewriteOptions inherit

  # Forbid any python source files from being served.

  RewriteRule \.(py|pyc|pyo|wsgi)$  -  [F]

  # To handle content views

  RewriteRule ^/User/Content/([a-zA-Z0-9]+)$  /User/Content/Index?C=$1  [PT,L]

  # WSGI Settings

  WSGIScriptAlias /CCRM          /home/jason/DevLevel.2/AMSTMT/Web/Dashboard/CCRM/index.wsgi

  WSGIScriptAlias /ShockBox      /home/jason/DevLevel.2/AMSTMT/Web/Dashboard/ShockBox/index.wsgi

  WSGIScriptAlias /User          /home/jason/DevLevel.2/AMSTMT/Web/Dashboard/User/index.wsgi

  WSGIScriptAlias /Integration   /home/jason/DevLevel.2/AMSTMT/Web/Dashboard/Integration/index.wsgi

  WSGIScriptAlias /              /home/jason/DevLevel.2/AMSTMT/Web/Dashboard/index.wsgi

  WSGIProcessGroup Port60011

  LogLevel info

  ErrorLog /home/jason/DevLevel.2/AMSTMT/apache-error.log

</VirtualHost>

[Graham Dumpleton]

It was fixed in 4.4.1.

http://modwsgi.readthedocs.org/en/master/release-notes/version-4.4.1.html

"""

1. Process crashes could occur when request content had been consumed by the WSGI application. The trigger was when the Python wsgi.input was still in existence after the web request had finished. The destruction of the wsgi.input object was accessing memory which had already been released back to the Apache memory pools and potentially reused. This could cause crashes or other unexplained behaviour. This issue was introduced in version 4.4.0 of mod_wsgi.

"""

The latest is 4.4.2.

I would have to dig back through email to try and see if I can find details of the maintainer of the IUS packages, but if you know how to notify them and point out the newer version that fixes the problem that would be great,

Graham

On 19/12/2014, at 6:32 AM, Jason Garber <ja...@gahooa.com> wrote:

Hi Graham,

We were upgraded to python33-mod_wsgi-4.4.0-1.ius.el6.x86_64 this morning and started getting strange errors on a major production site.

The message in the log was about Truncated or oversized response headers.  But upon further looking - it looks like a Segmentation Fault is being emitted to the main apache error log.

Here was the upgrade:

Dec 18 03:27:39 Updated: python33-mod_wsgi-4.4.0-1.ius.el6.x86_64

Here was the downgrade:

Dec 18 10:46:56 Installed: python33-mod_wsgi-4.3.2-1.ius.el6.x86_64

After the downgrade, the segfaults stopped happening. 

I traced it down to a `global` statement:

def parse_form_data(self):

  global Log  #SEGFAULT CAUSED HERE

<image.png>

--
You received this message because you are subscribed to the Google Groups "modwsgi" group.
To unsubscribe from this group and stop receiving emails from it, send an email to modwsgi+u...@googlegroups.com.
To post to this group, send email to mod...@googlegroups.com.
Visit this group at http://groups.google.com/group/modwsgi.
For more options, visit https://groups.google.com/d/optout.

[Jason Garber]

Hi Graham,

I read that but did not realize it was the problem.  I've asked IUS to upgrade to 4.4.2.

Thanks!

[Graham Dumpleton]

I am going to assume it is that. Best I have right now. :-)

Graham

[Jason Garber]

I've worked around the problem by removing the `global` statement that was unneeded anyway.  I'll let you know if it crops up again.

Thanks!