Django/CherryPy problem with POST data

[Mark]

Changeset [14394], which introduced the HttpRequest.read feature,
breaks my Django project running behind CherryPy 3.1.2 on Python 2.7
-- it hangs when trying to read request.POST data, specifically at:

django/http/__init__.py, line 144: self._raw_post_data = self.read()

Django development server works fine, and I confirmed that both worked
fine under revision 14393.

In the development server, wsgi.input is
django.core.handlers.wsgi.LimitedStream, while under CherryPy,
wsgi.input is cherrypy.wsgiserver.SizeCheckWrapper.

In revisions prior to 14394, SizeCheckWrapper.read is called with
CONTENT_LENGTH as the argument. In 14394, it is called with no
argument, leading to a default for size of -1 and apparently to the
hang. In 14394, if I pass CONTENT_LENGTH as the argument, it all seems
to work as before.

I am using admin login in my testing; I think this problem would occur
any time there is POST data present.

I do not have a good understanding of either the Django code in this
area, nor the CherryPy code, so I don't know whether a fix could or
should be supplied in Django, or CherryPy, or both. I am hoping
someone with better knowledge can take a look at this. I would be glad
to open on a ticket on this if appropriate.

Thanks,

--Mark

[Russell Keith-Magee]

Sounds like a ticket is called for.

To aid the debugging process, if you can provide a complete standalone
example (or at least a complete set of duplication instructions), that
would be most helpful.

Yours,
Russ Magee %-)

[Ivan Sagalaev]

On 11/01/2010 03:06 AM, Russell Keith-Magee wrote:
> Sounds like a ticket is called for.

Mark, can you also CC: me ('isagalaev') on this ticket. I have an idea
where it can be broken.

I remember that the initial patch was always passing a `size` parameter
into .read() of the underlying stream. Russell can you remember why you
changed it to pass or not pass the `size` according to how it was passed
into the outer .read()?

[Russell Keith-Magee]

The only changes I'm aware of making were related to getting
readline() to behave correctly. The patch as submitted didn't return
the right values for the readline(len) case. Which call to read() are
you saying is wrong?

Yours,
Russ Magee %-)

[Mark]

>Sounds like a ticket is called for.

Ticket #14594
<http://code.djangoproject.com/ticket/14594>

>To aid the debugging process, if you can provide a complete standalone
>example (or at least a complete set of duplication instructions), that
>would be most helpful.

Working on this now.

--Mark

[Mark]

Steps to reproduce Ticket #14594

I used a virtualenv running Python 2.7, but I don't think the Python
version matters

# Install or make sure you are using Django trunk >= 14394
pip install CherryPy # 3.1.2
django-admin.py startproject newbug
cd newbug
python manage.py runserver # verify on port 8000 that "It works!"
# Edit urls.py to enable the admin
# Edit settings.py to enable the admin
# Edit settings.py to create a database.
# I used sqlite3 and specified a path
python manage.py syncdb # setup the database and create a superuser
# Verify that you can login, using the development server and your new
superuser at
# localhost:8000/admin/
# Log out, stop the development server
python manage.py shell
>>>import cherrypy
>>>from django.core.handlers.wsgi import WSGIHandler
>>>cherrypy.tree.graft( WSGIHandler(), script_name="")
>>>cherrypy.quickstart() # this will start the CherryPy server on 8080
# The admin media will not be served, but this doesn't matter
# Go to localhost:8080/admin/
# Try to login with your superuser, or just click the login button
with blank input
# Hangs

--Mark

[Ivan Sagalaev]

On 11/01/2010 08:44 AM, Russell Keith-Magee wrote:
> The only changes I'm aware of making were related to getting
> readline() to behave correctly. The patch as submitted didn't return
> the right values for the readline(len) case. Which call to read() are
> you saying is wrong?

Sorry, my memory has failed me here. Indeed I was thinking about your
fixes in readline and they shouldn't have anything to do with the bug.

[Ivan Sagalaev]

On 11/01/2010 11:21 AM, Mark wrote:
> Ticket #14594
> <http://code.djangoproject.com/ticket/14594>

From a quick look it can be fixed in two ways:

- HttpRequest.raw_post_data can always explicitly ask for content_length
bytes. If content_length is absent just treat it as 0.

- WSGIRequest can wrap all wsgi.input into a LimitedStream (now it's
done only for known fragile sockets).

I don't like pure Python wrappers moving bytes in performance critical
places, so I'd go with the first approach. Does anyone foresee any bad
things with it?

[Russell Keith-Magee]

The former looks like the right approach to me -- in fact, it looks to
be a pretty close rendition of what the code used to do. I've made the
change in r14435.

Yours,
Russ Magee %-)