Python Paste and Chunked Transfers

[Chris Moyer]

Hello,

The paste.httpserver mentions that the server is not yet fully HTTP/
1.1 compliant because it does not yet support Chunked Encoding:
http://pythonpaste.org/modules/httpserver

Is there a work-around for this or an ETA on when this might be
implemented?

Thanks,
Chris Moyer

[Graham Dumpleton]

Are you asking about chunked request content or chunked response
content?

Even if it did support chunked request content, that will not
necessarily translate to a WSGI application running on top of it
supporting chunked transfer encoding on requests. This is because the
WSGI specification doesn't really support chunked request content.

If you want to step outside of the WSGI specification slightly and
ignore the requirement to only read what CONTENT_LENGTH specifies or
zero length if not defined, then you will be able to use Apache/
mod_wsgi version 3.0 to handle chunked request content.

Graham

[Chris Moyer]

I'm looking for chucked request content, sorry for the confusion.

I'm currently working on an implementation of a WebDav server using
paste and webob but I've discovered that OSX sends large files as
chunked transfers when using the finder. After digging around I found
the mention that the past server does not "yet" support this, which
made me assume it eventually will. The request objects as of now have
the ability to read in the body or provide a file-like object to read,
but these simply do not work for chunked encoding.

I've been trying to steer away from apache just for WSGI, but it
sounds like I'll have to stick with it for a while at least.

Thanks,

--
Chris Moyer

[Ian Bicking]

On Mon, Jul 6, 2009 at 8:54 PM, Chris Moyer <kope...@gmail.com> wrote:

I'm looking for chucked request content, sorry for the confusion.

I'm currently working on an implementation of a WebDav server using
paste and webob but I've discovered that OSX sends large files as
chunked transfers when using the finder. After digging around I found
the mention that the past server does not "yet" support this, which
made me assume it eventually will. The request objects as of now have
the ability to read in the body or provide a file-like object to read,
but these simply do not work for chunked encoding.

I've been trying to steer away from apache just for WSGI, but it
sounds like I'll have to stick with it for a while at least.

There's not much active work on the Paste http server, so barring a patch there won't be support soon.  I believe the CherryPy server does have support for that though (I'm assuming it just reads the request body in its entirety before invoking the WSGI application; is that also what mod_wsgi does?)

--
Ian Bicking  |  http://blog.ianbicking.org  |  http://topplabs.org/civichacker

[Graham Dumpleton]

On Jul 8, 2:50 am, Ian Bicking <ianbick...@gmail.com> wrote:

> On Mon, Jul 6, 2009 at 8:54 PM, Chris Moyer <koper...@gmail.com> wrote:
>
> > I'm looking for chucked request content, sorry for the confusion.
>
> > I'm currently working on an implementation of a WebDav server using
> > paste and webob but I've discovered that OSX sends large files as
> > chunked transfers when using the finder. After digging around I found
> > the mention that the past server does not "yet" support this, which
> > made me assume it eventually will. The request objects as of now have
> > the ability to read in the body or provide a file-like object to read,
> > but these simply do not work for chunked encoding.
>
> > I've been trying to steer away from apache just for WSGI, but it
> > sounds like I'll have to stick with it for a while at least.
>
> There's not much active work on the Paste http server, so barring a patch
> there won't be support soon.  I believe the CherryPy server does have
> support for that though (I'm assuming it just reads the request body in its
> entirety before invoking the WSGI application;

Yes, that is what CherryPy does. Not sure what measures it has to stop
really big posts given that no way to stream data to application.

> is that also what mod_wsgi does?)

No, that is not what Apache/mod_wsgi does.

In Apache/mod_wsgi it implements return of end sentinel for input
stream. Ie., provides empty string for read() when no more data. Thus,
if you want to ignore the WSGI application requirement of not reading
more than CONTENT_LENGTH, then you can just read until no more data.

I have always thought it rather stupid that WSGI application has that
requirement because it means that WSGI applications have to count the
amount of data read and ensure they do a partial read on last bit to
avoid reading more than CONTENT_LENGTH. With the end sentinel
implemented like normal file like object, you could instead just say:

s = wsgi.input.read(BLKSIZE)
while s:
... process s
s = wsgi.input.read(BLKSIZE)

The only time one would need to consult CONTENT_LENGTH is where you
want to provide a request entity too large error response. If you want
to be able to handle chunked content, currently though would also need
to look at HTTP_TRANSFER_ENCODING since WSGI doesn't have a separate
way of saying that request content length is unknown, as opposed to
zero length.

Graham