Transfer-Encoding bug

[Peter Ward]

Hi,

I'm currently trying to get my notify.io client to work behind a proxy, and have run into a bug.

The proxy I'm behind will send a HTTP/1.0 request to the api server, but will be responded to with a HTTP/1.1 response containing a "Transfer-Encoding: chunked" header.
This causes my client to try to decode a unencoded stream, resulting in a bunch of errors. :(

According to RFC2616 Section 3.6 (http://www.w3.org/Protocols/rfc2616/rfc2616-sec3.html#sec3.6):

A server MUST NOT send transfer-codings to an HTTP/1.0 client.

A simple request which will demonstrate the behaviour:
GET /v1/listen/deb9a08f6d91cfe9159e4170c7fc6427507b9b96 HTTP/1.0
Host: api.notify.io

Something else which may be related to this bug is that when a HTTP/1.1 request is send, the "Transfer-Encoding: chunked" header is sent twice.

I've had a poke through the source code, but I haven't worked out why this happens.
Could someone more knowledgeable about twisted take a look?

Thanks,
  Peter

--
Peter Ward
http://it.usyd.edu.au/~pwar3236/

[Jeff Lindsay]

Ooo, thanks for the report. Do you know what would happen if your 1.0 client receives the stream as-is without the Transfer-Encoding header?

Not sure about the double header, but I'm less concerned about it. :\

-jeff

--
Jeff Lindsay
http://progrium.com

[Peter Ward]

In both cases, the stream is sent normally (not chunked).
If the Transfer-Encoding was not sent for HTTP/1.0 (which I don't think it should be), my client will work fine (tested).

Here's the headers on each stage:

1. Original request to proxy server.
GET http://api.notify.io/v1/listen/{LISTEN_URL} HTTP/1.1
Accept-Encoding: identity
Host: api.notify.io
Proxy-Authorization: Basic {PROXY_AUTH_DATA}
Connection: close
User-Agent: libnotify.io/1.1

2. Request from proxy to api.notify.io server.
GET /v1/listen/{LISTEN_URL} HTTP/1.0
Accept-Encoding: identity
User-Agent: libnotify.io/1.1
Host: api.notify.io
Via: 1.1 www-cache.it.usyd.edu.au:8000 (squid/2.5.STABLE1), 1.0 www-cacheE.usyd.edu.au:8080 (squid/2.6.STABLE5)
X-Forwarded-For: 129.78.8.237, 129.78.8.171
Cache-Control: max-age=259200
Connection: keep-alive

3. Response from api.notify.io to proxy.
HTTP/1.1 200 OK
Server: nginx/0.6.35
Date: Tue, 02 Mar 2010 07:21:45 GMT
Content-Type: application/json
Connection: close
Transfer-Encoding: chunked

{"text": "Hello World", "icon": "http:\/\/flowblok.selfip.net:8001\/img\/hg.png", "source": "Peter Ward", "title": "Blah"}
{"text": "hello", "icon": "http:\/\/www.gravatar.com\/avatar\/0263cded7f48e42a23cc519d8fdffb10", "source": "Peter Ward", "title": "hello"}

4. Response from proxy to client.
HTTP/1.0 200 OK
Server: nginx/0.6.35
Date: Tue, 02 Mar 2010 07:23:13 GMT
Content-Type: application/json
Transfer-Encoding: chunked
X-Cache: MISS from www-cacheE.usyd.edu.au
X-Cache-Lookup: MISS from www-cacheE.usyd.edu.au:8080
Via: 1.0 www-cacheE.usyd.edu.au:8080 (squid/2.6.STABLE5)
X-Cache: MISS from www-cache.it.usyd.edu.au
Proxy-Connection: close

{"text": "Hello World", "icon": "http:\/\/flowblok.selfip.net:8001\/img\/hg.png", "source": "Peter Ward", "title": "Blah"}
{"text": "hello", "icon": "http:\/\/www.gravatar.com\/avatar\/0263cded7f48e42a23cc519d8fdffb10", "source": "Peter Ward", "title": "hello"}

It appears that everyone's doing a bit of mangling on the way - ah, the "joys" of proxies. :(

[Jeff Lindsay]

Cool. Can you add HTTP/1.0 support to the UserVoice? When I get a chance I'll drop the Transfer-Encoding for 1.0.

[Peter Ward]

(quick email, I'm about to go out)

Take a look at this bit of Twisted:
http://twistedmatrix.com/trac/browser/trunk/twisted/web/http.py#L936

As long as you don't set a Content-Length, twisted will automagically put the HTTP stream into chunked mode.
Therefore, removing the

request.setHeader('Transfer-Encoding', 'chunked')

line from api/server.py:143 should make everything work properly.

(this is all untested, of course!)

Peter

--
Peter Ward
SID: 309203368
2nd Year BCST @ USyd
http://it.usyd.edu.au/~pwar3236/

[Jeff Lindsay]

Good find! Thanks.