Content-Length: 0

[Mat]

Have had obscure problems with a site returning a different page in
Scrapy that what I got in Firefox (it appeared to ignore URL
parameters). With the help of Wireshack I have identified the cause as
the presence of a Content-Length: 0 header in the GET request. Sending
the same request with netcat narry the Content-Length header returns
the expected result.

The only reference to 'Content-Length' that I can find in the Scrapy
source is in contrib/downloadermiddleware/common.py, but this does not
seem to be active for a GET request. Printing headers out here shows
no Content-Length header. Following through to core/downloader/
handlers.py, the Content-Length header is not present inside
create_factory. So, I presume it handled by something downstream of
HTTPClientFactory creation. Is that into the realms of Twisted?

Anyone else come across this one? I took a while to figure out the
cause as I was scraping inside a website I needed to login to, so
thought the problem was something to do with that. Can I code around
this with monkeypatches.py?

[Mat]

Amazing how you fix the problem immediately after posting about it. A
short walk does wonders for fixing problems!

It seems that Twisted's HTTPClientFactory sets Content-Length unless
body is None. Scrapy explicitly changes a Request body from None to an
empty string in scrapy.http.request. Patch to resolve this follows,
but I am unsure of its wider effect.

Index: __init__.py
===================================================================
--- __init__.py (revision 826)
+++ __init__.py (working copy)
@@ -49,7 +49,7 @@
elif isinstance(body, unicode):
self._body = body.encode(self.encoding)
elif body is None:
- self._body = ''
+ self._body = None
else:
raise TypeError("Request body must either str or unicode.
Got: '%s'" % type(body).__name__)
body = property(lambda x: x._body, set_body)

[Daniel Graña]

Hi Mat,

take a look at http://dev.scrapy.org/changeset/840

thanks
Daniel