Getting raw request headers

[Davíð Steinn Geirsson]

Hi all,

Is there a good way to get at the raw request headers sent by
scrapy 1.0.x? I know I can recreate a functionally equivalent
request from response.request.(cookies,headers,...) but for my
application I really need the exact bytes that went out on the
wire.

It seems that with the old HTTP/1.0 downloader this could be done
by setting DOWNLOADER_HTTPCLIENTFACTORY and overriding
ScrapyHTTPPageGetter. However the new HTTP/1.1 downloader uses a
twisted Agent.

The HTTP/1.1 downloader seems to listen to the DOWNLOAD_HANDLERS
setting, but I don't see a way to get the same effect overriding
HttpDownloadHandler without duplicating a whole bunch of code
from both scrapy and twisted.

Any pointers would be much appreciated.

Best regards,
Davíð