Scrapy with https proxy
Oana Goga
I am trying to use scrapy to access https web pages over a proxy and I have some problems getting it to work.
When I am trying to fetch/view https://www.paypal.com with scrapy I am getting the 501 error (Not Implemented), but when I fetch the page with wget everything is working well. Here are the steps that I am doing:
$ export http_proxy="http://us.proxymesh.com:31280"
$ export https_proxy="http://us.proxymesh.com:31280"
$ scrapy view https://www.paypal.com
2011-08-25 19:41:43-0700 [scrapy] INFO: Scrapy 0.12.0.2545 started (bot: nice_bot)
2011-08-25 19:41:43-0700 [scrapy] DEBUG: Enabled extensions: FeedExporter, TelnetConsole, SpiderContext, WebService, CoreStats, MemoryUsage, CloseSpider
2011-08-25 19:41:43-0700 [scrapy] DEBUG: Enabled scheduler middlewares: DuplicatesFilterMiddleware
2011-08-25 19:41:43-0700 [scrapy] DEBUG: Enabled downloader middlewares: HttpProxyMiddleware, HttpAuthMiddleware, DownloadTimeoutMiddleware, UserAgentMiddleware, RetryMiddleware, DefaultHeadersMiddleware, RedirectMiddleware, CookiesMiddleware, HttpCompressionMiddleware, DownloaderStats
2011-08-25 19:41:43-0700 [scrapy] DEBUG: Enabled spider middlewares: HttpErrorMiddleware, OffsiteMiddleware, RefererMiddleware, UrlCanonicalizerMiddleware, UrlLengthMiddleware, DepthMiddleware
2011-08-25 19:41:43-0700 [scrapy] DEBUG: Enabled item pipelines:
2011-08-25 19:41:43-0700 [scrapy] DEBUG: Telnet console listening on 0.0.0.0:6023
2011-08-25 19:41:43-0700 [scrapy] DEBUG: Web service listening on 0.0.0.0:6080
2011-08-25 19:41:43-0700 [default] INFO: Spider opened
2011-08-25 19:41:43-0700 [scrapy] DEBUG: Cookie: None for https://www.paypal.com
2011-08-25 19:41:44-0700 [scrapy] INFO: Set-Cookie: [] from https://www.paypal.com
2011-08-25 19:41:44-0700 [default] DEBUG: Crawled (501) <GET https://www.paypal.com> (referer: None)
2011-08-25 19:41:44-0700 [default] INFO: Closing spider (finished)
2011-08-25 19:41:48-0700 [default] INFO: Spider closed (finished)
$ wget https://www.paypal.com
--2011-08-25 19:44:08-- https://www.paypal.com/
Resolving us.proxymesh.com... 184.106.76.204
Connecting to us.proxymesh.com|184.106.76.204|:31280... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: `index.html'
I have scrapy 0.12.0.2545 , twisted 11.0.0 and python 2.7.
After some investigation, it appears that scrapy instead of issuing a CONNECT method and then doing a GET it is only issuing a GET requests which causes the fetch to fail.
Do you have any idea why this happens and how it can be fixed?
Thanks,
Oana
Pablo Hoffman
http://dev.scrapy.org/ticket/159
> 2011-08-25 19:41:44-0700 [default] *DEBUG: Crawled (501) <GET
> https://www.paypal.com>* (referer: None)
> 2011-08-25 19:41:44-0700 [default] INFO: Closing spider (finished)
> 2011-08-25 19:41:48-0700 [default] INFO: Spider closed (finished)
>
>
> $ wget https://www.paypal.com
> --2011-08-25 19:44:08-- https://www.paypal.com/
> Resolving us.proxymesh.com... 184.106.76.204
> Connecting to us.proxymesh.com|184.106.76.204|:31280... connected.
> Proxy request sent, awaiting response*... 200 OK*
> Length: unspecified [text/html]
> Saving to: `index.html'
>
> I have scrapy 0.12.0.2545 , twisted 11.0.0 and python 2.7.
>
> After some investigation, it appears that scrapy instead of issuing
> a CONNECT method and then doing a GET it is only issuing a GET
> requests which causes the fetch to fail.
>
> Do you have any idea why this happens and how it can be fixed?
>
> Thanks,
> Oana
>
>
>
>
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "scrapy-users" group.
> To post to this group, send email to scrapy...@googlegroups.com.
> To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.
>
Palash Jain
陈伟伟
在 2011年8月26日星期五 UTC+8上午11:04:23，Oana Goga写道：
Does Scrapy work with HTTP proxies?
HttpProxyMiddleware.