conditional settings ?

[Duy Nguyen]

I want to enable http proxy for some spiders, but disable http proxy for others. Can i do something like this:

proxy_spiders = ['a1' , b2']

if spider in proxy_spider: #how to get spider name ???
HTTP_PROXY = 'http://127.0.0.1:8123'

DOWNLOADER_MIDDLEWARES = {

     'myproject.middlewares.RandomUserAgentMiddleware': 400,

     'myproject.middlewares.ProxyMiddleware': 410,

     'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None

}
else:

DOWNLOADER_MIDDLEWARES = {

     'myproject.middlewares.RandomUserAgentMiddleware': 400,

     'scrapy.contrib.downloadermiddleware.useragent.UserAgentMiddleware': None

}

If the above is not possible, are there any work-around ?

[Rolando Espinoza La Fuente]

You can define your own proxy middleware, something straightforward like this:

from scrapy.contrib.downloadermiddleware import HttpProxyMiddleware

class ConditionalProxyMiddleware(HttpProxyMiddleware):

    def process_request(self, request, spider):

        if getattr(spider, 'use_proxy', None):

            return super(ConditionalProxyMiddleware, self).process_request(request, spider)

Then define the attribute use_proxy = True in the spiders that you want to have the proxy enabled. Don't forget to disable the default proxy middleware and enable your modified one.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.