Per-proxy throttling

[Aru Sahni]

Hello,

I'm currently assigning random proxies to requests via middleware. I'd like to key download throttling to the proxy that the request is using, but as far as I tell this is only possible when tied to domains or IPs.  Has anyone done this before? Any pointers would be appreciated.

Regards,
~Aru

[Mikhail Korobov]

Hi Aru,

You can use 'download_slot' Request.meta parameter to make throttle per-proxy. It is a string that Downloader uses as a key for a slot. If not specified in request meta, the host is used (or ip, if CONCURRENT_REQUESTS_PER_IP is True).

вторник, 16 сентября 2014 г., 3:30:35 UTC+6 пользователь Aru Sahni написал:

[Aru Sahni]

That did it! Thanks, Mikhail.

~Aru

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.