How to enable the Scrapy's duplicate urls filter for start_urls?

[Antoine Brunel]

Hello,

I found out that Scrapy's duplicate url filter RFPDupeFilter is disabled for urls set in start_urls. 

How can I enable it?

Thanks!

[张昊]

It will not work as you expect. You have to get rid of duplicate urls by your own code, maybe the "in" operation can help you. The RFPDupeFilter only works  in visited urls.

在 2016年5月3日星期二 UTC+8上午4:04:34，Antoine Brunel写道：

[Paul Tremberth]

Hi Antoine,

you can override the start_requests method of your spider.

the default is this (explicitly disabling filtering):

    def start_requests(self):

        for url in self.start_urls:

            yield self.make_requests_from_url(url)

    def make_requests_from_url(self, url):

        return Request(url, dont_filter=True)

You can change it to (default for Request is dont_filter=False):

    def start_requests(self):

        for url in self.start_urls:

            yield Request(url)

Regards,

Paul.

[Antoine Brunel]

It works perfectly Paul, thank you very much!