disable duplicate URL filtering

[Kumar Abhishek]

I want to disable the duplicate URL filtering feature. I think it's triggered by DUPEFILTER_CLASS which is set to a default value of 'scrapy.dupefilter.RFPDupeFilter'. Is there a way to disable this feature/setting ?

[Pablo Hoffman]

You can change DUPEFILTER_CLASS to scrapy.dupefilter.BaseDupeFilter which doesn't filter anything, or create your Request objects with dont_filter=True, which causes them to be ignored by the dupe filter.

On Sun, Aug 4, 2013 at 8:24 PM, Kumar Abhishek <kr.a...@gmail.com> wrote:

I want to disable the duplicate URL filtering feature. I think it's triggered by DUPEFILTER_CLASS which is set to a default value of 'scrapy.dupefilter.RFPDupeFilter'. Is there a way to disable this feature/setting ?

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.