Scrapy RabbitMQ Redis Library

35 views

Skip to first unread message

Rakesh Chawda

unread,

Jun 2, 2016, 7:51:23 AM6/2/16

to scrapy-users

Hey guys,

I've combined a library from two current libraries: scrapy-redis and scrapy-rabbitmq: https://github.com/rdcprojects/scrapy-mq-redis

I found that scrapy-rabbitmq incorrectly uses rabbitmq for duplicate filtering which it cannot do. So, I added the dupefilter from scrapy-redis to it and made some other fixes. One of the fixes is sending acknowledgement to rabbitmq when the Request is processed so that RabbitMQ can mark the message as finished and remove it from queue.

I plan to add one more feature which lets you add more urls while the spiders are running.