scrapy + redis = awesome

[Rolando Espinoza La Fuente]

I've published a scrapy+redis integration. This allows to:
* run many crawlers for the same spider and share the workload
* run many post-processing workers to consume the items
* persist request queue, therefore pause/resume crawling

Certainly this is best suited for CPU-bound scrapers.

Requires latest development version and scrapy and haven't been
tested on production.

Source code: https://github.com/darkrho/scrapy-redis

Regards,

~Rolando

[Daniel Graña]

great work :)

so, to use redis as scheduler backend:

- set SCHEDULER to scrapy_redis.scheduler.Scheduler

- set SCHEDULER_PERSIST to True

and to store scraped items in redis for futher post processing:

- add scrapy_redis.pipelines.RedisPipeline to ITEM_PIPELINES

right?

[Rolando Espinoza La Fuente]

Yes. But SCHEDULER_PERSIST is optional, set to True only if you want the
pause/resume feature.

The RedisPipeline serialize the item using json, so the
post-processing can be done
in any language that support redis and json.

Regards

~Rolando

[shahidashraff]

how to set the server adress in the scheduler and how to run it 

[Andres Douglas]

Rolando, thanks for sharing this is really interesting. How stable is this at this point? It seems like it's been a while since you published it, but the code in github still has a warning about not being production ready? 

[Tuấn Lê]

Hi,

I'm research about scrapy-redis. But I don't know how it work. I need a example step by step.
Can you help me?

Thank you