Callback from pipeline or between spiders

25 views
Skip to first unread message

Billy John

unread,
Mar 26, 2017, 11:54:14 AM3/26/17
to scrapy-users
Hi. How can I make a callback from a pipeline? How can I make a callback to different spider when spider 1 has finished extraction? Say in first cycle I generate ids. Then I split them into chunks of 200 and create one huge URL for each chunk. In my understanding this can be accomplished from pipeline or by using 2 all spiders.

Lhassan Baazzi

unread,
Mar 26, 2017, 12:30:59 PM3/26/17
to scrapy...@googlegroups.com
Hi,

I think you can do it in one spider a callback that generate ids and then call the second callback to construct the huge URL.
If you want to do it using two spiders you need to run them programmatically http://kirankoduru.github.io/python/running-scrapy-programmatically.html


Best Regards.
Lhassan.


2017-03-26 7:24 GMT+02:00 Billy John <arch...@gmail.com>:
Hi.  How can I make a callback from a pipeline?  How can I make a callback to different spider when spider 1 has finished extraction? Say in first cycle I generate ids. Then I split them into chunks of 200 and create one huge URL for each chunk. In my understanding this can be accomplished from pipeline or by using 2 all spiders.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users+unsubscribe@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at https://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

Billy John

unread,
Mar 27, 2017, 5:52:36 AM3/27/17
to scrapy-users
Yeah, but how do I access all generated id,s to make new call back? That is the main question. Id,s are stored in item, so there should be a way to make call back from pipeline. Or do I miss smth? Please show example of your logic?
Reply all
Reply to author
Forward
0 new messages