some error in scrapy redis

35 views
Skip to first unread message

crawler

unread,
Jun 3, 2015, 6:45:21 AM6/3/15
to scrapy...@googlegroups.com
hi,when i use scrapy redis ,i found this problem:
i write two spider,the first spider parse list urls and then obtain detail url then push detail url to redis schedule,
when i run the second spider ,
i want the second spider call some function,like def start_requets(self): (this method is login method,and get cookies,after get cookies can visit detail url),this spider didn't call def start_request(self): function,this spider get detail url from redis schedule,and then this spider request detail url,and response code 302(because the second spider not cookies).

def start_requests(self):
url = 'http://XXXXX'

yield FormRequest(url=url,formdata=data, callback=self.get_checkcode,dont_filter=True)


hope ,some one give me advice.

thank you .

crawler

unread,
Jun 3, 2015, 7:09:46 AM6/3/15
to scrapy...@googlegroups.com
hi,when i use scrapy redis ,i found this problem:
i write two spider,the first spider parse list urls and then obtain detail url then push detail url to redis schedule,
when i run the second spider ,
i want the second spider call some function,like def start_requets(self): (this method is login method,and get cookies,after get cookies can visit detail url),this spider didn't call def start_request(self): function,this spider get detail url from redis schedule,and then this spider request detail url,and response code 302(because the second spider not cookies).

i want to call some function before my spider get url from redis schedule , after spider run these function ,this spider get url from redis schedule,then request these url.
Reply all
Reply to author
Forward
0 new messages