yield item issue in errback or in case of 302 Request

123 views
Skip to first unread message

akhter wahab

unread,
Feb 7, 2012, 3:38:22 PM2/7/12
to scrapy-users
i have an issue using scrapy

yield Request(a_url[0],
meta={'item': aitem}, dont_filter=True,
callback=self.redeem_url, errback=self.error_page)


def redeem_url(self, response):
item = response.request.meta['item']
item['Click_to_Redeem_URL'] = response.url
yield item
aitem is populated before doing a_url[0] request some times i get
302,404,301 status from Request so what i want is if i can't get 200
response from a_url[0] request item should yield i didn't found any
way because when i got 302 scrapy retry on this request and not go to
error_page errback, and if in 404 it goes to error_page what would be
the way to yield item over there because what i know in errback i
can't get response object but a failure object that not contains item
in meta

thanks in advance

vitsin

unread,
Feb 8, 2012, 10:56:49 AM2/8/12
to scrapy-users
You can implement your own middleware to handle non-200 HTTP
responses,
than such HTTP response not even need to go down to your below code -
handle errors in middleware.

--vs

Pablo Hoffman

unread,
Feb 11, 2012, 4:49:57 PM2/11/12
to scrapy...@googlegroups.com
Check the documentation for the following Request meta keys:
  • dont_redirect
  • http_handlestatus_all
Reply all
Reply to author
Forward
0 new messages