Been a while I'm working on a project based off scrapy. I happen to come at a point where I need to make a recursive call originating from the same parse function.
def parse_search_result(self, response):
try:
# import pdb; pdb.set_trace()
next_page = response.xpath("//div[@class='align_center']//node()[following::span and not(@class='noborder')]/@href").extract()[0]
next_page = str(next_page).translate(None, delete)
next_page_url = '{0}{1}'.format(self.base_url[0], next_page)
yield FormRequest(next_page_url, method="GET", callback = self.parse_search_result)
except:
pass
yield FormRequest(response.url, method="GET", callback = self.parse_applications)
I don know if I am looking at this piece of logic in a correct way?