No results with middle function

Skip to first unread message

Nick Gilmour

Jun 12, 2017, 9:59:48 AM6/12/17
to pyspider-users
Hi all,

I wanted to have the response object in the first page, so I have created a new function named middle() between the functions on_start() and index_page(). Like this:

from pyspider.libs.base_handler import *

class Handler(BaseHandler):
    crawl_config = {

    @every(minutes=24 * 60)
    def on_start(self):
        self.crawl('', callback=self.middle)

    @every(minutes=24 * 60)
    def middle(self,reponse):
        self.crawl('', callback=self.index_page)
    @config(age=10 * 24 * 60 * 60)
    def index_page(self, response):
        for each in response.doc('a[href^="http"]').items():
            self.crawl(each.attr.href, callback=self.detail_page)

    def detail_page(self, response):
        return {
            "url": response.url,
            "title": response.doc('title').text(),

When I run this in the debugger mode everythin seems to be working fine - links are generated correctly and I can see the end results. When I set this script in DEBUG mode and let it run, I get no erros but there are no results. Am I missing something or is this a bug?

Nick Gilmour

Jun 12, 2017, 12:27:42 PM6/12/17
to pyspider-users

def middle(self,response):

Roy Binux

Jun 14, 2017, 10:06:06 PM6/14/17
to Nick Gilmour, pyspider-users

Are you actually crawling same URL?

You received this message because you are subscribed to the Google Groups "pyspider-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
To post to this group, send email to
To view this discussion on the web visit
For more options, visit
Reply all
Reply to author
0 new messages