Help with link extractor

[Tim Fitzhardinge]

Hi

I want to crawl through extracted url links on a webpage. I was trying to incorporate creating a spider template within the code rather than the genspider command line in the prompt 

The code with dummy urls are below.

What am I missing with my code. 

Thank you for your help.

import scrapy

from scrapy.spiders import BaseSpider

from scrapy.linkextractors import LinkExtractor

from scrapy.spiders import CrawlSpider, Rule

from bid.items import BidItem

class testSpider(CrawlSpider):

    name = 'test'

    allowed_domains = ['example.com.au']

    start_urls = ['http://www.example.com.au']

    rules = [

        Rule(LinkExtractor(allow_domains(example.com.au), follow=True, callback='parse_product'),

        ]

    def parse_product(self, response):

        title = response.xpath('//title/text()').extract()[0]

        print title

[Sayth Renshaw]

What is the error you receive?

Sayth

[Tim Fitzhardinge]

The website I was crawling was the asx.com.au. So change example.com.au with asx. When a run the spider I get the following in the console window. Why does it not any pages.

2016-11-06 16:12:20 [scrapy] INFO: Spider opened

2016-11-06 16:12:20 [scrapy] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)

2016-11-06 16:12:20 [scrapy] DEBUG: Telnet console listening on 127.0.0.1:6023

2016-11-06 16:12:20 [scrapy] DEBUG: Crawled (200) <GET http://www.asx.com.au/robots.txt> (referer: None)

2016-11-06 16:12:20 [scrapy] DEBUG: Crawled (200) <GET http://www.asx.com.au/> (referer: None)

2016-11-06 16:12:20 [scrapy] INFO: Closing spider (finished)

2016-11-06 16:12:20 [scrapy] INFO: Dumping Scrapy stats:

{'downloader/request_bytes': 434,

 'downloader/request_count': 2,

 'downloader/request_method_count/GET': 2,

 'downloader/response_bytes': 41080,

 'downloader/response_count': 2,

 'downloader/response_status_count/200': 2,

 'finish_reason': 'finished',

 'finish_time': datetime.datetime(2016, 11, 6, 5, 12, 20, 752027),

 'log_count/DEBUG': 3,

 'log_count/INFO': 7,

 'response_received_count': 2,

 'scheduler/dequeued': 1,

 'scheduler/dequeued/memory': 1,

 'scheduler/enqueued': 1,

 'scheduler/enqueued/memory': 1,

 'start_time': datetime.datetime(2016, 11, 6, 5, 12, 20, 329100)}

2016-11-06 16:12:20 [scrapy] INFO: Spider closed (finished)