One little question ( import)

[field....@gmail.com]

Hey, everyone. I'm new to scrapy and there are things I do not understand. 

The question I have is this.

When I create a spider how to figure out what to import? Maybe I'm a Python really do not know, do not argue.

For example.

rules = (
    Rule(LinkExtractor(allow=(r"href=\"\/courses\/\d*\"", )), callback='parse'),
)

Doesn't work for me, if I don't do from scrapy.contrib.spiders import CrawlSpider, Rule

Here's how I understand what and when to import? 

Thank you.

[Paul Tremberth]

Hi there,

welcome to the scapy community!

The online documentation has lots of code snippets with valid import statements.

For example for CrawlSpider, you can check https://docs.scrapy.org/en/latest/topics/spiders.html#crawlspider-example

which shows these imports:

import scrapy
from scrapy.spiders import CrawlSpider, Rule
from scrapy.linkextractors import LinkExtractor

You also have the Index (https://docs.scrapy.org/en/latest/genindex.html) to go to the different classes and methods.

(Most classes are documented on https://docs.scrapy.org and listed in the index)

I was about to take LinkExtractor as an example, but in fact it's an alias for

scrapy.linkextractors.lxmlhtml.LxmlLinkExtractor

See https://docs.scrapy.org/en/latest/topics/link-extractors.html#module-scrapy.linkextractors

Let's take Rule as example.

In the index, you can find it on https://docs.scrapy.org/en/latest/genindex.html#R

Then you follow https://docs.scrapy.org/en/latest/topics/spiders.html#scrapy.spiders.Rule

and you can see

classscrapy.spiders.Rule(link_extractor, callback=None, cb_kwargs=None, follow=None, process_links=None, process_request=None)

So to use the Rule class and instances in your code, you'll have to import something like this:

from scrapy.spiders import Rule

Hope this helps.

/Paul.

[field....@gmail.com]

Paul, thank you very much. I figured out how to import all thanks to you. You answer is so painted that the questions and left. Thanks again.