How can I use spider-specific settings

[zdbei...@gmail.com]

Hi,

There are several spiders in my one project. I've written my basic global settings in my project's settings.py and I don't know how to override some settings for different spider.

Just like spider1 can override settings: ITEM_PIPELINES = {'myproject.pipelines.UserImagesPipeline':1,'myproject.pipelines.WeibospiderPipeline':300}

while spider2 can stll use the settings: ITEM_PIPELINES = {'myproject.pipelines.UserImagesPipeline':None,'myproject.pipelines.WeibospiderPipeline':300} in the  project's settings.py

I've found the following method in the Core API of the Scrapy documentation, and I once try to use this method like this:

Solution1:

from scrapy.settings import Settings

class MySpider(BaseSpider):

      myset = Settings()

      myset.set('ITEM_PIPELINES',{'myproject.pipelines.UserImagesPipeline':1,'myproject.pipelines.WeibospiderPipeline':300},priority='spider')

Solution2:

from scrapy.conf import settings

class MySpider(BaseSpider):

      settings.set('ITEM_PIPELINES',{'myproject.pipelines.UserImagesPipeline':1,'myproject.pipelines.pipelines.WeibospiderPipeline':300},priority='spider')

However,these didn't work for me. Besides, Solution2 can change all the spider's settings, and actually module scrapy.conf is deprecated now.

Is there a way to use different settings for spider1 and spider2 separately?

Thanks! 

Bill     

[Jeremy D]

May be worth giving more information as to why these solutions didn't work (error messages, what happened vs what you expected etc) specifically for solution 1. 

Additionally, if the only thing you're looking to change is to not send images through, could not just set the `image` variable in your item class to None?

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/d/optout.

[Paul Tremberth]

Scrapy 1.0 has support for per-spider settings:

http://doc.scrapy.org/en/latest/topics/settings.html#settings-per-spider

just add a `custom_settings` attribute to your spider class,

and you should be good.

--

[Paul Tremberth]

From the release notes: http://doc.scrapy.org/en/latest/news.html?highlight=custom_settings#per-spider-settings-gsoc-2014

[zdbei...@gmail.com]

Thank you for your help！^_^

在 2015年10月7日星期三 UTC+8下午9:09:34，Jeremy D写道：

[zdbei...@gmail.com]

Thank you so much! It's helpful~~

在 2015年10月7日星期三 UTC+8下午9:18:19，Paul Tremberth写道：