Scrapy custom_settings cant be seen in a log. How do I know they are applied?

123 views
Skip to first unread message

Billy John

unread,
Aug 21, 2017, 2:11:02 AM8/21/17
to scrapy-users

I use scrapy 1.3 According to documentation I can declare settings for each specific spider simple enough. Here is my code

class BhSpider(scrapy.Spider):
name = "code"

custom_settings = {'CONCURRENT_REQUESTS' : '20',
                   'FEED_EXPORT_FIELDS' :['price', 'stock','partnumber','sku', 'name' ,'manufacture','attribute','distributor','upc','descr', 'p_url','main_image','images']} 



allowed_domains = ***

But according to logs project settings are used. How do i know project settings are overcomed by this specific spider settings?

here is the log

 2017-08-20 13:47:51 [scrapy.utils.log] INFO: Scrapy 1.3.2 started (bot: pc)
2017-08-20 13:47:51 [scrapy.utils.log] INFO: Overridden settings: {'NEWSPIDER_MODULE': 'pc.spiders'
, 'FEED_URI': 'test_output1.csv', 'CONCURRENT_REQUESTS': 200, 'SPIDER_MODULES': ['pc.spiders'], 'BO
T_NAME': 'pc', 'USER_AGENT': 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) C
hrome/59.0.3071.115 Safari/537.36', 'FEED_FORMAT': 'csv', 'FEED_EXPORT_FIELDS': ['price', 'stock',
'partnumber', 'sku', 'name', 'manufacture', 'attribute', 'distributor', 'upc', 'descr']}
2017-08-20 13:47:51 [scrapy.middleware] INFO: Enabled extensions:

In the output file Export fields seem to be correct( as in custom settings, though log does not show that). How can i know that specific settings are enabled?

Reply all
Reply to author
Forward
0 new messages