scrapy raise exception run from out side the project directory

[Masood Rehman]

I have a scrapy project 'tutorial' in the following directory

C:\wamp64\www\tutorial>

the project directory structure is given below.

I want to run the spider through my custom python script runspiders.py

      from __future__ import print_function
      import scrapy
      from scrapy.crawler import CrawlerProcess
      from scrapy.utils.project import get_project_settings


      def main():
          process = CrawlerProcess(get_project_settings())
          process.crawl("quotes")
          process.start()

      if __name__ == '__main__' : main()

The spider runs correctly when the custom python script run from inside the scrapy project folder e.g

C:\wamp64\www\tutorial>python runspiders.py

But the scrapy raise the following exception when the custom python script run from outside the project folder e.g

C:\wamp64\www>python tutorial/runspiders.py

File "C:\Python27\lib\site-packages\scrapy\spiderloader.py", line 43, in load
raise KeyError("Spider not found: {}".format(spider_name))
KeyError: 'Spider not found: quotes'

Thank you in advance.

[Erik Dominguez]

The reason is because Scrapy looks for the scrapy.cfg file, which contains your project structure and project name. Running it outside your project folder where there isn't a scrapy.cfg will not work.