Re: Scrapyd Deployment

[Steven Almeroth]

Deploying your project into a Scrapyd server typically involves two steps: 1.) building a Python egg of your project. This is called “eggifying” your project and 2.) uploading the egg to the Scrapyd server.

https://scrapy.readthedocs.org/en/latest/topics/scrapyd.html?highlight=scrapyd#deploying-your-project

On Friday, November 9, 2012 8:42:45 AM UTC-6, Haider Mahmood wrote:

I have a set of projects which use a file MyBaseFile (from MyLibs.Common.MyBaseFile import MyBaseFileClass) which are present outside the scope of project. 

Using the following command: scrapy deploy -d scrapyd2 -p projectname, I get the error following error:

Server response (200):

{"status": "error", "message": "ImportError: No module named MyLibs.Common.MyBaseFile"}

How can I tell scrapyd where MyLibs is so there is no issue deploying the project?

Regards,

Haider

[Haider Mahmood]

Yes, I know. But my actual problem is this:

I have a Project named MyScrapers whose spider classes uses an import statement as follows:

from mylibs.common.my_base_spider import MyBaseSpider

The path to my_base_spider is /home/myprojectset/mylibs/common/my_base_spider

While setting environment variable PYTHONPATH=$HOME/myprojectset/, I am able to run MyScrapers using scrapy command: scrapy crawl MyScrapers.

But when I use scrapyd for deploying MyScrapers by following command: scrapy deploy scrapyd2 -p MyScrapers, I get the following error:

Server response (200): {"status": "error", "message": "ImportError: No module named mylibs.common.my_base_spider"}

Please tell how to make deployed project to use these libs?

Regards,

Haider

[Haider Mahmood]

The thing is mylibs is outside my project. In development, I used PYTHONPATH and scrapy crawl MyScraper ran with no issues. But scrapyd wont pick PYTHONPATH, so I need some info on how can I use mylibs on deployed project?

[Robert Forkel]

You'll have to install mylibs as regular python package in the correct python site-packages of the machine running scrapyd.

Am 11.11.2012 10:15 schrieb "Haider Mahmood" <haider....@arbisoft.com>:
The thing is mylibs is outside my project. In development, I used PYTHONPATH and scrapy crawl MyScraper ran with no issues. But scrapyd wont pick PYTHONPATH, so I need some info on how can I use mylibs on deployed project?

On Friday, November 9, 2012 7:42:45 PM UTC+5, Haider Mahmood wrote:
>

> I have a set of projects ...

To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/7Y2eWJfwxecJ.

To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group,...

[Haider Mahmood]

Thanks Robert. That solved my problem :)

[Steven Almeroth]

It is more portable to install libraries as eggs: http://doc.scrapy.org/en/latest/topics/scrapyd.html#eggs-dir