scrapyd deployment to remote server

271 views
Skip to first unread message

Michael Pastore

unread,
Apr 6, 2014, 10:53:29 AM4/6/14
to scrapy...@googlegroups.com
Forgive the confusion, but I was hoping that someone could clear up the following. 

I would like to run/manage several spider instances on a remote server using scrapyd, but I am developing the spiders on local system.  So, if I am developing spiders on machineA and want to deploy them and run them using scrapyd on machineB, do I need scrapyd installed on both machines?  For example, the scrapyd deploy command  scrapyd-deploy scrapyd -p project1 implies you are on the same instance in which scrapyd is running.  If that is the case, then the example command also implies project1 is a scrapy project on that same instance.

Thanks in advance for any clarification.


Michael Pastore

unread,
Apr 13, 2014, 1:47:19 PM4/13/14
to scrapy...@googlegroups.com
After trial and error, finally was able to deploy my crawler to a remote EC2 instance using scrapyd.  Details are here: http://bgrva.github.io/blog/2014/04/13/deploy-crawler-to-ec2-with-scrapyd/
Reply all
Reply to author
Forward
0 new messages