Anyone using PyCharm IDE for their scrapy spider development and testing?

5,120 views
Skip to first unread message

colorado5280

unread,
Oct 16, 2013, 8:25:51 PM10/16/13
to scrapy...@googlegroups.com
I use PyCharm IDE for all of my Python development and, of late, have run into a strange behavior when trying to build and debug multiple spiders that are contained in a project.  First, some background:

1) In my project I have 10 or so spiders - all Python code
2) As I build a new spider I set up a PyCharm configuration as follows: 

3) Next I hit debug to run the code and Pycharm indicates that its running the first alphetically named spider in the same directory.  

4) This seems to be something with how Pycharm 3.0 is working as I had done lots of development on other spiders and was able to write and debug each one individually before.  


Anyone using PyCharm to do their development?  If so can you share how you set up your configuration to be able to debug each spider individually?

Thanks for any assistance provided!


Mark


William Kinaan

unread,
Feb 14, 2014, 3:12:36 PM2/14/14
to scrapy...@googlegroups.com
I have the same questions. Have you known the answer ?

colorado5280

unread,
Feb 14, 2014, 3:18:30 PM2/14/14
to scrapy...@googlegroups.com
I've not received any follow up.  I've tried a number of things but still get other crawlers running when I only want to debug one.  

For now I have two projects.  When I need to develop/debug a crawler I put it in one project where it is the only crawler.  I do my work and then move it to the other project that has all of them in place.  

I'd love to understand why this is happening and find a solution.  It certainly makes development/debugging difficult.

Akhter Wahab

unread,
Feb 19, 2014, 3:21:59 AM2/19/14
to scrapy...@googlegroups.com
i am using pycharm 

in

Script : path of scrapy 

script parameters : crawl spiderName

Working directory : directory path where scrapy.cfg file exists 

bol...@gmail.com

unread,
Apr 20, 2015, 6:01:42 PM4/20/15
to scrapy...@googlegroups.com
Reply all
Reply to author
Forward
0 new messages