scrapy class in bat file

[d4v1d]

Hello,

I have several class scrapy (scrapy crawl crawl1; scrapy crawl crawl2 ...) I would like to start one after the other (with a bat, for example.).

I tried this (in a .bat file) :

cd C:\perso\crawler_test\tutorial
scrapy crawl crawl1
scrapy crawl crawl2
scrapy crawl crawl3

This don't work because it execute only the first class

Thanks for you help

Regards

[Luar Roji]

If I remember correctly, you can use the command "START" to spawn background processes..

something like:

START scrapy crawl crawl1

START scrapy crawl crawl2

...

You can check the syntax of the command START to see special options than suit best to you.

Best luck,

Luar 

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scrapy-users...@googlegroups.com.
To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.

[d4v1d]

Hello,

Thank you, It is an interesting idea but I don't think that answer to my problem

I would like to execute first: scrapy crawl crawl1

and as soon as it is finished, execute: scrapy crawl crawl2

The START command execut the 2 scripts at the same time

Can i ask the execution of "scrapy crawl crawl2" directly in the scrapy code?

thanks for your help

[d4v1d]

I find this :

http://doc.scrapy.org/en/0.20/topics/practices.html#running-multiple-spiders-in-the-same-process

I think this is what I need

But i don't really understand how this can work with 3 differents spider classes

have you got an example ?

thanks

[Bill Ebeling]

Are you sure you want to run the spiders all in the same process, and not just 'at the same time'.

If you just want to kick spiders off on a schedule then you want cron (if running in linux).  If you're using windows there is a scheduler like cron..  called task scheduler, or some such.

[David LANGLADE]

Hello

Thanks for your feed back

I am sorry, my explanations are not clear

I have 3 spiders that I execute like this: 

scrapy crawl crawl1 

scrapy crawl crawl2 

scrapy crawl crawl3

I would like to create a script that execute one after the other (when the fisrt is finish, the second start...)

Whith a planned task this don't work beacause I don't know the processing time of the first task

thanks for your help

Regards

2014/1/16 Bill Ebeling <bille...@gmail.com>

Are you sure you want to run the spiders all in the same process, and not just 'at the same time'.

If you just want to kick spiders off on a schedule then you want cron (if running in linux).  If you're using windows there is a scheduler like cron..  called task scheduler, or some such.

--
You received this message because you are subscribed to a topic in the Google Groups "scrapy-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scrapy-users/9gns7VyScEE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to scrapy-users...@googlegroups.com.

To post to this group, send email to scrapy...@googlegroups.com.
Visit this group at http://groups.google.com/group/scrapy-users.
For more options, visit https://groups.google.com/groups/opt_out.

--
David LANGLADE
5 rue du patuel
42800 Saint martin la plaine
Tel : 06.49.42.38.85

[Bill Ebeling]

What OS are you doing this with?

[Bill Ebeling]

with .bat's must be windows (I've done very little with windows scripting). 

Check this out, see if that helps: http://stackoverflow.com/questions/8177695/how-to-wait-for-a-process-to-terminate-to-execute-another-process-in-batch

[David LANGLADE]

ok it seem working with CALL 

thanks

2014/1/16 Bill Ebeling <bille...@gmail.com>