Scrapyd: Scheduling spiders on weekly basis?

[Haider Mahmood]

Is there a way where we can schedule my spider using scrapyd on weekly basis? Any config file available for this?

Regards,

Haider Mahmood

[Pablo Hoffman]

No, Scrapyd does not currently support any mechanism for configuring recurring/periodic crawls of any kind. What you typically do is setup a cron job that calls schedule.json with your desired schedule (daily, weekly, custom, etc). Running this from cron gives you a lot of flexibility on how to configure the schedule, although having something on Scrapyd to drop the dependency with cron would be nice too.

--
You received this message because you are subscribed to the Google Groups "scrapy-users" group.
To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/4gsZBIBCVlMJ.
To post to this group, send email to scrapy...@googlegroups.com.
To unsubscribe from this group, send email to scrapy-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/scrapy-users?hl=en.

[Pablo Hoffman]

Yes, you could just run "scrapy crawl" from cron, as Scrapy is not coupled (in any way) with Scrapyd.

Scrapyd is an application (not a programming framework, like Scrapy) that lets you manage your running spiders more conveniently by providing a HTTP api to schedule jobs and download data, and a web UI (although still basic, but would improve over time) for visualizing spiders running, pending and completed, as long as a way to control/limit how many spiders can run in parallel (in order not to overload the machine).

On Mon, Nov 19, 2012 at 7:33 AM, Spadez <block...@gmail.com> wrote:

Sorry for the basic question and possible hi-jack, but what is the point of scrapyd, couldnt you just call the command line tool from cron instead?

To view this discussion on the web visit https://groups.google.com/d/msg/scrapy-users/-/PQTezVWRoY0J.