What is the purpose of "version" parameter in addversion.json call to scrapyd?

53 views
Skip to first unread message

Sergey Zhemzhitsky

unread,
Jan 24, 2016, 2:52:03 PM1/24/16
to scrapy-users
Hi there,

I cannot figure out what is the purpose of "version" parameter in "addversion.json" call to scrapyd.
Although it is possible to add and delete versions with "addversion.json" and "delversion.json" it is not possible to specify a version when scheduling a crawling with "schedule.json" that seems to be pretty natural.

If, let's say, I would like to create two versions "1.0.0" and "2.0.0" both of which contain spider with the same name "MySpider" and then to schedule this spider, spider from what egg is going to be scheduled?

$ curl http://localhost:6800/addversion.json project=my_spiders -F version=1.0.0 -F egg=@my_spiders-1.0.0-py2.7.egg
$
curl http://localhost:6800/addversion.json project=my_spiders -F version=2.0.0 -F egg=@my_spiders-1.0.0-py2.7.egg
$ curl http://localhost:6800/schedule.json -d project=my_spiders -d spider=MySpider

Maybe I'm missing something? Could you please help?

Kind regards,
Sergey

Nikolaos-Digenis Karagiannis

unread,
Jan 25, 2016, 3:08:16 AM1/25/16
to scrapy-users
Although it looks like it's made to serve scheduling spiders in multiple versions,
the reason is quite elementary.
When uploading a new version, there may still be spiders from an older version running.

Do you need to be able to schedule multiple versions?
It should be possible to build such a feature.

Sergey Zhemzhitsky

unread,
Feb 1, 2016, 10:14:58 AM2/1/16
to scrapy-users
>> Do you need to be able to schedule multiple versions?
Yep. it seems to be pretty natural.
If older spider versions are running I'd like to start new versions and let the old ones to complete and shutdown gracefully.

Dimitris Kouzis - Loukas

unread,
Feb 2, 2016, 9:23:42 PM2/2/16
to scrapy-users
Check-out this Pull Request on a way to do this: https://github.com/scrapy/scrapyd/pull/123 

Sergey Zhemzhitsky

unread,
Feb 6, 2016, 7:10:12 AM2/6/16
to scrapy-users
Hi Demitris,

Thanks a lot! Still no luck - scrapyd's rest api does not handle _version parameter or I'm missing something.

curl 'http://localhost:6800/listspiders.json?project=myproject&_version=1_0_0' always return spiders from the latest version.

Dimitris Kouzis - Loukas

unread,
Feb 8, 2016, 3:35:12 PM2/8/16
to scrapy-users
Hey Sergey, no worries... follow-up on that PR... it will likely be in the next version of Scrapyd, but with little effort, you should will be able to patch your existing version and get this feature working right now.

Sergey Zhemzhitsky

unread,
Feb 9, 2016, 7:57:43 AM2/9/16
to scrapy-users
Hi Dimitris,

Thanks to you, we've already patched locally installed scrapyd. New version support works perfectly.

Dimitris Kouzis - Loukas

unread,
Feb 10, 2016, 6:00:48 AM2/10/16
to scrapy-users
Awesome! That's great news, happy to help :))
Reply all
Reply to author
Forward
0 new messages