Progress on WikiTeam and dumpgenerator API usage

18 views
Skip to first unread message

Federico Leva (Nemo)

unread,
Feb 10, 2020, 5:55:18 PM2/10/20
to wikiteam...@googlegroups.com
In the last few days I've rewritten large part of the so called
--xmlrevisions option, whose purpose is to allow using the API only
(rather than Special:Export) and to batch requests so that they are both
faster and more reliable.

Hopefully we'll manage to reduce the most annoying crashes due to memory
errors and similar, as well as unnecessary failures due to poor
interaction with some MediaWiki versions or some webserver
configurations. For instance, it shouldn't be up to the user to notice
that one webserver wants to receive API requests via POST and another
via GET. I don't want to hack such things in the code on the run, either.

I've closed milestone 0.3 and I added to the milestone version "0.4" the
bugs and features which I think can be verified and fixed around this
sort of changes.
https://github.com/WikiTeam/wikiteam/milestones

I'm not sure how big an improvement such changes will be for the average
user, frankly. (For istance I keep having zero motivation to fix issues
with Windows.) My benchmark is first of all whether I manage to archive
400k wikis, and secondarily if I manage to archive the large majority of
the wikis we already know about (without too much manual work).

Federico
Reply all
Reply to author
Forward
0 new messages