Script to upload downloaded wikis
Federico Leva (Nemo)
the archives of downloaded wikis in a directory (with a list of API
URLs?) and upload them to archive.org properly? I have about 1000
TaskForce wikis to upload, already 100+ GB of stuff to upload, and I
can't do all those tedious steps listed in
<https://code.google.com/p/wikiteam/wiki/NewTutorial#Publishing_the_dump> manually.
;-)
I'm uploading the biggest wikis to free some disk space, e.g.
http://archive.org/details/wiki-en.ecgpedia.org .
Nemo
emijrp
Could someone *please* write a simple script to automatically fetch all the archives of downloaded wikis in a directory (with a list of API URLs?) and upload them to archive.org properly? I have about 1000 TaskForce wikis to upload, already 100+ GB of stuff to upload, and I can't do all those tedious steps listed in <https://code.google.com/p/wikiteam/wiki/NewTutorial#Publishing_the_dump> manually. ;-)
I'm uploading the biggest wikis to free some disk space, e.g. http://archive.org/details/wiki-en.ecgpedia.org .
Nemo
--
Federico Leva (Nemo)
> There problem is, is there an API for that or whatever? How can I make a
> script that uploads + fills the IA form? The current upload button at IA
> is a flash app.
There's S3: http://www.archive.org/help/abouts3.txt
You only have to do a script that fetches the info and produces a
suitable curl command.[1]
Or, it could just output the info to a metadata.csv and then let you run
the bulk uploader: this sounds more complex and perhaps can't be
automated as easily as the former, but has built-in checks for for
completion, various errors etc. Perhaps it's better even for single-file
uploads? https://wiki.archive.org/twiki/bin/view/Main/IAS3BulkUploader
Nemo
[1] Like this:
curl --location --header 'x-amz-auto-make-bucket:1' \
--header 'x-archive-queue-derive:0' \
--header 'x-archive-size-hint:9638436173' \
--header "authorization: LOW vRwVKdY7hwEWUQOY:$redacted_by_me" \
--header 'x-archive-meta-mediatype:web' \
--header 'x-archive-meta-collection:opensource' \
--header 'x-archive-meta-title:Wiki - ECGpedia' \
--header 'x-archive-meta-description:<a
href="http://en.ecgpedia.org/" rel="nofollow">ECGpedia,</a>: a free
electrocardiography (ECG) tutorial and textbook to which anyone can
contribute, designed for medical professionals such as cardiac care
nurses and physicians. Dumped with <a
href="http://code.google.com/p/wikiteam/" rel="nofollow">WikiTeam</a>
tool.' \
--header 'x-archive-meta-subject:ecg; ECGpedia; wiki;
wikiteam; MediaWiki' \
--header
'x-archive-meta-licenseurl:http://creativecommons.org/licenses/by-nc-sa/3.0/'
\
--header
'x-archive-meta-rights:http://en.ecgpedia.org/wiki/Frequently_Asked_Questions'
\
--header
'x-archive-meta-originalurl:http://en.ecgpedia.org/api.php' \
--upload-file
/home/federico/ArchiveTeam/WikiTeam/enecgpediaorg-20120419-wikidump.7z \
http://s3.us.archive.org/wiki-en.ecgpedia.org/enecgpediaorg-20120419-wikidump.7z
> WT.log
emijrp
How can we get a "public" authorization key to hardcode it in the uploader GUI?
Also, have people to accept any terms (checkbox) before upload?
emijrp, 20/04/2012 20:11:
Federico Leva (Nemo)
> WOW, that is very cool.
>
> How can we get a "public" authorization key to hardcode it in the
> uploader GUI?
I don't know if it's a good idea. It quite surely can't be completely
public. For now, we don't have many downloaders and most of them already
have upload access.
> Also, have people to accept any terms (checkbox) before upload?
They have to register, that's the requirement to get a s3 key on
http://www.archive.org/account/s3.php (I don't know if you get one
automatically, I looked at it only after being told by an IA sysadmin).
Nemo
emijrp
emijrp, 20/04/2012 21:22:
Hydriz Wikipedia
Alex Buie
With regard to "public" s3 keys, IA has no notion of that. What we'd probably do is create an account for all uploaders to use, and then share the s3 keys through a private channel.
Alex
Federico Leva (Nemo)
> Nemo: If you are really dying of space, temporarily store them in an
> identifier that is personal to use (at least I am doing it now
> temporarily on http://archive.org/details/hydriz).
Thank you, but I don't like the idea much for various reasons.
I could download some more batches of wikis, but I had to stop
everything because of citywiki.ugr.es whose images will be well over 100
GiB (I downloaded half).
Now I'm only bringing to completion the biggest wikis (with many pages)
left in the other batches, until I can upload the 120 GB of the first
thousand wikis downloaded.
emijrp wrote a stub of
https://code.google.com/p/wikiteam/source/browse/trunk/uploader.py ; I
think he might use some help, in particular if someone knows how to
fetch information from the API, what curl exit codes to check for etc.
Nemo