Task force wikis uploader

[Federico Leva (Nemo)]

After a few very busy weeks I'm going to start a new bunch of wikis
download. http://code.google.com/p/wikiteam/wiki/TaskForce
I'm currently at 4982 7z archives to upload (so about 2500 wikis) for a
total of 147 GB. We still need to write the tool to upload the to
archive.org; emijrp started it at
http://code.google.com/p/wikiteam/source/browse/trunk/uploader.py but we
need someone to complete it, in particular with the metadata fetch from
the wikis and the error checks (to avoid missing/partial uploads or even
worse deletion of non-uploaded wikis). Please help!

Nemo

[emijrp]

I have coded a bit more of the uploader script, and written instuctions here http://code.google.com/p/wikiteam/wiki/TaskForce#footer

We are testing the script, it is not 1.0.

2012/5/27 Federico Leva (Nemo) <nemo...@gmail.com>

After a few very busy weeks I'm going to start a new bunch of wikis download. http://code.google.com/p/wikiteam/wiki/TaskForce
I'm currently at 4982 7z archives to upload (so about 2500 wikis) for a total of 147 GB. We still need to write the tool to upload the to archive.org; emijrp started it at http://code.google.com/p/wikiteam/source/browse/trunk/uploader.py but we need someone to complete it, in particular with the metadata fetch from the wikis and the error checks (to avoid missing/partial uploads or even worse deletion of non-uploaded wikis). Please help!

Nemo

--

Emilio J. Rodríguez-Posada. E-mail: emijrp AT gmail DOT com

Pre-doctoral student at the University of Cádiz (Spain)

Projects: AVBOT | StatMediaWiki | WikiEvidens | WikiPapers | WikiTeam

Personal website: https://sites.google.com/site/emijrp/

[Hydriz Wikipedia]

Wait, stop, stop!

I think the item identifier is wrong, we need to fix that first...

--

Regards,

Hydriz

We've created the greatest collection of shared knowledge in history. Help protect Wikipedia. Donate now: http://donate.wikimedia.org

[emijrp]

for the dots in the name or what?

2012/8/7 Hydriz Wikipedia <ad...@alphacorp.tk>

[Hydriz Wikipedia]

Yeah, the domain name. The directories (that is converted to underscores) too (unless we agree to that?)...

[emijrp]

Is it really important the item name? If there is no way to move, now we have about 700 wikis without dots, and a few with dots in domain.

2012/8/7 Hydriz Wikipedia <ad...@alphacorp.tk>

[Mike Dupont]

On Tue, Aug 7, 2012 at 1:23 PM, Hydriz Wikipedia <ad...@alphacorp.tk> wrote:
> eam/source/browse/trunk/uploader.py but we need someone to complete it, in
> particular with the metadata fetch from the wikis and the error checks (to
> avoid missing/partial uploads or even worse deletion of non-uploaded wikis).
> Please help!

Well a short glance at the code,
http://code.google.com/p/wikiteam/source/browse/trunk/uploader.py#188
why are you not using boto instead ? Are you sure it is a good idea to
reinvent the s3 api?


--
James Michael DuPont
Member of Free Libre Open Source Software Kosova http://flossk.org
Saving wikipedia(tm) articles from deletion http://SpeedyDeletion.wikia.com
Contributor FOSM, the CC-BY-SA map of the world http://fosm.org
Mozilla Rep https://reps.mozilla.org/u/h4ck3rm1k3

[Hydriz Wikipedia]

If currently it is being uploaded this way, then just continue with it this way (unless of course you can change it for future dumps) :)

[Hydriz Wikipedia]

We have been using this format of code (using curl to upload) for quite some time. I am hoping to implement boto though, as it supports multipart uploading nicely, errors too.

But, can you help out in this? :P

[Mike Dupont]

Yes i already committed code in my branch that does uploading, did you see?

On Tue, Aug 7, 2012 at 1:57 PM, Hydriz Wikipedia <ad...@alphacorp.tk> wrote:
> But, can you help out in this? :P

[Hydriz Wikipedia]

Oh yes, you said that before *kicks myself*

Hopefully I can implement that in my own scripts too :)

[Mike Dupont]

But there is still lots to do. I like the idea of creating buckets
based on the description of the wiki, I dont have anything for that at
all yet. I am working on some other code right now, working on a bit
torrent tracker in perl.
https://github.com/h4ck3rm1k3/BitTorrent-TrackerCGI-sqlite when I have
it working nice I will come back to do some more wikiteam hacking.,
mike

[Federico Leva (Nemo)]

emijrp, 08/07/2012 03:47 PM:

> Is it really important the item name? If there is no way to move, now we
> have about 700 wikis without dots, and a few with dots in domain.

I had some problems so I'm reading emails only now. I thought we had
switched to the new naming because of some discussion here in the
meanwhile. The original instructions underscor forwarded us asked the
dots, but later we've been made aware that domain-like item names can
give problems so I guess this is not a tragedy, let's just update the docs.

Nemo

[Hydriz Wikipedia]

Hmm, thinking about it, I do need to confirm a few stuff first:

1. Is boto a default python package for all Unix systems?
2. Does Windows have support for boto?

IMO we were using this hackish method of using curl as we need to care about our Windows users, and perhaps not all systems have boto installed (and can't install them).

If any of the above questions are a "no", then I don't think boto can be implemented :(

[Platonides]

On Sat, Aug 11, 2012 at 5:55 PM, Hydriz Wikipedia <ad...@alphacorp.tk> wrote:
> Hmm, thinking about it, I do need to confirm a few stuff first:
>
> Is boto a default python package for all Unix systems?
> Does Windows have support for boto?
>
> IMO we were using this hackish method of using curl as we need to care about
> our Windows users, and perhaps not all systems have boto installed (and
> can't install them).
>
> If any of the above questions are a "no", then I don't think boto can be
> implemented :(

boto package is not installed in the toolserver. Relevant for the
question if it's a "default everywhere", I'm sure TS roots would
gladly install it.

[Hydriz Wikipedia]

I just checked with the Sourceforge shell service, it does not have boto as well :(

So, we can just cast this proposal aside (though it might be picked up later on when we deal with multipart uploading)

[Mike Dupont]

I just included it in my directory, you dont need to make it default.
Windows? who uses that? no idea. Why dont you try my scripts they
should work.

[Mike Dupont]

It is in my h4ck3rm1k3 home dir, experiments wikiteam, easy.

/home/h4ck3rm1k3/experiments/wikiteam/boto it just works.

[Hydriz Wikipedia]

Hmm, we can package boto inside our repository, thats a good idea...

Windows? AFAIK emijrp does (I think) :P

[Mike Dupont]

look, as i said, just put it in your source dir with your files.

[Hydriz Wikipedia]

Issue 54 :)

[Alex Buie]

I've found that curl under cygwin can enter into conditions that cause
it to corrupt files greater than 4GB. I really would recommend
double-triple checking MD5s of the uploads if you're uploading via
curl on Windows.

(just a warning)

[emijrp]

I don't use Windows, but I would like that our script works on it.

2012/8/11 Hydriz Wikipedia <ad...@alphacorp.tk>

[Mike Dupont]

People,
lets have a discussion about facts here, test it, and report bugs,
lets not get into too much conjecture.
thanks
mike