wikiteam-discuss Google Group

Wikimedia Commons grab

nemow...@gmail.com (Federico Leva (Nemo)) — Wed, 22 May 2013 14:07:00 UT

Hydriz, as regards
[link] , it
really makes no sense to stop the Commons archiving "in the process of
trying to optimize bandwidth and resource usage".
From what I understood, the blockers were special characters and some
database corruption. If that's been fixed or worked around, we must

Re: [wikiteam-discuss:640] Move Wikia dumps to Internet Archive

nemow...@gmail.com (Federico Leva (Nemo)) — Sat, 04 May 2013 09:40:43 UT

Found a method stupid enough for me: I downloaded the HTML and JSON with
wget,
<[link]>
Grep tells me there are now 37440 wikis, 29502 last dumped in 2012 and
6111 in 2013, 3988 in April.

Nemo

Re: [wikiteam-discuss:640] Move Wikia dumps to Internet Archive

nemow...@gmail.com (Federico Leva (Nemo)) — Fri, 03 May 2013 10:48:07 UT

Emilio J. Rodr�guez-Posada, 03/05/2013 09:29:
> I don't see many new dumps in [link] but I haven't
> counted.
>
>
> count it ...

I will at some point.

> But we have dumps only of 10 % of their wikis, which are about 340k.
>
>
> Probably 90% of them are empty (just the mainpage).

Re: [wikiteam-discuss:640] Move Wikia dumps to Internet Archive

emi...@gmail.com (Emilio J. Rodríguez-Posada) — Fri, 03 May 2013 07:29:03 UT

2013/5/2 Federico Leva (Nemo) <nemow...@gmail.com>

count it ...

Probably 90% of them are empty (just the mainpage).

If someone can write a simple bot to scrape ?Special:Statistics page and
read the good=XXX value to count how many wikis have real content...

Re: [wikiteam-discuss:640] Move Wikia dumps to Internet Archive

nemow...@gmail.com (Federico Leva (Nemo)) — Thu, 02 May 2013 17:08:18 UT

Emilio J. Rodr�guez-Posada, 02/05/2013 18:49:

I don't see many new dumps in [link] but I haven't
counted. The help page is wrong, by the way, autoconfirmed is enough.

But we have dumps only of 10 % of their wikis, which are about 340k.

Nemo

Re: [wikiteam-discuss:640] Move Wikia dumps to Internet Archive

emi...@gmail.com (Emilio J. Rodríguez-Posada) — Thu, 02 May 2013 16:49:07 UT

How many dumps are being generated by sysops?

We need to know how many wikis have been edited since last backup and no
new dump is available.

Anyway, Wikia wikis were saved in December 2012, so it is not an emergency
by now.

2013/5/1 Federico Leva (Nemo) <nemow...@gmail.com>

Re: [wikiteam-discuss:640] Move Wikia dumps to Internet Archive

nemow...@gmail.com (Federico Leva (Nemo)) — Wed, 01 May 2013 16:27:47 UT

Ah, I hadn't seen it: now dumps can only be requested by wiki sysops.
<[link]>
What should we do, run a bot to request all sysops to press the button?
Just start crawling all wikis directly at last?

Nemo

Our collections have moved

ad...@alphacorp.tk (Hydriz Wikipedia) — Sat, 20 Apr 2013 14:14:55 UT

Hi all,
In case you have not noticed, all collections related to wikis ("wikiteam"
and "wikimediadownloads") have all been moved to a new collection called
"wikicollections", though still under the web collection. I have already
sent an email to the staff regarding the rationale behind this change, and

Fwd: [Xmldatadumps-l] making imports suck less, the sequel

nemow...@gmail.com (Federico Leva (Nemo)) — Wed, 10 Apr 2013 21:21:17 UT

I think this is important for Wikiteam. The dumps we produce are of
little value if we don't ensure they can actually be used/read!

Nemo

-------- Messaggio originale --------
Oggetto: [Xmldatadumps-l] making imports suck less, the sequel
Data: Wed, 10 Apr 2013 23:40:48 +0300
Mittente: Ariel T. Glenn

Re: [wikiteam-discuss:642] trying to generate a dump of wikilawschool.net for import into wikireader

emi...@gmail.com (emijrp) — Tue, 09 Apr 2013 10:17:03 UT

Looks like Special:Export is disabled
[link] which is the method we use
to export the content.

But API allows content extraction
[link]Main%20Page&rvprop=timestamp|u ser|comment|contentAlthought

trying to generate a dump of wikilawschool.net for import into wikireader

ggar...@twentyseven.com (Gustavo Garcia) — Tue, 09 Apr 2013 05:34:38 UT

I am trying to generate a dump of:
python dumpgen.py --api=[link] --xml
--curonly --path=dump5 --delay=8
The titles work ok but then when it tries to retrieve the XML, I get:
Server is slow... Waiting some seconds and retrying...
An error have occurred while retrieving "Main_Page"

Re: [wikiteam-discuss:611] Move Wikia dumps to Internet Archive

emi...@gmail.com (emijrp) — Thu, 28 Mar 2013 20:58:17 UT

Great, I see you did another backup
[link]

I have added it to [link]

34,000 wikis saved, yeah! Nice work.

2012/11/11 Federico Leva (Nemo) <nemow...@gmail.com>

Re: [wikiteam-discuss:638] xml dump of private mediawiki

emi...@gmail.com (emijrp) — Thu, 28 Mar 2013 14:08:23 UT

I'm watching
[link] and
login through API is a bit a mess...

Also, Special:Export allows login? We export text from that special page,
not API. We use API only for titles.

2013/3/28 Federico Leva (Nemo) <nemow...@gmail.com>

Re: [wikiteam-discuss:638] xml dump of private mediawiki

nemow...@gmail.com (Federico Leva (Nemo)) — Thu, 28 Mar 2013 13:19:58 UT

Scott Kraft, 28/03/2013 14:02:

No, not yet. [link]

Nemo

xml dump of private mediawiki

sakra...@gmail.com (Scott Kraft) — Thu, 28 Mar 2013 13:02:12 UT

I have a private mediawiki (a login is required to view and edit pages)
installation on my ftp server. I am looking to do an xml dump before I
upgrade the mediawiki version (I already backed up the database). I am
trying to use this:
[link]