New enwiki-popular

[timo.hors...@gmail.com]

With the wikistats files [1] and some modification to Igor's slob tool, I managed to compile a few enwiki-popular dictionaries from MHBraun's complete enwiki-20141201.

There are now dictionaries with different number of titles ranging from 500 K to 2 M. The aim is to provide enwiki dictionaries even for devices with smaller memory, like many smartphones lacking a micro-sd slot.

Thus, the references and infoboxes have been removed to make the dictionaries even a bit more compact.

The files are located on my Mega account in the enwiki-popular folder:

https://mega.co.nz/#F!HFQyEAwL!pKgoB4yOleG-yNcnoPRcqQ

The subfolder enwiki-popupar/wikisort contains a simple ipytohn notebook to generate a sorted list of popular articles out of the wikistats files, as well as the resulting wordlist (enwiki_sortet.txt) and my modified slob.py for [2].

Cheers,

Timo

[1] http://dumps.wikimedia.org/other/pagecounts-ez/

[2] https://github.com/itkach/slob

[itkach]

On Saturday, December 13, 2014 9:49:35 AM UTC-5, timo.hors...@gmail.com wrote:

With the wikistats files [1] and some modification to Igor's slob tool, I managed to compile a few enwiki-popular dictionaries from MHBraun's complete enwiki-20141201.

There are now dictionaries with different number of titles ranging from 500 K to 2 M. The aim is to provide enwiki dictionaries even for devices with smaller memory, like many smartphones lacking a micro-sd slot.

Thus, the references and infoboxes have been removed to make the dictionaries even a bit more compact.

The files are located on my Mega account in the enwiki-popular folder:

https://mega.co.nz/#F!HFQyEAwL!pKgoB4yOleG-yNcnoPRcqQ

The subfolder enwiki-popupar/wikisort contains a simple ipytohn notebook to generate a sorted list of popular articles out of the wikistats files, as well as the resulting wordlist (enwiki_sortet.txt) and my modified slob.py for [2].

Very useful, thank you.

If I may suggest - Github is a much better place for code than Mega. Anyone looking at https://github.com/itkach/slob would be able to discover your fork. It's also much easier to see differences. Same goes for for ipython notebooks - they are awesome, but looking at a simple script (and running) in version control repository is much easier.  

[mhbraun]

Great. Well done.