Wikipedia on LibraryBox?

[Don Clark]

Hi,

I am wondering about the possibility of adding a snapshot of Wikipedia to LibraryBox.  I could just add a snapshot for people to download for use on a laptop.  However, I'd prefer something minimally interactive that people on mobile devices could use without having to install anything.

Ideally it'd be as simple as adding kiwix-serve and a snapshot of Wikipedia in whichever language I'd want but I can't find anything simple.

There's a feature request for a Piratebox/KiwixPlug crossover here:
http://sourceforge.net/p/kiwix/feature-requests/294/

But, as far as I can tell, there's been no work on the project.

There is a project called wikifunken which seems close but complicated and scrapes Wikipedia directly.  See http://github.com/heikoheiko/wikifunken/blob/master/README.md for info.

I noticed this post:  https://groups.google.com/d/msg/librarybox/0oZB8rBtkRM/q7X0JbC9X8UJ from the LibraryBox group mentioning a Wikipedia install but, there are no details.

What would it take to get Wikipedia on a LibraryBox and, is it even feasible on the typical router hardware we are using?

Thanks for your help.

Regards,

Don

[Jason Griffey]

That absolutely should work. As a matter of fact, I've played around with a couple of different versions of offline-wikipedia, the easiest to implement is the Wikipedia for Schools found here:

http://www.sos-schools.org/wikipedia-for-schools

I've got a test LibraryBox with a 64GB USB that's running this right now, as a matter of fact. As long as you can pull down a static version of the content, it should run.

Jason Griffey

Creator & Director

The LibraryBox Project

http://librarybox.us

--
You received this message because you are subscribed to the Google Groups "LibraryBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to librarybox+...@googlegroups.com.
To post to this group, send email to libra...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/librarybox/607b9108-1f76-46d5-9308-800f17c9a5af%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Don Clark]

Thanks.  That looks interesting.  I'll download a copy and see how it works.

I am interested in Wikipedia in another language too so I'll see if I can find out how they put their offline version together.

Regards,

Don

[T Gillett]

Hi

We have made some translations of the popular "Wikipedia for Schools" content into French, Spanish and Portuguese.

Each translations adds around 350MB storage requirement, so for an additional 1GB you can have the content in these three languages as well as the original English.

The translations are intended to be installed in their own directories along side the original English language directory called "wikipedia_for_schools".
The translations do not include the large "images" directory, but link to that directory within the English language directory to save duplicating the storage space.

The translations have been done automatically (think ~ Google translate) and so are perhaps not idiomatically perfect, but we think they are good enough for a first cut and can be edited over time to correct any significant issues.

Following are links to zip files (~100MB each) for the translations.

We also have test versions in Malay and Khmer languages.

Feedback welcome.

Regards
Terry


Spanish
    http://1drv.ms/1Jq7FLw

Portuguese
     http://1drv.ms/1IbJ3vj

French
     http://1drv.ms/1NrnEMB

[Christophe Rhein]

Hello
Is there a solution to use the full version off Wikipedia on the librarybox?
Thank you
Christophe

[Don Clark]

I haven't seen any easy solutions for low-powered hardware.  Here's a good overview of offline versions of Wikipedia:
https://en.wikipedia.org/wiki/Wikipedia:Database_download

XowA (https://gnosygnu.github.io/xowa/requirements.html) needs a minimum of a 1.5 GHz processor and 512MB of memory.
OWI (http://owi.sourceforge.net/#requirements) recommends a computer that is a least a couple of years old.

According to the creator of the wikifunken project, kiwix-serve takes too much power.  There is a project to make Kiwix useable on low-powered systems but it is dormant as far as I can tell (http://sourceforge.net/p/kiwix/feature-requests/294/).

It'd be amazing if we could get something working with the Wikipedia database dumps.  Even if only very simple features (such as indexing) were available.  Updating would be easier that way.

Anyone know anyone working on this in the Piratebox or OpenWRT communities?

[Jason Griffey]

The best option I've found is the Wikipedia for Schools version that's hosted by the RACHEL project. 

http://dev.worldpossible.org/mods/

That works beautifully.

Jason

--

You received this message because you are subscribed to the Google Groups "LibraryBox" group.
To unsubscribe from this group and stop receiving emails from it, send an email to librarybox+...@googlegroups.com.
To post to this group, send email to libra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/librarybox/7573d3fa-0e68-4da8-97d7-12558968b828%40googlegroups.com.

[Campbell Smythe]

Yeah just confirming what Jason's just said - the Wikipedia for Schools is probably the best around at the moment.

If you cant download 6GB, contact them through http://www.sos-schools.org/wikipedia-for-schools

Get the download from here, unzip, open the folder and click the "index" file.  Copy to Shared resources or in your interface setup a Menu link straight to it.

Theyll send out a USB if youre unable to download. I was having a lot of trouble downloading a few months back.  The file would stop downloading at the 4GB mark  Said it was finished but would always be corrupted/incomplete unable to be unzipped.

Guys there were really helpful.

Campbell Smythe

On Tuesday, September 8, 2015 at 9:31:06 AM UTC+10, Don Clark wrote:

[Don Clark]

I've noticed that sector size can make a big difference in how much space Wikipedia for Schools uses on the drive as there are a lot of tiny files.

The version I downloaded is 5.9 GB.

With 32 KB/sector the size on disk is 9.54 GB.  With 8192 bytes/sector the size on disk is 6.72 GB.  The smaller sector size gives you 2.82 GB extra to use for something else.  Just for comparisson, with 256 KB/sector (typical for external hard drives) the size on disk is 48.5 GB.

The sector size can be selected when you format the drive.

[T Gillett]

Thanks for the heads up on this.

I have found that on standard 32GB USB and SD memories WfS takes around 6.7 GB also. So I guess they are using 8192 byte also.

--
You received this message because you are subscribed to a topic in the Google Groups "LibraryBox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/librarybox/KtcRdj8cUWo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to librarybox+...@googlegroups.com.

To post to this group, send email to libra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/librarybox/cbb21176-c908-4a67-9e22-c881219ea4f2%40googlegroups.com.

[T Gillett]

For low powered devices, you might be interested in the Village Telco implementation of the RACHEL library.

     http://wiki.villagetelco.org/RACHEL_on_MP2

There is firmware available to run on a range of low powered devices including the TP Link MR3020 and MR3040 as well as larger devices such as the WR842, WDR4300 and the  VT MP02.

The devices will support up to 35 connected clients.

You can re-flash the TP Link devices and add your own library modules on USB memory.

The VT MP02 device will be available from the Village Telco on line shop.

This is a robust, out-of-the-box solution that includes the wifi router device and the RACHEL library (including WfS in several languages) on USB-SD memory, ready to go.

The MP02 is powered from 12V so is easy to use with solar power systems.

You can also connect any of the devices to an upstream network to share an internet connection alongside the local content.

The larger devices are planned to support 3G/4G USB modems and caching of internet content in the next firmware release.

--
You received this message because you are subscribed to a topic in the Google Groups "LibraryBox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/librarybox/KtcRdj8cUWo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to librarybox+...@googlegroups.com.

To post to this group, send email to libra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/librarybox/7573d3fa-0e68-4da8-97d7-12558968b828%40googlegroups.com.

[Don Clark]

Thanks for the info on the Village Telco project.  They have some interesting stuff on their site that I want to investigate further.

[Don Clark]

Hi Terry,

Thanks for translating Wikipedia for Schools!  I am particularly interested in a Bahasa Indonesia version but will be happy to use the Malay version.  I did a quick search for a download link for it but couldn't find one.  Would you please post a link for the Malay version?

Would I be able to assist in getting a Bahasa Indonesia version done?  If so, let me know how I might help.

Thanks.

Regards,

Don

[T Gillett]

Hi Don

The RACHEL project has kindly hosted some of the translations on their download site as below.

They are also available for download as .zip files as below, including Malay.

We have not done the Bahasa translation as yet, but if you are interested in using it we can have a look at doing it.

Please let me know how you go with it.

Terry

RACHEL Downloads
http://dev.worldpossible.org/cgi/rachelmods.pl

French
http://dev.worldpossible.org/cgi/viewmod.pl?module_id=41

Spanish
http://dev.worldpossible.org/cgi/viewmod.pl?module_id=42

Portuguese
http://dev.worldpossible.org/cgi/viewmod.pl?module_id=43

ZIP Downloads

Spanish
    http://1drv.ms/1Jq7FLw

Portuguese
     http://1drv.ms/1IbJ3vj

French
     http://1drv.ms/1NrnEMB

Malay
     http://1drv.ms/1j5WxgT

When you download and expand the translation directory for a particular language it will be around 300MB in size.

It does not contain the "images" directory, so if you open index.html you will see all the text as usual, but no pictures.

The html in the translations expects to find the images directory in the English language directory alongside named "wikipedia_for_schools".

--
You received this message because you are subscribed to a topic in the Google Groups "LibraryBox" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/librarybox/KtcRdj8cUWo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to librarybox+...@googlegroups.com.
To post to this group, send email to libra...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/librarybox/fb5039f8-3c2a-40c5-a09b-b33161c3ec8b%40googlegroups.com.

[Don Clark]

Thanks for the links Terry.  I have downloaded the Malay version.

I am setting up LibraryBoxes in Indonesia and am currently using a snapshot of Wikipedia Bahasa Indonesia from 2008.  Unfortunately, it doesn't have any images.  So, an Indonesian version of Wikipedia for Schools would be great.  There are enough differences between Malay and Bahasa Indonesia that the additional translation would be much appreciated by the LibraryBox users.

Don

[T Gillett]

We are currently looking at the Bahasa version.

To view this discussion on the web visit https://groups.google.com/d/msgid/librarybox/ba780aa2-4dde-4cfe-a6c5-9bc9e7e36371%40googlegroups.com.

[T Gillett]

Don

Did you get the Malay version running OK with pictures from the English language images directory?

[Don Clark]

Hi Terry,

I did get it working.  I renamed the English directory "wikipedia_for_schools" and all of the pictures showed up.  It looks great!

Thank you for your help.

Regards,

Don