USB File Size difference

[Pascal Laurent]

Hi,

 I am having a peculiar problem. I can't seem to figure out how to address it. I have downloaded the en-wikipedia_for_schools-static.zip module for the DL. As shown in the image the files copied from the source to the Destination (USB drive) are not of the same size. The files size on the USB drive are much larger than the original files (from the source). I have played around with different unit sizes when formatting the USB drive and the problem still persisted. The 5GB en-wikipedia_for_schools-static.zip file ends up occupying over 100GB+ on the USB drive. Any idea the best way to resolve this issue? Thanks

[Steve Song]

Hi Pascal,

It looks like you are on a Windows platform. Perhaps your USB drive is formatted as FAT32 which only supports file sizes up to 4GB.  For OpenWRT, you would ideally format the usb as ext4 for optimum performance.  Do you have a linux computer that you could use to format the usb and transfer the files?

Cheers... Steve

--
You received this message because you are subscribed to the Google Groups "Village Telco Development Community" group.
To unsubscribe from this group and stop receiving emails from it, send an email to village-telco-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/village-telco-dev/CAJfpDpzyCJFsEzE3Pnag2oo62ZOVivoLFnZkxgQe1Xnb9cUevA%40mail.gmail.com.

--

Steve Song

+1 902 529 0046

http://villagetelco.org

[T Gillett]

Hi Pascal

That is a very strange problem.

I suggest that you try to unpack the zip file on to your PC hard disc first so that you see what is going on. You will need 6GB of space.

It may be that the zip file is corrupted in some way or that there is some problem with the usb drive.

Can you tell us the details of the USB drive please - brand and type?

The USB can be formatted as ext2 or ExFAT for use with the DL. For ExFAT you will need to specify 4k block size (128k is the default).

The WfS module should occupy around 6GB on disc.

You might also try copying a few GB of other files from your PC on to the USB to check that they are stored correctly.

Regards

Terry

[Pascal Laurent]

Hi Steve,

The USB drive I am using is formatted as exFAT. I would ext4 a try. 

Thanks, 

Pascal

To view this discussion on the web visit https://groups.google.com/d/msgid/village-telco-dev/CAD_CWO2dYsJ7YRUAJ9oTO6PTBw-j4U98xtmicPhCyKxVCo0nGg%40mail.gmail.com.

[T Gillett]

Hi Pascal

Can you confirm exactly what the memory device is that you are using please?

I understand that it is a 128GB USB memory stick (ie not a SD card). Is that correct.

If so, what is the brand and type please?

The USB memory should work fine on exFAT, and changing to ext2 should not make any great difference to the storage capacity.

But if you have a Linux machine, then it would be a way to check what is going on.

One thing to sanity check is that the USB has been exFAT formatted with Allocation unit size of 4096 bytes, not 4096 kilobytes.

If you use ext2 format you will automatically get 4096 byte block size.

But you will not be able to read the memory on a Windows system.

On a Ubuntu / LInux system you should be able to read either format.

Also you will have rsync available which provides fast and robust copying.

Regards

Terry

To view this discussion on the web visit https://groups.google.com/d/msgid/village-telco-dev/CAJfpDpzvZ%3Dqs%2BKFE_nGBt3NoxSPSwBK35Qoo9-rB0PqxckbYRQ%40mail.gmail.com.

[T Gillett]

Attached screenshot of Windows 10 Format screen with Allocation unit size of 4096 bytes selected.

[Pascal Laurent]

Hi Terry,

This is the hardware i am using and how the drive is formatted:

• 128GB flash drive
• Formatted: exFAT
• Allocation unit size: 4096 bytes
• GL-AR150

The module I was using is: en-wikipedia_for_schools-static.zip. Sure I could use Linux. I am afraid this could impact other community members with no access to linux. Once again, thank you for your assistance.

Regards,

--Pascal

To view this discussion on the web visit https://groups.google.com/d/msgid/village-telco-dev/CAD58Ekzkqkzj6-i5uLkgJ9jT2XWFokf%3DKhxefS2Wmw%3DV40KAHg%40mail.gmail.com.

[T Gillett]

Hi Pascal

I agree it will be best to sort the problem out with ExFAT and Windows if that is the environment that is used in your community.

It appears that the issue is just around the USB memory, and nothing to do with the AR150, or the DL firmware.

I suspect that the issue may be that the particular USB memory you have may not accept the 4096 byte block size format. It may be pre-set to some larger value.

Do you have another, smaller (eg 8 or 16GB) USB memory that you can test with?

If so, copy some files on to it and check the 'size' compared to the 'size on disk'.

Use a few hundred MB just to keep it simple.

Then try reformatting to exFAT / 4096 byte block size and repeat the test.

Compare the results and compare with the same test on the 128GB USB.

Also if you put the DL template and the WfS module onto a smaller USB drive, you can use that in the Digital Library as a starting point.

My preference for DL memory devices is to use SD cards with a USB adapter rather than USB memory sticks.

I use Sandisk and Samsung 100Mbps speed cards specifically.

Reputable brand SD cards are well specified and branded for their performance, unlike USB sticks which generally have little or no performance specs.

Also be aware of counterfeit SD cards and USB sticks.

Please let us know the results of your tests. 

Regards

Terry

To view this discussion on the web visit https://groups.google.com/d/msgid/village-telco-dev/CAJfpDpwxD0hZToWq%3DWo0WRP9g%3DTyTwqht8dgu5fhABja3ZP_9Q%40mail.gmail.com.

[T Gillett]

Hi Pascal

That is good news.

Regarding managing connection times, the only time mechanism that is in place is the DHCP lease time.

The client devices will receive a DHCP lease for the IP Address that is assigned.

The lease expiry time is set to 2 hours for the main 'lan' network, and 1 hour for the 'guest' network.

The lease will be renewed after expiry if the client device is still alive and makes a request.

But that doesn't really manage long connection times.

There is also the limit on the maximum number of concurrent wifi connections for both WiFi Access Points, which manages the total number of concurrent users.

Beyond that, I think you may be in the territory of setting up a captive portal and managing individual accounts for your clients, which is a whole different ballgame.

But if you have a separate portal system to manage users, you can use the Digital Library as a networked resource behind the portal. Just disable the Access Points and connect it via Ethernet cable.

Regards

Terry

On Tue, 19 Jan. 2021, 10:29 Pascal Laurent, <tip...@gmail.com> wrote:

Hi Terry,

It works great using the usb-sdcard.

 

Is there a way to allocate time per connection and to end inactive connections. This is a way to prevent abuse. Thanks

Regards,

--Pascal

On Thu, Jan 14, 2021 at 1:45 AM T Gillett <tgil...@gmail.com> wrote:

Hi Pascal

OK thanks.

Please let me know if there is anything I can do to help.

And just for interest, can you tell me where you are located please?

Regards

Terry

On Thu, Jan 14, 2021 at 1:13 PM Pascal Laurent <tip...@gmail.com> wrote:

Hi Terry,

Thank you for your question. I have ordered a USB to sdcard adapter. I would share with you how everything goes this weekend. 

Regards, 

Pascal

[T Gillett]

---------- Forwarded message ---------
From: T Gillett <tgil...@gmail.com>

Date: Tue, Mar 2, 2021 at 4:08 PM
Subject: Re: USB-to-SDCARD issue
To: Pascal Laurent <tip...@gmail.com>

Hi Pascal

A few points:

1. The first thing I notice is that the memory card appears as "/dev/sda"  in your df command on the DL device.

This is what you should see in the /dev directory:

root@Digital-Library:~# ls -l /dev/sd*
brw-------    1 root     root        8,   0 Jan  1  1970 /dev/sda
brw-------    1 root     root        8,   1 Jan  1  1970 /dev/sda1

"/dev/sda1" is the first partition on the "/dev/sda" memory device

The library memory should appear as "/dev/sda1". It will then be mounted as "/mnt/sda1"

This is the df command output on a working DL system:

root@Digital-Library:~# df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/root                 3840      3840         0 100% /rom
tmpfs                    62432        84     62348   0% /tmp
/dev/mtdblock6           10816       456     10360   4% /overlay
overlayfs:/overlay       10816       456     10360   4% /
tmpfs                      512         0       512   0% /dev
/dev/sda1             61387804  56862020   1407448  98% /mnt/sda1

I suspect that something has gone astray in the formatting / partitioning of your card.

You should have a single ext2 formatted partition on the card.

I usually use the GParted graphical utility on Ubuntu to format SD cards and USB sticks. Just use the ext2 selection to format.

Perhaps you could try this out on a smaller USB memory stick just to get things sorted out before attempting it on the 64GB SD card.

2. The contents in the top level of the SD card don't look quite correct.

This is what a working card looks like:

 --------------------------
root@Digital-Library:~# cd /mnt/sda1
root@Digital-Library:/mnt/sda1# ls -l
-rw-rw-rw-    1 1000     1000             0 Jan  1  2014 ##LIBRARY##
drwxrwxrwx    2 1000     1000          4096 Nov 16  2020 art
-rw-rw-rw-    1 1000     1000          1367 Jun 15  2014 common.css
drwxrwxrwx    2 1000     1000          4096 Sep 15  2013 css
-rwxrwxrwx    1 1000     1000         47602 Jan 18  2021 index.html
drwxr-xr-x    2 1000     1000          4096 Nov 28  2020 local
drwxrwxrwx    2 1000     1000          4096 Jan  1  2014 logs
drwxrwxrwx   46 1000     1000          4096 Jan 17  2021 modules
-rw-rw-rw-    1 1000     1000          4354 Jan 18  2021 style.css

-----------------------------

The sizes shown for the directory entries on your card (eg art, css, local etc) are much larger than they should be (4,194,304 compared to 4096 on ext2)

Also I see a "System Volume Information" directory is present. Where has this come from??

Looks a bit MS Windows like...

This makes me wonder if the card is actually formatted as ext2 with its default 4k block size.

It looks a bit like it might be the original exFAT format that the card would likely have had on it from the factory.

I usually use the graphical GParted utility on Ubuntu to format SD cards and USB sticks.

Just use the ext2 selection to format.

3. The Home screen you are getting when the card is plugged into the DL device is clearly not from your "index.html" file on the SD card. )

(It is missing your extra link, and the icon for Wikipedia for Schools.)

It looks like the default screen you will get if there is no library memory card installed. You might like to check what you see with no memory installed and compare.

This confirms that the SD card in the USB adapter is not being recognised by the DL, as above.

Again, I suggest that you experiment with a smaller USB memory device (eg 8GB)  to get the basic formatting sorted out, and the basic library files in place and working before moving on to the larger SD card.

Regards

Terry

On Tue, Mar 2, 2021 at 11:14 AM Pascal Laurent <tip...@gmail.com> wrote:

Thanks for your quick response:

SD-Card:

https://www.amazon.com/gp/product/B08879MG33/ref=ppx_yo_dt_b_asin_title_o03_s00?ie=UTF8&psc=1

USB-SD card adapter

https://www.microcenter.com/product/485280/inland-usb-30-card-reader-with-otg

Screenshot of USB/SD inserting in DL device

Troubleshooting Steps as recommended:

1) DL Page when sdcard connected to the DL device

2) Try opening the index.html file with the web browser on the Linux box and you should see the DL Home page.

This is the page when opening the DL page from the sdcard connected to my linux box. As you can see I added an extra link on the sdcard. I would expect this one to display, not the one on step 1.

3) What linux os am i using?

I am using ubuntu 18.04

Regards,

--Pascal

On Mon, Mar 1, 2021 at 10:45 AM T Gillett <tgil...@gmail.com> wrote:

Hi Pascal

All good here, thanks.

Your process looks ok at first glance.

Could you send details of the SD card and the USB adapter please?

We need to check what is actually present on the SD card.

As a quick sanity check, plug the USB/SD into your Linux box and run the " ls -l" command to show what files and directories are present at the top level of the file system on the card.

Please send me the output of the command e.g. screenshot.

There should be an index.html file present, plus a few directories including the "modules" directory.

Try opening the index.html file with the web browser on the Linux box and you should see the DL Home page.

If the contents of the file system does not look correct, remove the SD card from the USB adapter and instal it directly in your Linux box and try again.

Once we have it working on the Linux box, we can move it to the DL device.

And just for interest, what Linux distribution are you using e.g. Ubuntu, Mint etc, and the version.

Regards

Terry

On Tue, 2 Mar. 2021, 00:36 Pascal Laurent, <tip...@gmail.com> wrote:

Hi Terry,

I am hoping you are well. I have noticed something super weird. I have formatted my sd-card  using ext2. Then I rsync it with the module on dev.worldpossible.org from my linux box. Then I inserted the sdcard into the USB-to-SDCARD dongle, then into the Digital Library device. Unfortunately, the home page of the Digital Library is not loaded.  Would you please educate me the steps I could take to troubleshoot this issue?

Here are the steps i followed to prepare the sdcard:

## Preparing the USB stick for content
-- Format the drive using Linux ext2
    sudo umount /dev/sdb
    mke2fs -t ext2 -b 4096 /dev/sdb

-- Mount the USB drive to linux machine
    sudo mount /dev/sdb /mnt

## Adding Content to the Drive
-- Download  https://github.com/villagetelco/digital-library/blob/Ver-1/DIGITAL-LIBRARY-USB.zip
-- Extract the content of DIGITAL-LIBRARY-USB.zip  to the root of (i.e /mnt/sdb)

## Download individual modules
-- Khan Academy On a Stick

   cd /mnt/module/kaos-en

    cd module/kaos-en
    rsync -Pavz rsync://dev.worldpossible.org/rachelmods/en-kaos ./

## Plug sdcard into DL

Insert sdcard into usb-sdcard device

plugged in usb-sdcard device to the Digital Library device

power up the Digital Library device

Regards,

--Pascal

[T Gillett]

Hi Pascal

Two thoughts as to how you might handle the situation with resolving the "pas01" network name.

1. Simplistically, you could add an entry for "pas01.jeremyonline.com" to the Local Hosts file (/etc/hosts) using the 'General' DL admin screen, or by directly editing the file.

2. Nameserver resolution is controlled by /tmp/resolv.conf.auto which is generated on startup by the dnsmasq process, based on /etc/resolv.conf and other parameters.

See this discussion: https://forum.openwrt.org/t/what-is-repeatedly-resetting-etc-resolv-conf/12838/2

"In the default configuration, /etc/resolv.conf (/tmp/resolv.conf) always points to 127.0.0.1.

The locally running dnsmasq initially handles all resolutions.

That process reads /tmp/resolv.conf.auto to find the next upstream nameserver.

The file resolv.conf.auto is written by the DHCP client on the wan,or by "option nameserver" settings in the network."

If you look at /tmp/resolv.conf.auto on a running DL system with default configuration it will look like this:

------------

root@Digital-Library:~# cat /tmp/resolv.conf.auto
# Interface lan
nameserver 208.67.220.123

------------

The nameserver IP address shown above is created from the "DNS Filter" setting on the Network system admin page

(10.10.10.1/admin -> Network  or http://10.10.10.1/cgi-bin/system-network.has).

The value is stored in /etc/config/dhcp as below:

------------

root@Digital-Library:~# cat /etc/config/dhcp

config dnsmasq 'setup'
...

...

list server '208.67.222.123'
list server '208.67.220.123'

-----------

This "server" setting forces name resolution to be done by the nominated DNS server, thus implementing the DNS Filtering function.

(Note: Port 53 is captured by a firewall rule to enforce this DNS resolution path)

You could make the DL use your nameserver at 10.2.1.1 by changing the DNS Filter setting, and / or editing the /etc/config/dhcp file.

Be aware that if you edit the config file directly, subsequent Save operations on the DL Admin - General page may overwrite your changes,

unless you disable the DNS Filter setting.

By default, the DNS Filter setting in DL points to an OpenDNS "filtered" nameserver so that students using the system don't have access to malicious and inappropriate sites. If you switch it off, access to the Internet will be unfiltered, unless you have filtering applied at your upstream nameserver.

Regards

Terry

On Mon, Mar 8, 2021 at 2:07 AM Pascal Laurent <tip...@gmail.com> wrote:

It works now.

Here is how the dl looks now:

Another question: How do I resolve a local DNS when connecting to the DL AP? the IP of the local DNS is 10.2.1.1?

from the DL AP, when I got this result from issuing this command:

But when i added  nameserver 10.2.1.1 to /etc/resolv.conf I got this

When I reboot the DL the changes to resolv.conf is gone. How do I make it permanent?

Thanks,

--Pascal

On Mon, Mar 1, 2021 at 10:45 AM T Gillett <tgil...@gmail.com> wrote:

Hi Pascal

All good here, thanks.

Your process looks ok at first glance.

Could you send details of the SD card and the USB adapter please?

We need to check what is actually present on the SD card.

As a quick sanity check, plug the USB/SD into your Linux box and run the " ls -l" command to show what files and directories are present at the top level of the file system on the card.

Please send me the output of the command e.g. screenshot.

There should be an index.html file present, plus a few directories including the "modules" directory.

Try opening the index.html file with the web browser on the Linux box and you should see the DL Home page.

If the contents of the file system does not look correct, remove the SD card from the USB adapter and instal it directly in your Linux box and try again.

Once we have it working on the Linux box, we can move it to the DL device.

And just for interest, what Linux distribution are you using e.g. Ubuntu, Mint etc, and the version.

Regards

Terry

On Tue, 2 Mar. 2021, 00:36 Pascal Laurent, <tip...@gmail.com> wrote:

Hi Terry,

[T Gillett]

Hi Pascal

Some further information around dnsmasq operation here:

    https://openwrt.org/docs/guide-user/base-system/dhcp

Search references to "upstream" on the page.

In particular this option in  /etc/config/dhcp  may be useful:

Option            Type          Default

allservers        boolean     0

"By default, when dnsmasq has more than one upstream server available, it will send queries to just one server.

Setting this parameter (to 1) forces dnsmasq to send all queries to all available servers.

The reply from the server which answers first will be returned to the original requester."

Regards

Terry

[Pascal Laurent]

Thanks a bunch. I would try your suggestions