Scripted download of maps on Linux

[We Ka]

Hello,

I am using Osmand+ (paid version) on several devices and I think it's highly inefficient to download each map on each device, unnecessary increasing data usage. Therefore, I'd like to download updated maps to my pc and copy them onto the sdcards afterwards. I know I can do "wget http://download.osmand.net/download.php?standard=yes&file=Germany_baden-wuerttemberg_europe_2.obf.zip" to get one map, but there are two problems with that:

1. Automatic file name completion doesn't work, so I can't use a command like "wget http://download.osmand.net/download.php?standard=yes&file=Germany_*.zip", I have to use a wget for each single file I want to download.

2. After downloading, the files are stored as "download.php?standard=yes&file=Germany_bremen_europe_2.obf.zip" to the local hard disk, and I have to remove the part in front of "Germany".

How can I solve these two problems? Is there any other way of limiting the download to a minimum and still get the maps onto every device?

[CP]

BASE_URL='http://dl3.osmand.net/download.php?standard=yes&file='
STOR=/mnt/vol1/algemeen/OsmAnd/maps
COUNTRIES="Italy Germany Austria Netherlands Switzerland Us_new-york Us_south-carolina"

curl http://download.osmand.net/list.php | sed -e 's/<tr>/<tr>\'$'\n/g' > osm_maps.html
for COUNTRY in ${COUNTRIES}; do
    grep ${COUNTRY} osm_maps.html |sed -e 's/^.*file=//' | sed -e 's/">.*$//' |
    while read MAP; do
        echo "${MAP}"
        ( curl ${BASE_URL}${MAP} > ${STOR}/${MAP} 2>/dev/null )
        if [[ ${MAP} =~ .zip$ ]]; then
            unzip -o -d ${STOR} ${STOR}/${MAP} && rm ${STOR}/${MAP}
        fi;
    done
done

Have fun!

CP

Op 11-03-18 om 18:01 schreef We Ka:

--
You received this message because you are subscribed to the Google Groups "Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[We Ka]

Hey, that is a pretty cute script. Thank you very much! :)

Yet, there are a few things that needed to be corrected to make it work:

1. the URL's hostname needs to be "download", not "dl3", in order to be resolved. (I guess there's some load balancing happening there.)

2. the line containing the grep command needs to end with a "\ ". (That's two characters: a backslash followed by a space, but without quotes around them.)

3. curl doesn't seem to be able to handle the actual URL containing the "&" (ampersand). I had to replace curl by wget in order to make the download work:

           wget "${BASE_URL}${MAP}" -O "${STOR}/${MAP}"

Apart from that: very handy! Now I can get the list of files every week or so, compare it to the version from the last check, and if there's a difference, I can download the maps and distribute them to all of my Android devices inside my home network using "OwnCloud" and "foldersync".

Thank you once more for the script!

[Poutnik]

As I once wrote and occasionally use
a cygwin/wget based batch for maps download in Windows,
I do not see a big deal in writing a script for Linux.

As inspiration ( older, but should work, even if not checked for a while )
Github : /Brouter-profiles/master/OSMANd_wget_maps
( will be displayed as the CMD content and can be directly saved as CMD file )

Dne 11/03/2018 v 18:01 We Ka napsal(a):

-- 
Poutnik ( The Wanderer )

My Brouter profiles 
https://github.com/poutnikl/Brouter-profiles/wiki

[Poutnik]

I see this batch version uses an obsolete wget version from Gnuwin32 project.
Windows users of cygwin can easily switch to the cygwin wget version.

Dne 13/03/2018 v 09:36 Poutnik napsal(a):

As I once wrote and occasionally use
a cygwin/wget based batch for maps download in Windows,
I do not see a big deal in writing a script for Linux.

As inspiration ( older, but should work, even if not checked for a while )
Github : /Brouter-profiles/master/OSMANd_wget_maps
( will be displayed as the CMD content and can be directly saved as CMD file )

[CP]

Op 13-03-18 om 09:27 schreef We Ka:

Hey, that is a pretty cute script. Thank you very much! :)

You're welcome.

Yet, there are a few things that needed to be corrected to make it work:

1. the URL's hostname needs to be "download", not "dl3", in order to be resolved. (I guess there's some load balancing happening there.)

Agreed. In my reality download works just fine.

2. the line containing the grep command needs to end with a "\ ". (That's two characters: a backslash followed by a space, but without quotes around them.)

No, that is not needed. If you copy this to a bash script it works. I think you've fallen victim to a local issue of command line and/or shell

3. curl doesn't seem to be able to handle the actual URL containing the "&" (ampersand). I had to replace curl by wget in order to make the download work:

           wget "${BASE_URL}${MAP}" -O "${STOR}/${MAP}"

This is probably related to the curl version. Depending on your distribution you might get a different curl.
Mine is:

koos@zwaluw:~> curl --version
curl 7.37.0 (x86_64-suse-linux-gnu) libcurl/7.37.0 OpenSSL/1.0.2j zlib/1.2.8 libidn/1.28 libssh2/1.4.3

Apart from that: very handy! Now I can get the list of files every week or so, compare it to the version from the last check, and if there's a difference, I can download the maps and distribute them to all of my Android devices inside my home network using "OwnCloud" and "foldersync".

Yup :-)

[Rob]

For some time I've been thinking to do something like this but never got round to it. I was planning to set up a caching proxy and configure my home router to redirect only traffic for the Osmand map server there. Similar idea but less overhead - just download the maps on one device and they are automatically cached on the local network for subsequent devices. I hope this would also be a cross platform solution (iOS/Android) but I'm not sure whether the 2 apps use the same map files.

Curious to know if anyone out there has already tried this setup?

R

[CP]

You cannot cache HTTPS traffic. That goes against the design of HTTPS.
I you want to distribute maps locally and automatically , your best option is some kind of p2p setup (syncthing/torrent)

CP

Op 15-03-18 om 00:16 schreef Rob:

--

[Sabine Mayer]

Hi,

thats a cute script and does look much better than one I currently use!

But I also to download the srtm-files and the wiki files but the links do no longer work.

srtm used to be at http://download.osmand.net/srtm-countries/

and wiki at http://download.osmand.net/wiki/

do you know where these files were moved to?

cheers,

sabby

[Poutnik Fornntp]

Note also if one installs termux, a very good Linux terminal emulator, one can write wget or curl based Bash scripts for direct download to the final Android placement, optionally skipping downloads if no update is available, or if local files are not older than chosen threshold.

Note that it is by default sandboxed, must be explicitly configured to have access to Android storage.

I use it for BRouter RD5 file updates and Mapsforge compatible maps for LocusMap.(OpenAndroMaps globally, or osm.paws.cz for middle Europe.), using iteration overthe script arguments, where each argument is a code for particular downloadm defined withing tge script.

E.g. argument 0 means an update of my Brouter "homegrid" 5x5deg RD5 file to be downloaded to

~/storage/shared/Brouter/segments4

[Poutnik Fornntp]

Attaching my Bash script for illustrative and inspirative purposes.

Note that I am not particularly good at Linux scripting, so it is surely possible to write in in more elegant or more powerful way. I have just written it recently, so updates are expected.

Dne 9. června 2019 17:33:45 Poutnik Fornntp <poutni...@gmail.com> napsal: