How to: Build your own offline OSMAND maps using completely free tools

[Andrew Davie]

Thank you to the developers of OSMAND, and to the community who have supported it.

I would like to contribute a "HOW TO" - because this took me quite some time to figure out and perfect.

Essentially, I'm going to try to demonstrate how to go from georeferenced TIFF source files (many) to an OSMAND compatible file which has all of those source files as a displayable zoomable map.  And, although there are some commercial software packages out there already that do this, they are not free - and some are far from cheap. I'm going to show you how to do it completely free. I'll try, too, to make these instructions platform-agnostic - that is, should work for Windows, OSX, or Linux - although I'm using OSX myself and this is untested on those other platforms. YMMV.

Step 1 - Maps

As source material, I'm assuming you have a bunch of TIF format files which you want to get displaying in your OSMAND app.  Now I'm not going to cover how to georeference those TIF files (that is, specify the exact place in the world they represent).  That's another subject, and basically although I did check that out a while back, I'm starting with georeferenced files already.  You can download these pretty cheap - for example, the file covering my capital city can be had for about US$1.50 so hey, why not.  I am sure you can also find tons of georeferenced map files on the interweb, too.

But, let's say we have a directory of files... as I do.  50 will be a nice round number - it doesn't matter how many; the point being there are lots.

Now, the files I am using have been scanned from physical maps - that is, paper printouts, scanned, and then georeferenced. It doesn't really matter - but the point being there are areas in the files which are genuine map stuff, and there are areas which are NOT map stuff (around the edges where there's no map).  These areas are usually represented in the TIF files as transparent areas - and that's important for what's to come.  

So, if you want the "mangle all the maps together" to work properly in this how-to, you will want your source files to have transparent areas where there's "nothing".

So, first step is to convert each individual TIF file into a hierarchy of sub-images, with each level of the heirarchy being higher and higer magnification. This is all to do with how the data is stored so that OSMAND can access it efficiently - esentially each level is 2x the dimensions on each axis of the lower level.  Each 'tile' is just a square 256x256 pixel slice from the original image, but the scaling on the tile is different from level to level.  These scaled image tiles are stored in a directory hierarchy, with the top level representing the zoom/scale/level, whatever you want to call it.  0 1 2 3 4.... etc.  Now I'm really only interested in levels 7 or so to about 15 or so - that's from a scale of hundreds of kilometers down to tens of metres.

But we don't really need to know much about all of that - it's all done automatically for you by our first tool - 'gdal2tiles.py'

You will need to download and install a package called GDAL - get that at http://www.gdal.org

There are direct binary downloads for this at https://trac.osgeo.org/gdal/wiki/DownloadingGdalBinaries and tht's what I used.

We're really only interested in a couple of the GDAL tools - the first being gdal2tiles.py and the second being 'gdaltranslate'.

When I did the install, the .py file was hidden in /Library/Frameworks/GDAL.framework/Programs/gdal2tiles.py  so you may need to hunt for it after installation.  For me, gdal_translate was in the path and so just worked.  Once you have checked that those are findable, then you can cut/paste the following and save it as a script (call it something like 'tile.sh' if you're on OSX or Linux).  For windows users, you will have to figure out how to iterate over files, so that's a bit of homework for you.  Sorry, I don't do windows. OSX/Linux,  make sure it's executable (from a terminal window, type 'chmod +x tile.sh).  Windows should be OK as is.

Here's the code...

#!/bin/bash

FILES="*.tif"

for f in $FILES

do

echo "Processing ${f%.*}"

gdal_translate -of vrt -expand rgba $f  ${f%.*}.vrt

python /Library/Frameworks/GDAL.framework/Programs/gdal2tiles.py -z 7-14 ${f%.*}.vrt

done

OK, so just to explain;  it iterates over every '.tif' file in the current directory, and for each it calls first gdal_translate, and then gdal2tiles.py

The gdal_translate is just converting the file format form tif to some format that is recognised by the second part.

It's the gdal2tiles.py that does all the hard work.  It takes one source image and splits up it into smaller and smaller tiles, creating a directory for each level of detail.  We are specifying those levels (zooms) by that "-z 7-14" at the end there. You can change it to whatever range you want, but over 14 and you start gettting tens of thousands of images (literally) and the resultant database will be huge.  So, stick with this range for now; seems about right for general use.

This will take a fair while - about a minute per source image, say.  So, just wait patiently for it to finish.  Once it's done, you will see a new directory for each of the source TIF files.  So, if your tif was called '8513.tif' then you will see a directory called '8513' and inside that you will see subdirectories 7 and 8 and 9 and 10 ... etc... up to 14.  Inside each of THOSE you will see more directories with odd numbering, and inside THOSE you will see individual image files with png extension, also with strange numbering.  As an aside, the numbering is actually an encoding of the position of each of the tiles. No need to really worry/understand about this, but that's what's happening.  Encoded directory structure and filename --> location of png in the real world.

Right, so if all has gone OK we now have all of our TIF files processed and we've created a bunch of directories with the sliced PNG tiles inside each.  We now need to combine those directories and PNG files into a single heirarchy, representing the merged maps.  There are a couple of issues here - firstly, any image tile name combined with its parent directory name uniquely represents where that tile is, and what scale it is.  If we have two original TIF files which are adjacent, or even overlap, then it's possible we will have duplicate PNG files.  For the same physcial location. So, the PNG files need to be merged, in that case - that is where the files parent name is the same, and the filename is the same.  This merging process involves copying the non-transparent pixels from each PNG into a destination PNG (again, of the same name as the source PNG files).  Fortunately, Python to the rescue here - I wrote a short python program to do just that - that is, traverse ALL the directories, and find ALL of the images and merge all the duplicates and recreate the tree structure as a new 'merged' tree.

Here's the python program.  Copy this code into, say, 'tree.py'...

import os
from PIL import Image
import shutil


def merge(destination,source):

    print 'merging ',destination,'and',source
    srcd = Image.open(source)
    src = srcd.convert('RGBA')
    dst = Image.open(destination).convert('RGBA')
    dst.paste(src,(0,0),src)
    dst.save(destination)


rootdir = '/Users/boo/Downloads/TasTOPO_100K'   # TODO: change this to your source directory
dest = '/Users/boo/Desktop/pymerge'            # TODO: change this to your destination directory


# Now merge everything...

directories = next(os.walk(rootdir))[1]
directories = sorted(directories)
               
if os.path.exists(dest):
    shutil.rmtree(dest)

for dir1 in directories:
    for root, dirs, files in os.walk(rootdir+'/'+dir1):
        for f in files:
            if 'png' in f:
                dix = root[len(rootdir+'/'+dir1)+1:]
                fkey = root+'/' + f
                ddir = dest + '/' + dix
                if not os.path.isfile(ddir+'/'+f):
                    try:
                        if not os.path.exists(ddir):
                            os.makedirs(ddir)
                    except:
                        break
                    shutil.copyfile(root + '/' + f, ddir + '/' + f)
                else:
                    merge(ddir+'/'+f,root+'/'+f)

OK, this is a bit of a quick'n'dirty - you see the comment where it says "change this to your ...". Well, you gotta do that.

Obviously this could be a stand-alone python program with the paths as an input, but hey - there's some homework for you.

As it stands, it's good enough to do the job.

What it does is grab a list of directories, and then 'walks' down the heirarchy of these to find the PNG files.  Each PNG file is either new (not yet in the new tree), or it already exists. If it already exists then it must be pixel-merged with the existing PNG file - and that's where "PIL" magic comes into action.  That little subroutine 'merge' is doing just that with just a single line of code. Amazing.

Anyway, this will take a while to run.  If you want to go into the destination directory as it's running and watch one of the images you can actually see it being progressively upgraded as more and more images are merged into it.  That is, if you have multiple source TIFs.  Spellibinding.  Might take a half hour or so to run through all the tens of thousands of images, so go get a cuppa.

BUT, once that's done - now you're sitting on data in the right format for the next tool.

Before I get to that, though - I need to mention something that's going to be problematic for those in the Southern Hemisphere - there is something I don't quite understand with the earlier tool 'gdal_translate' or possibly a bug - it does not seem to understand the meaning of 'southern' and so all my maps were treated as if they were in the northern hemisphere.  That is, all of the tiles were shifted from south to north but otherwise in the correct latitude.  So, shifted north and vertically stacked the wrong way.  Immedate thoughts "uh oh!" but it turns out that that file naming convention I talked about earlier is now advantageous to us, because it's quite easy to write a quick python program that goes through and renames all the tiles so that they are correct names for where they SHOULD be (that is, shift from the wrong northern hemisphere numbering to the correct southern hemisphere numbering).

So, here's that little fixup FOR SOUTHERN HEMISPHERE MAPS ONLY!!!.  Northern hemispherians skip this next bit...

OK, first just add this to the top part of the file - it's a new function to fix the naming.

import math

def fixNorthSouthNaming(dir):

    directories = next(os.walk(dir))[1]
    directories = sorted(directories)

    for level in directories:
        base = math.pow(2.,float(level)-1)
        for root,dirs,files in os.walk(dest+'/'+level):
            for f in files:
                if 'png' in f:
                    num = float(f.split('.')[0])
                    num = base + (base-num) - 1
                    print 'level ',level,'rename ',f,' to ',num
                    os.rename(root+'/'+f,root+'/'+str(int(num))+'.png')

Lastly, at the very bottom of the file we want to add the call...

fixNorthSouthNaming(dest)

That's it.  If you've done things as instructed, then the original directory walk will first create the merged directory, then this new addition will go through and rename the tiles to fixup that 'bug' in the northern/southern hemisphere thing.  I'm sure that there is a better way to do this, but this is the hack I used, and it WORKS, so there's that.

Right, everyone should be here - both northern and southern folks.  We have a MERGED directory containing the processed tiles generated from multiple input georeferenced TIF files.  Just one thing to add here - those with eagle eyes might have noticed that I sorted the directories in the merging code.  There's a very specific reason for that.  Say we had high resolution tifs of some parts of the map.  Our original maps might be 1:100000 and we might have one or two areas at 1:25000.  We might only want to display the high resolution maps at zoom levels (say, 12, 13, 14).  Here's where we manually intercede. BEFORE running the merging code, find the directory with the tiles for each (remember, the directory name is the same as the TIF file name).  Go into it and DELETE the levels you do NOT want to see.  So, I would delete all directories 0,1,2,3,4,5,6,7,8,9,10,11 and keep 12,13,14.  That's the first step - the next step is to RENAME the directory so that its name comes after all of the other directories in an alphabetical sense.  So, call it 'zzz" or put 'zzz' in front of the name.  Basically that will force that directory to be processed last in the merge, and the upshot is that any images with pixels will also be processed last - and so overwrite the low-resolution pixels that may already be in the merged directory. Trust me, it works.

Well, here we are ready to actually create our database.  We'll do that with another free tool... MOBAC.  That's Mobile Atlas Creator, available from http://mobac.sourceforge.net/

Fantastic tool, but horrible UI/design and confusing to learn.  We will use this tool to create a 'sqlite' file that's needed by OSMAND to display our maps.

So, download and install MOBAC from the sourceforge project page linked from the previous URL, unzip it and there you go...  it's a java program, but on OSX at least you start it by opening a terminal and runinng 'start.sh' from the directory where you put it. BUT, don't do that yet!!!!

MOBAC is designed out of the box to connect to various online map sources, and grab the data from those. We're NOT doing that. We have custom maps already waiting in our merged directory and we need to point MOBAC to those. To do that you have to add a short file to one of the MOBAC subdirectories, to specify where the data is.  I created a file called 'local.xml' and its contents are...

<?xml version="1.0" encoding="UTF-8"?>

<localTileFiles>

   <name>ZoomTas</name>

   <sourceType>DIR_ZOOM_X_Y</sourceType>

   <sourceFolder>/Users/boo/Desktop/pymerge</sourceFolder>

   <invertYCoordinate>false</invertYCoordinate> <!-- optional -->

   <backgroundColor>#000000</backgroundColor>

</localTileFiles>

See where it says 'sourceFolder'?  That's where you need to put the destination directory you used for the merged stuff we just did. So, fix that up, and put this file in the MOBAC subdirectory 'mapsources'.  You can also name your source - here i have called mine 'ZoomTas' - call it whatever you like. Now we're ready to go.  Fire up MOBAC with ./start.sh from its root directory.

Ugly UI.  But lovely program. OK, let's get this happening.

Click on Atlas/new Atlas.  Give it a name, and select "OSMAND SQLite" as the format.  Very important you get this right!   If everything has worked, then we should see the name of the source that we put in the local.xml file (in my example, 'ZoomTas') appearing in the list of sources in the window top left of MOBAC.   Select it, and everything working OK, our map should actually load up in the view window on the right hand side (the one with all the red crosses that made us think nothing was working).  If you can't see anything, try changing that zoom slider to a small number. You can also click on the map area and scroll around with the arrow keys.  But hopefully, you will be able to find your map and even zoom in and out with that slider bar.  Almost there!

So if we actually see our map, all we need to do now is package the data into a database for OSMAND. That involves selecting an area on the map, selecting the zoom levels we want to include, and adding that selection to the atlas.  So here's where the UI is confusing. Essentially that "Zoom levels" area with all the tickboxes is only relevant just before you click on "Add Selection".  It tells MOBAC which zoom levels you want to include.  After you've done that, the ticks are meaningless - so don't try and changing them and go "export" - because export exports what is in the atlas, not what is shown on the tick-boxes.

So, go ahead - let's put a square around the entire area you want to export by selecting it in the map area with your mouse/button and drag. You will see a reddish square over the area.  Now click the zoom levels you want.  Something lick 7-14 will do for now. You can choose less or more.  But if you want a real quick test to see if it's working on your phone, just select a medium-zoom (perhaps 11 or so).  OK, select as mentioned, click the zoom levels tick-boxes as mentioned, and NOW click "Add Selection". In the Atlas Content window, where you see the atlas with the name you gave it, you will now see under that a "Layer" which lists the zoom levels you ticked.  This is pretty much all you HAVE to do. However, you can if you wish select other parts of your map, and repeat.  In other words, you can add multiple selections.

Right. almost done... now we generate the map database.  Click on 'Create Atlas'.  A window with lots of progress barring and time estimates will come up, and after a reasonable amount of time you will see it's finished and the "Open Atlas Folder" button is now clickable.  That's where your file is, baby!

Now it's time to get that file onto your phone.  Naturally you will need to have enough spare space on the phone to take this file. If you've been silly and included gazillion levels of a large area, then your file is going to be big. Smaller areas, or less levels are the way to go. Or, a big SD card :)  In any case, for the whole of my home state, the database file is about 1GB. That's very usable. We need to connect our phone via USB connection, and when it comes up look at the directories with a file browser or somesuch.  OSMAND is installed in /osmand - how nice - and under that is a directory called 'tiles'. THAT directory is where you dump the .sqlite file that MOBAC just created for you. So, copy it over... and now start OSMAND.

You want to tell OSMAND to USE the map you have just created.  I typically put it as an underlay map, but you can choose other options. Not really the point of this tutorial to show you how to use osmand, but instead just how to get a map.  But in short, do the following..  top left you will see an icon representing layers of maps. Looks a bit like a chesse sandwich. Click that. and provided you have a map source already setup, you will see just under that menu option "Overlay map" and "underlay map". CLick on underlay map and in the selection of maps that comes up you SHOULD see the map name of the sqlite file you just created.  Select that, and exit back to the main screen.

Now, don't be disappointed!  It seems that OSMAND needs a fair bit of time to update/convert that map into an internal format - perhaps indexing it or something, so that it can access it quicly.  So, scroll around a bit, or click the zoom levels.  One thing you will see if it's all OK is a small blue circle slider bar bottom middle. This is a cross-fade.  When you play with that you can fade between vector maps (I have my offline vectors as my main map, and as noted my tiled tif map as the underlay). So, slide that back and forth, and eventually -- once OSMAND has finished converting your database file.. it should ALL WORK!!!

Now obviously this first pass of instructions will have a few unclear areas. I wanted to get this out there as a thank-you to all those who have written tools such as MOBAC and the GDAL stuff. Fantastic stuff, and I hope that my own contribution here helps others in turn.

Cheers

A

[Andrew Davie]

You may need to install PIL - Python Image Library if you don't already have it!

 

 

 

[Max Erickson]

It won't do the nice merging of tiles that you have done, but you can use gdalbuildvrt to create a source mosiac:

http://www.gdal.org/gdalbuildvrt.html

This creates a vrt file that is essentially an index of the other files, and you can point anything using GDAL at the file and it will take care of fetching the data from the appropriate file.


Max

[Andrew Davie]

On Saturday, 5 March 2016 05:04:49 UTC+11, Max Erickson wrote:

It won't do the nice merging of tiles that you have done, but you can use gdalbuildvrt to create a source mosiac:

http://www.gdal.org/gdalbuildvrt.html

This creates a vrt file that is essentially an index of the other files, and you can point anything using GDAL at the file and it will take care of fetching the data from the appropriate file.

I will check that out. But, you are wrong about the merging - in my world the merging is beautiful - many of the merging is seamless, but where there are hue differences in the source maps they are evident. Even with overlaps, to my eye it looks fantastic. Here are two completely random examples...  the second shows a 4-way 'join'. Looks good to me!

 

[Andrew Davie]

I think I misunderstood your comment :)  Sorry about that. I thought you were saying that my solution wouldn't work :P

No more early-morning forum posts for me!

[Reinaldo Javier Menendez]

Hello, I need some help. I am making a project using Raspberry and GPS, so I want to read coordinates and show dots in a maps in live, just like osmand does. I found very interesting this post, but I can't find where to download TIF files for my country. I live in Cuba. Could you help me?

[xmd5a]

Personally I prefer to use QGis for preparing tiffs and then making tiles with QMetaTiles plugin. Then move tiles to OsmAndMapCreator working dir/tiles instead of tiles which were downloaded by MapCreator (I mean replace all tiles). Then open MapCreator and create sqlite. Next step is to open sqlite with some sqlite browser and delete column "url" in "info" table. All done.

[Reinaldo Javier Menendez]

Where can I donwload TIFFs files for my country (Cuba)?????

> --
> You received this message because you are subscribed to a topic in the
> Google Groups "Osmand" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/osmand/8iLr2fZKMpc/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> osmand+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>