Creation of hill shading tiles with color-coded inclination

[Florian Achleitner]

Hi everybody!

I really fancy the hill shading feature together with contour lines! Thanks for adding this cool thing!

It's especially useful for mountain sports and it simply looks great and led me to another idea..

Out of curiosity I tried to find the tools that create the hillshade tiles

in the osmand git repos and read some threads in this group, but I couldn't

find how they are actually made.

Can you give me a hint?

Are you going to change it to some vector format?

My idea was to add another hill shading feature that displays slope inclination,

like an additional layer that displays different inclination ranges in different colors. 

This would quite useful for mountaineering and especially for ski touring where you are 

interested in how steep a hill actually is.

Thanks,

Florian

[yvecai]

The tools used to generate contour tiles are here:
https://github.com/osmandapp/OsmAnd-tools/tree/master/obf-generation/contours/V3
The core tool being Gdalcontour.

For hillshading, all the primary work to build a big hillshading file was made 'by hand', in Osmand repo there is just the splitting script to generate countries extracts.
https://github.com/osmandapp/OsmAnd-tools/blob/master/obf-generation/contours/V3/extractSqlite.py
The big sqlite was done with Gdaldem and a modified gdal2tiles script.

I had a request for such a slope layer, but haven't much time nor serverpower to do this right know. If you have 200GB and 2-3 weeks computing power available, I can guide you and provide the data.

[Florian Achleitner]

Let's try how far I get.

I don't have a lot of 'serverpower', but time and space shouldn't be the problem. Do you create those layers for the complete world at once?

It would probably nice to only try it for the alps or so ..

Can you give me some guide for the 'by hand' work?

I'm not yet familiar with the tools you mentioned. 

Thanks!

Florian

[Florian Achleitner]

Cool, gdaldem can already create a slope raster map, containing angle values from some DEM input..

[yvecai]

Please send me an email in private so that I can provide a download link for the DEM (~20GB).
mypseudo at gmail.com

I'll clean a little my script and put them in github latest tomorrow.

Basically you will need gdal and python on a linux machine.

[yvecai]

Here are all what you need (except DEM):
http://github.com/osmandapp/OsmAnd-tools/tree/master/obf-generation/hillshade

It's much more simpler than it looked when I built the hillshading layer, I should have committed earlier :)

Yes gdaldem slope is the right tool for what you want.
For the hillshading layer I used a mix between hillshading and slopes, so you'll find what you want.

For the record, I was asked before to provide a layer such as in the Swiss maps with areas with slopes > 30° colored in pink on pistes-nordiques.org, so I will be glad to provide the result there :)
I think this should be do-able, but some tests should be done on a smaller area.
Also, given the accuracy of the DEM, a BIG disclaimer should be advertised for ski-tourers : we don't want to kiill anybody.

Yves

[Florian Achleitner]

My first question is now, how can I restrict the area to a small part of the
world for testing? So that it doesn't take weeks before I know if it works.
And how can I look at the data and verify it?

I'm curious how accurate the output will be, the disclamer will be needed
anyways ;).

Florian

[yvecai]

You downloaded the all set, so you can copy some of the files in a separte directory.
Something like

cp N4*E00* subset/

for a part of the alps.
To look at the data, quantumGis is handy.

[Florian Achleitner]

I played around with gdaldem and the first part of your script and created
quite what I wanted. A combination of hillshade and color coded slope
inclination.
(Only for some tiles in Austria currently.)
It looks nice in qgis, but I'd like to see it together with osmand maps.
Related question: Is there any way to render osmand's obf files with some
overlay to get a preview on the PC?

The lower part of your script merges everything in a huge tiff and transforms
coordinates. I guess this is neccessary to be compatible with osmand's
requirements.
Then it creates those sqlite files (which takes by far the most time).

What modifications did you make in ./gdal2tiles_gray2alpha_sqlite.py? Does it
some channel mapping? This could be important because my hillshade is no
longer a single gray channel image, but RGB color.

And finally: Is it possible, as long as the complete file is small, to feed the
sqlite produced by this step into osmand, skipping extractSqlite.py?
It currently fails when trying to create some file for Russia, probably because
it expects the whole world to be available ;).

Thanks!
Florian

[yvecai]

On Wednesday, March 27, 2013 3:48:18 PM UTC+1, Florian Achleitner wrote:

I played around with gdaldem and the first part of your script and created
quite what I wanted. A combination of hillshade and color coded slope
inclination.
(Only for some tiles in Austria currently.)
It looks nice in qgis, but I'd like to see it together with osmand maps.
Related question: Is there any way to render osmand's obf files with some
overlay to get a preview on the PC?

I use VMLite on virtualbox for testing, it works great. http://www.vmlite.com/
Also, you can use MOBAC to view the sqlite (when in BigPlanet / Osmand format, see below).

The lower part of your script merges everything in a huge tiff and transforms
coordinates. I guess this is neccessary to be compatible with osmand's
requirements.
Then it creates those sqlite files (which takes by far the most time).

What modifications did you make in ./gdal2tiles_gray2alpha_sqlite.py? Does it
some channel mapping? This could be important because my hillshade is no
longer a single gray channel image, but RGB color.

Hmm hmm, probably not. It takes a single band image and save it to the alpha channel of black png tiles.
There is some work to be done here.

And finally: Is it possible, as long as the complete file is small, to feed the
sqlite produced by this step into osmand, skipping extractSqlite.py?

 

No, because of the numbering scheme of the tiles. My modified version of gdal2tiles save tiles coordinates in TMS, Osmand uses BigPlanet, I'll make a small script for this so that you can skip extractSqlite.py for testing.

 

[yvecai]

I've just added the sqlite converter in osmand-tools/hillshade

The gdal2tile issue is a bigger one. Do you have a screenshot of the TIFF you obtained that please you?

[Florian Achleitner]

Am Donnerstag, 28. März 2013 19:09:18 UTC+1 schrieb yvecai:

I've just added the sqlite converter in osmand-tools/hillshade

Thanks! I'll try it now..

 

The gdal2tile issue is a bigger one. Do you have a screenshot of the TIFF you obtained that please you?

 

.. and look into gdal2tile then. I attach a small crop of the generated TIFF. The color from yellow to violet represents slope angles from 30 to >45 degrees.

I haven't yet seen this as an overlay on a rendered map. Probably a rather white winter rendering style would be nice for the map. Wasn't something like this already 

available? My osmand doesn't have it currently.

Florian

[Florian Achleitner]

I have now some demo screenshots for the slope inclination and hillshading layer on top of the winter rendering style.

I restructured the creation process a little. The alpha-channel mapping originally done in yvecai's modified gdal2tiles is 

now done by imagemagick's convert when creating the tiles at the beginning.

Then, I found that the creation of a whole giant world tiff and reprojecting it is not necessary, because gdal2tiles can read a vrt 

file and transform the projection on the fly. Saves a lot of disk space, along with enabling lossless LZW compression for the tiff format.

I use TMS tiles created by gdal2tiles in the corresponding directory structure instead of a giant sqlite and create country-specific

sqlites from that files at the end. Hence, no patched gdal2tiles is needed, and the creation could probably be parallelized easier.

However, there's still a problem I can't track down yet. The resolution of the hillshading layer created by myself is lower than what can 

be downloaded from the osmand site. The tile zoom levels seem to match, gdal2tiles creates zoom from 0..11, and in the sqlite

they seem to be same, too (9..13 in BigPlanet format). On the screenshots you can see the smooth original shading layer and the course

one I created. Any tips are welcome! 

Florian

[Hardy]

Hm . but am I right that we are trying to produce something more like shown as "color tint" on this web site ?

http://www.esri.com/news/arcwatch/0312/learn-a-new-method-for-displaying-hillshades-and-elevation-tints.html 

 

I think this is the one common way of depicting altitudes which OsmAnd is still missing ... although itwill  heaviliy collide with all other map features for normal use which we depict mostly by color-coding areas ...

[Florian Achleitner]

What I try to produce doesn't color-tint altitudes, but gradients.
the color should tell the steepness of the terrain. Only areas steeper than 30
degrees are tinted at all, changing color every 5 degrees.
This would be useful for ski touring and other mountain sports.

The colors tend to collide with the underlying map, thats right. It looks best
on top of the white/blue winter rendering style. I think users who are not
interested in slope inlination angle will not use it anyways.

[yvecai]

Vrt : the issue here will be artifacts every 1 degree (between dem files). The only way I found was to merge in wgs84 then reproject in Mercator.

For the issue with resolution, you should use gdal2tiles with -antialias from z11.

For tiles vs sqlite: try once to play 'rm -rf' on the all world at z11 :)

[yvecai]

What about gdalpolygonize to get these data in vector format ?That would be rather efficient.

[Florian Achleitner]

You see, I've only tried it on a 4-tiles-area yet. :)
Do you have an idea, why the shading I produced is of such a coarse
resolution? It's the same if I create the intermediate whole-world images.
Did you use a higher resolution DEM? Or is it just a better resampling
interpolation.

On Sunday 31 March 2013 10:16:28 yvecai wrote:
> Vrt : the issue here will be artifacts every 1 degree (between dem files).
> The only way I found was to merge in wgs84 then reproject in Mercator.

How do those artifacts look like? They might have fixed it in gdal (?).

> For the issue with resolution, you should use gdal2tiles with -antialias
> from z11.
>
> For tiles vs sqlite: try once to play 'rm -rf' on the all world at z11 :)

That's a filesystem vs. sqlite benchmark :) Looking forward to try it ;)

[yvecai]

Use original gdal2tiles with --resampling=antialias for better looking resampling. That's what I've done for hillshading.

The artifacts came from resampling the tile edges, it's better if gdal know the neighbour pixel. It's hard to find a recent gdal package, and complicated to compile all dependencies.

For the benchmark, you really don't want to try, it take a few minutes to copy a big sqlite, hours for all these all pkg.

But given the small areas with steep slopes, a vector layer could be better handled by Osmand, and much lighter than tiles. Off course you won't have the smooth coloring effect, but you can style the area for better visibility.

[Florian Achleitner]

Am Montag, 1. April 2013 00:13:55 UTC+2 schrieb yvecai:

Use original gdal2tiles with --resampling=antialias for better looking resampling. That's what I've done for hillshading.

The artifacts came from resampling the tile edges, it's better if gdal know the neighbour pixel. It's hard to find a recent gdal package, and complicated to compile all dependencies.

The reason for the difference in smoothness was, that the modified gdal2tiles_gray2alpha_sqlite applied a blur-filter when resizing the tiles. 

I was using 'antialias' anyways.

 

But given the small areas with steep slopes, a vector layer could be better handled by Osmand, and much lighter than tiles. Off course you won't have the smooth coloring effect, but you can style the area for better visibility.

Could give it a try, 

[Florian Achleitner]

Am Dienstag, 2. April 2013 11:39:12 UTC+2 schrieb Florian Achleitner:

 

But given the small areas with steep slopes, a vector layer could be better handled by Osmand, and much lighter than tiles. Off course you won't have the smooth coloring effect, but you can style the area for better visibility.

Could give it a try, 

Doesn't look very promising. I tried gdal_polygonize on the hillshading layer and the slope inclination layer in the form they are produced by gdaldem.

Converting the hillshading layer takes a huge amount of time (like 2hours for 2x2 degrees) and results in a huge file.

The slope layer is empty in many areas and therefore works better, but still the xml polygon file is twice the size of the compressed tif.

The problem is that both layers contain only very small continuous areas that can be converted to bigger polygons efficiently. 

On the hillshading layer almost every pixel has a value different from it's neighbor's. On the slope layer there are slightly

bigger areas, but still only a few pixels in size. 

[yvecai]

I think polygonize should be used on two or three values slope layer, like white 0-30deg., gray 30-40, black 40+.

[Florian Achleitner]

Am Dienstag, 2. April 2013 19:38:18 UTC+2 schrieb yvecai:

I think polygonize should be used on two or three values slope layer, like white 0-30deg., gray 30-40, black 40+.

Yes, I use four colors. 0-30 transparent, 30-35 yellow, 35-40 orange, 40-45 red, 45+ violet.

But regions of the same color are usually not much larger than a few pixels.

[yvecai]

I made a test and obtained good results with this:

gdaldem slope /home/yves/DEV/N44E00/N44E006.tif slopes.tif -s 111120.0 -of GTiff
gdaldem color-relief color_slope2.txt 30degree_slopes.tif -of GTiff
##~ color_slope2.txt :
#~ 0 0 0 0
#~ 30 0 0 0
#~ 30.0001 255 255 255
#~ 90 255 255 255
gdal_contour -i 255 30degree_slopes.tif 30degree_slopes.shp

With gdal_contour, there is no staircase effect. With only one set of slope>30°, the file size are more reasonable.
But then the shapefile is made of linestrings, that should be converted to polygons, then bufferized a little for a bit of roundness, then converted to osm, then to obf.

Do-able.

[Florian Achleitner]

I've now managed created the tiles for this layer for the whole world. Still to be done are the per-country sqlites. 

Would you prefer the vector-data solution?

[Victor Shcherb]

Hillshade tiles are not supposed to be vector solution or am I missing? If you have something interesting we can easily put it somewhere here for example or even consider it to distribute, here http://download.osmand.net/hillshade/.

Best Regards,

Victor

On Thu, Jun 27, 2013 at 9:05 AM, Florian Achleitner <florian.achl...@gmail.com> wrote:

I've now managed created the tiles for this layer for the whole world. Still to be done are the per-country sqlites. 

Would you prefer the vector-data solution?

--
You received this message because you are subscribed to the Google Groups "Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Hardy]

I would really like to see it as vector data, had suggested that some months ago! How does it look in size and performance-wise? Could we try a small portion, just to check it out?

 

 

[yvecai]

Victor, Hardy,
It's not strictly Hillshading, read the start of the thread :)
Is it available somewhere ?

[Hardy]

True! Thanks, Yves! Still interested in seeing stuff like this in vector format, it could give us more flexibilty here, could maybe revolutionize things a bit ... :-)

 

[Florian Achleitner]

I've created the sqlites for Europe as a test now. Of course I could upload
them somewhere!? The highest zoom level would look better after applying some
blur.

[Victor Shcherb]

To Hardy : 

Definitely not vector format :) But it could be better understandable by program. Our vector format for contour lines takes 3 times more space than raw data. If we can process raw data faster then unpack vector data we need to look at raw data format.

For example to determine heights and optimize routing TIF data is even more suitable (grayscale to represent height) than whatever vector format (I mean equally suitable and of course raw data is easier to distribute).

So it is question for specific task which format is better to use, I'm not sure if our user regrets that hillshade data is in sqlitedb format rather than obf.

Regards,

Victor

On Fri, Jun 28, 2013 at 8:46 PM, Hardy <hm.gg...@gmail.com> wrote:

True! Thanks, Yves! Still interested in seeing stuff like this in vector format, it could give us more flexibilty here, could maybe revolutionize things a bit ... :-)

 

--