Hillshading for Mapsforge

[Maki]

Il giorno sabato 22 novembre 2014 09:43:49 UTC+1, Ludwig ha scritto:

On 22 November 2014 at 07:49, Emux <deve...@gmail.com> wrote:

It would be handy too in this case to have the ability to store bitmap tiles in the map file, and control them via rendertheme.

You mean raster tile images inside the map file that can be controlled via the render theme and render along with the vector layers.

I don't know if that's the best solution, to mix them in the map file format, which is optimized already for vector data.

We have support for offline raster tiles pyramid via the TileStoreLayer, which can be put along in z-order with common vector layers, but via code.

Plus changing the map file format is really a different discussion, this was more about establishing a common standard for making maps and designing render themes so that we can achieve (at least for basic functionality) compatibility. 

Fair enough, let's start a new discussion. :-)

I think that any non-urban map needs hillshading, and I think it should be supported in the library.

In the discussion on OM forum http://oruxmaps.com/foro/viewtopic.php?f=8&t=1756 we examined various options, at page 2 I posted a simulation that I think shows how much better map readability is with relief shading.

One option is rendering on the fly from DEMs which is what Orux is trying to implement with some success (I think Locus already does it, but I haven't tried). Ludwig some time ago said it's a slow process for a mobile device. As we discovered experimenting that's true just to an extent. At higher zoom levels it can actually be done with ease, at lower zoom levels it can't, not with 1" DEMs anyway (but I'm convinced that with 3" DEMs it can be done). Other than rendering times, there is the problem of needing an external resource. Plus the installed DEMs may not match the map contours, giving a weird effect. Plus we can only apply the effect to everything, which has side effects.

Another option is pre-rendering the shades, which solves the speed problem, but not the others.

The third option is direct support in the map files, which is what I'm asking because I'm convinced it's the best overall solution. The map ships with the same data it was used to generate the contours, so everything matches, and the user doesn't have to hunt around for DEMs. Is this going to make the file size explode? I would have said yes, but it turns out the opposite is true. A 1°x1° tile from a 1" DEM can be JPEG compressed to 1-1.5MB, from a 3" DEM we talk about 300-400 KB. The tile should be stored at its native resolution and scaled up and down on the fly with a simple bilinear interpolation, no need to store tiles at different zoom levels, that's just a waste of space.

As I said in the other thread for best results shading should be applied only to terrain features, not highways & stuff, and should be controllable from the rendertheme. So we would need to have some control on layers and embedded pre-rendered shade data (blending mode and opacity).

I won't say it's urgent, but I think it's worth considering the first time the map creator is modified.

Maki

[Ludwig]

Maybe if someone could produce a set of hill-shaded tiles for a limited (but interesting areas, maybe something like Sicily) I could try to see what is actually required. 

I am not so sure if I would want to change the mapfile format, but I see that some sort of mixed layer could be interesting, e.g. a mixture out of our TileLayerStore with prerendered tiles together with data rendered and blended in some sort of way (where directives could direct the order and the blending mode). 

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/1ad68193-f1c0-4f2e-8ba7-9d089a555ee7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emux]

Are we absolutely sure that we cannot have the same result with the existing or *improved* vector drawing?

I'm not in favor either for changing the map file format, but I understand the need for including elevation data.

--
Emux
Cruiser - Atlas

[Ludwig]

The hill-shading Maki is talking about is really raster data as it gives for every single point a slope aspect that is coded as a greyscale value (as if the sun was shining at the hills from a certain angle). So IMHO we are not talking about contour-lines here. 

From the discussion at http://oruxmaps.com/foro/viewtopic.php?f=8&t=1756&start=10 that Maki mentioned, I would understand the rendering process as this:

• render area map data onto tile (like forests)
• overlay it by hillshaded tile and multiply for best appearance
• render road etc features on top of that.

I would not want to change the map file format either and I think that the main concern that hill-shading and contours can be ill-matched could be addressed in some other way. 

But that is early days, this is more something to play with. 

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/54775486.40708%40gmail.com.

[Emux]

The hill-shading Maki is talking about is really raster data as it gives for every single point a slope aspect that is coded as a greyscale value (as if the sun was shining at the hills from a certain angle). So IMHO we are not talking about contour-lines here.

Yes actually I know the data. I have builded tools and produced several sets of elevation pyramids years ago, before Android, in order to use it in home made Java (OpenGL) 2D / 3D map viewers.

[fzk]

Locus Pro and Oruxmap are using hgt-files in order to display hill shading. The result is something like this (Swiss, Vierwaldstätter See):

You can find such hgt-files for example here: http://www.viewfinderpanoramas.org/

BTW: It's not very helpful to display hill-shading in very high or very low zoom level. IMHO it's best to render hill-shading only in the zoom level range 9-16.

Klaus

[Ludwig]

IIRC, hgt is just a raster file format, with the disadvantage of being relatively hard to decode (obviously someone wrote a decoder for gdal)

The SRTM files in .hgt format are at the end just elevation files, computing the hill-shading from them takes another two steps: first computation of aspect, then shading, the aspect is always fixed, the shading depends on your sun model (lightsource elevation and angle) but I personally think that there is no point in a variation of this as it is just visually confusing. 

Precomputing the hill-shading makes sense as the resulting file size is the same as the input data (as long as you encode the resulting grey-scale with a maximum of 32000 colors). 

The viewfinder panoramas have a strange non-liberal license, which would not allow this data to be used in many commercial contexts. I personally tend to use straight SRTM data from NASA (no restrictions) with plain interpolated hole filling, which is good enough for most purposes. The popular hole-filled CGIAR data is not free. 

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/96c0045b-f472-416e-8b04-61b6327bd459%40googlegroups.com.

[Maki]

Il giorno giovedì 27 novembre 2014 17:28:57 UTC+1, Ludwig ha scritto:

Maybe if someone could produce a set of hill-shaded tiles for a limited (but interesting areas, maybe something like Sicily) I could try to see what is actually required. 

Not sure this is what you mean, but you can download a geotiff with Sicily's hill-shading and two mbtiles files, one at ZL10 the other at ZL11, from here https://app.box.com/s/m15m9imi5hisly62p8lf 

Hillshading tiles are smaller than corresponding HGT files, because they are only 8 bit grayscale.

To answer Emux, in theory we can do this in vectors, the same tools used to extract contour lines from elevation data can find areas of homogeneous color in an hillshaded geotiff. I did it with the file above and the shapefile describing 32 shades of grey is 116MB (46 if zipped), 64 shades is roughly double that; so it's big and will probably render in an unpleasant way.

The screenshot from Klaus shows two problems of applying the shade on top of everything: the same road on a south face is darker than a similar one on a north face, and, especially, the shade doesn't match the lake contour, it looks like the land is floating in the air. With water and highways put on top of everything else it would look a lot better. But, if Mapsforge can output RGBA bitmaps it can be done with only two layers (if the sea is above the land, that is).

As I said above it's not urgent at all, I was just "thinking forward". At some point I think it will be needed as a core feature. Thinking even more forward a generic mix of vector and raster data will be useful, there's stuff that just cannot be done in vectors. As storage space, processing power and bandwidth continue to grow, eventually it will be easy to do.

Maki

[Ludwig]

Not sure this is what you mean, but you can download a geotiff with Sicily's hill-shading and two mbtiles files, one at ZL10 the other at ZL11, from here https://app.box.com/s/m15m9imi5hisly62p8lf 

Thanks that is what I wanted, I understand that the tif file is a full-resolution hill-shade (i.e. not down-sampled from the original SRTM data), I have not yet looked at the mtb tiles. Are they down-sampled or just cut into tiles?

Ludwig

[Maki]

Il giorno venerdì 28 novembre 2014 06:51:12 UTC+1, Ludwig ha scritto:

Not sure this is what you mean, but you can download a geotiff with Sicily's hill-shading and two mbtiles files, one at ZL10 the other at ZL11, from here https://app.box.com/s/m15m9imi5hisly62p8lf 

Thanks that is what I wanted, I understand that the tif file is a full-resolution hill-shade (i.e. not down-sampled from the original SRTM data), I have not yet looked at the mtb tiles. Are they down-sampled or just cut into tiles?

Yes, the tif is straight out from QGIS (gdaldem, actually) from 3" DEMs from viewfinderpanorams. For reference, a single 1°x1° tile gives a 1201x1201 pixels tile.

The mbtiles are generated with Tilemill. AFAIK tilemill considers ZL0 the whole planet and than splits each tile in 4 at every next ZL. I doubt it matches exactly the geotiff so there is almost surely some interpolation. Visually the ZL11 file isn't showing more detail than the ZL10 one. Now that I think about it I didn't set any resampling algorithm so nearest neighbor is used. I'll look better at it this evening, but it's just a (marginal) matter of image quality.

Maki

[Ludwig]

Just thinking aloud here.

1. To minimize work on the client, the raster data should be projected into spherical mercator, epsg:900913
2. Implementing a reader for any sort of geotiff is actually pretty difficult, there are libraries out there, but they tend to pull in an enormous amount of other stuff, so starting from a tiled source would be an easier start (but of course it increases the amount of data to be shipped).
3. The source of the tile data could at least initially be our TileStoreLayer (which could later be specialized into a method that operates on a geotiff or whatever file to produce tiles on the fly, would likely involve https://developer.android.com/reference/java/nio/channels/ByteChannel.html, but an external solution would be preferable).
4. Then have something like a 'SandwichLayer' that consists of multiple tile renderer layers and tile layers, where each feeds its results as the bitmap onto which the next layer draws (together with a blending instruction). So each of the renderer layers would have their own rendertheme (e.g. the lowest one only instructions for rendering areas, the top one only roads that are then drawn on top.) Between them would be a tile layer, that reads in the hill-shading data and blends according to instruction. So in pattern-talk, the SandwichLayer is a Composite, a Layer made up from Layers with some instructions as to how the composition works).
5. Done.

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/a6252abb-adcb-40c3-bd98-400d430a0d62%40googlegroups.com.

[Maki]

I'm not familiar with the map creator, let alone the map format internals, so I can't really comment on what's best. My idea was to feed the map creator with a geotiff and get back tiles that are stored in the map file at the most appropriate zoom level. All the client has to do is to assemble/crop/resample tiles to cover the needed area at the chosen zoom level. If I haven't screwed up the maths, assuming a standard 256px tile, for the typical 3"DEM the most appropriate ZL is 11 that uses 1456 px to cover 1° vs 1201px outputted by gdaldem. For 1" DEMs ZL12 should be enough (2912px for a 3601px source is plenty acceptable, ZL 14 would be a waste of space).

I proposed to store the raster tiles into the map file for user convenience, but using a sidecar file would work just as well without the need to touch the map file format. Under a certain point of view it would be even better since one doesn't need to download it every time a map is updated, but in most cases the overhead should be relatively small. All in all, in the long term I think a single file is overall the best solution, but it's not needed at all to experiment.

Maki

[Richard Z]

On Thu, Nov 27, 2014 at 02:52:48PM -0800, Maki wrote:

> Il giorno giovedě 27 novembre 2014 17:28:57 UTC+1, Ludwig ha scritto:
> >
> > Maybe if someone could produce a set of hill-shaded tiles for a limited
> > (but interesting areas, maybe something like Sicily) I could try to see
> > what is actually required.
> >
>
> Not sure this is what you mean, but you can download a geotiff with
> Sicily's hill-shading and two mbtiles files, one at ZL10 the other at ZL11,
> from here https://app.box.com/s/m15m9imi5hisly62p8lf
>
> Hillshading tiles are smaller than corresponding HGT files, because they
> are only 8 bit grayscale.
>
> To answer Emux, in theory we can do this in vectors, the same tools used to
> extract contour lines from elevation data can find areas of homogeneous
> color in an hillshaded geotiff. I did it with the file above and the
> shapefile describing 32 shades of grey is 116MB (46 if zipped), 64 shades
> is roughly double that; so it's big and will probably render in an
> unpleasant way.

could the height lines included in some map files be used to do hillshading ?

The data is already there but I assume it would be a costly computation at
rendering time?


Richard

---
Name and OpenPGP keys available from pgp key servers

[Ludwig]

could the height lines included in some map files be used to do hillshading ?

No, the hill-shading really needs an aspect value (the slope degree and orientation) per pixel, in a second step then a light source is imagined at a certain angle and hight shining on the hills. 

The aspect is of course in a way embedded in contour lines, but only in a very crude way, and if you refined the contour information to the degree to produce a nice hill-shading it would take up much more space than a raster file. 

It is more the other way round: contour lines could be embedded in the hill-shading raster file without requiring additional space (if a grey value is used), but I am not sure how that would look.

Ludwig