Theme extension for hillshading?
691 views
Skip to first unread message
usrusr
Feb 6, 2017, 5:48:11 PM2/6/17
to mapsforge-dev
Hi mapsforge,
my name is Ulf and I am currently trying my luck at adding offline hillshading to mapsforge, from .hgt files like the hillshading in Orux.
(Background: toy project depending on mapsforge, then got lost scratching this particular itch)
So far I am seeing good progress, but I am not sure if my current theme extension (yes, unfortunately...) is the best approach.
The reason why I even started to mess with ThemeHandler is this: the hillshading should be painted on top of areas like forests, but under non-area features like roads. Since the drawing order within a layer, the level, is determined from the order of rules in the theme, inserting a hillshading element between top-level rules seemed to be the clearest way forward. This is what the current implementation in my branch does, and it uses the opportunity to set zoom-min, zoom-max and the depth of color-mangling from the theme (technically, it should even be possible to use separate elements for multiple zoom ranges with different parameter values or drawing order positions, but that wasn't a deliberate goal).
From an integration perspective, I dislike the requirement for updated themes, particularly since they involve an updated schema that would make them incompatible with older builds.
The alternatives I have been able to come up with myself are worse though:
a) use some heuristic to determine the level, e.g. find the highest rule that matches k=landuse and paint on top of that
b) abuse rules with some butchered magic value so that themes with hillshading are at least don't crash older versions, e.g. trigger on some magic nonsense like k="~~hillshading:6-14:64~~"
I think that both a) and b) would be terrible as permanent solutions, but I do consider having an optional preprocessor XmlRenderTheme implementation that does a) to insert a default hillshading element on the fly. I already did something very close in my very first prototype implementation, just need to port it over from scala (well, more like "pig-scala" actually...).
Any comments/suggestions?
PS: oh, and the mysterious layer byte: I did not find much about it in mapsforge, is that something inherited directly out of OSM? Is it OK to hardwire the layer or does it have to be cleverly inferred, e.g. during waymatching? (where I would be able to see the layer of ways from the map base)
my name is Ulf and I am currently trying my luck at adding offline hillshading to mapsforge, from .hgt files like the hillshading in Orux.
(Background: toy project depending on mapsforge, then got lost scratching this particular itch)
So far I am seeing good progress, but I am not sure if my current theme extension (yes, unfortunately...) is the best approach.
The reason why I even started to mess with ThemeHandler is this: the hillshading should be painted on top of areas like forests, but under non-area features like roads. Since the drawing order within a layer, the level, is determined from the order of rules in the theme, inserting a hillshading element between top-level rules seemed to be the clearest way forward. This is what the current implementation in my branch does, and it uses the opportunity to set zoom-min, zoom-max and the depth of color-mangling from the theme (technically, it should even be possible to use separate elements for multiple zoom ranges with different parameter values or drawing order positions, but that wasn't a deliberate goal).
From an integration perspective, I dislike the requirement for updated themes, particularly since they involve an updated schema that would make them incompatible with older builds.
The alternatives I have been able to come up with myself are worse though:
a) use some heuristic to determine the level, e.g. find the highest rule that matches k=landuse and paint on top of that
b) abuse rules with some butchered magic value so that themes with hillshading are at least don't crash older versions, e.g. trigger on some magic nonsense like k="~~hillshading:6-14:64~~"
I think that both a) and b) would be terrible as permanent solutions, but I do consider having an optional preprocessor XmlRenderTheme implementation that does a) to insert a default hillshading element on the fly. I already did something very close in my very first prototype implementation, just need to port it over from scala (well, more like "pig-scala" actually...).
Any comments/suggestions?
PS: oh, and the mysterious layer byte: I did not find much about it in mapsforge, is that something inherited directly out of OSM? Is it OK to hardwire the layer or does it have to be cleverly inferred, e.g. during waymatching? (where I would be able to see the layer of ways from the map base)
Emux
Feb 7, 2017, 3:41:16 AM2/7/17
to mapsfo...@googlegroups.com
Interesting feature to discuss!
There could be two ways to "show" hillshading in vector maps.
- The easy (ready) way is to add a bitmap tile layer on top with pre-rendered hillshaded semi-transparent png tiles containing the shaded reliefs (levels of gray).
e.g. OpenMapSurfer hillshading
This already works in both Mapsforge + VTM without changes anywhere. :)
(see attached screenshots with Mapsforge and VTM renderers)
- The "vector" way is to perform the rendering in real-time on client using e.g. height files.
That could involve two steps:
1. A reader / parser of the height files is needed to extract the information
2. Somehow the extracted height information has to be rendered on screen combined with the map
How do you expect to visualize those heights, like actual 3D terrain ...in Mapsforge (software rendered) MapView ?
Or just colorize the shaded reliefs by altitude (like 1st method with bitmaps tiles)?
(BTW Locus / Orux could use a custom map renderer)
And we are not against theme schema additions, if they're necessary and being the "cleanest" way to include a feature.
Since heuristics based on tag names are usually a recipe for future problems, so we should not rely on fixed names..
Still we prefer any API changes to be in small distinct steps (i.e. pull requests) in order to better study and test them.
There could be two ways to "show" hillshading in vector maps.
- The easy (ready) way is to add a bitmap tile layer on top with pre-rendered hillshaded semi-transparent png tiles containing the shaded reliefs (levels of gray).
e.g. OpenMapSurfer hillshading
This already works in both Mapsforge + VTM without changes anywhere. :)
(see attached screenshots with Mapsforge and VTM renderers)
- The "vector" way is to perform the rendering in real-time on client using e.g. height files.
That could involve two steps:
1. A reader / parser of the height files is needed to extract the information
2. Somehow the extracted height information has to be rendered on screen combined with the map
How do you expect to visualize those heights, like actual 3D terrain ...in Mapsforge (software rendered) MapView ?
Or just colorize the shaded reliefs by altitude (like 1st method with bitmaps tiles)?
(BTW Locus / Orux could use a custom map renderer)
And we are not against theme schema additions, if they're necessary and being the "cleanest" way to include a feature.
Since heuristics based on tag names are usually a recipe for future problems, so we should not rely on fixed names..
Still we prefer any API changes to be in small distinct steps (i.e. pull requests) in order to better study and test them.
--
Emux
Emux
gro...@mail.ulf-schreiber.de
Feb 9, 2017, 7:34:44 PM2/9/17
to mapsfo...@googlegroups.com
Am 07.02.2017 um 09:41 schrieb Emux:
> Interesting feature to discuss!
>
> There could be two ways to "show" hillshading in vector maps.
>
> - The easy (ready) way is to add a bitmap tile layer on top with
> pre-rendered hillshaded semi-transparent png tiles containing the shaded
> reliefs <http://wiki.openstreetmap.org/wiki/Relief_maps> (levels of gray).
> Interesting feature to discuss!
>
> There could be two ways to "show" hillshading in vector maps.
>
> - The easy (ready) way is to add a bitmap tile layer on top with
> pre-rendered hillshaded semi-transparent png tiles containing the shaded
> e.g. OpenMapSurfer <http://korona.geog.uni-heidelberg.de/> hillshading
>
> This already works in both Mapsforge + VTM without changes anywhere. :)
> (see attached screenshots with Mapsforge and VTM renderers)
This tends to make it difficult to see roads that lie "in the shadow"
> This already works in both Mapsforge + VTM without changes anywhere. :)
> (see attached screenshots with Mapsforge and VTM renderers)
(compare the Cruiser screenshots with the attached samples from my
branch build). With the theme extension for selecting/reserving a
painting level, prerendered tiles could be spliced into the rendering
just as well as internally rendered ("the vector way"). But if they come
from a remote source one would want to render an unshaded version and
then repaint once the data arrives, but that amount tile-management is
beyond my current understanding of mapsforge internals.
> - The "vector" way is to perform the rendering in real-time on client
> using e.g. height files.
>
> That could involve two steps:
> 1. A reader / parser of the height files is needed to extract the
> information
> 2. Somehow the extracted height information has to be rendered on screen
> combined with the map
repository of prerendered shading tiles would be smaller (well optimized
image formats and 8 bit alpha vs. 16 bit height model), but there are
more usecases for an offline height model than just using it as input
for relief shading.
The implementation for #1 is still quite rudimentary. It does not even
use any trigonometric functions yet, i just wanted to have something
recognizable for verifying the rest (not visible in the screenshots, but
the current version is really inadequate for less dramatic landscapes).
As you can see in the first screenshot (most prominently on the
south/west lake shore between Mandello del Lario and Valmadrera), #2
still has some alignment issues. I think that this is because currently
the 1°-SRTM files are scaled between would-be pixel positions that may
lie far outside the tile, I hope this will be resolved when i project
the visible intersection into the SRTM coordinates and calculate the
bitmap transformation so that the corners of the visible parts are correct.
Also, #2 on Android currently does not look nice on high zoom levels, as
the default bitmap canvas seems to lack any form of interpolation when
upscaling. I try to do API level switch to do this part of painting in
renderscript on API 17+.
[..]
> Since heuristics based on tag names are usually a recipe for future
> problems, so we should not rely on fixed names..
I could not agree more... If I take the compatibility route with an
> problems, so we should not rely on fixed names..
optional XmlRenderTheme preprocessor I will make it a (tiny) separate
module.
> Still we prefer any API changes to be in small distinct steps (i.e. pull
> requests) in order to better study and test them.
and would not send a PR before reaching at least "good enough". If WIP
PR help, making them is sure easier than reading ;)
Emux
Feb 10, 2017, 4:13:26 AM2/10/17
to mapsfo...@googlegroups.com
On 10/02/2017 02:35 πμ,
gro...@mail.ulf-schreiber.de wrote:
This tends to make it difficult to see roads that lie "in the shadow" (compare the Cruiser screenshots with the attached samples from my branch build).
Yes, I just blended (one) hillshading source with Mapsforge map,
using the existing default theme, for demonstrating the feature.
Also depends on the gray levels the tile source has been used in the shaded reliefs.
Using a different theme (or better a theme style) if hillshading enabled is actually the easiest part.
But that's more app responsibility. :)
Also depends on the gray levels the tile source has been used in the shaded reliefs.
Using a different theme (or better a theme style) if hillshading enabled is actually the easiest part.
But that's more app responsibility. :)
With the theme extension for selecting/reserving a
painting level, prerendered tiles could be spliced into the rendering
just as well as internally rendered ("the vector way"). But if they come
from a remote source one would want to render an unshaded version and
then repaint once the data arrives, but that amount tile-management is
beyond my current understanding of mapsforge internals.
That could get overcomplicated and we better focus for simple steps
in the start?
- Height reader
- Height renderer
- Potential theme changes
(That's why I mentioned compact PR, each for each functionality added on top, no need for complex changes all at once, that would make it very difficult to review / merge)
- Height reader
- Height renderer
- Potential theme changes
(That's why I mentioned compact PR, each for each functionality added on top, no need for complex changes all at once, that would make it very difficult to review / merge)
That is basically what created the attached screenshots.
Thanks for those, nice result!
Also, #2 on Android currently does not look nice on high zoom levels, as the default bitmap canvas seems to lack any form of interpolation when upscaling. I try to do API level switch to do this part of painting in renderscript on API 17+.
Minor note that Mapsforge having a multi-platform focus, e.g.
'map-reader' is a platform agnostic module, so expecting any core
feature to not rely on some specific OS functionality.
So far I just toil on in the branch, occasionally merging from master, and would not send a PR before reaching at least "good enough". If WIP PR help, making them is sure easier than reading ;)
Sure we can discuss more in a PR actual code, when you feel there is
some "workable" result.
And I could merge it in a separate branch for start so more users to test it.
And I could merge it in a separate branch for start so more users to test it.
--
Emux
Emux
gro...@mail.ulf-schreiber.de
Feb 11, 2017, 2:38:06 PM2/11/17
to mapsfo...@googlegroups.com
Am 10.02.2017 um 10:13 schrieb Emux:
> Yes, I just blended (one) hillshading source with Mapsforge map, using
> the existing default theme, for demonstrating the feature.
> Also depends on the gray levels the tile source has been used in the
> shaded reliefs.
The amount of shading certainly makes a big difference. I call that
> Yes, I just blended (one) hillshading source with Mapsforge map, using
> the existing default theme, for demonstrating the feature.
> Also depends on the gray levels the tile source has been used in the
> shaded reliefs.
parameter "magnitude" in my code so far because I want to avoid
referencing specific blending modes.
For AWT I have implemented blending
in HSV space to keep more of the colors of the original theme intact,
but I am not sure yet if I can achieve the same on Android (for both API
and efficiency reasons). See the attached screenshot for a comparison
between flat, alpha blending and the luminance/lightness blending that
keeps the saturation and chroma of the original colors. I think this is
important because even when themes would be rewritten to look good under
strong blending, they should still work well in absence of hill shading.
> That could get overcomplicated and we better focus for simple steps in
> the start?
> - Height reader
> - Height renderer
> - Potential theme changes
>
> (That's why I mentioned compact PR, each for each functionality added on
> top, no need for complex changes all at once, that would make it very
> difficult to review / merge)
the architecture forced upon mapsforge by its multiplatform nature
should make those elements easily distinguishable nonetheless. The wide
range of files touched mostly stems from the requirement to follow the
existing flow of styling information all the way from theme parsing to
drawing on the tile bitmap, just to hook into the painting order at the
right point in time. Little of that would make sense in isolation.
> Minor note that Mapsforge having a multi-platform focus, e.g.
> 'map-reader' is a platform agnostic module, so expecting any core
> feature to not rely on some specific OS functionality.
on single-channel bitmaps, to make them somewhat cache-friendly
(hillshading might still not be for the weakest devices though). The
existing architecture does quite a good job of forcing changes to stay
platform-agnostic. (But I often felt very tempted to do a separate,
unrelated round of refactoring to eradicate all static references to
GraphicsFactory instances and methods, I really believe that it should
be something that a client application should be able to override)
Emux
Feb 12, 2017, 3:06:33 AM2/12/17
to mapsfo...@googlegroups.com
On 11/02/2017 09:38 μμ,
gro...@mail.ulf-schreiber.de wrote:
even when themes would be rewritten to look good under strong blending, they should still work well in absence of hill shading.
Indeed it'd be nice if we have not to change the render theme
coloring in case of hillshading, there are themes with too many
rules and complicated.
It's true, the PR I put together today touches quite a few files.
Thanks for the PR, I'll review it!
The existing architecture does quite a good job of forcing changes to stay platform-agnostic. (But I often felt very tempted to do a separate, unrelated round of refactoring to eradicate all static references to GraphicsFactory instances and methods, I really believe that it should be something that a client application should be able to override)
Mapsforge's architecture is one of its strengths, being able to
serve more than one platform.
We can always enhance its graphics API, though already users can get original platform classes and perform more complicated things in draw calls.
e.g. with AndroidGraphicFactory static getters
We can always enhance its graphics API, though already users can get original platform classes and perform more complicated things in draw calls.
e.g. with AndroidGraphicFactory static getters
--
Emux
Emux
Emux
Feb 12, 2017, 1:57:28 PM2/12/17
to mapsfo...@googlegroups.com
I merged the PR in Mapsforge hillshading
branch (with minor improvements).
Anyone wanting to try the real time hillshading can use the Android or Desktop samples.
Provided there is a folder with SRTM HGT files for the loaded map, see HillshadingMapViewer for Android path or pass it as argument on Desktop.
My first remarks would be:
- The area patterns appear with wrong color in zooms <13
- At large zooms there are artifacts at tile borders
BTW the result is very promising!
Anyone wanting to try the real time hillshading can use the Android or Desktop samples.
Provided there is a folder with SRTM HGT files for the loaded map, see HillshadingMapViewer for Android path or pass it as argument on Desktop.
My first remarks would be:
- The area patterns appear with wrong color in zooms <13
- At large zooms there are artifacts at tile borders
BTW the result is very promising!
--
Emux
Emux
Longri, Andre Höpfner
Feb 18, 2017, 7:44:20 AM2/18/17
to mapsfo...@googlegroups.com
I have found a interest real time Hilshade for OpenGL!
--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mapsfo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mapsforge-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/98e3e454-3b3c-856d-3b8d-2ddea4b5b461%40gmail.com.
gro...@mail.ulf-schreiber.de
Feb 19, 2017, 2:05:39 PM2/19/17
to mapsfo...@googlegroups.com
Sorry for the late reply, I haven't disapeared, just changed pace.
> - The area patterns appear with wrong color in zooms <13
Red military zones (default theme)? Somehow it seems like,
despite the switch back to the previous composite at the end of
shadeBitmap(), the AwtLuminanceShadingComposite influences the way
the drawing of the
<area fill="#15DF003F" /> lines
is applied (the switch at zoom 13 happens because of the
additional layer of military.png that is painted right on top of
them). The easy way out would be sticking to alpha compositing,
which would be advisable for consistency with android anyways.
Desaturation isn't too bad at low magnitude, and if we generally
add a dummy flat shading where hill data is not available, or
deactivated for the zoom level, themes could compensate by
increasing saturation (without getting too colorful in absence of
hill data/configuration).
> - At large zooms there are artifacts at tile borders
I noticed the visible borders too and think that they are caused
by the upscale blending not using the shading pixels outside the
clip region (the drawBitmap method used on android explicitly
mentions clamp mode in its documentation and the AWT result looks
consistent with that too). Maybe rounding to full shading pixels
(which are huge on high zoom) also plays a role, but i'm not sure
of that. On AWT, both issues should be avoidable by using
matrix-defined scaling. Then only shading tile borders would be
visible (typically full degrees latitude/longitude), which is rare
at high zoom.
On Android, I would probably need a larger-than-tile temporary
bitmap for upscaling at a similar quality level. But I think this
could only be efficient with rendering workers (including
platform-specific state) that are reused between jobs. I think
this could also reduce allocations elsewhere. Maybe it could be as
simple as reusing CanvasRasterer instances? (after taking care for
proper state reset)
--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
Emux
Feb 26, 2017, 12:34:33 PM2/26/17
to mapsfo...@googlegroups.com
Since regular Mapsforge API can
continue working as usual and for broader user visibility..
I merged the hillshading PR on master (with minor improvements).
I merged the hillshading PR on master (with minor improvements).
Note:
On Desktop Samples the 1st argument can be used as dem folder.
--
Emux
Emux
Tobias
Jul 21, 2017, 4:32:03 PM7/21/17
to mapsforge-dev
I experimented with hillshading a bit since it's included in the latest OruxMaps beta. I like the general idea and the options, thanks for that.
I found that the tinting of flat areas or the "bright" side of the slopes is pretty strong, so everything white or light gray gets tinted and will be some middle gray. The theme colors are changed a lot, no matter what magnitude. For better results flat areas and bright side of the slopes should not be affected or only very light.
I found that the tinting of flat areas or the "bright" side of the slopes is pretty strong, so everything white or light gray gets tinted and will be some middle gray. The theme colors are changed a lot, no matter what magnitude. For better results flat areas and bright side of the slopes should not be affected or only very light.
0 new messages