Missing water: a work-around solution

[Ludwig]

The currently missing processing for the coastlines is probably the most annoying issue in mapsforge at the moment as it makes all map areas with coastlines look bad.

However, there is a work-around that requires no real changes to the map-writer nor the map-reader as it just utilises the render-theme mechanism.

This is how I do it:

First of all we need a good set of either land or sea areas. OSM data does not natively have this as coastlines, as opposed to all other data for areas, are just lines. The main OSM renderers all employ some sort of intermediate processing to get the sea and land areas right. I am using the dataset that Jochen Topf has provided on his openstreetmapdata.com site. I am using just this set:  http://data.openstreetmapdata.com/land-polygons-split-4326.zip (I think the non-tiled version would also work).

First of all I cut the data down a bit, using ogr2ogr with the bounds for the desired map:

ogr2ogr -clipsrc $LEFT $BOTTOM $RIGHT $TOP output /data/coastline/land-polygons-split-4326/land_polygons.shp

 

This produces a, depending on your bounds, a much smaller coastline shapefile. I usually extend the boundaries of this file a bit over the actual area for the map to make sure I do not get any funny cut-offs.

Then I convert the resulting shapefile with shp2osm.py with slight modifications (as are always required with shp2osm). I set 

fixed_tags = {

  'natural': 'nosea',

}  

which will attach this tag to all polygons. I also changed the starting id for the OSM ids, as otherwise there is chance of collision with real OSM ids, which will create strange artifacts, likes lines running through the map. I also changed the setting for the maximum length of ways, which does not seem to cause a problem.

I now have an OSM XML file with land represented as polygons with the tag "natural" -> "nosea". I agree, it is a stupid name and I first used the tag "natural" -> "land" but this is generally used to denote islands in lakes and such and will not play well with some rendering rules later.

The following steps could be taken in one, but I separated them out (a testing legacy):

I now merge the resulting osm file with the land polygons with my real osm data file:

$OSMOSIS_HOME/bin/osmosis --rb file=data.pbf  --rx file=land.osm --sort --merge --wb file=out.pbf omitmetadata=true

out.pbf can now be fed into the mapfile-writer:

$OSMOSIS_HOME/bin/osmosis --rb file=out.pbf --mapfile-writer file=out.map  map-start-position=$LAT,$LON bbox=$BOTTOM,$LEFT,$TOP,$RIGHT

The bounding box parameters are now those for the desired map, this will cut off any excess land polygons. 

For this to work correctly, only the tag-mapping.xml file needed to be modified by adding:

<osm-tag key="natural" value="nosea" zoom-appear="5" />

This ensures that the new land polygons areas are included in the resulting file.

This finishes the map creation process.

On the reader side, the only change required is to display the land areas correctly by changing the render theme:

<?xml version="1.0" encoding="UTF-8"?>

<rendertheme xmlns="http://mapsforge.org/renderTheme" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

xsi:schemaLocation="http://mapsforge.org/renderTheme ../renderTheme.xsd" version="1" map-background="#b5d6f1">

        <rule e="way" k="natural" v="nosea">

                <area fill="#f8f8f8" stroke="#f8f8f8" stroke-width="0.1" />

        </rule>

...

The first change is to paint the map on a blue (=sea) background. The next rule, at the very top of the file, makes sure all the nosea areas get painted first, here in some sort of grey. 

The one short-coming here is that areas outside the map area all appear blue=sea. I guess that could be fixed by also extracting the real sea areas from the openstreetmapdata data set and painting them as well while having a different background, but this remains as an exercise to the reader.

Attached a few screenshots that show complicated coastline areas are all rendered correctly. I have not found a single area where I thought something was missing.

I am happy to answer questions about this. Sorry, this is not a full script to do this as my scripts tie into a few more infrastructure things that are difficult to factor out.

Ludwig 

[Christian Kernbeis]

Hi Ludwig,

This is a _real_ stunning aproach to the Sea-Problem!
I tried the one with the water_polygons but with questionable success.

Please could you provide me with additional the informations:
-there are various versions of shp2osm.py around, which version do you use?
-what is your choice for the starting point of the ID's?
-do you think that merging the planet file with the land-shapes will work

best regards
Christian

[Ludwig]

Hi there, 

I have attached my modified version of shp2osm.py, I just cannot remember which one I originally used.

Technically, you do not need to merge first, you could just use osmosis in pipeline mode and merge two OSM input files while you are feeding into the mapfilewriter. I sometimes get memory problems, splitting the steps makes restarting scripts easier.

To find out a safe starting id try something like this:

http://www.openstreetmap.org/browse/node/18951459320

I think in the script the starting id is, for whatever reason, 22951459320.

The script supports splitting long ways into multipolygons and into multiple files, I have tried to set the parameters so that this does not happen. So the default values for the main function are then just fine.

I then run like this:

ogr2ogr -clipsrc $ELEFT $EBOTTOM $ERIGHT $ETOP $PROJECT /data/coastline/land-polygons-split-4326/land_polygons.shp

python /data/coastline/shape2osm.py $PROJECT/land_polygons.shp

I have not tried to do this with the whole planet, but since you cannot get the whole planet into one mapfile, I am not sure why you would want to do this.

HTH

Ludwig

[Christian Kernbeis]

Hi Ludwig,


After I managed to install the gdal-phyton bindings on Windows (Oh heaven!!) I tried both approaches to render the sea.
The result for a big map like Great Britain is that the water-polygons.osm contain 389 relations while there are zero with the land_polygons.osm...

The BBox is
ogr2ogr.exe -clipsrc -10.706, 48.999, 3.225, 61.136

The mapsforge writer stucks at 10% Level2 after rendering for ages, the land tiles reder fast and the result is perfect at all zoom-levels.

While I would prefer to render the water tiles (these would cover faulty contourlines and the switch to a new render-theme would be painless),
your workaround is definitly the way to go till someone provides water_polygons without relations.


Thanks again
and
best regards
Christian

[Christian Kernbeis]

Hi Ludwig,

The first maps with your solution are rendered.

There is one modification I made:
The Background in the render.xlm stays white, I generate a dummy sea.osm that contains only one polyline with
$ELEFT $EBOTTOM $ERIGHT $ETOP as nodes and taged as natural=sea, rendered in blue
and the land_polygons are rendered in white.

This ensures compatibility of the theme with all Maps and there is no need to include the land_polygons in
maps where sea/coastline is not involved.

The original coastline handling in the theme is slightly modified, I skipped the area-fill.
Basicly this coastline handling could be skiped completly...

Thanks again and best regards
Christian

www.openandromaps.org

Am Sonntag, 22. Juli 2012 11:38:55 UTC+2 schrieb Ludwig:

[Ludwig]

Hi Christian, 

good that this works. The addition of the background polygon certainly makes sense and is a very simple solution to the background problem. I will add that to my scripts. 

In a cleaned-up version I think this could be the general solution for the coastline problem in mapsforge. While it is not the easiest solution for developers to install (I regularly have trouble with my GDAL installations on various platforms as well), any other solution would have mapsforge re-implementing a coastline polygonizing algorithm as well as some sort of extraction, and I think it would be better to build on what is available already. 

I am happy to help with getting this solution into mapsforge proper, but the first step would be for the mapsforge owners to have a look at the concept and decide if they want to go down this road.

Ludwig

[Christian Kernbeis]

Hi Ludwig,

I have found 2 minor issues with the dummy sea.osm.

) There are (very) thin blue lines runnig across the map like it was with the contour lines in the beginning at low zoom levels.
) The coverage of the sea.osm should be slightly smaller than the coverage of the land_polygones otherwise there are problems
at the map borders (see screenshot)

However, this for fine tuning only, the approach itself is perfect OK.

regards
Christian

[Ludwig]

I have seen these thin lines, do you have any solution for it or know where they actually come from? (I actually assumed for a while they come from something I have done, like defining area outlines, but I do not think this is actually the case).

[Christian Kernbeis]

Hi Ludwig,

I suppose its the same Problem as with the contourlines in the beginning, this was solved by adding "force-polygon-lines=true" to the tag-mapping.
But this does NOT work in this case.

The lines itself seem to represent the border of the tiles, IMO this is an issue of the polygon-handling in mapsforge, Im fighting this problem since I have started mapping with mapsforge (glacier polys, contourlines, aso..)

There is an other issue I exerienced in aeras with high data density like south of GreatBritain where there a white (empty) tiles in the first zoom-interval (0-7) so for these maps the water-rendering should start at level 8.
I have had  this problem with the cyclemap for Netherlands too (even without water-rendering), for me it seems to be a memory problem and occours both with Oruxmaps and Locus.
There is no solution besides changing the zoom-intervalls from 3 to 4 and carefully adjusting the tag-mapping to these intervals, BUT for automated mapmaking this is realy a pain....

I hope that the mapsforge developers will solve these Problems (memory + polygonhandling) in the next release.

best regards
Christian

[Christian Kernbeis]

Hi Ludwig,

Try this:

            <rule e="way" k="natural" v="nosea">

                <area fill="#f8f8f8" stroke="#f8f8f8" stroke-width="1.0" />
            </rule>

I'm not shure, but for me it seems to that it minimize the problem a bit.

regards/grüsse

Christian

Am Samstag, 4. August 2012 11:10:14 UTC+2 schrieb Ludwig:

[Christian Kernbeis]

Hi Ludwig,

On which Layer do you place the land/sea-polygons?

regards
Christian

Am Samstag, 21. Juli 2012 06:14:34 UTC+2 schrieb Ludwig:

[Ludwig]

I just draw the land polygons as the first thing in the theme file, I do not attach a layer mark to the land polygons. 

Do you think this is an issue for something?

[Christian Kernbeis]

Hi Ludwig,

Unfortunately this seems to be a big issue, all Objekts that have a layer attached lower than the default layer
do not render anymore, best to be seen on tunnels, they usually have a layer="-1" tag attached.

The layer issue seems to exist since the begining of Mapsforge:
http://www.openstreetmap.org/?lat=48.12122&lon=14.8735&zoom=16&layers=M

The two tunnels right in the west of the railway station do not render in mapsforge,
they are covered by a landuse=residential.
This occurs even with the mapsforge 3.0 map with the osmarender theme,
where no land/sea-polys exist.

While the landuse is rendered long bevore the tunnels in every theme
(as with the land/sea) the tunnels do not render.

This would mean that all objekts with a layer tag <0 will not render with the land/sea polys,
as they do (not) actually when covered by eg. a landuse tag (or other area-fill).

Can you confirm this?

best regards
Christian

[Ludwig]

Yes, I can confirm this. 

I did not actually look at the spot you pointed me to, but I quickly found another spot where conveniently two sides of a motorway run side by side through a mountain. One way is tagged with layer=-1, the other one has the layer tag missing. They are otherwise identical. Detailed info below.

The one without the layer tag is rendered, the one with the layer tag is missing. 

I have not had time to investigate this further, but is this a rendering problem (meaning the android code needs to change) or a mapwriter problem (meaning the writer needs to change)?

What would happen if the layers would all be moved up by 5 (meaning that in OSM there should be no 'underground' layers, while retaining the stack order?

Ludwig

Info about the two ways:

Way: 129615212

  State: has direction keys

  Data Set: 7e3bfb66

  Edited at: 2011-09-11T07:02:34Z

  Edited by: _kendell (68934)

  Version: 1

  In changeset: 9268053

  Tags: 

    "layer"="-1"

    "oneway"="yes"

    "ref"="G56"

    "highway"="motorway"

    "tunnel"="yes"

  Bounding box: 102.6070089, 24.9803898, 102.6177964, 24.9881034

  2 Nodes: 

    1429732143

    1429732142

  Part of: 

    Relation: 杭瑞高速 (909082)

Way: 39158400

  State: has direction keys

  Data Set: 7e3bfb66

  Edited at: 2011-09-11T07:02:55Z

  Edited by: _kendell (68934)

  Version: 3

  In changeset: 9268053

  Tags: 

    "converted_by"="Track2osm"

    "oneway"="yes"

    "ref"="G56"

    "highway"="motorway"

    "tunnel"="yes"

  Bounding box: 102.6073299, 24.9799499, 102.6189673, 24.9877306

  2 Nodes: 

    253483375

    253483374

  Part of: 

    Relation: 杭瑞高速 (909082)

[Ludwig]

Thinking aloud here:

I assume that the layering actually works, meaning that higher levels obscure lower ones. The problem is just that things like tunnels should be shown. This suggests to me that this could be fixed by adjusting the layer tag in the map file writer, rather than changing the rendering library (which has the advantage that the maps will look ok on third-party mapsforge renderers). 

In OSMUtils.java in the mapfile writer is this method:

public static SpecialTagExtractionResult extractSpecialFields(Entity entity, String preferredLanguage) {

Its function is to do some 'massaging' of tags, including the layer tag. 

So would it be possible in that method to adjust the layer tag, so that areas are pushed down to the lowest layer? Or do we get into an epic hack trying to decide what is an area in OSM?

Ludwig

[Christian Kernbeis]

/*I assume that the layering actually works, meaning that higher levels obscure lower ones. The problem is just that things like tunnels should be shown.*/

I totaly aggree with this, actually I transform all tunnels to layer 0 (by tagtransform)
and will test what side effects ist could cause to lower the land/sea to level -5.
At a first glance i think that only nature reservats or military areas would be incolved in this.

First thing is to have a look which objekts are usualy affected by this whole layer issue.

regards
Christian

[Ludwig]

I tried a little patch to the OSMUtils.java class, attached.

The patch works by attaching a layer tag to natural and landuse areas and shifts them down to the lowest level. Points are omitted as they should show on top of areas even if they are underground. 

Tunnels are now showing and I have not observed any adverse side effects, but I have not had the time to look at too many areas yet.

Where this will fail is where, incorrectly, landuse and natural areas in OSM are stacked on top of each other by means of the layer tag. Eg. someone painting an island on a lake with the layer tag. Generally I would assume that this would be done by adding a +1 layer to the island. So technically the downshifting could happen by subtracting 5 from the layer value set in the OSM data (assuming a default of 0), but I have not tried this.

[Ludwig]

Attached a hopefully slightly improved version of the patch, which preserves layer stacking for the natural/landuse layers that are being pushed down to the bottom.

This works by having a layeradjustment that is subtracted from any existing layer value, so that if there are areas where the areas are drawn on top of each other are still drawn in the same order.

This will not work if the layers are already assigned negative layer values, but I would not expect them to be frequent.

Comments/test cases welcome

Ludwig

[Steven Low]

Hi Ludwig,

Can you explain more on statement below?

"To find out a safe starting id try something like this:

http://www.openstreetmap.org/browse/node/18951459320

I think in the script the starting id is, for whatever reason, 22951459320."

I have tried to merge the land file with my orginal map xml, and I get the following error 

"SEVERE: Thread for task 1-rx failed

org.openstreetmap.osmosis.core.OsmosisRuntimeException: Pipeline entities are no

t sorted, previous entity type=Way, id=160440781, version=1 current entity type=

Way, id=47829549, version=5."

Is this related to the id_counter set in script?

Can you advice how should I able get a suitable starting id?

Regards,

Steven.

[Ludwig]

Are you following my instructions to the dot?

$OSMOSIS_HOME/bin/osmosis --rb file=data.pbf  --rx file=land.osm --sort --merge --wb file=out.pbf omitmetadata=true

I think you need to have the --sort operation enabled here. The error message has nothing to do with starting numbers, in fact the error comes (look at the numbers) from the original OSM data file. If you have duplicate node ids you will just get lines all over your output file.

Ludwig

[voldy]

Hi Guys,

I am in creation own maps with blue sea areas. I’m following our solution with little change. For this reason I’d like to ask you: Ludwig or Christian print blue background or blue polygon and this polygon is covered with “white” land areas. 

OpenStreetMap Data [1] offers also water polygons so I used your workaround only for water polygons. Mapsforge renders only sea areas and the land is rendered in standard way. It seems that this solution works well but is this solution correct? Do I miss something? Why do you render land areas over the blue background if is it possible to render only sea areas?

Thanks

Petr

[1] http://openstreetmapdata.com/data/water-polygons

[Ludwig]

Do not follow the route suggested in this thread, it was a workaround solution, good until a solution more integrated with mapsforge was available. Fortunately, this is the case now. 

Have a look at the following post https://groups.google.com/d/topic/mapsforge-dev/t-_zxJ0if-s/discussion

While, AFAIK, this is not yet integrated into the main branch, it was very up-to-date with the rest of the changes until very recently. I have give it a try and did not find any issues with it. I would actually advocate to integrate it into the main branch asap, so that more people will start testing it and any issues that arise will be fixed before the next official release.

Following this route will save you the effort of changing over later. Please report any issues that you find to the mailing list.

Ludwig

[Christian Kernbeis]

Hi,

The reason why I render land_polygons above a blue background is because the usual tools used for creating osm-water_polygons out of the shapefile are creating relations (multipolygons) and this is still a problem in Mapsforge, especially in Levels 0-11 with unpredictable results.

best regards
Christian

[Daniele Germano]

Hello

I am try to add your solution to my map of  Barcelona. Unfortunately all the background become blue, not just the sea. i don't know what i have done wrong. i am quite sure that i have follow the all the instructions as you describe.

This is what i have done:

download the land-polygons-split-4326

cut the area of Barcelona from land_polygons.shp

convert the above result to osm

add a new nosea line to the tag mapping.xml

merge the previous osm file with the osm file of Barcelona

convert the new Barcelona.osm.pbf to Barcelona.map

Change the background color of the osmarender.xml and add the new rule as you describe.

But i get all background in blue.

What i am doing wrong?

[Ludwig]

Do you do this?

Then I convert the resulting shapefile with shp2osm.py with slight modifications (as are always required with shp2osm). I set 
fixed_tags = {
  'natural': 'nosea',
}  

On 25 July 2014 16:28, Daniele Germano <danieleg...@gmail.com> wrote:

Hello

I am try to add your solution to my map of  Barcelona. Unfortunately all the background become blue, not just the sea. i don't know what i have done wrong. i am quite sure that i have follow the all the instructions as Ludwig describe in his post.

This is what i have done:

download the land-polygons-split-4326

cut the area of Barcelona from land_polygons.shp

convert the above result to osm

add a new nosea line to the tag mapping.xml

merge the previous osm file with the osm file of Barcelona

convert the new Barcelona.osm.pbf to Barcelona.map

Change the background color of the osmarender.xml and add the new rule as Ludwig describes.

But i get all background in blue.

What i am doing wrong?

Il giorno sabato 21 luglio 2012 06:14:34 UTC+2, Ludwig ha scritto:

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/1c673898-4851-4484-8cd7-ff7742a4de48%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniele Germano]

yes, i have use the shp2osm.py you provide which incluse this modification

[Emux]

You can open the osm file with an xml editor or JOSM to check if it has indeed the tag
<tag k="natural" v="nosea"/>

--
Emux
Cruiser - Atlas

[Daniele Germano]

yes, i have checked. i have that tag

[Daniele Germano]

I found the error...a very stupid error..

I have  add the nosea tag in the tag mapping, in the pois section, instead of the ways section!

i was think, the layer is grab from the land_polygons.shp. But if i  take the layer from the water_polygons.shp, leave the background grey, and set color blue to the nosea rule (renamed as sea), i will get the same result?

[Emux]

i was think, the layer is grab from the land_polygons.shp. But if i  take the layer from the water_polygons.shp, leave the background grey, and set color blue to the nosea rule (renamed as sea), i will get the same result?

Yes you can follow the opposite process.
Actually above someone else asked the same question.
See Christian's answer which explains the issue.

[Vladimir Abramov]

Nice way to add coastline.

But i've found some small issue. After i add land polygons and adjusted theme, i'm lost some small islands in Montengro boka bay.  
I change background colour and found that they are filled same colour as sea area should also overlapping land polygon. Here a part of theme responsible for that (closed="yes" part)  

<!-- natural -->

<rule e="way" k="natural" v="*">

<rule e="way" k="natural" v="coastline">

<rule e="way" k="*" v="*" closed="no">

<line stroke="#708599" stroke-width="2.0" />

</rule>

<rule e="way" k="*" v="*" closed="yes">

<area fill="#b5d6f1" stroke="#b5d6f1" stroke-width="0.4" />

</rule>

</rule>

Is it a bug in theme or bug in map? Can i freely modify this part of theme so it can be displayed correctly? 

суббота, 21 июля 2012 г., 6:14:34 UTC+2 пользователь Ludwig написал:

The currently missing processing for the coastlines is probably the most annoying issue in mapsforge at the moment as it makes all map areas with coastlines look bad.

[Emux]

The map creation has changed to include this procedure and the render themes have extra lines for that.
You can check Osmarender and RenderThemeV4 highlighted lines.

I took screenshots with latest map and lib / themes and the island's land is visible.

[Vladimir Abramov]

Thank you Emux. I have confused and thought someone used closed coastlines for drawing lakes. But i've read osm wiki and found its strongly not recommended. 


четверг, 12 февраля 2015 г., 22:40:09 UTC+1 пользователь Emux написал: