Why complains mapwriter on some polygons?

301 views
Skip to first unread message

fzk

unread,
Sep 24, 2014, 3:52:04 PM9/24/14
to mapsfo...@googlegroups.com
Today I have build a map (based on the geofabrik extract from 24.09.2014) for "lower saxonia" (Niedersachsen) and map writer (0.3.1) complains on some MPs. OSM user "wambacher" has analysed one MP but was unable to find any reason for the complaint.

I have attached the debug file (see attachment DebugOuterWay_29413042.txt) from mapwriter. The analysis shows, that it is a simple administrative border relation (https://www.openstreetmap.org/relation/897895) with only one way (https://www.openstreetmap.org/way/29413042) for the german island "Juist".



Questions:
- How is it possible to find the cause of problem?
- Is it possible to write the OSM ids into the debug file?

Klaus


DebugOuterWay_29413042.txt

Emux

unread,
Sep 24, 2014, 4:48:04 PM9/24/14
to mapsfo...@googlegroups.com
Are there any exceptions during the build process?

You can try also the release 0.4, it's compatible.

--
Emux
Cruiser - Atlas

Ludwig

unread,
Sep 25, 2014, 12:52:30 AM9/25/14
to mapsfo...@googlegroups.com
Yes, can you elaborate a bit what you mean with 'complains', what actually happens? Is the MP dropped? 
I looked at it and the OSM data there looks fine (no self-intersections which in the past have given errors).

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/54232E0E.9070205%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

fzk

unread,
Sep 25, 2014, 1:46:06 AM9/25/14
to mapsfo...@googlegroups.com
Here's the output from mapwriter. My understanding is that the way is dropped (but I will crosscheck this in the resulting map data).

...
INFO: start writing file...
Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.util.GeoUtils toJtsGeometry
WARNING: outer way of multi polygon is not a polygon, skipping it: 1200045271
Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.util.GeoUtils toJtsGeometry
WARNING: outer way of multi polygon is not a polygon, skipping it: 1200047707
Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.util.GeoUtils toJtsGeometry
WARNING: outer way of multi polygon is not a polygon, skipping it: 1200047706
Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.cache.WayPreprocessingCallable call
WARNING: cannot create geometry for way with id: 1200045271
Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.cache.WayPreprocessingCallable call
WARNING: cannot create geometry for way with id: 1200047707
Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.cache.WayPreprocessingCallable call
WARNING: cannot create geometry for way with id: 1200047706
Sep 24, 2014 7:04:44 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 100% of sub file for zoom interval index 0
Sep 24, 2014 7:04:44 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 10.0% of sub file for zoom interval index 1
Sep 24, 2014 7:04:45 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 20.0% of sub file for zoom interval index 1
Sep 24, 2014 7:04:45 PM org.mapsforge.map.writer.util.GeoUtils toJtsGeometry
WARNING: outer way of multi polygon is not a polygon, skipping it: 29413042
Sep 24, 2014 7:04:45 PM org.mapsforge.map.writer.cache.WayPreprocessingCallable call
WARNING: cannot create geometry for way with id: 29413042
Sep 24, 2014 7:04:47 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 30.0% of sub file for zoom interval index 1
...

Klaus 

fzk

unread,
Sep 25, 2014, 1:50:11 AM9/25/14
to mapsfo...@googlegroups.com
I have added the output from mapwriter in the post above. What has changed between 0.3 and 0.4? Something essential? My understanding was that only technical changes were made in order to make the plugin compatible with the current osmosis version. Is that not correct?

Klaus

Emux

unread,
Sep 25, 2014, 1:55:36 AM9/25/14
to mapsfo...@googlegroups.com
I remember there were some changes in the code too (probably in the master).
But it's always best to use the latest release or master, as the development continues there.

Ludwig

unread,
Sep 25, 2014, 2:03:59 AM9/25/14
to mapsfo...@googlegroups.com

Maybe the polygon is clipped in the geofabrik data. Isn't that a Dutch island?

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

fzk

unread,
Sep 25, 2014, 2:57:16 AM9/25/14
to mapsfo...@googlegroups.com
It's Juist a german island which is part of lower saxonial (beween Borkum and Norderney). And it's a popular holiday ressort for families with small children because there is no motor vehicle traffic allowed.

Klaus

fzk

unread,
Sep 25, 2014, 3:05:48 AM9/25/14
to mapsfo...@googlegroups.com
Yes, I agree. But I use the mapwriter version with integrated coastline support (0.3.1). I think I mentioned in another thread how important mapwriter is for the map builders. Version 0.4 has no proper coastline support and that's is for me a knockout citeria.

My wishlist for mapwriter:
- proper coastline support
- better performance
- less memory consumption
- better relation support (e.g. polygons)

Klaus

PS: Could someone with a working 0.4 mapwriter envirionment runs a build with the current lower saxony extract from geofabrik? Are there also problems with some ways?

Emux

unread,
Sep 25, 2014, 3:51:07 AM9/25/14
to mapsfo...@googlegroups.com
I think none official map writer has coastline support.
We're talking about the work announced by Jürgen here that was based on the map writer version of that era.

I have tried it, but due to the lack of type=hd mode for complex/large maps, I'm playing with the solution mentioned by Ludwig here.
I think that Christian in OpenAndroMaps is using that method too?

fzk

unread,
Sep 25, 2014, 2:51:42 PM9/25/14
to mapsfo...@googlegroups.com
This (found by OSM user couchmapper) looks very similar to the dropped way:


Was a fix for it applied to version 0.4?

Klaus

Emux

unread,
Sep 25, 2014, 3:28:59 PM9/25/14
to mapsfo...@googlegroups.com
I remember Ludwig has pushed a map writer change for the polygon identification, it's currently also in master.
https://code.google.com/p/mapsforge/source/detail?r=b3ddd8d7bc9503e2d5bee8fc8491f8bb6202db8c

Message has been deleted

fzk

unread,
Sep 28, 2014, 11:51:06 AM9/28/14
to mapsfo...@googlegroups.com
"couchmapper" mentioned this: https://wiki.openstreetmap.org/wiki/Overpass_turbo/Polygon_Features

Maybe a more sophisticated algorithm.

Klaus

fzk

unread,
Sep 28, 2014, 11:54:32 AM9/28/14
to mapsfo...@googlegroups.com
I have run some tests with mapwriter 0.3.1 and 0.4 and got different results. I have created two incorrect MPs.
One with crossing outer lines and one with an overlapping of inner and outer ring:

Mapwriter 0.4 cuts the outer line at the corssing point and mapwriter 0.3.1 drops the outer:

My expection was, that mapwriter would write a (debug) file with the problem MPs. But that's not the case. What am I doing wrong?

Klaus
Message has been deleted
Message has been deleted

Emux

unread,
Sep 28, 2014, 12:19:39 PM9/28/14
to mapsfo...@googlegroups.com
Hi Klaus,

Can you attach a small osm file from the area and write the bounding box or other map-writer parameters you used for these tests?
That could help us reproduce and examine the issue.

fzk

unread,
Sep 28, 2014, 3:28:44 PM9/28/14
to mapsfo...@googlegroups.com
Hi emux,

that's the build call:

sh /home/kto/Freizeitkarte-Entwicklung-Android-1410a/tools/osmosis_043/bin/osmosis --read-pbf /home/kto/Freizeitkarte-Entwicklung-Android-1410a/work/Freizeitkarte_MUENSTER/Freizeitkarte_MUENSTER.transformed.osm.pbf --mapfile-writer file=/home/kto/Freizeitkarte-Entwicklung-Android-1410a/install/Freizeitkarte_MUENSTER/Freizeitkarte_MUENSTER.map type=hd debug-file=false tag-conf-file=/home/kto/Freizeitkarte-Entwicklung-Android-1410a/theme/tag_mapping.xml comment="(c) Map: FZK project (free for private use); Map data: OpenStreetMap contributors; Contour data: U.S. Geological Survey or J. de Ferranti."
Sep 28, 2014 8:22:44 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.43.1
Sep 28, 2014 8:22:45 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Sep 28, 2014 8:22:46 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>
INFO: mapfile-writer version: mapsforge-map-writer-0.4.0
Sep 28, 2014 8:22:46 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>
INFO: mapfile format specification version: 3
Sep 28, 2014 8:22:46 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Sep 28, 2014 8:22:46 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Sep 28, 2014 8:22:46 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask process
INFO: start reading data...
Sep 28, 2014 8:23:45 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFO: completing read...
Sep 28, 2014 8:26:11 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFO: start writing file...
Sep 28, 2014 8:26:13 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 100% of sub file for zoom interval index 0
Sep 28, 2014 8:26:13 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 10.0% of sub file for zoom interval index 1
Sep 28, 2014 8:26:14 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 20.0% of sub file for zoom interval index 1
...
INFO: Finished writing file.
Sep 28, 2014 8:30:18 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFO: finished...
Sep 28, 2014 8:30:18 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFO: estimated memory consumption: 1,140.84MB
Sep 28, 2014 8:30:18 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline complete.
Sep 28, 2014 8:30:18 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Total execution time: 454008 milliseconds.

You will find the osm data (70 MB) here:

It would be very helpful if mapwriter could write out any problem that leads to a loss or a drop of data. And with the knowledge of the corresponding OSM object it's possible to verify or repair the data ...

Klaus

openandromaps osm

unread,
Oct 4, 2014, 2:16:02 PM10/4/14
to mapsfo...@googlegroups.com
Hi EMUX,

I'm using the shapefile from Jochen Topf processed with ogr2ogr and shape2osm, merged into the main osm-file.
Works fine and I prefer it over any other method cause its easy to repair the polys in Josm bevore merging them.

Regards, Christian

Emux

unread,
Oct 4, 2014, 2:22:10 PM10/4/14
to mapsfo...@googlegroups.com
Hi Christian,

Thanks for the feedback!

fzk

unread,
Oct 5, 2014, 2:40:33 AM10/5/14
to mapsfo...@googlegroups.com
Could you provide a working example of your tool chain?

Klaus
Message has been deleted

openandromaps osm

unread,
Oct 6, 2014, 9:22:39 AM10/6/14
to mapsfo...@googlegroups.com
Hmm, posted an answer yesterday, seems to be gone....

Anyway, again:  I use the fwtools (ogr2ogr) for cutting out the land-polygons.
The see is a simple rectangle prozessed by a script with the outbounds of the bbox overlayed by the land-polygones.

Second step is to convert the land-polygones (still shp after ogr2ogr) to .osm with shape2osm.py

Here is the batch for Azores:

SET FWTOOLS_DIR=m:\tools\FWTools
call
%FWTOOLS_DIR%\bin\setfwenv.bat
md m
:\osm_data\shapes\land\europe\Azores
del m:\osm_data\shapes\land\europe\Azores\land_polygons.*

ogr2ogr
.exe -clipsrc -31.5851140, 36.7101106, -24.5201647, 39.9317387 m:\osm_data\shapes\land\europe\Azores m:\osm_data\shapes\land\land_polygons.shp

pushd m
:\osm_data\shapes\land\europe\Azores\
m
:\tools\fwtools\python\python m:\osm_data\shapes\land\shape2osm.py m:\osm_data\shapes\land\europe\Azores\land_polygons.shp

copy poly_output
.1.osm m:\Mapsforge\europe\Azores\land.osm
popd


Best regards,
Christian
www.openandromaps.org

fzk

unread,
Oct 8, 2014, 10:54:40 AM10/8/14
to mapsfo...@googlegroups.com
Christian, thanks for the clarification.

I have run some evaluation tests. After installing ogr2ogr and ogr2osm I have cut out an area around the azores. I have cut out the water-polygons which results in one huge mp-relation, with a rectangle as outer and the islands as inner (holes). That's the result in JOSM:


After adding the tag "natural=huge_water" I have tried to build a map (without adding further data). The surprisingly result was, that it took 55 minutes (!) with map writer 0.4 to build the map. It seems that mapwriter has a general program with large mp relations. Conclusion: It doesn't makes sense to use this approach for "real" countries with complex coastline (eg. norway or sweden).

The next try was to use the land-polygons in order to render them over a blue background (sea). The out cuted result is totally different. You didn't get a single mp-relation, you get only a few simple mps which are additional also tiled (see the lines in vertical direction). With this input it takes only a minute to build the map.


I have attached both osm files for debugging. 

Question: Has map writer a performance issue with large oder huge mp-relations?

Klaus

azoren-wasser-7.osm
azoren-land.osm

Emux

unread,
Oct 8, 2014, 12:36:06 PM10/8/14
to mapsfo...@googlegroups.com
I'd recommend the 2nd method too, use land instead of water polygons, and paint the background blue in the render theme.
See also Christian's thought about that here.

Certainly in case of land inside the sea, the land polygons are easier to process than the whole remaining sea polygons around them.

OpenStreetMap Data provides land polygons split and not split, have you used split?
I prefer the split, it's a lot faster to handle and as we fill them the seams are invisible.

Also I have seen tremendous increase at build speed by using the type=ram instead of type=hd at map writer, providing that your Ram can handle the process (playing with the famous Java Xmx parameter).

fzk

unread,
Oct 8, 2014, 2:56:07 PM10/8/14
to mapsfo...@googlegroups.com
type=ram: was used for building both map
split polygons: were used for the water- and the land-polygons

Concerning the map writer performance I typically see during the build process the usage of 1 - 1.5 cpu, sometimes 2 cpu, but not more.

Klaus

fzk

unread,
Oct 9, 2014, 10:47:46 AM10/9/14
to mapsfo...@googlegroups.com
With the verbose option (-v in osmosis) I get a lot of messages (errors?, warnings?, infos?):

FINE: constructed outer polygon in relation has no known tags: 3615429

Oct 09, 2014 4:39:57 PM org.mapsforge.map.writer.BaseTileBasedDataProcessor$RelationHandler execute

FINE: relation contains dangling ways which could not be merged to polygons: 32214

Oct 09, 2014 4:40:01 PM org.mapsforge.map.writer.util.JTSUtils repairInvalidPolygon

FINE: unable to repair invalid polygon

Oct 09, 2014 4:40:01 PM org.mapsforge.map.writer.util.GeoUtils mapWayToTiles

FINE: unable to create geometry from way: 253258437

Oct 09, 2014 4:40:02 PM org.mapsforge.map.writer.util.JTSUtils repairInvalidPolygon

FINE: unable to repair invalid polygon

Oct 09, 2014 4:40:02 PM org.mapsforge.map.writer.util.GeoUtils mapWayToTiles

FINE: unable to create geometry from way: 272940243

Oct 09, 2014 4:40:04 PM org.mapsforge.map.writer.util.JTSUtils repairInvalidPolygon

FINE: unable to repair invalid polygon

Oct 09, 2014 4:40:04 PM org.mapsforge.map.writer.util.GeoUtils mapWayToTiles

FINE: unable to create geometry from way: 190450852


Where can I find further information about the messages (eg. something to correlate the internal IDs with the ones from OSM)?

Klaus

fzk

unread,
Oct 9, 2014, 10:56:34 AM10/9/14
to mapsfo...@googlegroups.com
Sorry, it seems that the IDs are already from OSM. 

Christian Kernbeis

unread,
Oct 9, 2014, 2:03:44 PM10/9/14
to mapsfo...@googlegroups.com
Hi Klaus,

The mapwriter has a _serious_ problem with big relations.
Its imposible to render japan with the forrest/wood, the Northern Territories in Canada (water), it take ages to render NewZealand (forrest/wood) aso..
All MPs with more than 1000 members are a problem.

And yes, the land_polygons above a simple blue rectangle is the best solution, I use this since I started the Openandromaps.
Dont forget to set the Layer to -5 or -4 for these items or you loose all objects tagged below Layer "0"

Best regards, Christian

Ludwig

unread,
Nov 13, 2014, 3:01:21 PM11/13/14
to mapsfo...@googlegroups.com

The mapwriter has a _serious_ problem with big relations.
Its imposible to render japan with the forrest/wood, the Northern Territories in Canada (water), it take ages to render NewZealand (forrest/wood) aso..
All MPs with more than 1000 members are a problem.

Christian, can you recheck with the latest map writer in dev? 
I pushed a change that should make dealing with large relations much better.

Ludwig 

openandromaps osm

unread,
Nov 15, 2014, 3:02:53 PM11/15/14
to mapsfo...@googlegroups.com
Hi Ludwig,

Tobias informed me about the new snapshot, just installed it on my test-equipment and its running fine.
On Monday the Rendermachine should have finished the current regular update and i will test the snapshot with the problematic countrys.

Best regards,
Christian

Emux

unread,
Nov 15, 2014, 3:37:28 PM11/15/14
to mapsfo...@googlegroups.com
That's great, thanks Christian.
We're waiting your feedback too.

Christian Kernbeis

unread,
Nov 16, 2014, 8:59:04 AM11/16/14
to mapsfo...@googlegroups.com
I gave Sweden ( with 2 MPs > 1000 Members) a try on my notebook and it rendered in 7 hours.
Usualy it takes >24h on my rendermachine (3 times faster)

The result seems to be perfect OK, not more brocken MPs that the usual ones in the forrest areas.
The two big natural=water MPs look fine.
Rel 1239458;1020members
Rel 1433877;1062members
Map size is pretty much the same as with the V0.3 Mapwriter.

Looking forward to give Japan a try with the full tagset.
Relations:
ID;Members
949087;1466
961523;2096
964756;626
967915;623
969180;3065
1103869;679
1328589;940
1328622;1218
1328635;1781
1330957;1210
1333283;2574
1333455;696
1336890;639
1337911;722
1340805;1366
1341460;740
1956189;633
1968525;660
1978268;863
2942521;844
3596674;1324

Best regards,
Christian

Emux

unread,
Nov 16, 2014, 10:27:04 AM11/16/14
to mapsfo...@googlegroups.com
Thanks Christian, keep us updated.
I suppose you're using the default zoom-interval-conf ?

openandromaps osm

unread,
Nov 17, 2014, 10:36:18 AM11/17/14
to mapsfo...@googlegroups.com
Hi,

Yes, for the maps in question I use the standard base zoom levels.
Japan and Canada rendered fine.

However there is a strange Problem with white tiles on standard mapsforge V3 viewers.

The white tiles appear at places where I would never expect them:
eg http://www.openstreetmap.org/#map=16/43.9376/-79.9915&layers=C (ontario.map)
or http://www.openstreetmap.org/#map=16/60.3542/29.4117&layers=C (russia north-west-1.map)

and the behavior is strange too:
Ok level 1-7
white at zoom-level 8,9,10 (big white tile)
OK at level 11
white at level 12-14 (small white tile in center? of big white tile)
Ok from level 15 on

The two maps are available at:
ftp://ftp.openandromaps.org/pub/maps/

There are several areas in Japan too with these big/small white tiles.

Best regards, Christian

fzk

unread,
Nov 17, 2014, 11:34:03 AM11/17/14
to mapsfo...@googlegroups.com
I have build a map for "fd russia northwest" some days ago and couldn't find any problem in the mentioned area. Neither with Cruiser Beta nor with Locus Pro.

Could you provide a screenshot showing the "white tiles"?

Klaus




 

openandromaps osm

unread,
Nov 17, 2014, 2:16:25 PM11/17/14
to mapsfo...@googlegroups.com
You are perfect right, I tested it on VectorialMap  - on Locus its perfect OK.

Best regards, Christian

Ludwig

unread,
Nov 17, 2014, 2:19:16 PM11/17/14
to mapsfo...@googlegroups.com
I have not yet had time to look at the maps in question, but with much older viewers (aka 0.3.0) there is a further issue that was fixed in https://github.com/mapsforge/mapsforge/commit/c63af3e9237af51f1bcfea45fa5c40dd0698649f . In that case tiles that had somehow more than a certain number of nodes were dropped, for no good reason. 

If you still have the logs for the creation of these maps, there is some unconditional logging in there that reports on very large geometries (more than 5000 nodes in the unclipped form). If you still have the logs, can you grep over them and extract lines with 'Large geometry' in them? The log line will also report to what size the geometry was cut down to. We should be generally be pretty small now, but maybe something slips through the lines.

Ludwig






--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/614f460b-59a4-46d3-aba3-5071e2bbf735%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Ludwig

unread,
Nov 18, 2014, 5:19:59 AM11/18/14
to mapsfo...@googlegroups.com
Without screenshots it is a bit difficult reproducing the problem, but since newer map viewers are reported to be ok and the symptoms you are reporting are consistent with what you might get without the mentioned fix, I think the problems lie with the 0.3 map viewers and not the map.

and the behavior is strange too:
Ok level 1-7
white at zoom-level 8,9,10 (big white tile)
OK at level 11
white at level 12-14 (small white tile in center? of big white tile)
Ok from level 15 on

Detailed explanation for the behaviour (can be skipped):
For one reason or another there are probably too many pois or way nodes in the area. 
For the first zoom interval (0-7) this is not an issue, because some data is not mapped into that segment. For levels 8-10 the whole tile data is read in (actually multiple subtiles), causing the tile rendering to return early, with the result of a white tile. For level 11 only way/pois that are part of that tile are returned by the reader, so that tile is ok again. For level 12-14 again there is the overflow, from level 15 on again only part of the data is read in, so the rendering is ok again.

tl;dr
So far no issues with new fix to map writer, issues reported are with old libraries, will promote to master later. 

Thanks to all for the testing.

Ludwig

Ludwig

unread,
Nov 18, 2014, 7:14:24 AM11/18/14