Why complains mapwriter on some polygons?

[fzk]

Today I have build a map (based on the geofabrik extract from 24.09.2014) for "lower saxonia" (Niedersachsen) and map writer (0.3.1) complains on some MPs. OSM user "wambacher" has analysed one MP but was unable to find any reason for the complaint.

I have attached the debug file (see attachment DebugOuterWay_29413042.txt) from mapwriter. The analysis shows, that it is a simple administrative border relation (https://www.openstreetmap.org/relation/897895) with only one way (https://www.openstreetmap.org/way/29413042) for the german island "Juist".

Questions:

- How is it possible to find the cause of problem?

- Is it possible to write the OSM ids into the debug file?

Klaus

[Emux]

Are there any exceptions during the build process?

You can try also the release 0.4, it's compatible.

--
Emux
Cruiser - Atlas

[Ludwig]

Yes, can you elaborate a bit what you mean with 'complains', what actually happens? Is the MP dropped? 

I looked at it and the OSM data there looks fine (no self-intersections which in the past have given errors).

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/54232E0E.9070205%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[fzk]

Here's the output from mapwriter. My understanding is that the way is dropped (but I will crosscheck this in the resulting map data).

...

INFO: start writing file...

Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.util.GeoUtils toJtsGeometry

WARNING: outer way of multi polygon is not a polygon, skipping it: 1200045271

Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.util.GeoUtils toJtsGeometry

WARNING: outer way of multi polygon is not a polygon, skipping it: 1200047707

Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.util.GeoUtils toJtsGeometry

WARNING: outer way of multi polygon is not a polygon, skipping it: 1200047706

Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.cache.WayPreprocessingCallable call

WARNING: cannot create geometry for way with id: 1200045271

Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.cache.WayPreprocessingCallable call

WARNING: cannot create geometry for way with id: 1200047707

Sep 24, 2014 7:04:43 PM org.mapsforge.map.writer.cache.WayPreprocessingCallable call

WARNING: cannot create geometry for way with id: 1200047706

Sep 24, 2014 7:04:44 PM org.mapsforge.map.writer.MapFileWriter writeSubfile

INFO: written 100% of sub file for zoom interval index 0

Sep 24, 2014 7:04:44 PM org.mapsforge.map.writer.MapFileWriter writeSubfile

INFO: written 10.0% of sub file for zoom interval index 1

Sep 24, 2014 7:04:45 PM org.mapsforge.map.writer.MapFileWriter writeSubfile

INFO: written 20.0% of sub file for zoom interval index 1

Sep 24, 2014 7:04:45 PM org.mapsforge.map.writer.util.GeoUtils toJtsGeometry

WARNING: outer way of multi polygon is not a polygon, skipping it: 29413042

Sep 24, 2014 7:04:45 PM org.mapsforge.map.writer.cache.WayPreprocessingCallable call

WARNING: cannot create geometry for way with id: 29413042

Sep 24, 2014 7:04:47 PM org.mapsforge.map.writer.MapFileWriter writeSubfile

INFO: written 30.0% of sub file for zoom interval index 1

...

Klaus 

[fzk]

I have added the output from mapwriter in the post above. What has changed between 0.3 and 0.4? Something essential? My understanding was that only technical changes were made in order to make the plugin compatible with the current osmosis version. Is that not correct?

Klaus

[Emux]

I remember there were some changes in the code too (probably in the master).
But it's always best to use the latest release or master, as the development continues there.

[Ludwig]

Maybe the polygon is clipped in the geofabrik data. Isn't that a Dutch island?

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/5423AE68.3020800%40gmail.com.

[fzk]

It's Juist a german island which is part of lower saxonial (beween Borkum and Norderney). And it's a popular holiday ressort for families with small children because there is no motor vehicle traffic allowed.

Klaus

[fzk]

Yes, I agree. But I use the mapwriter version with integrated coastline support (0.3.1). I think I mentioned in another thread how important mapwriter is for the map builders. Version 0.4 has no proper coastline support and that's is for me a knockout citeria.

My wishlist for mapwriter:
- proper coastline support
- better performance
- less memory consumption
- better relation support (e.g. polygons)

Klaus

PS: Could someone with a working 0.4 mapwriter envirionment runs a build with the current lower saxony extract from geofabrik? Are there also problems with some ways?

[Emux]

I think none official map writer has coastline support.
We're talking about the work announced by Jürgen here that was based on the map writer version of that era.

I have tried it, but due to the lack of type=hd mode for complex/large maps, I'm playing with the solution mentioned by Ludwig here.
I think that Christian in OpenAndroMaps is using that method too?

[fzk]

This (found by OSM user couchmapper) looks very similar to the dropped way:

https://code.google.com/p/mapsforge/issues/detail?id=431

Was a fix for it applied to version 0.4?

Klaus

[Emux]

I remember Ludwig has pushed a map writer change for the polygon identification, it's currently also in master.
https://code.google.com/p/mapsforge/source/detail?r=b3ddd8d7bc9503e2d5bee8fc8491f8bb6202db8c

[fzk]

"couchmapper" mentioned this: https://wiki.openstreetmap.org/wiki/Overpass_turbo/Polygon_Features

Maybe a more sophisticated algorithm.

Klaus

[fzk]

I have run some tests with mapwriter 0.3.1 and 0.4 and got different results. I have created two incorrect MPs.

One with crossing outer lines and one with an overlapping of inner and outer ring:

Mapwriter 0.4 cuts the outer line at the corssing point and mapwriter 0.3.1 drops the outer:

My expection was, that mapwriter would write a (debug) file with the problem MPs. But that's not the case. What am I doing wrong?

Klaus

[Emux]

Hi Klaus,

Can you attach a small osm file from the area and write the bounding box or other map-writer parameters you used for these tests?
That could help us reproduce and examine the issue.

[fzk]

Hi emux,

that's the build call:

sh /home/kto/Freizeitkarte-Entwicklung-Android-1410a/tools/osmosis_043/bin/osmosis --read-pbf /home/kto/Freizeitkarte-Entwicklung-Android-1410a/work/Freizeitkarte_MUENSTER/Freizeitkarte_MUENSTER.transformed.osm.pbf --mapfile-writer file=/home/kto/Freizeitkarte-Entwicklung-Android-1410a/install/Freizeitkarte_MUENSTER/Freizeitkarte_MUENSTER.map type=hd debug-file=false tag-conf-file=/home/kto/Freizeitkarte-Entwicklung-Android-1410a/theme/tag_mapping.xml comment="(c) Map: FZK project (free for private use); Map data: OpenStreetMap contributors; Contour data: U.S. Geological Survey or J. de Ferranti."

Sep 28, 2014 8:22:44 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Osmosis Version 0.43.1

Sep 28, 2014 8:22:45 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Preparing pipeline.

Sep 28, 2014 8:22:46 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>

INFO: mapfile-writer version: mapsforge-map-writer-0.4.0

Sep 28, 2014 8:22:46 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>

INFO: mapfile format specification version: 3

Sep 28, 2014 8:22:46 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Launching pipeline execution.

Sep 28, 2014 8:22:46 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Pipeline executing, waiting for completion.

Sep 28, 2014 8:22:46 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask process

INFO: start reading data...

Sep 28, 2014 8:23:45 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete

INFO: completing read...

Sep 28, 2014 8:26:11 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete

INFO: start writing file...

Sep 28, 2014 8:26:13 PM org.mapsforge.map.writer.MapFileWriter writeSubfile

INFO: written 100% of sub file for zoom interval index 0

Sep 28, 2014 8:26:13 PM org.mapsforge.map.writer.MapFileWriter writeSubfile

INFO: written 10.0% of sub file for zoom interval index 1

Sep 28, 2014 8:26:14 PM org.mapsforge.map.writer.MapFileWriter writeSubfile

INFO: written 20.0% of sub file for zoom interval index 1

...

INFO: Finished writing file.

Sep 28, 2014 8:30:18 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete

INFO: finished...

Sep 28, 2014 8:30:18 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete

INFO: estimated memory consumption: 1,140.84MB

Sep 28, 2014 8:30:18 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Pipeline complete.

Sep 28, 2014 8:30:18 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Total execution time: 454008 milliseconds.

You will find the osm data (70 MB) here:

https://dl.dropboxusercontent.com/u/1677057/Freizeitkarte_MUENSTER.transformed.osm.pbf

It would be very helpful if mapwriter could write out any problem that leads to a loss or a drop of data. And with the knowledge of the corresponding OSM object it's possible to verify or repair the data ...

Klaus

[openandromaps osm]

Hi EMUX,

I'm using the shapefile from Jochen Topf processed with ogr2ogr and shape2osm, merged into the main osm-file.
Works fine and I prefer it over any other method cause its easy to repair the polys in Josm bevore merging them.

Regards, Christian

[Emux]

Hi Christian,

Thanks for the feedback!

[fzk]

Could you provide a working example of your tool chain?

Klaus

[openandromaps osm]

Hmm, posted an answer yesterday, seems to be gone....

Anyway, again:  I use the fwtools (ogr2ogr) for cutting out the land-polygons.
The see is a simple rectangle prozessed by a script with the outbounds of the bbox overlayed by the land-polygones.

Second step is to convert the land-polygones (still shp after ogr2ogr) to .osm with shape2osm.py

Here is the batch for Azores:

SET FWTOOLS_DIR=m:\tools\FWTools
call %FWTOOLS_DIR%\bin\setfwenv.bat
md m:\osm_data\shapes\land\europe\Azores
del m:\osm_data\shapes\land\europe\Azores\land_polygons.*

ogr2ogr.exe -clipsrc -31.5851140, 36.7101106, -24.5201647, 39.9317387 m:\osm_data\shapes\land\europe\Azores m:\osm_data\shapes\land\land_polygons.shp

pushd m:\osm_data\shapes\land\europe\Azores\
m:\tools\fwtools\python\python m:\osm_data\shapes\land\shape2osm.py m:\osm_data\shapes\land\europe\Azores\land_polygons.shp

copy poly_output.1.osm m:\Mapsforge\europe\Azores\land.osm
popd

Best regards,
Christian
www.openandromaps.org

[fzk]

Christian, thanks for the clarification.

I have run some evaluation tests. After installing ogr2ogr and ogr2osm I have cut out an area around the azores. I have cut out the water-polygons which results in one huge mp-relation, with a rectangle as outer and the islands as inner (holes). That's the result in JOSM:

After adding the tag "natural=huge_water" I have tried to build a map (without adding further data). The surprisingly result was, that it took 55 minutes (!) with map writer 0.4 to build the map. It seems that mapwriter has a general program with large mp relations. Conclusion: It doesn't makes sense to use this approach for "real" countries with complex coastline (eg. norway or sweden).

The next try was to use the land-polygons in order to render them over a blue background (sea). The out cuted result is totally different. You didn't get a single mp-relation, you get only a few simple mps which are additional also tiled (see the lines in vertical direction). With this input it takes only a minute to build the map.

I have attached both osm files for debugging. 

Question: Has map writer a performance issue with large oder huge mp-relations?

Klaus

[Emux]

I'd recommend the 2nd method too, use land instead of water polygons, and paint the background blue in the render theme.
See also Christian's thought about that here.

Certainly in case of land inside the sea, the land polygons are easier to process than the whole remaining sea polygons around them.

OpenStreetMap Data provides land polygons split and not split, have you used split?
I prefer the split, it's a lot faster to handle and as we fill them the seams are invisible.

Also I have seen tremendous increase at build speed by using the type=ram instead of type=hd at map writer, providing that your Ram can handle the process (playing with the famous Java Xmx parameter).

[fzk]

type=ram: was used for building both map

split polygons: were used for the water- and the land-polygons

Concerning the map writer performance I typically see during the build process the usage of 1 - 1.5 cpu, sometimes 2 cpu, but not more.

Klaus

[fzk]

With the verbose option (-v in osmosis) I get a lot of messages (errors?, warnings?, infos?):

FINE: constructed outer polygon in relation has no known tags: 3615429

Oct 09, 2014 4:39:57 PM org.mapsforge.map.writer.BaseTileBasedDataProcessor$RelationHandler execute

FINE: relation contains dangling ways which could not be merged to polygons: 32214

Oct 09, 2014 4:40:01 PM org.mapsforge.map.writer.util.JTSUtils repairInvalidPolygon

FINE: unable to repair invalid polygon

Oct 09, 2014 4:40:01 PM org.mapsforge.map.writer.util.GeoUtils mapWayToTiles

FINE: unable to create geometry from way: 253258437

Oct 09, 2014 4:40:02 PM org.mapsforge.map.writer.util.JTSUtils repairInvalidPolygon

FINE: unable to repair invalid polygon

Oct 09, 2014 4:40:02 PM org.mapsforge.map.writer.util.GeoUtils mapWayToTiles

FINE: unable to create geometry from way: 272940243

Oct 09, 2014 4:40:04 PM org.mapsforge.map.writer.util.JTSUtils repairInvalidPolygon

FINE: unable to repair invalid polygon

Oct 09, 2014 4:40:04 PM org.mapsforge.map.writer.util.GeoUtils mapWayToTiles

FINE: unable to create geometry from way: 190450852

Where can I find further information about the messages (eg. something to correlate the internal IDs with the ones from OSM)?

Klaus

[fzk]

Sorry, it seems that the IDs are already from OSM. 

[Christian Kernbeis]

Hi Klaus,

The mapwriter has a _serious_ problem with big relations.
Its imposible to render japan with the forrest/wood, the Northern Territories in Canada (water), it take ages to render NewZealand (forrest/wood) aso..
All MPs with more than 1000 members are a problem.

And yes, the land_polygons above a simple blue rectangle is the best solution, I use this since I started the Openandromaps.
Dont forget to set the Layer to -5 or -4 for these items or you loose all objects tagged below Layer "0"

Best regards, Christian

[Ludwig]

The mapwriter has a _serious_ problem with big relations.
Its imposible to render japan with the forrest/wood, the Northern Territories in Canada (water), it take ages to render NewZealand (forrest/wood) aso..
All MPs with more than 1000 members are a problem.

Christian, can you recheck with the latest map writer in dev? 

I pushed a change that should make dealing with large relations much better.

Ludwig 

[openandromaps osm]

Hi Ludwig,

Tobias informed me about the new snapshot, just installed it on my test-equipment and its running fine.
On Monday the Rendermachine should have finished the current regular update and i will test the snapshot with the problematic countrys.

Best regards,
Christian

[Emux]

That's great, thanks Christian.
We're waiting your feedback too.

[Christian Kernbeis]

I gave Sweden ( with 2 MPs > 1000 Members) a try on my notebook and it rendered in 7 hours.
Usualy it takes >24h on my rendermachine (3 times faster)

The result seems to be perfect OK, not more brocken MPs that the usual ones in the forrest areas.
The two big natural=water MPs look fine.
Rel 1239458;1020members
Rel 1433877;1062members
Map size is pretty much the same as with the V0.3 Mapwriter.

Looking forward to give Japan a try with the full tagset.
Relations:
ID;Members
949087;1466
961523;2096
964756;626
967915;623
969180;3065
1103869;679
1328589;940
1328622;1218
1328635;1781
1330957;1210
1333283;2574
1333455;696
1336890;639
1337911;722
1340805;1366
1341460;740
1956189;633
1968525;660
1978268;863
2942521;844
3596674;1324

Best regards,
Christian

[Emux]

Thanks Christian, keep us updated.
I suppose you're using the default zoom-interval-conf ?

[openandromaps osm]

Hi,

Yes, for the maps in question I use the standard base zoom levels.
Japan and Canada rendered fine.

However there is a strange Problem with white tiles on standard mapsforge V3 viewers.

The white tiles appear at places where I would never expect them:
eg http://www.openstreetmap.org/#map=16/43.9376/-79.9915&layers=C (ontario.map)
or http://www.openstreetmap.org/#map=16/60.3542/29.4117&layers=C (russia north-west-1.map)

and the behavior is strange too:
Ok level 1-7
white at zoom-level 8,9,10 (big white tile)
OK at level 11
white at level 12-14 (small white tile in center? of big white tile)
Ok from level 15 on

The two maps are available at:
ftp://ftp.openandromaps.org/pub/maps/

There are several areas in Japan too with these big/small white tiles.

Best regards, Christian

[fzk]

I have build a map for "fd russia northwest" some days ago and couldn't find any problem in the mentioned area. Neither with Cruiser Beta nor with Locus Pro.

Could you provide a screenshot showing the "white tiles"?

Klaus

 

[openandromaps osm]

You are perfect right, I tested it on VectorialMap  - on Locus its perfect OK.

Best regards, Christian

[Ludwig]

I have not yet had time to look at the maps in question, but with much older viewers (aka 0.3.0) there is a further issue that was fixed in https://github.com/mapsforge/mapsforge/commit/c63af3e9237af51f1bcfea45fa5c40dd0698649f . In that case tiles that had somehow more than a certain number of nodes were dropped, for no good reason. 

If you still have the logs for the creation of these maps, there is some unconditional logging in there that reports on very large geometries (more than 5000 nodes in the unclipped form). If you still have the logs, can you grep over them and extract lines with 'Large geometry' in them? The log line will also report to what size the geometry was cut down to. We should be generally be pretty small now, but maybe something slips through the lines.

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/614f460b-59a4-46d3-aba3-5071e2bbf735%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Ludwig]

Without screenshots it is a bit difficult reproducing the problem, but since newer map viewers are reported to be ok and the symptoms you are reporting are consistent with what you might get without the mentioned fix, I think the problems lie with the 0.3 map viewers and not the map.

and the behavior is strange too:
Ok level 1-7
white at zoom-level 8,9,10 (big white tile)
OK at level 11
white at level 12-14 (small white tile in center? of big white tile)
Ok from level 15 on

Detailed explanation for the behaviour (can be skipped):

For one reason or another there are probably too many pois or way nodes in the area. 

For the first zoom interval (0-7) this is not an issue, because some data is not mapped into that segment. For levels 8-10 the whole tile data is read in (actually multiple subtiles), causing the tile rendering to return early, with the result of a white tile. For level 11 only way/pois that are part of that tile are returned by the reader, so that tile is ok again. For level 12-14 again there is the overflow, from level 15 on again only part of the data is read in, so the rendering is ok again.

tl;dr

So far no issues with new fix to map writer, issues reported are with old libraries, will promote to master later. 

Thanks to all for the testing.

Ludwig

[Ludwig]

As announced earlier, the changes to the writer (and also the new display directive for rendertheme V4) have been pushed to master. 

I will leave that there for another few days for testing before making another 0.5 release candidate. 

Ludwig

[openandromaps osm]

Hello Ludwig,

I tested the maps with several Apps including Fahrradcomputer, Orux beta, Locus and the result is fine.
I will move the snapshot writer to production machine cause the speed benefit is incedible and even the NorthWest Territories in Canada with a 8000 Members MP renders in reasonable time.

Thanks and best regards, Christian

[Ludwig]

Thanks for the feedback.

I have also tested this in production with quite a few maps and the results seem fine. I have also moved that writer into production for our maps on the download server. 

Ludwig

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/7fd5c6c1-48e1-4b3c-a2d8-ef4b75aaa5cc%40googlegroups.com.

[Emux]

I have also moved that writer into production for our maps on the download server.

+1