Request for comments on new coastline-aware mapfile-writer

[Jürgen Broß]

As most of you know, appropriate rendering of coastlines and oceans has
been a problem in mapsforge for a while. We even removed coastline
handling completely with the last release. Some workaround solutions
have been posted on this list since then.

We have been working on this issue to provide a “native” solution that
allows for easy integration of coastlines, thus making any fiddlier
workaround obsolete. We would like to announce that a beta version of
the new coastline-aware mapfile-writer is now publicly available. Before
integrating the code into one of the next official releases, we would
like to invite you to test and play around with the new mapfile-writer.

You find all necessary documentation at [1], including a description on
how to compile and setup the new writer. The writer itself is based on
the current 0.3.1-SNAPSHOT and can be checked out from the Git
repository at [2] (please read the documentation on how to checkout and
compile before).

Have fun playing around with the new coastline-aware writer. We are much
interested in your feedback. For general questions or comments (e.g.
“how to compile and setup”) please use this mailing list, for more
technical comments please use the issue tracker at [3].

Best regards,

Philipp and Jürgen


[1]
http://mapsforgecoastline.wikia.com/wiki/Mapsforge_(Coastline-Enabled)_Wiki
[2] git://git.assembla.com/mapsf_coastline.git
[3] http://code.google.com/p/mapsforge/issues/list


--


Jürgen Broß
Institute of Computer Science
Databases and Information Systems
Freie Universität Berlin
Takustr. 9
D-14195 Berlin, Germany
phone: +49 30 838-75108
email: juerge...@fu-berlin.de

[Ludwig]

I am just giving this a quick try, so a few incomplete comments here.

The code checked out from git ok and compiled without problems. The mapfile-writer jar seems to have tripled in size to now 15MB. Expecting that this would now run against osmosis 0.41 I had upgraded osmosis as well, but that was a mistake as the fix in the old writer has not yet been applied to this new branch. 

I then changed my scripts around to use the new library. That also worked fine. The only change I had to make was to remove the -XX:+UseCompressedOops as that is an instruction for 64 bit environments only and the machine I use for map compilation is a 32-bit system. Map compiled fine. I had been concerned about memory consumption as that machine has limited RAM, but I was able to compile the map in less than 3GB of memory. 

I had hoped to be able to use the map in a 0.30-based mapviewer, but apparently now the rendertheme has changed to version 2. As the mapformat itself has not changed, I think I should be able to use a version 1 rendertheme, but I have not yet looked into the differences. Maybe someone could advise if that is impossible because of some missing features in version 1, so I can save myself the effort.

However, with a build of the AdvancedMapViewer from the same project I was able to correctly display the new map and it looked good. Some screenshots attached.

The area here is Cambodia and parts of the Mekong Delta, plenty of islands and two different stretches of coastline. (The boxes shown in the text labels are Khmer-script place names, apparently the device does not have the glyphs). 

Well done, this looks like it has solved a long-open issue. 

Ludwig

[Ludwig]

Forgot to add that I was using the shapefile mode. 

I have not tried the other mode and I have my doubts whether it is worth investing the effort of maintaining it. JoTo's data set is probably as good as it gets, many have spent ages trying to get this right. I think it would be more beneficial to focus efforts on new things rather than solving old problems again in a different package. One working solution is sufficient. 

Ludwig

[Phil]

The latest version does now support (and require) Osmosis 0.41.

As for the CompressedOops JVM-Flag, I kind of assumed the use of a 64-bit system, regarding the memory requirements.

Of course you are right that the native (non-shapefile) mode will require some effort to maintain, but it also has some advantages:

1. It runs out-of-the-box, with no additional external data required (and no depedencies whatsoever).

2. The performance of both modes are almost the same, if not in favor of the native implementation.

3. The shapefiles are not always up-to-date. This however, is mostly due to severe errors in the coastline data and sometimes may
algorithm may be able to cope with them, sometimes it may not.

Phil

[voldy]

Hi,

I’ve tried to test new mapfile-writer in shapefile mode=1 (thanks Ludwig for information about new mapwriter).

It seems that for “type=ram” everything works correctly. Unfortunately I got some errors (see error stack in attachment) for “type=hd”. Is there any solution how solve this issue? The converted pbf file had about 150MB. 

Thank you
Petr

[vklein]

Hi,
I´m very interested in the coastline aware writer, as I want to display seamarks from Openseamap.
I have a little testmap from Warnemünde but processing with the normal mapsforge writer the 030 library does not display the water.
As I could see on the pictures from Ludwig the water is nicely displayed in blue.

Problem: my build of the coastline aware writer failed.

I followed the advice in [1]  but the build of the writer failed., the others are ok

this is what I got:

[INFO] ------------------------------------------------------------------------

[INFO] Reactor Summary:

[INFO]

[INFO] mapsforge ......................................... SUCCESS [2:11.965s]

[INFO] mapsforge-core .................................... SUCCESS [7.622s]

[INFO] mapsforge-map-reader .............................. SUCCESS [2.395s]

[INFO] mapsforge-render-theme ............................ SUCCESS [3.790s]

[INFO] mapsforge-map ..................................... SUCCESS [6.833s]

[INFO] mapsforge-map-writer .............................. FAILURE [5:31.858s]

[INFO] ------------------------------------------------------------------------

[INFO] BUILD FAILURE

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 8:05.072s

[INFO] Finished at: Wed Nov 07 10:44:42 CET 2012

[INFO] Final Memory: 39M/118M

[INFO] ------------------------------------------------------------------------

[ERROR] Failed to execute goal on project mapsforge-map-writer: Could not resolv

e dependencies for project org.mapsforge:mapsforge-map-writer:jar:0.3.1-SNAPSHOT

: Failed to collect dependencies for [org.mapsforge:mapsforge-core:jar:0.3.1-SNA

PSHOT (compile), net.sf.trove4j:trove4j:jar:3.0.3 (compile), com.vividsolutions:

jts:jar:1.12 (compile), com.google.guava:guava:jar:11.0.2 (compile), osmosis:osm

osis-core:jar:0.41 (system), org.geotools:gt-shapefile:jar:8.0 (compile), org.ge

otools:gt-swing:jar:8.0 (compile), junit:junit:jar:4.10 (test)]: Failed to read

artifact descriptor for com.google.guava:guava:jar:11.0.2: Could not transfer ar

tifact com.google.guava:guava:pom:11.0.2 from/to central (http://repo.maven.apac

he.org/maven2): Connection to http://repo.maven.apache.org refused: Connection t

imed out: connect -> [Help 1]

[ERROR]

[ERROR] To see the full stack trace of the errors, re-run Maven with the -e swit

ch.

[ERROR] Re-run Maven using the -X switch to enable full debug logging.

[ERROR]

[ERROR] For more information about the errors and possible solutions, please rea

d the following articles:

[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/DependencyReso

lutionException

[ERROR]

[ERROR] After correcting the problems, you can resume the build with the command

[ERROR] mvn <goals> -rf :mapsforge-map-writer

Help appreciated

Victor

[Jürgen Broß]

To build the mapsforge library, maven needs to download required libs
from the central maven repository. As the error message from maven reads
"Connection to http://repo.maven.apache.org refused", please make sure
that your internet connection is working properly.

Use your favorite browser and check if you can access
http://repo.maven.apache.org/
Maybe you are behind a firewall that blocks access?

>> email: juerge...@fu-berlin.de <javascript:>

[vklein]

Sucess!!
I downloaded via a 3G-Connection, some files we not found.
Today I made a new "mvn clean install"  and everything was ok.

I installed osmosis0.41 and started . The plugin was automatically found ( I wonder there was no plugin directory inthe osmosis-0.41-directory)
The warnemuende.map was created.
It shows the blue water of the baltic sea.

The wiki explains the necessary things perfect.
Thank you very much for this piece of software and the help.
I was using the online-version with Mapnik, so from now I can use offline maps.

Victor


Victor


[INFO] Executed tasks

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]

[INFO] mapsforge ......................................... SUCCESS [3.838s]
[INFO] mapsforge-core .................................... SUCCESS [7.815s]
[INFO] mapsforge-map-reader .............................. SUCCESS [2.730s]
[INFO] mapsforge-render-theme ............................ SUCCESS [7.005s]
[INFO] mapsforge-map ..................................... SUCCESS [8.580s]
[INFO] mapsforge-map-writer .............................. SUCCESS [5:10.191s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 5:40.876s
[INFO] Finished at: Wed Nov 07 19:00:03 CET 2012
[INFO] Final Memory: 51M/155M
[INFO] ------------------------------------------------------------------------

[vklein]

Schwerwiegend: Execution aborted. Inconsistency In Coastline-Data!

I have a osm-file that covers the bounding box from Lübeck to Gedser bbox=53.51,10.33,54.35,12.30
 I start the mapwriter with
D:\josmData\osmosis-0.41>bin\osmosis --rx file=D:\josmData\osmosis_input\luebeck_gedser.osm
--mw file=D:\josmData\osmosis401_output\luebeck_gedser41_01.map type=ram coastline-mode=0
tag-conf-file=D:\josmData\osmosis_input\my_opensea_tag-mapping030_401_134.xml bbox=53.51,10.33,54.35,12.30

but the execution is aborted.

there seems to be a problem with TileCoordinate [x=2169, y=1309, zoomlevel=12 .

I tried to use a shape-file and followed your link to  http://openstreetmapdata.com/data/water-polygons  but the shapefile lines.shp provides there did not work.

How can I get a shapefile for the coast for the bbox I use?

with another osm File FehmarnRostock everything is ok and I get a nice map with some seamarks :)).
I got the osm files from   http://www.overpass-api.de

Nov 21, 2012 11:10:26 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Osmosis Version 0.41
Nov 21, 2012 11:10:27 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Preparing pipeline.
Nov 21, 2012 11:10:27 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <ini
t>
Information: mapfile-writer version: mapsforge-map-writer-0.3.1-SNAPSHOT
Nov 21, 2012 11:10:27 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <ini
t>
Information: mapfile format specification version: 3
Nov 21, 2012 11:10:27 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Launching pipeline execution.
Nov 21, 2012 11:10:27 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Pipeline executing, waiting for completion.
Nov 21, 2012 11:10:31 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask prep
rocessCoastlines
Information: Starting Coastline Processing...
Nov 21, 2012 11:10:32 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTa
skManager waitForCompletion
Schwerwiegend: Thread for task 1-rx failed
java.lang.IllegalStateException: CPP: Inconsistency In Coastline-Data!
No Coastlines In Read Data But External Coastline-Data Suggests Otherwise!
Inner Tile At TileCoordinate [x=2169, y=1309, zoomlevel=12] Claims Mixed Tile
        at org.mapsforge.map.writer.coastline.WaterAreaPreprocessor.checkWaterAr
ea(WaterAreaPreprocessor.java:135)
        at org.mapsforge.map.writer.osmosis.MapFileWriterTask.preprocessCoastlin
es(MapFileWriterTask.java:202)
        at org.mapsforge.map.writer.osmosis.MapFileWriterTask.complete(MapFileWr
iterTask.java:139)
        at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:113)
        at java.lang.Thread.run(Unknown Source)

Nov 21, 2012 11:10:32 PM org.openstreetmap.osmosis.core.Osmosis main
Schwerwiegend: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed
.
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForComple
tion(Pipeline.java:146)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Laun
cher.java:329)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
a:239)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
uncher.java:409)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
352)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

[Christian Kernbeis]

Hi Phil,

Is there a chance that the "type=HD" option will work in future,
I would like to switch to the new SVN-Version, couse the use of water_shapes has some advantages over the work-around using land_shapes over blue background.

Unfortunately its not possible to build maps out of a 1,5GB pbf-file in RAM (tibet, thailand, alps with contourlines), so would need the hd-option.

Thanks for all your efforts
Christian

[Phil]

Am Mittwoch, 21. November 2012 23:28:59 UTC+1 schrieb vklein:

Schwerwiegend: Execution aborted. Inconsistency In Coastline-Data!

I have a osm-file that covers the bounding box from Lübeck to Gedser bbox=53.51,10.33,54.35,12.30
 I start the mapwriter with
D:\josmData\osmosis-0.41>bin\osmosis --rx file=D:\josmData\osmosis_input\luebeck_gedser.osm
--mw file=D:\josmData\osmosis401_output\luebeck_gedser41_01.map type=ram coastline-mode=0
tag-conf-file=D:\josmData\osmosis_input\my_opensea_tag-mapping030_401_134.xml bbox=53.51,10.33,54.35,12.30

but the execution is aborted.
 

there seems to be a problem with TileCoordinate [x=2169, y=1309, zoomlevel=12 .

I tried to use a shape-file and followed your link to  http://openstreetmapdata.com/data/water-polygons  but the shapefile lines.shp provides there did not work.

How can I get a shapefile for the coast for the bbox I use?

If the files are all named "lines.*" instead of "water_polygons.*" than you got the wrong file. OpenStreetMapData provides waterpolygons, landpolygons and coastlines. You need the first. Please try this file instead: http://data.openstreetmapdata.com/water-polygons-split-4326.zip

The error message "Inconsistency In Coastline-Data! No Coastlines In Read Data But External Coastline-Data Suggests Otherwise!" implies pretty much what it says. Your input file "luebeck_gedser.osm" does not seem to contain any coastlines, therefore an external file has to be used to determine if it's all water or land. But the external file claims that coastlines should be present in that area. Since your bounding-box should definitely contain coastlines, your osm-input-file might be incomplete or broken. That said, utilizing the shapefile mode should bypass this problem.

[Phil]

Am Donnerstag, 22. November 2012 10:08:46 UTC+1 schrieb Christian Kernbeis:

Hi Phil,

Is there a chance that the "type=HD" option will work in future,
I would like to switch to the new SVN-Version, couse the use of water_shapes has some advantages over the work-around using land_shapes over blue background.

Unfortunately its not possible to build maps out of a 1,5GB pbf-file in RAM (tibet, thailand, alps with contourlines), so would need the hd-option.

Thanks for all your efforts
Christian

Unfortunately coastline processing requires a new data structure that needs to persist in memory with no HD-mode support. Though the coastlines in itself, through the new structure, aren't memory intensive in themselves, the rest of the map-generation gets bloated in the process. Aside from that, we are still unhappy about the performance with coastline-processing in general. Therefore i am currently working on using PostGIS for that. If that succeeds HD-mode could be reenabled. It is also an option to use PostGIS for the whole programm but that would require a serious reimplementation of the map-writer.

So yes, there is definitely a chance, although i cannot say when specifically.

[vklein]

Hi Phil,
a shapefile for my self-created map is not available to me.
I found geofabrik could do this at heavy costs. Is there a software that I could do it for myself?
I need only a small testfile for the mapping process to see if my seamarks are correctly put in the map file and rendered by my rendertheme.

I have tried to use the shape option with the denmark files from geofabrik:  denmark.osm.pbf and denmark.shp/waterways.shp
but the writer aborts with a multipoligon error.

I start with:
bin\osmosis --rb file=D:\input\denmark.osm.pbf --mw file=D:\output\denmark01.map type=ram
coastline-mode=1 shapefile=D:\input\shapefiles\denmark.shp\waterways.shp
tag-conf-file=D:\input\my_opensea_tag-mapping030_401_134.xml

thats the output of the writer:
...
Information: start reading data...
Nov 22, 2012 10:53:22 AM org.mapsforge.map.writer.osmosis.MapFileWriterTask prep

rocessCoastlines
Information: Starting Coastline Processing...

Nov 22, 2012 10:53:24 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTa
skManager waitForCompletion
Schwerwiegend: Thread for task 1-rb failed
java.lang.ClassCastException: Type Conversion Error! Shapefile Feature Is Not Mu
ltiPolygon! Was: MultiLineString
        at org.mapsforge.map.writer.coastline.ShapefilePreprocessor.readFromShap
efile(ShapefilePreprocessor.java:158)

Do you have any hints for me?

Thanks

Victor

[vklein]

Success!!!
A made a new try with the shape file you told me http://data.openstreetmapdata.com/water-polygons-split-4326.zip

Maybe there was an unrecognized error in the parameters I supplied in the previous tries.

Thanks

Victor

 this is the command I used successfully:
I wonder about the negative value in current max way id: -9223372036854775793.

D:\josmData\osmosis-0.41>bin\osmosis --rx file=D:\josmData\osmosis_input\luebeck_gedser.osm --mw file=D:\josmData\osmosis401_output\luebeck_gedser03.map type=ram coastline-mode=1 shapefile=D:\josmData\water-polygons-split-4326\water-polygons-split-4326\water_polygons.shp tag-conf-file=D:\josmData\osmosis_input\my_opensea_tag-mapping030_401_134.xml bbox=53.51,10.33,54.35,12.30

Nov 24, 2012 1:05:17 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Osmosis Version 0.41
Nov 24, 2012 1:05:18 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Preparing pipeline.
Nov 24, 2012 1:05:18 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init
>
Information: mapfile-writer version: mapsforge-map-writer-0.3.1-SNAPSHOT
Nov 24, 2012 1:05:18 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init

>
Information: mapfile format specification version: 3

Nov 24, 2012 1:05:18 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Launching pipeline execution.
Nov 24, 2012 1:05:18 PM org.openstreetmap.osmosis.core.Osmosis run

Information: Pipeline executing, waiting for completion.

Nov 24, 2012 1:05:21 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask prepr

ocessCoastlines
Information: Starting Coastline Processing...

Nov 24, 2012 1:05:27 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask prepr
ocessCoastlines
Information: Finished Coastline Processing!
Nov 24, 2012 1:05:27 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask compl
ete
Information: current max way id: -9223372036854775793
Nov 24, 2012 1:05:27 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask compl
ete
Information: completing read...
Nov 24, 2012 1:05:28 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask compl
ete
Information: current max way id: -9223372036854775793
Nov 24, 2012 1:05:28 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask compl
ete
Information: start writing file...
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 100% of sub file for zoom interval index 0
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 10.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 20.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 30.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 40.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 50.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 60.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 70.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 80.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 90.0% of sub file for zoom interval index 1
Nov 24, 2012 1:05:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 100% of sub file for zoom interval index 1
Nov 24, 2012 1:05:30 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 10.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:30 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 20.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:30 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 30.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 40.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 50.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 60.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 70.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 80.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 90.0% of sub file for zoom interval index 2
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
Information: written 100% of sub file for zoom interval index 2
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeFile
Information: JTS Geometry cache hit rate: 1.0
JTS Geometry total load time: 0
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.MapFileWriter writeFile
Information: Finished writing file.
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask compl
ete
Information: finished...
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask compl
ete
Information: estimated memory consumption: 632,98MB
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask compl
ete
Information: maximum monitored memory consumption: 0MB
Nov 24, 2012 1:05:31 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask compl
ete
Information: estimated runtime: 12854ms = 12.854s
Nov 24, 2012 1:05:31 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Pipeline complete.
Nov 24, 2012 1:05:31 PM org.openstreetmap.osmosis.core.Osmosis run
Information: Total execution time: 13401 milliseconds.

D:\josmData\osmosis-0.41>

Am Donnerstag, 22. November 2012 13:48:48 UTC+1 schrieb Phil:

[Sergei]

Thanks a lot for a great job!

I'm trying to display very minimalistic map with white background with the coastline shown as a solid line.

Unfortunately along with the coastline itself I also see vertical and horizontal lines in the middle of the water apparently parts of the water polygons that coincide with map grid at some scale level.

I would appreciate your advice how to rid of these lines if it's possible at all

Here is my render rule for the coast line:

<rule e="way" k="natural" v="coastline">

  <rule e="way" k="*" v="*" closed="no">

    <line stroke="#000000" stroke-width="5.0" />

  </rule> 

  <rule e="way" k="*" v="*" closed="yes">

   <area fill="#ffffff" stroke="#000000" stroke-width="5.0" />

 </rule>

</rule>

Attachment contains the screenshot of the map. (I also enabled the grid lines display) 

Thanks a lot,

  Sergei

[fzk]

I'm using the "coastline snapshot" map writer for a long time.
But now I ran into this problem concering the swiss (with the current geofabrik extract):

...

java.lang.IllegalStateException: CPP: Inconsistency In Coastline-Data!
No Coastlines In Read Data But External Coastline-Data Suggests Otherwise!

First Inner Tile At TileCoordinate [x=2116, y=1428, zoomlevel=12] Claims Land Tile
Another Inner Tile At TileCoordinate [x=2120, y=1450, zoomlevel=12] Claims Sea (Water) Tile
      at org.mapsforge.map.writer.coastline.WaterAreaPreprocessor.checkWaterArea(WaterAreaPreprocessor.java:142)
      at org.mapsforge.map.writer.osmosis.MapFileWriterTask.preprocessCoastlines(MapFileWriterTask.java:202)
      at org.mapsforge.map.writer.osmosis.MapFileWriterTask.complete(MapFileWriterTask.java:139)
...

It seems that map writer tries to handle something with sea tiles, but the swiss has no coast and no coastlines.
Any ideas ?

Cheers Klaus

PS: The coastline switch is set to "coastline-mode=0".

[Carlos Alberto Martínez Gadea]

I got the same problem with several regions. And I've checked that there is not SEA (but probably water). I was extracting a region from Sweden, Dalarnas Lan. It comes from a map file from Geofabrik, dated last week. Any help would be appreciated.

may 09, 2013 1:20:31 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.41
may 09, 2013 1:20:32 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
may 09, 2013 1:20:32 AM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>
INFO: mapfile-writer version: mapsforge-map-writer-0.3.1-SNAPSHOT
may 09, 2013 1:20:32 AM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>
INFO: mapfile format specification version: 3
may 09, 2013 1:20:32 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
may 09, 2013 1:20:32 AM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
may 09, 2013 1:20:32 AM org.mapsforge.map.writer.osmosis.MapFileWriterTask process
INFO: start reading data...
may 09, 2013 1:20:45 AM org.mapsforge.map.writer.osmosis.MapFileWriterTask preprocessCoastlines
INFO: Starting Coastline Processing...
may 09, 2013 1:20:46 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion
SEVERE: Thread for task 1-rx failed

java.lang.IllegalStateException: CPP: Inconsistency In Coastline-Data!
No Coastlines In Read Data But External Coastline-Data Suggests Otherwise!

First Inner Tile At TileCoordinate [x=2187, y=1139, zoomlevel=12] Claims Land Tile
Another Inner Tile At TileCoordinate [x=2215, y=1169, zoomlevel=12] Claims Sea (Water) Tile

    at org.mapsforge.map.writer.coastline.WaterAreaPreprocessor.checkWaterArea(WaterAreaPreprocessor.java:142)
    at org.mapsforge.map.writer.osmosis.MapFileWriterTask.preprocessCoastlines(MapFileWriterTask.java:202)
    at org.mapsforge.map.writer.osmosis.MapFileWriterTask.complete(MapFileWriterTask.java:139)

    at org.openstreetmap.osmosis.xml.v0_6.XmlReader.run(XmlReader.java:113)
    at java.lang.Thread.run(Thread.java:722)

[Phil]

Some clarification:

If the OSM-Data doesn't contain coastlines, the Coastline-Algorithm

can/ has to assume that there is either no ocean water in the

extract or that everything is completly filled with ocean

water. In any case, the algorithm has to make use of external data

to figure out which is the right case for the data you are trying

to convert. And that is what the oceantiles_12.dat is used for.

Generally it is a converted png file of the world which indicates

for a given area (pixel) that it is land, sea or mixed.

Problem is, even though the definition of the oceantiles implies

that sea pixels should only be ocean areas (since it's about

coastlines), many seas are wrongfully marked as sea pixels (=ocean)

which would than mean that coastlines have be in the osm-extract,

hence the data inconsistency.

We used a rather old oceantiles png file as with newer files and

enhanced precision these errors seam to increase a lot.

tl;dr

If anyone else runs into this problem 

=> CPP: Inconsistency In Coastline-Data <=

try this tool:

http://wiki.openstreetmap.org/wiki/Png2tileinfo

http://svn.openstreetmap.org/applications/rendering/png2tileinfo/

to fix (mostly remove blue pixels) and convert the png file below,

and use it/replace it with the one provided in mapsforge.

http://mapsforgecoastline.wikia.com/wiki/File:Oceantiles_12.png

P.S.: This only applies to "coastline-mode=0"

[fzk]

I'm trying to generate maps for norway, sweden, finland and russia (european part) from the related geofabrik extracts. In all cases I get this exception:

kto@orion1974:~/Freizeitkarte-Entwicklung-Android-1402$ sh /home/kto/Freizeitkarte-Entwicklung-Android-1402/tools/osmosis/bin/osmosis --read-pbf /home/kto/Freizeitkarte-Entwicklung-Android-1402/work/Freizeitkarte_FIN/Kartendaten_Freizeitkarte_FIN.osm.pbf --mapfile-writer file=/home/kto/Freizeitkarte-Entwicklung-Android-1402/install/Freizeitkarte_FIN/Freizeitkarte_FIN.map type=ram tag-conf-file=/home/kto/Freizeitkarte-Entwicklung-Android-1402/theme/tag_mapping.xml comment="(c) Map: FZK project (free for private use); Map data: OpenStreetMap contributors; Contour data: U.S. Geological Survey or J. de Ferranti." coastline-mode=1 shapefile=/home/kto/Freizeitkarte-Entwicklung-Android-1402/water-polygons/water_polygons.shp

java -cp /home/kto/Freizeitkarte-Entwicklung-Android-1402/tools/osmosis/lib/default/plexus-classworlds-2.4.jar -Dapp.home=/home/kto/Freizeitkarte-Entwicklung-Android-1402/tools/osmosis -Dclassworlds.conf=/home/kto/Freizeitkarte-Entwicklung-Android-1402/tools/osmosis/config/plexus.conf org.codehaus.classworlds.Launcher

Dec 25, 2013 12:12:03 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Osmosis Version 0.41

Dec 25, 2013 12:12:04 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Preparing pipeline.

Dec 25, 2013 12:12:05 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>

INFO: mapfile-writer version: mapsforge-map-writer-0.3.1-SNAPSHOT

Dec 25, 2013 12:12:05 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>

INFO: mapfile format specification version: 3

Dec 25, 2013 12:12:05 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Launching pipeline execution.

Dec 25, 2013 12:12:05 PM org.openstreetmap.osmosis.core.Osmosis run

INFO: Pipeline executing, waiting for completion.

Dec 25, 2013 12:12:05 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask process

INFO: start reading data...

Dec 25, 2013 12:13:07 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask preprocessCoastlines

INFO: Starting Coastline Processing...

Dec 25, 2013 12:26:08 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion

SEVERE: Thread for task 1-read-pbf failed

java.lang.IllegalArgumentException: Comparison method violates its general contract!

at java.util.ComparableTimSort.mergeLo(ComparableTimSort.java:714)

at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:451)

at java.util.ComparableTimSort.mergeCollapse(ComparableTimSort.java:376)

at java.util.ComparableTimSort.sort(ComparableTimSort.java:182)

at java.util.ComparableTimSort.sort(ComparableTimSort.java:146)

at java.util.Arrays.sort(Arrays.java:472)

at java.util.Collections.sort(Collections.java:155)

at com.vividsolutions.jts.operation.buffer.SubgraphDepthLocater.getDepth(SubgraphDepthLocater.java:66)

at com.vividsolutions.jts.operation.buffer.BufferBuilder.buildSubgraphs(BufferBuilder.java:280)

at com.vividsolutions.jts.operation.buffer.BufferBuilder.buffer(BufferBuilder.java:156)

at com.vividsolutions.jts.operation.buffer.BufferOp.bufferFixedPrecision(BufferOp.java:341)

at com.vividsolutions.jts.operation.buffer.BufferOp.bufferReducedPrecision(BufferOp.java:329)

at com.vividsolutions.jts.operation.buffer.BufferOp.bufferReducedPrecision(BufferOp.java:293)

at com.vividsolutions.jts.operation.buffer.BufferOp.computeGeometry(BufferOp.java:285)

at com.vividsolutions.jts.operation.buffer.BufferOp.getResultGeometry(BufferOp.java:272)

at com.vividsolutions.jts.operation.buffer.BufferOp.bufferOp(BufferOp.java:148)

at com.vividsolutions.jts.geom.Geometry.buffer(Geometry.java:1157)

at org.mapsforge.map.writer.coastline.ShapefilePreprocessor.preprocessCoastlines(ShapefilePreprocessor.java:116)

at org.mapsforge.map.writer.osmosis.MapFileWriterTask.preprocessCoastlines(MapFileWriterTask.java:263)

at org.mapsforge.map.writer.osmosis.MapFileWriterTask.complete(MapFileWriterTask.java:139)

at crosby.binary.osmosis.OsmosisBinaryParser.complete(OsmosisBinaryParser.java:35)

at crosby.binary.file.BlockInputStream.process(BlockInputStream.java:37)

at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)

Question: Does someone has an idea what causes the exception ?

Other countries with complex coastlines (eg. italy, greece, great britain) are working fine. Only the nordic countries are not working. A good test seems to be the geofabrik extract from finland becauses it crashes after a few minutes. I'm using the coastline from Jochen Topf (http://openstreetmapdata.com/data/water-polygons) from 25.12.2013 and have tried it also with coastline from october (same result).

Cheers Klaus

[vklein]

Hi Klaus,
did you ever try to build a map with a rectangle bb for (e.g) sweden?
I create the .pbf from a europe-latest.pbf with a rectangle box and the coastline aware map-filewriter runs successfull.
Victor

[fzk]

Hi Victor,

> did you ever try to build a map with a rectangle bb for (e.g) sweden?

no, I haven't tried this ... but it sounds promising.

Questions:
- Could you post the coordinates of your bounding box?
- Is the north part of sweden included?

End of last year I mailed Phil concerning this issue. I didn't got an answer ... it seems that he also retired from the project. I regret this because the wide interest in the rescue project shows the general demand for the mapsforge lib. And I hope that the "coastline-aware mapfile-writer" finds new maintainers ... because it is a very import piece of software in the toolchain for creating maps.

Klaus

[vklein]

Hi Klaus,

I made a map from Simrishamn to Nynaeshamn south of Stockholm. I used an enhanced  tag-mapping to add support for the navigation-lines
The map is available from osm.franken.de/offline-maps.
There is a additional xx_seamarks.dat file, which contains the describing data for the semarks in this region. Its an xml extract of an .osm file.
The bbox is in the second line of the .dat file.

Yor can view the map with the AISOSEAMPLOTTER or the Openseamapviewer from the play store.

There are some other maps from NL and DE on the osm.franken.de/offline-maps.

Comments, error or crash reports are welcome.

Regards

Victor

[fzk]

I have tried your suggestion and cut out my own map polygon (poly-file) for south norway. After 14 hours of (hard) processing map writer produces a resulting map.

Mapwriter is a very important piece in my tool chain and I hope that it finds (better to say it urgent needs) a new maintainer. There is room for a lot of improvements mainly bug fixing and the reduction of processing time and ram consumption.

Cheers Klaus

PS: What's about the former maintainers Phil and Jürgen?

[vklein]

Is it possible to get the the map for south norway and the map polygon?

I would like to create a seamarks file and test if it could be used in the AISOSEAMPlotter app


Regards Victor

[fzk]

The image shows the course of the poly file:

You can find the poly file here: https://dl.dropboxusercontent.com/u/1677057/fzk_nor%2Bsouth.poly

What do you mean with "the map"? The resulting map file (365 MB)?

Klaus

BTW: The rendering of south sweden wasn't successful ... crash (same exception as above) after half an hour.

[vklein]

I think I can create an extract from the europa-latest.pbf by filtering with osmosis and the polyfile  to get all nodes and ways with seamark tags.

Can you provide a link to the map-file?

Victor

[fzk]

> Can you provide a link to the map-file?

You can download the map (365 MB) here:

http://easyclasspage.de/maps/files/Freizeitkarte_NOR+SOUTH.map

The link only valid within the next few weaks.

The map requires a special theme which can be found here:

http://download.freizeitkarte-osm.de/android/1402/freizeitkarte.zip

Cheers Klaus