Question regarding map creation

[Sheng Huang]

I am a Canadian user that uses MapsForge to render offline OSM maps in our Android application. The application works very well.

However, we need to generate custom country, region and city maps to be rendered in our application. The map writer program of MapsForge seems to work very slow, it takes hours to generate a single country map out of a continent map if it will ever complete.

Also since maps are updated weekly and some countries have huge amount of data like USA and Germany thus we want to reduce the map size to MBs instead of GBs by eliminating some details, we are thinking about finding a service provider to generate the maps for us at a reasonable cost. Does MapsForge offer this service or do you recommend a company/organization that offers MapsForge format map creation?

Please advise. Thanks in advance for your help.

Regards,
Sheng 

[Ludwig]

To reduce the map size, you can supply a custom tag-mapping.xml file that governs what data is encoded in map files, see https://github.com/mapsforge/mapsforge/blob/master/docs/Getting-Started-Map-Writer.md

But be reminded that our map file format has some overhead corresponding to the area covered, so even for empty areas there is some size overhead.

We provide some areas as map downloads (albeit with the well-known water problem) at download.mapsforge.org. 

To facilitate your own server, you can install the following from github:  https://github.com/jjbross/mapsforge-mapcreator.

I am not aware of any commercial offerings for map creation.

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/64c5abd9-651c-497c-898e-e1b702fc7532%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emux]

Yes the key point is a custom tag-mapping.xml that can guide the map writer what specific vector data to insert into the map.

Also I advice to try with type=ram instead of hd, it speedups significantly.

About the water issue you can check the solution here.

--
Emux
Cruiser - Atlas

[Sheng Huang]

Thanks a lot Emux and Ludwig, I have checked download.mapsforge.org before but the country/region list doesn't meet all our needs. This is the reason I posted this question. Also thanks for the link 

https://github.com/jjbross/mapsforge-mapcreator, we can possibly layout our map generation server following the same strategy.

I tried the type=ram option, however, even with JAVACMD_OPTIONS=-Xmx1920m, the following command:

/osmosis/bin/osmosis --rb file=/home/mapservice/africa.map :q--mapfile-writer file=/home/mapservice/countries/CO.map bbox=-4.225869,-81.728111,13.380502,-66.869835 type=ram throws the exception:

SEVERE: Thread for task 1-rb failed

java.lang.OutOfMemoryError: Java heap space

        at gnu.trove.map.hash.TLongObjectHashMap.rehash(TLongObjectHashMap.java:158)

So shall the memory size be at least 10 times as big as the source map file? For a continent map like europe-latest.osm.pbf of 14GB, to get the map of any country in Europe, is the memory required to be 140GB? This is a lot and I don't think our server can be configured to have such as big memory space.

Please advise.

Thanks,
Sheng

[Emux]

Usually I use the following Java options for my custom maps (depending on Java version and with type=ram).
-XX:+AggressiveOpts -XX:+UseCompressedOops -Djava.util.Arrays.useLegacyMergeSort=true -XX:PermSize=256m -XX:MaxPermSize=256m -Xmx8192m -Xms8192m -server

You can check also some other map sources:
- OpenAndroMaps (great coverage for outdoor use too)
- AndroidMaps
- Freizeitkarte

[Sheng Huang]

Thanks a lot Emux. I tried the type=ram option with the following JAVA options:

export JAVACMD_OPTIONS="-XX:+AggressiveOpts -XX:+UseCompressedOops -Djava.util.Arrays.useLegacyMergeSort=true -XX:PermSize=512m -XX:MaxPermSize=768m -Xmx6144m -Xms4096m -server"

 

So for my VM with 8GB RAM, I allocated 6GB (Xmx6144m) to the osmosis task. Then running the task to generate the map of France from the 13.3GB Europe map that I downloaded from http://download.geofabrik.de/europe.html:

bin/osmosis --rb file=/mapservice/europe-latest.osm.pbf --mapfile-writer file=/mapservice/countries/france.map bbox=41.333333,-5.133333,51.083333,9.533333 type=ram

The command still gave the "out of memory" error:

INFO: start reading data...

Nov 25, 2014 8:47:40 AM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion

SEVERE: Thread for task 1-rb failed

java.lang.OutOfMemoryError: Java heap space

        at gnu.trove.map.hash.TLongObjectHashMap.rehash(TLongObjectHashMap.java:157)

        at gnu.trove.impl.hash.THash.postInsertHook(THash.java:388)

        at gnu.trove.map.hash.TLongObjectHashMap.doPut(TLongObjectHashMap.java:248)

        at gnu.trove.map.hash.TLongObjectHashMap.put(TLongObjectHashMap.java:222)

        at org.mapsforge.map.writer.RAMTileBasedDataProcessor.addNode(RAMTileBasedDataProcessor.java:80)

        at org.mapsforge.map.writer.osmosis.MapFileWriterTask.process(MapFileWriterTask.java:157)

        at crosby.binary.osmosis.OsmosisBinaryParser.parseDense(OsmosisBinaryParser.java:138)

        at org.openstreetmap.osmosis.osmbinary.BinaryParser.parse(BinaryParser.java:124)

        at org.openstreetmap.osmosis.osmbinary.BinaryParser.handleBlock(BinaryParser.java:68)

        at org.openstreetmap.osmosis.osmbinary.file.FileBlock.process(FileBlock.java:135)

        at org.openstreetmap.osmosis.osmbinary.file.BlockInputStream.process(BlockInputStream.java:34)

        at crosby.binary.osmosis.OsmosisReader.run(OsmosisReader.java:45)

        at java.lang.Thread.run(Thread.java:745)

Nov 25, 2014 8:47:40 AM org.openstreetmap.osmosis.core.Osmosis main

SEVERE: Execution aborted.

org.openstreetmap.osmosis.core.OsmosisRuntimeException: One or more tasks failed.

        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.waitForCompletion(Pipeline.java:146)

        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:92)

        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)

        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

        at java.lang.reflect.Method.invoke(Method.java:606)

        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Launcher.java:329)

        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:239)

        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:409)

        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:352)

        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

So how can generate the country maps I want? Are you also generating country/region maps from the planet and/or continent maps? Your suggestion would be greatly appreciated.

Regards,
Sheng

[Emux]

Well eventually all depend on the machine / memory you can use.

France is a big country so either you need to allocate more memory or use the type=hd.
(There is a reason some countries are split into regions at Geofabrik or map servers)

Also usually there is no need to use as input a whole continent osm.
You can download from Geofabrik just the country / region you need to convert.

(And if I don't mentioned, make sure to use 64bit Java to see the whole memory)

[Sheng Huang]

Thanks for the update Emux. I need to re-generate some country maps out of continent maps because the region/destination boundaries of our travel app are similar to country boundaries but sometimes are different. And I cannot find a single source that provides maps for all countries.

I noticed for the type=hd option, temp space needs to be around 10 to 11 times the size of the source map file. What about the type=ram option? Does it bind to the same constraint? If that is the case, do I need about 140GB RAM to generate maps out of a 12.5 GB source map? 

Thanks,

Sheng

[Emux]

Well we write at Wiki the 10 times rule for memory and relative small maps, as a general guide.

But it's better to experiment and if you want to use a huge continent as input,
extract first the needed bounding box into another osm.pbf and then use it with the writer.

In any case I recommend to start with memory as it has great speed
(allocating as much as you can to your 64bit Java).
If that is not enough then you can resort to type=hd using for best results a solid state disk.

[Sheng Huang]

Thanks again Emux.

Finally our system administrator granted 180GB RAM to my SLES 11 Linux server, and I found Java can use a maximum of 130GB by trying commands like "java -Xmx125g -version", which returns right away by either displaying the version of my 64-bit Server VM or showing the error message "Could not reserve enough space for object heap".

Even with 130MB RAM, the reading phase can complete successfully but the map still cannot be created. The error I got is:

INFO: Pipeline executing, waiting for completion.

Nov 26, 2014 1:45:35 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask process

INFO: start reading data...

Nov 26, 2014 3:09:36 PM org.openstreetmap.osmosis.core.pipeline.common.ActiveTaskManager waitForCompletion

SEVERE: Thread for task 1-rb failed

java.lang.IllegalStateException: No free or removed slots available. Key set full?!!

        at gnu.trove.impl.hash.TLongHash.insertKeyRehash(TLongHash.java:326)

        at gnu.trove.impl.hash.TLongHash.insertKey(TLongHash.java:275)

        at gnu.trove.map.hash.TLongObjectHashMap.rehash(TLongObjectHashMap.java:164)

        at gnu.trove.impl.hash.THash.postInsertHook(THash.java:388)

        at gnu.trove.map.hash.TLongObjectHashMap.doPut(TLongObjectHashMap.java:248)

        at gnu.trove.map.hash.TLongObjectHashMap.put(TLongObjectHashMap.java:222)

        at org.mapsforge.map.writer.RAMTileBasedDataProcessor.addNode(RAMTileBasedDataProcessor.java:80)

 

The JVM options I have is "export JAVACMD_OPTIONS="-XX:+AggressiveOpts -Djava.util.Arrays.useLegacyMergeSort=true -XX:PermSize=512m -XX:MaxPermSize=768m -Xmx130g -Xms50g -server"". I dropped the "-XX:+UseCompressedOops" option as JVM showed "Java HotSpot(TM) 64-Bit Server VM warning: Max heap size too large for Compressed Oops".

Using type=hd seems not to help as well because it takes over 8 hours to generate the map with my dual core Intel Xeon CPU @ 2.66GHz. I tried several times and it never finishes during work hours. Given that we have over 180 maps to generate, the speed is not acceptable. So only the in-memory solution could potentially work for us.

Now I am wondering how those country maps online were generated by those organizations/companies. Are they using a super-computer? 

Regards,
Sheng

[Emux]

Hmm the exception indicates a data overflow in a Trove library's map/set.

If I remember correctly Germany map takes a whole day on Mapsforge's server (Ludwig can help more here).

Also Christian has mentioned that he works with a SSD on its render machine (regular hard disks can be very slow for type=hd).
You can post a question to its OpenAndroMaps forum too.

I don't think that there is need for a super computer.
But you may have to leave it to work and find the result the next day for large maps.

Also some techniques may help.
Like I have said above, if you absolutely have to use a continent as input for producing a country map, split the process to more steps in order to handle / debug it better.
You can start by cutting with Osmosis the continent required bounding box and creating a new smaller osm.pbf.
Then feed that smaller osm.pbf to the map writer, to avoid many Osmosis pipes that can potentially fill the memory.

Check also the --read-pbf-fast task and play with the 'workers' option.

The tag-mapping xml is important too, a structure that permits many heavy vector data to occupy small zoom levels (0-7 or 8-11) where the simplifications take place can lead to long build times (and large map files).

[Ludwig]

I am not sure what your exact chain is, but I would recommend first to extract the area and write it out to a file before feeding it into osmosis with the mapsforge writer again. 

The second suggestion that I have is to use the osmium tool http://wiki.openstreetmap.org/wiki/Osmium to first extract the data you are interested in to reduce the map file size, then feed into osmosis with the mapfile writer. Osmium is magnitudes faster than osmosis, but you will have to write some code yourself. 

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/547644F9.9060901%40gmail.com.

[Ludwig]

In https://github.com/mapsforge/mapsforge/issues/544 there was also the suggestion to increase certain constants in the writer, the exceptions that were reported are not the same as yours so this might not help, but maybe there is a way to configure trove to allocate more key space.

[Sheng Huang]

Hi Emux,

Could you provide more details on "The tag-mapping xml permits many heavy vector data to occupy small zoom levels (0-7 or 8-11) where the simplifications take place can lead to long build times (and large map files)"? 

For example, the following tags configured in the default tag-mapping.xml:

<!-- HIGHWAY TAGS -->
	<pois>
		<osm-tag key="highway" value="bus_stop" zoom-appear="17" />
		<osm-tag key="highway" value="mini_roundabout" zoom-appear="17" />
		<osm-tag key="highway" value="traffic_signals" zoom-appear="17" />
		<osm-tag key="highway" value="turning_circle" zoom-appear="15" />
	</pois>

I only see a couple of tags under zoom level 10, like:

		<osm-tag key="natural" value="coastline" zoom-appear="0" />
		<osm-tag key="natural" value="land" zoom-appear="8" />
		<osm-tag key="natural" value="water" zoom-appear="8" />
		<osm-tag key="natural" value="wood" zoom-appear="8" />
		<osm-tag key="admin_level" value="1" force-polygon-line="true" zoom-appear="4">
			<zoom-override key="boundary" value="administrative" />
		</osm-tag>
How can I change them to make the map generation faster? 

Thanks a lot,
Sheng

[Emux]

I suggest to take a look to map file wiki. I copy from there:
"The map file consists of several sub-files, each storing the map objects for a different zoom interval. Zoom intervals are non-overlapping groups of consecutive zoom levels. Each zoom interval is represented by a single member of the group, the so-called base zoom level. "

Map writer simplifies the vector data until base zoom level 12. From there no simplification is computed.
Take a also look at map writer wiki, at 'simplification-factor' and 'zoom-interval-conf'.

So you must carefully select from which zoom level each layer is to appear.
e.g. if you try to put in tag-mapping the whole road network of a country from small zoom levels,
the simplification process will try to parse all of it and include it in the map file in all sub-files
(much process time and large map file).

These are advices if you're building your own tag-mapping file.
If you're using the default one with OSM data then it's already taken care of.