Please enlighten me about the mapsforge-mapcreator

[M. Dietrich]

Hi,

there was a discussion about creating mapsfiles in another thread with a different topic. let's continue the discussion here.

last status was that i got an errno2 which i interpreted as a missing target directory. instead the tool ogr2ogr was missing (which you have to install on debian separately, the package is named gdal-bin, maybe it would be good to list that dependency in the README?). after that was fixed i get an java error:

java.lang.RuntimeException: XPathFactory#newInstance() failed to create an XPathFactory for the default object model: http://java.sun.com/jaxp/xpath/dom with the XPathFactoryConfigurationException: javax.xml.xpath.XPathFactoryConfigurationException: No XPathFctory implementation found for the object model: http://java.sun.com/jaxp/xpath/dom

        at javax.xml.xpath.XPathFactory.newInstance(XPathFactory.java:72)

        at org.mapsforge.map.writer.OSMTagMapping.<init>(OSMTagMapping.java:191)

xalan is missing in /etc/osmosis/plexus.conf in that case (-> README?). now i have the exception:

java.lang.LinkageError: loader (instance of  org/codehaus/plexus/classworlds/realm/ClassRealm): attempted  duplicate class definition for name: "org/apache/xerces/jaxp/datatype/DatatypeFactoryImpl"

...any idea, what's wrong?

[M. Dietrich]

i installed osmosis from tar and that version does not have the xml* problem.

furthermore it now uses more than just 100%, it jumped up to 700% (using 7 CPUs) which is nice.

anyway it again terminates with an error in the end...  OutOfMem, probably because i didn't set the type=hd. 

now it takes quite long with setting hd but the load is very low which probably means it's stuck in io...

[Ludwig]

The mapsforge map writer osmosis plugin is extremely slow, for testing purposes it is best to start with a small area and a small original pbf file, otherwise you will spend days waiting for each iteration. 

I wish I could offer more assistance with the osmosis installation, but basically we have one on the server that works and that I do not want to touch unless there is a reason.

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/62996534-2c95-4788-aabd-acee0ffc7af7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Emux]

type=hd is certainly slower than type=ram, specially if you don't use a SSD for it.

BTW have you set the JVM (I hope 64bit) environment for Osmosis to use more memory via the known parameters (e.g. Xmx) ?

At my tests and depending on Java version I have tried:

JAVACMD_OPTIONS="-XX:+AggressiveOpts -XX:+UseCompressedOops -Djava.util.Arrays.useLegacyMergeSort=true -XX:PermSize=256m -XX:MaxPermSize=256m -Xms8g -Xmx8g -server"

--
Emux

[M. Dietrich]

Am Samstag, 19. September 2015 10:36:01 UTC+2 schrieb Emux:

type=hd is certainly slower than type=ram, specially if you don't use a SSD for it.

which data is mainly held in ram, the target?

 

BTW have set your JVM (I hope 64bit) environment to use more memory via the known parameters (e.g. Xmx) ?

i added -d64 to force 64bit now

 

At my tests and depending on Java version I have tried:

JAVACMD_OPTIONS="-XX:+AggressiveOpts -XX:+UseCompressedOops -Djava.util.Arrays.useLegacyMergeSort=true -XX:PermSize=256m -XX:MaxPermSize=256m -Xms8g -Xmx8g -server"

i added this too (in  ~/.osmosis, right?) and switched back to type=ram, lets see. do you use ram or hd with your setting (or does that depend on the map?)?

[Emux]

On 19/09/2015 11:44 πμ, M. Dietrich wrote:

i added this too (in  ~/.osmosis, right?) and switched back to type=ram, lets see. do you use ram or hd with your setting (or does that depend on the map?)?

Modify the JVM parameters based on your actual RAM.

The 'type' depends on the map, the default is 'ram'.
If you search the example config, you'll see where we need 'hd' (I see Germany and Japan).

(And of course all these play for the specific machine configuration)

--
Emux

[Ludwig]

The more ram the better, I only switched those to HD that fail because of oom

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/55FD21DD.4030906%40gmail.com.

[M. Dietrich]

i was now able to create a nrw map file which is rendered fine by mapsforge. now i changed to render rectangles but i wonder about the meanin of two files that are used:

 - data/europe-germany-nordrhein-westfalen-sea.osm

 - data/europe-germany-nordrhein-westfalen.osm

what do these files express? how are they used? are they needed for / what happens if i omit those for now?

[Emux]

Two intermediate OSM procedures take place (in landextraction.py) for a map creation:

- Sea rectangle creation (to be put at the bottom)

- Land download (OpenStreetMap Data) / extraction / conversion (to be put above sea rectangle)

(that is because the coastline issues in OSM data)

Above them we put the official OSM planet parts.

--
Emux

[M. Dietrich]

yes, thats what i read from the code, but why is that needed and what is achived? or more important: if i do not have polynomes by states or countries, what should i provide then? what exactly do these files fix? what is a "Sea rectangle"?

as you remember i try to create just rectangles of map files for the world (ok, nrw for now)...

i see the following log entries:

Sep 23, 2015 10:09:22 AM org.mapsforge.map.writer.util.GeoUtils clipToTile

WARNING: invalid geometry prior to tile clipping, trying to repair 306081789

Sep 23, 2015 10:09:22 AM org.mapsforge.map.writer.util.GeoUtils clipToTile

WARNING: invalid geometry even after attempt to fix 306081789

is this somehow alarming or just a warning that can be ignored?

and: i gave the osmosis 24GB of ram (just noticed the box has 32) which is speeding up the process alot, a 700MB map file is created in less than an hour!

i wonder why the above area is white while the left area is gray around my rectangle:

any idea why that happens?

[Emux]

Like mentioned above, we prefer using the OpenStreetMap Data land polygons, as they have many corrections (you can check the site for details and search the forum for previous discussions about that).

A sea rectangle is a simple osm xml to have blue sea below our land.
(We don't paint anymore map background blue in render themes)

The extraction from land polygons is done with the map bbox (also the sea).

The poly files are used to extract the data from planet osm, here you might need to use rectangle poly for your maps.

The warnings are from complex geometries during parsing (many can be ignored).

About the image, I suspect something wrong on map bbox and/or poly used to get data from planet osm.

--
Emux

[Ludwig]

RE that image: I think on the left side an area outside the map coverage is shown, while on the top part of a tile is shown that does not have any data. 

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/5602865E.2040705%40gmail.com.

[M. Dietrich]

Am Mittwoch, 23. September 2015 13:00:49 UTC+2 schrieb Emux:

Like mentioned above, we prefer using the OpenStreetMap Data land polygons, as they have many corrections (you can check the site for details and search the forum for previous discussions about that).

A sea rectangle is a simple osm xml to have blue sea below our land.
(We don't paint anymore map background blue in render themes)

The extraction from land polygons is done with the map bbox (also the sea).

so mapsforge first painsts all blue (sea), than a white shape (the country) on that and the osm planet vectors on top?

The poly files are used to extract the data from planet osm, here you might need to use rectangle poly for your maps.

The warnings are from complex geometries during parsing (many can be ignored).

About the image, I suspect something wrong on map bbox and/or poly used to get data from planet osm.

i think i found it. there was a second map installed (sachsen anhalt) which has a bb which is "higher north". it seems mapsforge just build a joined bb around both and paints that white, everything around that gray.

[Emux]

On 23/09/2015 02:20 μμ, M. Dietrich wrote:

so mapsforge first painsts all blue (sea), than a white shape (the country) on that and the osm planet vectors on top?

That's right.

--
Emux

[M. Dietrich]

Am Mittwoch, 23. September 2015 13:20:49 UTC+2 schrieb M. Dietrich:

i think i found it. there was a second map installed (sachsen anhalt) which has a bb which is "higher north". it seems mapsforge just build a joined bb around both and paints that white, everything around that gray.

yes, i deleted the map file for sa and just have my bb-created one and the background is now gray. so the other map file influences rendering here... strange 

[Emux]

On 23/09/2015 02:33 μμ, M. Dietrich wrote:

yes, i deleted the map file for sa and just have my bb-created one and the background is now gray. so the other map file influences rendering here... strange

It can depend if your app is made to use more than one map files (filling the void with white, depending the theme).

--
Emux

[Ludwig]

caching issue?

or as Emux said if you are using the MultiMapDataStore data is combined from two or more mapfiles.

L

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/5602900A.1090204%40gmail.com.

[M. Dietrich]

Am Mittwoch, 23. September 2015 15:19:27 UTC+2 schrieb Ludwig:

caching issue?

no,

 

or as Emux said if you are using the MultiMapDataStore data is combined from two or more mapfiles.

yes, it does (is there a app out there that  does not? how so?) and it was another map that extended the bb. it seems mapsforge does only support one single bb and considers everything inside different from something outside the bb.

[Emux]

On 23/09/2015 04:35 μμ, M. Dietrich wrote:

(is there a app out there that  does not? how so?)

All those which didn't integrate the new feature. :)

it seems mapsforge does only support one single bb and considers everything inside different from something outside the bb.

The map view uses the combined bounding box of all opened maps.

--
Emux

[M. Dietrich]

Am Mittwoch, 23. September 2015 15:41:25 UTC+2 schrieb Emux:

All those which didn't integrate the new feature. :)

it's an amazing feature! :D

i created my first "map vector tile" (a reactangle map) and choose to use degrees, so i create a reactangle from for example 51 to 52°. in the middle of germany this give files of 36MB per tile.

question for oeffie would be if that is acceptable,

question for mapsforge would be if there is an upper limit of the number of maps i add to the MultiMapDataStore -is it consuming alot of mem? how is the performance impacted?

[Emux]

On 23/09/2015 05:44 μμ, M. Dietrich wrote:

i created my first "map vector tile" (a reactangle map) and choose to use degrees, so i create a reactangle from for example 51 to 52°. in the middle of germany this give files of 36MB per tile.

It's worth checking how two adjacent map "tiles" be seen together, e.g. roads, blocks seams.

--
Emux

[M. Dietrich]

sure, but till now i have just that one ;) i assume it's similar to how the country border is seen

[M. Dietrich]

Am Mittwoch, 23. September 2015 17:54:55 UTC+2 schrieb Emux:

It's worth checking how two adjacent map "tiles" be seen together, e.g. roads, blocks seams.

indeed, i now have 9 rectangles for testing which show strange forrest-borders on map-boundary (i currently do not add the continent shape bg, so all is sea/blue): 

any chance to fix/improve that?

[Ludwig]

This might be caused by incorrect clipping of the polygons, try to set all the clipIncompleteXXX etc to false in osmosis. That works to a certain extend, but not always. It is one of the reasons why we do countries, not lat/long tiles.

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/4677712c-1703-4530-a1e5-ac8d057e7393%40googlegroups.com.

[Emux]

On 24/09/2015 05:16 μμ, Ludwig wrote:

This might be caused by incorrect clipping of the polygons, try to set all the clipIncompleteXXX etc to false in osmosis. That works to a certain extend, but not always. It is one of the reasons why we do countries, not lat/long tiles.

Indeed that maybe the reason.

BTW the relevant info is here with the Osmosis changes.

--
Emux

[M. Dietrich]

hi,

you can find the generated rectangle maps here: https://pyneo.org/maps/ which have lat/lon in their filename and cover one degree, so for example 052_007.map has a boundary box of 52,7, 53,8. 

it is amazing to see that even if nearly 50 maps are installed mapsforges rendering is not slowing down noticeable.

when scrolling through the maps i found this strange issue near emden which is not seen in openandromaps:

don't know where that reactangle is coming from. does someone?

[Emux]

On 26/09/2015 08:05 μμ, M. Dietrich wrote:

it is amazing to see that even if nearly 50 maps are installed mapsforges rendering is not slowing down noticeable.

:)

when scrolling through the maps i found this strange issue near emden which is not seen in openandromaps:

Note: OpenAndroMaps uses different tag-mapping xml than our default one.

What land polygons or coastline are you using, the official procedure or something different ?

BTW please when you refer to locations, give their full information (country, area, coordinates) as not all readers understand where to look for.

--
Emux

[M. Dietrich]

Am Samstag, 26. September 2015 19:10:46 UTC+2 schrieb Emux:

On 26/09/2015 08:05 μμ, M. Dietrich wrote:

when scrolling through the maps i found this strange issue near emden which is not seen in openandromaps:

Note: OpenAndroMaps uses different tag-mapping xml than our default one.

is this part of the themes only or already done in  the map itself? 

What land polygons or coastline are you using, the official procedure or something different ?

the one you are using as well, http://data.openstreetmapdata.com/land-polygons-split-4326.zip

 

BTW please when you refer to locations, give their full information (country, area, coordinates) as not all readers understand where to look for.

sorry, its  53.286009491745624,7.036471255070338.

[Ludwig]

Best to show tile boundaries, I think it's a clipping problem.

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/807ad34e-94e2-45d2-b26b-917be6ae08fe%40googlegroups.com.

[Emux]

On 27/09/2015 12:26 πμ, M. Dietrich wrote:

Note: OpenAndroMaps uses different tag-mapping xml than our default one.

is this part of the themes only or already done in  the map itself?

The tag-mapping is used by map-writer during map creation (it declares what data are included in map).

--
Emux

[M. Dietrich]

just read through it. did i understand that correctly that it's not a realy (from->to) mapping but attributes for the various psm tags like when it's rendered and if at all. is that correct?

i saw different sections of 'pois' or 'ways' which lists items, could that be in one 'pois'/'ways' tag as well? or does the separation tell something i did not get/understand from the docs?

[Ludwig]

I do not understand what you are trying to ask.

tag-mapping.xml determines what is contained in map file

rendertheme.xml determines how what is in the map file is rendered on map.

If something is not in the map file, it (naturally) will not be rendered

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/094a3264-5cea-411c-9c51-1fb9c9ebabf2%40googlegroups.com.

[Emux]

The tag-mapping declares the data that will be included in the map file, from what zoom level (important for zoom level intervals) and possibly what additional fields.

The render theme expands further the zoom levels handling (with min / max) and of course has the appearance to rule.

The tag-mapping order is grouped in pois and ways, with no specific order in them as the actual render order is handled in render theme.

--
Emux

[M. Dietrich]

Am Montag, 28. September 2015 12:50:48 UTC+2 schrieb Ludwig:

tag-mapping.xml determines what is contained in map file

tag-mapping sounds as if osm has different tags than mapsforge has. for example you could map something like

 building -> house

i understand the whole file in a way that you attribute tags but tags are not mapped to others, right?

 

rendertheme.xml determines how what is in the map file is rendered on map.

...mainly by the attribute to control from which zoom level on it is visible or if it it left our entirely, right?

 

If something is not in the map file, it (naturally) will not be rendered

ok, so defaults do not apply to stuff not mentioned.

but why are the tags in groups? 

[Emux]

On 28/09/2015 04:09 μμ, M. Dietrich wrote:

i understand the whole file in a way that you attribute tags but tags are not mapped to others, right?

tag-mapping maps osm data with map file without actual tag / value transformations.

For that there is the powerful TagTransform Osmosis plugin.

--
Emux

[M. Dietrich]

Hi,

i now want to play with tag mapping and wonder where the file is read from. to be sure i just deleted ../mapsforge/mapsforge-map-writer/target/classes/tag-mapping.xml and run the process but it still works fine. is this the wrong file/location or does the mapwrite have some built-in defaults?

[Emux]

The defaul tag-mapping is included in map-writer plugin.

Or you can pass an external one via the tag-conf-file parameter.

--
Emux

[M. Dietrich]

i no looked into the tag mapping and osm tags  maybe i should not play on that level ;) i disabled each and every POI because i'm mainly interested in roads. i learned that ways can be pois too (for example 

<ways>

    ....

    <osm-tag key="amenity" value="grave_yard" zoom-appear="14" />

) but after disabling all these i still see grave_yards. hm. 

additionally i wonder how i can get information on which tags could be (small) roads/ways/tracks that may be of interest for outdoor activities. how can i find those? the tag browser doesn't even tell if it's poi/node or way (or didn't i find it?) - btw: is it right that one could map node to poi?

[Emux]

There are cases of same OSM tags in points and polygons.

e.g. we could declare a amenity=hospital by its polygon bounds and draw it as an area with fill/stroke, and/or by its center point and draw it as a symbol image.

--
Emux

[M. Dietrich]

Am Mittwoch, 30. September 2015 14:58:54 UTC+2 schrieb Emux:

There are cases of same OSM tags in points and polygons.

e.g. we could declare a amenity=hospital by its polygon bounds and draw it as an area with fill/stroke, and/or by its center point and draw it as a symbol image.

thank you, that was the important hint, i found landuse=cemetery which explains my graveyard and the playground seems to be shown by an icon as well. this would mean i should dive into the theme as well :(

any idea on how to detect more "roads" than the current setup shows? 

[Ludwig]

I think roads are pretty much covered by highway tag, but see http://taginfo.openstreetmap.org/

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/40bbd7c3-445d-426d-856b-b7475f16d703%40googlegroups.com.