Displaying external data on Mapsforge map

[Mario Pavlic]

Hello everyone, 

I'm looking to build an Android app which displays a bunch of point and line data on the main map. My input file is in .json and it contains info about objects, points, lines and all that good stuff. I've successfully written a Java program which takes the .json in and generates an .osm file (basically mapping points to OSM Nodes and lines to OSM Ways). I've been using JOSM (Desktop app) to display this .osm file and it works fine.

Now, I want to achieve something similar on an Android app. I've just started working with Android and Mapsforge I'm looking for all the help I can get, since I got a little bit lost on Mapsforge's Github. The only thing I've managed to get to work is to display a map of, lets say, Berlin. (copy-pasted code from the GettingStarted.java file).

I'm wondering how to display this external data on top of the map. External data can be in .json, .osm and even .map (I've managed to convert the .osm to .map with MapWriter Osmosis plugin), whatever is easiest to load. How do I map Nodes and Ways in .osm file or Points and Lines in .json file to objects on Mapsforge map? Do I store them in some layer and then display the layer on top of the MapView? Will Nodes and Ways be visible on the map if their style is not defined in RenderTheme rules?

How does it all work?

Any guidance or suggestion would be appreciated.

Thanks :)

[Emux]

You need to extract the coordinates for the markers, lines, etc.

Then you can check OverlayMapViewer sample, which has examples for how to overlay markers, lines, etc.

--
Emux

[Mario Pavlic]

Thanks for the quick reply Emux. It was my mistake not to mention I've already tried using Markers and Polylines and it could work. Is this the only way to do it though?

I've been looking at MapReadResult class and list of Pois and list of Ways are pretty much everything I need (I can also add tags to Pois and Ways, which could be helpful in styling them?), but I'm wondering if there is a way to display Pois and Ways (or entire MapReadResult) on the map, since MapDataStore uses MapReadResult just for getting the data, but not setting it. May you give me some advice here?

Thanks again :)

[Emux]

If you do not want the easy way with the overlays,
then you can create a map file using osm as input.

You will need to read map-writer documentation and map creation guide
and probably use an external tag-mapping for your custom data.

Then read the render theme documentation for how to style them on the map.

--
Emux

[Mario Pavlic]

Thanks once again, I've already done these steps! Since I defined tags in .osm file, they are probably saved in the .map file. The only question I have now is how to properly set the RenderTheme. Can one of the InternalRenderThemes be modified (can I append some rules to them) and how? If not, I guess I need to go with ExternalRenderTheme. Can multiple RenderThemes be combined? Also, what is the best way to load a File and pass it to ExternalRenderTheme constructor? I've been using Intents and passing the file uri, but it doesn't work.

Thanks so much.

[Emux]

You can take the internal render theme files from repository and modify them.
(then you need to load them as external render themes)

To load external render themes with Android scoped storage,
you can check the MapsforgeMapViewer sample.

But you need to study the render theme documentation and render theme files
in order to see how the rules are set, so you can add / modify rules in a theme.

--
Emux

[Mario Pavlic]

Thanks for all the help. However, my XmlRenderTheme interface doesn't have setResourceProvider(...) method which is used in MapsforgeMapViewer sample (lines 191-193). Also, ContentRenderTheme only has these two constructors:
ContentRenderTheme(ContentResolver, String, Uri)
ContentRenderTheme(ContentResolver, String, Uri, XmlRenderThemeMenuCallback)

Perhaps I'm using the wrong mapsforge version in gradle dependencies? Currently I have:
dependencies {
implementation 'org.mapsforge:mapsforge-map-android:0.15.0'
implementation 'org.mapsforge:mapsforge-themes:0.15.0'
implementation 'org.mapsforge:mapsforge-poi-android:0.15.0'
...

[Emux]

Until next release, you will need to use the snapshot version.
(see the instructions in the integration guide)

Also it's recommended to use the zip render theme.
(the content render theme is very slow)

--
Emux

[Mario Pavlic]

Since I'm a rookie at Android I'm having troubles following the Snapshot part of the guide, I added:
maven { url "https://oss.sonatype.org/content/repositories/snapshots/" } to my project's build.gradle, and changed the module's build.gradle dependencies to:
implementation 'org.mapsforge:mapsforge-map-android:master-SNAPSHOT'
implementation 'org.mapsforge:mapsforge-themes:master-SNAPSHOT'
implementation 'org.mapsforge:mapsforge-poi-android:master-SNAPSHOT'

All of my imports are now invalid. Did I make a mistake or are there some extra steps?

Thanks for the tip about zip render theme :)

[Emux]

You can add all Mapsforge dependencies, as seen in integration guide.

For Android the needed ones are those in "Core" + "Android" sections.

Have you refreshed your Gradle configuration to read the changes in the build.gradle file?

You can post here your build.gradle file to see if they are declared correctly.

--
Emux

[Mario Pavlic]

Yes, added everything from Core and Android sections and synced build.gradle with my project. Here is the build.gradle:
plugins {
id 'com.android.application'
}

android {
compileSdkVersion 30
buildToolsVersion "30.0.3"

defaultConfig {
applicationId "com.example.mapsforgeandroiddemo"
minSdkVersion 19
targetSdkVersion 30
versionCode 1
versionName "1.0"

testInstrumentationRunner "androidx.test.runner.AndroidJUnitRunner"
}

buildTypes {
release {
minifyEnabled false
proguardFiles getDefaultProguardFile('proguard-android-optimize.txt'), 'proguard-rules.pro'
}
}
compileOptions {
sourceCompatibility JavaVersion.VERSION_1_8
targetCompatibility JavaVersion.VERSION_1_8
}
}

dependencies {
implementation 'org.mapsforge:mapsforge-core:master-SNAPSHOT'
implementation 'org.mapsforge:mapsforge-map:master-SNAPSHOT'
implementation 'org.mapsforge:mapsforge-map-reader:master-SNAPSHOT'
implementation 'org.mapsforge:mapsforge-themes:master-SNAPSHOT'
implementation 'net.sf.kxml:kxml2:2.3.0'
implementation 'org.mapsforge:mapsforge-map-android:master-SNAPSHOT'
implementation 'com.caverock:androidsvg:1.4'
implementation 'androidx.appcompat:appcompat:1.2.0'
implementation 'com.google.android.material:material:1.3.0'
implementation 'androidx.constraintlayout:constraintlayout:2.0.4'
testImplementation 'junit:junit:4.+'
androidTestImplementation 'androidx.test.ext:junit:1.1.2'
androidTestImplementation 'androidx.test.espresso:espresso-core:3.3.0'

[Emux]

You need to add also the:

repositories {
    maven { url "https://oss.sonatype.org/content/repositories/snapshots/" }
}

--
Emux

[Mario Pavlic]

That was it, thanks!

I still can't open the external theme so can you please look at the following code and see if there is something wrong:

onActivityResult:

protected void onActivityResult(int requestCode, int resultCode, Intent data) {
super.onActivityResult(requestCode, resultCode, data);

if(data != null && resultCode == Activity.RESULT_OK) {
Uri uri = data.getData();

if (requestCode == SELECT_MAP_FILE) {
openMap(uri);
} else if(requestCode == SELECT_THEME_FILE) {
openTheme(uri);
}
}
}

openTheme:

private void openTheme(Uri uri) {
XmlRenderTheme xmlRenderTheme = new ContentRenderTheme(getContentResolver(), uri);

TileCache tileCache = AndroidUtil.createTileCache(this, "mapcache",
mapView.getModel().displayModel.getTileSize(), 1f,
mapView.getModel().frameBufferModel.getOverdrawFactor());

MultiMapDataStore multiMapDataStore = new MultiMapDataStore(MultiMapDataStore.DataPolicy.RETURN_ALL);
multiMapDataStore.addMapDataStore(mapDataStore, true, true);
multiMapDataStore.addMapDataStore(dvDataStore, false, false);

TileRendererLayer tileRendererLayer = new TileRendererLayer(tileCache, multiMapDataStore,
mapView.getModel().mapViewPosition, AndroidGraphicFactory.INSTANCE);

tileRendererLayer.setXmlRenderTheme(xmlRenderTheme);

mapView.getLayerManager().getLayers().add(tileRendererLayer);
}

Method that starts Intent for theme picking:

public void loadThemeOnClick(View view) {
Intent intent = new Intent(Build.VERSION.SDK_INT >= Build.VERSION_CODES.KITKAT ? Intent.ACTION_OPEN_DOCUMENT : Intent.ACTION_GET_CONTENT);
intent.addCategory(Intent.CATEGORY_OPENABLE);
intent.addFlags(Intent.FLAG_GRANT_READ_URI_PERMISSION);
intent.setType("*/*");

startActivityForResult(intent, SELECT_THEME_FILE);
}

If I set the render theme to InternalRenderTheme.DEFAULT both .map files are displayed correctly.

Once again thanks for all the help!

[Emux]

You are still using content theme.
Content themes must first grant a folder permission and then open the xml file.
But they are very slow...

It is recommended to start with the zip theme, as it is much faster.
Please study carefully the sample's steps and compare with your code.

--
Emux

[Mario Pavlic]

Are zip themes just xml themes archived?

[Emux]

Like the Elevate theme of OpenAndroMaps.

--
Emux

[Mario Pavlic]

Ok, I have finally managed to load external theme. Thank you very much.

Now, I could finally test if my generated .map file is fine, and it looks like it's not. So, if I have an .osm file that looks like this:

<?xml version="1.0" encoding="UTF-8"?>

<osm version="0.6">

   <node id="666466176606664" version="1" timestamp="2021-04-18 12:38:49.172" lat="15.15" lon="15.15">

      <tag k="key1" v="val1" />

      <tag k="key2" v="val2" />

   </node>

   

   <node id="766466176606664" version="1" timestamp="2021-04-18 12:38:49.172" lat="20.15" lon="15.15">

      <tag k="key1" v="val1" />

      <tag k="key2" v="val2" />

   </node>

   

   <way id="224207122322524" version="1" timestamp="2021-04-18 12:38:49.239">

      <tag k="key1" v="val1" />

      <nd ref="666466176606664" />

      <nd ref="766466176606664" />

   </way>

</osm>

I use Osmosis to convert .osm to .map and it completes without errors. Should this data appear on the map if I add following rules to my external render theme:

<rule e="way" k="key1" v="val1">

<line stroke="#30000000" stroke-width="1" />

</rule>

<rule e="node" k="key1" v="val1">

<circle radius="1" scale-radius="true" fill="#A69D9D" />

</rule>

<rule e="node" k="key1" v="val1">

<circle radius="1" scale-radius="true" fill="#A69D9D" />

</rule>

[Emux]

You will first need to use a custom tag-mapping with map-writer for your custom tags.
(see map-writer documentation)

Then you will need a custom render theme.

Study the default tag-mapping and theme for known OSM tags to see how they work.

--
Emux

[Mario Pavlic]

Ok, I changed my .osm file to only have one node as such:

<?xml version="1.0" encoding="UTF-8"?>

<osm version="0.6">

   <node id="666466176606664" version="1" timestamp="2021-04-18 12:38:49.172" lat="15.85504" lon="15.888793">

      <tag k="type" v="xDot" />

      <tag k="id" v="123" />

   </node>   

</osm>

I added these tags to my custom mapping XML:
...

<pois>

<osm-tag key="type" value="xDot" />

</pois>

...

Generated a .map file with Osmosis and I added this rule to Elevate.xml theme:

<rule e="node" k="type" v="xDot">

<circle radius="1" scale-radius="false" fill="#FF0000" />

</rule>

Still no luck in displaying. Do you have a suggestion?

[Emux]

- Declare zoom-appear in tag-mapping (like the default demonstrates)

- Use a symbol in the render theme (it is easier to see while testing)

- Make sure your map-writer commands are correct building the map

--
Emux

[Mario Pavlic]

It works! Thank you so much :)