creating maps with non-standard tagging with OsmAndMapCreator

[Greg Troxel]

Following OSM doctrine, I am not adding parcel (cadastral) boundaries to
OSM. However, I want to see them on renderings on OsmAnd. (Also on
Garmin, via mkgmap, but that a) works and is b) off topic here.)

I have a file that is in .osm format with a bunch of polygons and the
important tag is "boundary=parcel". There are a bunch of other tags
from the conversion from a shapefile, but I think these are just noise.
The file is 7.3M; it has 2678 polygons.

Using OsmAndMapCreator on this file leads to a 219 byte file that
obviously lacks my 2K+ polygons. But it thought for a good long time
that seemed appropriate.

So, it seems there is someplace a set of rules that says (correctly)
that boundary=parcel is invalid and skipped, but I don't immediately see
where that is.

I realize that after making parcel allowable and converting, I need to
change the render xml to show the parcel boundaries somehow, but I'm on
the convert step so far.

Any hints would be appreciated; I'll be looking that the
OsmAndMapCreator sources in the meantime.

Thanks,
Greg

[Greg Troxel]

A followup question.

I checked out

g...@github.com:osmandapp/OsmAnd-tools.git

and am trying to build it, guessing that's on the path to changing the
boundary processing.

I'm on NetBSD 8 with openjdk8, which has worked well for building and
running splitter/mkgmap and running josm, and seems to run
OsmAndMapCreator fine as well.

I get an error:

> Task :OsmAndMapCreatorUtilities:compileJava FAILED
[parentdir]/OsmAnd-tools/java-tools/OsmAndMapCreatorUtilities/src/main/java/crosby/binary/BinaryParser.java:7: error: package com.google.protobuf does not exist
import com.google.protobuf.InvalidProtocolBufferException;
^
and don't see where this is declared; it's not in the gradle build. I
can get the jar and put it in CLASSPATH.

I didn't find a README explaining the build process, so I tried "gradle
build".

Any clues appreciated.

[A Thompson]

AFAIK, you need to get the file rendering_types.xml from github and put it in the right place for OsmAndMapCreator to read it. Then you can edit that file to control what is or isn't included in the map.

If you search the archives of this list for "rendering_types.xml" then you'll know as much as I do - I have no personal experience. Hope that helps!

[Harry van der Wolf]

Hi,

openjdk8 is fine. I use that as well (just as openjdk11 by the way as java8 is officially no longer supported).

You do not only need git, you also need the google repo tool. either download it or install it from your package manager.

I work on a 4GB Chromebook where I flashed different firmware onto and installed GalliumOS, a light-weight Ubuntu (18.04) derivative specially for Chromebook hardware. You're netBSD is not linux but a real BSD unix, but should work just as well.

I use below script:

============================  

#/bin/sh

libpath=$(pwd)
mkdir -p OsmAnd-all
cd OsmAnd-all
curpath=$(pwd)

repo init -u git://github.com/osmandapp/OsmAnd-manifest.git -m jenkins_tools.xml
repo sync -d -c -q

${curpath}/android/OsmAnd-java/collect_legacy_libs.sh

# Copy the libraries in
LIBDIR="${libpath}/lib"
cd ${curpath}/tools/java-tools/OsmAndMapCreator/
cp -v ${LIBDIR}/libOsmAndCoreWithJNI_standalone.so lib-native/
cp -v ${LIBDIR}/libOsmAndCore_ResourcesBundle_shared.so lib-native/
cp -v ${LIBDIR}/OsmAndCore_jni.jar lib-gl/OsmAndCore_jni.jar
rm lib-native/OsmAndCore_jni.jar

cd ${curpath}
cd tools/java-tools

#${curpath}/gradlew fatJar
./gradlew --info cleanTest test :OsmAndMapCreator:buildDistribution

cp OsmAndMapCreator/build/distributions/OsmAndMapCreator.zip ${curpath}/..

============================

Remarks:

- From my "software build scripts folder", I cd into "OsmAnd-all" where I build everything, but there is a very nasty detail. It is very important (!) to have a folder "lib" next to  "OsmAnd-all" (not inside) where you have all the jars AND "libOsmAndCore_ResourcesBundle_shared.so" and "libOsmAndCoreWithJNI_standalone.so", which you can take from a nightly build from OsmandMapCreator (lib versions are actually already quite old). Without these you will not be able to build a correct OsmAndMapCreator. Inside my script another script is called that is part of the OsmAnd-tools repo, but that script already assumes you have the necessary stuff available, which you normally don't have. And that script actually has an error as well, that's why my follow-up copying steps are in my script.

(- I use a comparable, but more complex, script for OsmAnd and OsmAnd+ WITH the same symlink to the libraries, otherwise OsmAnd does not build either).

In the end a zip will be created containing "everything" in the folder one level above "OsmAnd-all" . The zip is approx. 80Mb. If it is around 67MB (or so), you did something wrong with the libraries mentioned in remark 1.

Success.

Harry

[Andy Townsend]

On 17/02/2020 02:16, A Thompson wrote:
> AFAIK, you need to get the file rendering_types.xml from github and
> put it in the right place for OsmAndMapCreator to read it. Then you
> can edit that file to control what is or isn't included in the map.
>

Does https://www.openstreetmap.org/user/SomeoneElse/diary/391499 help?

I wrote that a bit ago trying to document a simple change to
default.render.xml .  The follow-up to that (which I haven't written
yet!) is intended to talk about changes to rendering_types.xml , but I
hit a problem doing some of what I was trying to do (tag transformations
- partly mentioned in
https://www.openstreetmap.org/user/SomeoneElse/diary/391484 ).

Best Regards,

Andy

[Harry van der Wolf]

To add some extra clarification (I hope).

The whatever.rendering.xml files change the display of the already defined map objects for car, pedestrian or whatever. These map objects need to be in the obf. roads-only maps don't contain houses. No matter what render.xml you use, you will not see them.

The rendering_types.xml defines which objects can be displayed (in the various ways the whatever.rendering.xml files define). This also means that you can have an osm with parcel polygons, or simply points where you peed against a wall, but as long as OsmAndMapCreator doesn't know the existence of such "objects" it cannot put them in an obf file.

This rendering_types.xml is used during map creation. Changes to this rendering_types.xml are only effective if you compile your own OsmandMapCreator with that modified rendering_types.xml (and I think that is what Gregs other question is about).

(Or you should make a pull-request and hope that the Osmand dev-team will add it to the repository.)

After having built his own OsmandMapCreator with modified rendering_types.xml, Greg can create obf files containing his parcels.

The finishing step is then indeed to create a parcel.render.xml based on the default.render.xml that does display these parcels now that they are in the obf.

From that moment on every OsmAnd with that parcel.render.xml can display Gregs parcels.obf.

Harry

Op ma 17 feb. 2020 om 12:33 schreef Andy Townsend <ajt...@gmail.com>:

--
You received this message because you are subscribed to the Google Groups "OsmAnd" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osmand/03694828-4de7-3a0b-b3e6-166e10ed1e41%40gmail.com.

[Philippe de Franclieu]

Hi Greg,

Converting shape files of cadastral parcels to OBF files is not that easy! 

1) create some custom rendering file;

2) create a translation file;

3) run some ogr2osm python command;

4) run OsmandMapCreator to create an OBF file from the above created OSM file (note that OMC does not handle relations. So you'll have to cut and paste in the OSM file);

5) select your rendering file in Osmand.

1) Custom rendering file? 

Mine looks as follows, designed to render the nature of cadastral parcels. Copy such 'custom.render.xml' file to the rendering dir on your Android device (android/data/net.osmand.plus/files/rendering): 

<renderingStyle name="NewPaPoPCn" depends="default" defaultColor="#F5FFFF" version="1">

<renderingAttribute name="shadowRendering">

<!-- 0 - no shadow, 1 - one step, 2 - blur shadow, 3 - solid shadow -->

<!-- next line corrects for shadowColor appearing very dark with solid shadow in day view, bridges would not stand out -->

<filter minzoom="12" nightMode="false" attrIntValue="3" shadowColor="#888888"/>

<filter minzoom="12" attrIntValue="3" shadowColor="#888888"/>

<!-- next line avoids melting close-together highways into one another - disadvantage is separator lines between segments and at intersections --> 

<filter minzoom="9" maxzoom="14" attrIntValue="3" shadowColor="#9e9e9e"/>

<!-- appearance equivalents: one-step #464646 equals solid #767676 -->

<!-- filter minzoom="9" maxzoom="13" attrIntValue="3" shadowColor="#767676"/ -->

<filter attrIntValue="0"/>

</renderingAttribute>

<renderingAttribute name="polygonMinSizeToDisplay">

<!-- Minimum polygon size in pixels -->

<filter attrIntValue="4"/>

</renderingAttribute>

<order>

        <!-- source: https://github.com/osmandapp/OsmAnd-resources/blob/master/rendering_styles/standalone-template.render.xml -->

        <!-- Point -->

<switch objectType="1" order="128">

</switch>

        <!-- Polygon -->

        <switch objectType="3">

            <case tag="boundary" value="cadastral" order="50"/>

        </switch>

        <!-- Line -->

        <switch objectType="2">

            <filter tag="highway" value="track"  order="52"/>

            <filter tag="highway" value="path"  order="51"/>

        </switch>

        

    </order>

<text>

<group>

            <filter tag="boundary" value="cadastral"  textOrder="8"/>

          <groupFilter textOnPath="true">

<filter minzoom="12" textColor="#202020" textHaloColor="#ffffff" textHaloRadius="1" textWrapWidth="12" textMinDistance="1" textSize="12"/>

</groupFilter>

</group>

</text>

    <point>    

    </point>

<polygon>

        

        <case minzoom="12" tag="boundary" value="cadastral" color="#55cccccc" color_2="#101010" strokeWidth_2="0.5"> <!-- Parcelle cadastrale -->

            <apply_if additional="papo:pc_styln=0" color="#70ffffc0"/> <!-- Sols, jardins, etc. -->

            <apply_if additional="papo:pc_styln=1" color="#ff80c080"/> <!-- Bois -->

            <apply_if additional="papo:pc_styln=2" color="#70ffc0ff" shader="quarry2"/> <!-- Carri?res -->

            <apply_if additional="papo:pc_styln=3" color="#70dedede"/> <!-- Chemins de fer,  -->

            <apply_if additional="papo:pc_styln=4" color="#700080ff"/> <!-- Eaux -->

            <apply_if additional="papo:pc_styln=5" color="#ffe0c0ff"/> <!-- Landes -->

            <apply_if additional="papo:pc_styln=6" color="#ffc0ffc0"/> <!-- Pr?s -->

            <apply_if additional="papo:pc_styln=7" color="#70ff8000"/> <!-- Terrains ? b?tir -->

            <apply_if additional="papo:pc_styln=8" color="#fff2d098"/> <!-- Terres -->

            <apply_if additional="papo:pc_styln=9" color="#70ffffff" shader="orchard"/> <!-- Vergers, vignes -->

            <apply_if additional="papo:pc_styln=10" color="#ffff8080"/> <!-- Mixte (j, k, etc.) -->

        </case>

    </polygon>

    

    <line>

    </line>

</renderingStyle>

2) Translation file? Mine looks as follows:

# -*- coding: utf-8 -*-

'''

Translation file to convert Parcelles & Polygones' shapefiles to OSM format.

The shapefile contains cadastral parcels shapes + attributes

 boundary:administrative

 admin_level: 12? (11 section?)

 

Tag translation

The DBF file contains :

 

PC_ID=Identifiant

PC_LABEL=Etiquette

...

PC_SURFM= Contenance cadastrale

PC_SURFV=Contenance calculée (ie <> contenance matrice cadastrale) exprimée en hectares

PC_PERIM=Périmètre exprimé en mètres

PC_STYLN=Index nature de culture (cf. PaPoPCn.render.xml)

...

PC_REM=Remarque(s)

etc.

'''

import re

def filterTags(attrs):

    if not attrs: return

    tags = {}

    

    tags.update({'boundary':'cadastral'})

    tags.update({'admin_level':'12'})

    tags.update({'name':attrs['PC_LABEL']})

    tags.update({'area':attrs['PC_SURFM']})

    

    tags.update({'PaPo:Type':'PC'})

    tags.update({'PaPo:PC_ID':attrs['PC_ID']})

    tags.update({'PaPo:PC_LABEL':attrs['PC_LABEL']})

    tags.update({'PaPo:PC_INSEE':attrs['PC_CODCO']})

    tags.update({'PaPo:PC_SECF':attrs['PC_SECF']})

    tags.update({'PaPo:PC_NUM':attrs['PC_NUM']})

    tags.update({'PaPo:PC_SURFM':attrs['PC_SURFM']})

    tags.update({'PaPo:PC_SURFV':attrs['PC_SURFV']})

    tags.update({'PaPo:PC_NATUR':attrs['PC_NATUR']})

    tags.update({'PaPo:PC_STYLN':attrs['PC_STYLN']})

    tags.update({'PaPo:PC_STYLP':attrs['PC_STYLP']})

    tags.update({'Source':'Parcelles et Polygones SARL - DGFIP'}) 

    

    return tags;

3) run some ogr2osm python command? 

My BATfile looks as follows:

REM Attn au "code page" du fichier DBF.  Ouvrir fichier avec DBF Commander - Tools - Set Code Page : 850

REM et vérifier que les fichiers de commandes python (ogr2osm.py et geom.py) sont dans le répertoire prévu (D:\PaPo\OSGeo4W64\bin\ ou E:\ANTEC\PaPo\OSGeo4W64\bin\)

REM Ensuite, avec OsmAndMapCreator, création du fichier *.OBF

REM Attention OsmAndMapCreator ne gère pas les tags "relation" ... Copier-Coller des données listées ("outer") dans la balise <way id dont la référence est indiquée dans la balise <member ref de la balise <relation id...

REM --epsg:

REM EPSG 27561 = Lambert I, 27571 = Lambert Zone 1 (Nord)

REM EPSG 27562 = Lambert II, 27572 = Lambert Zone 2 (Centre) (= II Etendu)

REM EPSG 27563 = Lambert III, 27573 = Lambert Zone 3 (Sud)

REM EPSG 27564 = Lambert IV, 27574  = Lambert Zone 4 (Corse)

REM EPSG 2154 = Lambert93

REM EPSG 3942:3950 = CC42:CC50

:: Chargement OSGeo4W (ie GDAL/OGR)

@TITLE Conversion format fichier SHP vers OSM - SHP2OSM

@ECHO OFF

call o4w_env.bat

IF ERRORLEVEL 1 GOTO :ERROR_OSGEO4W

:: Création fichier *.OSM

python "E:\ANTEC\PaPo\OSGeo4W64\bin\ogr2osm.py" -f -o "41-La Girolliere_PCn.osm" --positive-id --no-upload-false --add-version -t "\translations\NewPCn.py" -v --epsg=2154 ..\shp\PC_202001.shp

IF ERRORLEVEL 1 GOTO :ERROR_OGR2OSM

GOTO :EOF

::::::::::::

:::: ERREURS

::::::::::::

:ERROR_OSGEO4W

ECHO ERROR_OSGEO4W - Erreur lors de la configuration de l'environnement GDAL/OGR.

PAUSE

GOTO :EOF

:ERROR_OGR2OSM

ECHO ERROR_OGR2OSM - Erreur lors de la conversion de format du fichier.

PAUSE

GOTO :EOF

:EOF

ECHO ON

4) run OsmandMapCreator to create an OBF file, then copy it to your Android device (android/data/net.osmand.plus/files).

5) Custom renderer selection:

Menu-Configure map-Map style: custom rendering file.

For sure all this can be done in a more elegant way...

I hope this helps!

Regards,

Philippe

[Greg Troxel]

Harry van der Wolf <hvd...@gmail.com> writes:

> openjdk8 is fine. I use that as well (just as openjdk11 by the way as java8
> is officially no longer supported).

When you say "java8 is officially no longer supported", do you mean Java
8 the language flavor, or openjdk8, or jdk8 from Oracle, or ?

I also have openjdk11 installed, so I could use that, but it seems like
that is not my problem.

> You do not only need git, you also need the google repo tool. either
> download it or install it from your package manager.

Thanks for the hint. It would be really nice if someone caused the
README.md or a BUILDING.md in these repos to have build instructions.


Thank you very much for the scripts. Obviously I have some things to
work through, so I will not quote most of it and only when I have a
question.

> ${curpath}/android/OsmAnd-java/collect_legacy_libs.sh
>
> # Copy the libraries in
> LIBDIR="${libpath}/lib"
> cd ${curpath}/tools/java-tools/OsmAndMapCreator/
> cp -v ${LIBDIR}/libOsmAndCoreWithJNI_standalone.so lib-native/
> cp -v ${LIBDIR}/libOsmAndCore_ResourcesBundle_shared.so lib-native/
> cp -v ${LIBDIR}/OsmAndCore_jni.jar lib-gl/OsmAndCore_jni.jar
> rm lib-native/OsmAndCore_jni.jar

Do these .so files get built somehow by the previous steps? If so, I'll
figure things out.

> - From my "software build scripts folder", I cd into "OsmAnd-all" where I
> build everything, but there is a very nasty detail. It is very important
> (!) to have a folder "lib" next to "OsmAnd-all" (not inside) where you
> have all the jars AND "libOsmAndCore_ResourcesBundle_shared.so" and
> "libOsmAndCoreWithJNI_standalone.so", which you can take from a nightly
> build from OsmandMapCreator (lib versions are actually already quite old).

But I have different native ELF format and have to build these, not copy
some long-ago-built Linux binary.

> Without these you will not be able to build a correct OsmAndMapCreator.
> Inside my script another script is called that is part of the OsmAnd-tools
> repo, but that script already assumes you have the necessary stuff
> available, which you normally don't have. And that script actually has an
> error as well, that's why my follow-up copying steps are in my script.
> (- I use a comparable, but more complex, script for OsmAnd and OsmAnd+ WITH
> the same symlink to the libraries, otherwise OsmAnd does not build either).

Wow. This is more complicated than I expected. I wonder if there is
any chance of getting these fixes pushed upstream?

> In the end a zip will be created containing "everything" in the folder one
> level above "OsmAnd-all" . The zip is approx. 80Mb. If it is around 67MB
> (or so), you did something wrong with the libraries mentioned in remark 1.

Thanks very much for the detailed hints!

[Greg Troxel]

Andy Townsend <ajt...@gmail.com> writes:

> On 17/02/2020 02:16, A Thompson wrote:
>> AFAIK, you need to get the file rendering_types.xml from github and
>> put it in the right place for OsmAndMapCreator to read it. Then you
>> can edit that file to control what is or isn't included in the map.

A:

Thanks - I am now beginning to follow. You have said "put in right
place" and Harry indicates it is compiled in. Are you saying I can put
a modified version someplace and have the distributed OsmAnd read it?

> Does https://www.openstreetmap.org/user/SomeoneElse/diary/391499 help?
>
> I wrote that a bit ago trying to document a simple change to
> default.render.xml .  The follow-up to that (which I haven't written
> yet!) is intended to talk about changes to rendering_types.xml , but I
> hit a problem doing some of what I was trying to do (tag
> transformations - partly mentioned in
> https://www.openstreetmap.org/user/SomeoneElse/diary/391484 ).

Andy:

Yes, that does help. I was already aware that I needed to do something
like that, and this was seeming easier than getting my custom pbf, so I
hadn't worried about that step yet.

And from your diary entries I have found:

https://osmand.net/help/docs/Custom_Rendering_How-To.htm

which is quite helpful.

[Andy Townsend]

On 17/02/2020 17:52, Greg Troxel wrote:
>
> Thanks - I am now beginning to follow. You have said "put in right
> place" and Harry indicates it is compiled in. Are you saying I can put
> a modified version someplace and have the distributed OsmAnd read it?

I believe so, based on what I'm seeing on screen.  On a Linux system
with a display attached I'm running

java -jar -Xms1G -Xmx6G ~/src/osmandmapcreator/OsmAndMapCreator.jar

and then selecting "settings" from the menu.  That seems to allow me to
choose which rendering_types.xml it will use.  I'm not 100% convinced
yet though as just "creating a new name" in the tag transform and trying
to render that via "default.render.xml" didn't work - but that may be
just me doing it wrong (if I make the equivalent change for a Mapnik /
Carto map style I know I'd have to make versions of the same change in 4
different places, so OsmAnd isn't unusual here).   Of course if it is
effectively compiled in that would explain the problems that I was seeing!

You can run OsmAndMapCreator on a machine without a console (haven't
tried it in a while though), and if you do, you won't get any screens
displayed (obviously).

One other thing that I noticed was a "null pointer exception" while
processing a PBF (I was using
http://download.geofabrik.de/europe/great-britain/england/north-yorkshire.html
as it was in December, but got the error with several different files).

Best Regards,

Andy

[A Thompson]

Thanks to everyone for this thread - people have asked about this before without getting anywhere! 

Sorry if it's obvious, but there are also two low-effort solutions to the original problem:

1. Forget about making map files, convert the boundaries to .gpx tracks in JOSM, then OsmAnd can display them with no further effort. I gave an example here: https://groups.google.com/d/msg/osmand/3yY1zrJYXXc/1nulrqrIEAAJ

or

2. Deliberately tag the new boundaries with something that OsmAnd normally does display, e.g. a suitable boundary or even a type of highway or railway etc, then use OsmandMapCreator in the usual way and view in OsmAnd using the standard renderers. There could be confusion when viewing the map but sometimes this can be good enough.

A further snag with (2) is that boundaries often run down real highways, making them difficult to display. Option (1) is really easy, and can even give some extra leverage because of OsmAnd's rich features for working with .gpx tracks.

[Akkana Peck]

A Thompson writes:
> Thanks to everyone for this thread - people have asked about this before
> without getting anywhere!

Great thread, and I'm definitely going to try some of the tricks
that have been discussed.

> Sorry if it's obvious, but there are also two low-effort solutions to the
> original problem:
>
> 1. Forget about making map files, convert the boundaries to .gpx tracks in
> JOSM, then OsmAnd can display them with no further effort. I gave an
> example here:
> https://groups.google.com/d/msg/osmand/3yY1zrJYXXc/1nulrqrIEAAJ

I've used GPX track files (generated with Python scripts) several
times as a quick way of displaying boundaries in OsmAnd. The display
isn't ideal, but it's SO much easier than any of the other methods.

Another solution, not as easy (but maybe still easier than
OsmAndMapCreator plus a custom renderer) is using QGIS to make a
tiled map that you can use as an underlay or overlay map.
I wrote a couple of posts on that a while back:
http://shallowsky.com/blog/mapping/osmand-making-overlay-maps.html
http://shallowsky.com/blog/mapping/qgis-categorized.html

...Akkana

[Harry van der Wolf]

Op ma 17 feb. 2020 om 19:19 schreef Andy Townsend <ajt...@gmail.com>:

On 17/02/2020 17:52, Greg Troxel wrote:
>
> Thanks - I am now beginning to follow.  You have said "put in right
> place" and Harry indicates it is compiled in.  Are you saying I can put
> a modified version someplace and have the distributed OsmAnd read it?

I believe so, based on what I'm seeing on screen.  On a Linux system
with a display attached I'm running

java -jar -Xms1G -Xmx6G ~/src/osmandmapcreator/OsmAndMapCreator.jar

and then selecting "settings" from the menu.  That seems to allow me to
choose which rendering_types.xml it will use.  I'm not 100% convinced
yet though as just "creating a new name" in the tag transform and trying
to render that via "default.render.xml" didn't work - but that may be
just me doing it wrong (if I make the equivalent change for a Mapnik /
Carto map style I know I'd have to make versions of the same change in 4
different places, so OsmAnd isn't unusual here).   Of course if it is
effectively compiled in that would explain the problems that I was seeing!

I have done quite some testing/fiddling around with OsmAnd MapCreator since 2011, but I always overlooked that field/parameter. :)

(Maybe it is not so long in there, I will also check the commits in the git repo).

I will do some tests.

Thanks for this pointer!

Harry