Creation of OBF from SHP

[Adam Sherman]

Good Morning,

I've recently switched to using OSMand on a tablet for navigating through the backcountry forestry roads of Southwestern Quebec, Canada. I found an incredibly detailed free (as in beer) data source, provided by the Province of Quebec:

http://adressesquebec.gouv.qc.ca/aqreseauPlus.asp

They provide the data in multiple formats:

656M ESRI(FGDB)

894M ESRI(PGDB)

1.5G ESRI(SHP)

1.0G Geomedia(MDB)

1.5G MapInfo(TAB)

Since I would really like to get this data onto my tablet, I've been trying everything I can think of. Most recently, I used GPSbabel to cover the SHP file to OSM, then OSMand Map Creator to "Create .obf from osm file…". It processed for some time and produced a tiny obf of only 273B.

I'm running OS X on a MacBook Pro: 3 GHz Core i7 with 16G of RAM on a true SSD, so it should be reasonably fast. But what preprocessing should I do? I have the GDAL tools, QGIS, etc.

Any other ideas?

Thank you,

A.

[Harry van der Wolf]

And what do you get when you run inspector.sh against the obf?

Harry

--
You received this message because you are subscribed to the Google Groups "Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Adam Sherman]

On Wed, Sep 9, 2015 at 8:23 AM, Harry van der Wolf <hvd...@gmail.com> wrote:

And what do you get when you run inspector.sh against the obf?

$ ./inspector.sh -vmap ../../osmand/Aqreseau_plus.obf 

Sep 09, 2015 9:16:11 AM net.osmand.osm.MapPoiTypes init

INFO: Time to init poi types 22

Binary index Aqreseau_plus.obf version = 2 edition = Tue Sep 08 17:48:57 EDT 2015

1 Map data Aqreseau_plus - 27 bytes

Total map objects: 0

2 Routing data Aqreseau_plus - 15 bytes

Bounds (left top - right bottom) : 180, -90 NE - -180, 90 NE

3 Address data Aqreseau_plus - 63 bytes

3.1 Address part size=2 bytes

3.3 Address part size=2 bytes

3.2 Address part size=2 bytes

4 Poi data Aqreseau_plus - 45 bytes

5 Transport data Aqreseau_plus - 85 bytes

Bounds (left top - right bottom) : -180, 85.0511 NE - -180, 85.0511 NE

--

Adam Sherman
Directeur des opérations, Sauvetage bénévole Outaouais

Director of Operations, Ottawa Volunteer SAR
CTO, Versature Corp.
+1 613 797 6819

[Stephan75]

Hello Adam,

to get those shapefiles work in Osmand, you need to convert them in OSM data format (like te others mentioned already),

and that conversion result need not onlythe geometries, but also the correct tagging according to the OSM tagging schema.

So I recommend to get familiar with the OSM tagging of streets, areas, POIs, landuses, waters etc ... and you should find a solution to get these OSM attributes into your data.

Try this first with a small portion of shapefile data, just to see whether you come near your aim and what result you get from OsmandMapcreator and inside osmand app itself.

Also search for the process of Shapefile -> OSM data via OSM wiki or hrlp.openstreetmap.org

see http://wiki.openstreetmap.org/wiki/Shapefiles

and http://wiki.openstreetmap.org/wiki/Converting_map_data_between_formats

Success?

[Philippe de Franclieu]

Hi Adam,

Steps to create custom OBF vector files (with ogr2osm.py) from your SHP files :

1) Translation
You have to create a translation file to map your data using OSM tags (or create new ones). It will be something like \translations\MyTransFile1.py
and look like :

# -*- coding: utf-8 -*-

'''
Translation file to convert shapefiles to OSM format.

The shapefile contains ponds (ie natural:water objects)

Tag translation
The DBF file contains :
 
ID=Identifiant
LABEL=Etiquette
AREA=Contenance calculée (ie <> contenance matrice cadastrale) exprimée en hectares
LENGTH=Périmètre exprimé en mètres
REM=Remarque(s)
X= X(m)
Y= Y(m)
SRID=Code EPSG de la projection utilisée

'''
import re

def filterTags(attrs):
    if not attrs: return

    tags = {}
   
    tags.update({'natural':'water'})
    tags.update({'name':attrs['LABEL']})
    tags.update({'area':attrs['AREA']})
    tags.update({'length':attrs['LENGTH']})
   
    return tags;


2) Run a shp2osm.bat file
Such a batch file will create a *.osm file with the ogr2osm.py command :

Rem Attn au "code page" du fichier DBF.  Ouvrir fichier avec DBF Commander - Tools - Set Code Page : 850
Rem et vérifier que les fichiers de commandes python (ogr2osm.py et geom.py) sont dans le répertoire prévu (C:\OSGeo4W64\bin\)
:: Chargement OSGeo4W (ie GDAL/OGR)
@TITLE Conversion format fichier SHP vers OSM - SHP2OSM
@ECHO OFF
call o4w_env.bat
IF ERRORLEVEL 1 GOTO :ERROR_OSGEO4W

python "C:\OSGeo4W64\bin\ogr2osm.py" -f -o MyFile.osm --positive-id --no-upload-false --add-version -t "C:\OSGeo4W64\bin\translations\MyTransFile1.py" -v --epsg=999999 ..\shp\MyFile.shp
IF ERRORLEVEL 1 GOTO :ERROR_OGR2OSM
GOTO :EOF

::::::::::::
:::: ERREURS
::::::::::::
:ERROR_OSGEO4W
ECHO ERROR_OSGEO4W - Erreur lors de la configuration de l'environnement GDAL/OGR.
PAUSE
GOTO :EOF

:ERROR_OGR2OSM
ECHO ERROR_OGR2OSM - Erreur lors de la conversion de format du fichier.
PAUSE
GOTO :EOF

:EOF
EXIT

3) OSM to OBF with OsmandMapCreator

Edit the osm file and get rid of the <relation> tags if any. Then convert your OSM file to an OBF file with OsmandMapCreator.

4) Custom renderer
If you are using OSM tags then you are done!
If not then you will have to create a custom rendering file.

Bon courage!

Philippe

[Adam Sherman]

I had flagged Stephan's email to work on this over the weekend and now Philippe goes into even more depth! Merci beaucoup!

A.

--
You received this message because you are subscribed to a topic in the Google Groups "Osmand" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osmand/n2vaH_VH-GU/unsubscribe.
To unsubscribe from this group and all its topics, send an email to osmand+un...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Philippe de Franclieu]

Hi Adam,

 

To help you map your data and build your python translation file?

 

OSM map features :

http://wiki.openstreetmap.org/wiki/Map_Features

Enjoy!

or

http://wiki.openstreetmap.org/wiki/FR:%C3%89l%C3%A9ments_cartographiques

 

Bon week-end.

 

Philippe de FRANCLIEU

8, rue de Liège

F - 75009 Paris

Mobile :+33 (0)6 86 58 70 14

philipped...@gmail.com

 

---

De : osm...@googlegroups.com [mailto:osm...@googlegroups.com] De la part de Adam Sherman
Envoyé : jeudi 15 octobre 2015 16:49
À : osm...@googlegroups.com
Objet : Re: Creation of OBF from SHP

[Adam Sherman]

I'm starting to work on this project, finally!

First step, is to build the translation file. The data looks like this:

INFO: Open of `Reseau_routier_SNRC_3798_SNRC__031J11.shp'

      using driver `ESRI Shapefile' successful.

Layer name: Reseau_routier_SNRC_3798_SNRC__031J11

Geometry: Line String

Feature Count: 2319

Extent: (378317.549882, 290484.285769) - (418330.228096, 320549.554187)

Layer SRS WKT:

PROJCS["NAD83_MTQ_Lambert",

    GEOGCS["GCS_North_American_1983",

        DATUM["North_American_Datum_1983",

            SPHEROID["GRS_1980",6378137,298.257222101]],

        PRIMEM["Greenwich",0],

        UNIT["Degree",0.017453292519943295]],

    PROJECTION["Lambert_Conformal_Conic_2SP"],

    PARAMETER["standard_parallel_1",50],

    PARAMETER["standard_parallel_2",46],

    PARAMETER["latitude_of_origin",44],

    PARAMETER["central_meridian",-70],

    PARAMETER["false_easting",800000],

    PARAMETER["false_northing",0],

    UNIT["Meter",1]]

IdRte: String (32.0)

Version: String (10.0)

NomRte: String (60.0)

NoRte: Real (11.0)

ClsRte: String (40.0)

CaractRte: String (40.0)

Cls_CheFor: String (2.0)

An_Classi: String (4.0)

Gestion: String (254.0)

Source: String (10.0)

Notes: String (250.0)

AQRP_UUID: String (36.0)

SNRC: String (20.0)

SNRC_Name: String (254.0)

ZONEUTM: String (2.0)

I am working with a small piece, as was suggested.

The key fields would be NomRte, ClsRte, CaractRte.

A.

[Philippe de Franclieu]

Hi Adam,

 

Have a look at http://wiki.openstreetmap.org/wiki/Highway_tagging_samples (in or out of Town)

and let us know how you will map your key fields with OSM highway tags in your translation file?

This might help too :

https://github.com/pnorman/ogr2osm-translations/commit/9ded2e83536d28ce1e7b5257a662ddd19f165aa8

 

Bon courage!

 

Philippe de FRANCLIEU

8, rue de Liège

F - 75009 Paris

Mobile :+33 (0)6 86 58 70 14

philipped...@gmail.com

 

De : osm...@googlegroups.com [mailto:osm...@googlegroups.com] De la part de Adam Sherman

Envoyé : mardi 1 décembre 2015 16:24

À : osm...@googlegroups.com
Objet : Re: Creation of OBF from SHP

[Adam Sherman]

I'm making progress! Je fais du progrès!

One of the possible values for ClsRte is Régionale and, no matter what encoding parameter I use, cp1252 or latin-1, I get an error:

KeyError: u'R\xe3\xa9Gionale'

This is where I'm looking up the value in my dict. Maybe I should try setting the translation.py encoding to cp1252?

A.

[Philippe de Franclieu]

Adam,

 

What is the Code Page of your DBF file?

Set it to 850 International MS-DOS (with DBF Commander -Tools - Set Code Page) and give it another try.

 

Bon courage!

 

Philippe de FRANCLIEU

8, rue de Liège

F - 75009 Paris

Mobile :+33 (0)6 86 58 70 14

philipped...@gmail.com

 

De : osm...@googlegroups.com [mailto:osm...@googlegroups.com] De la part de Adam Sherman

Envoyé : mardi 1 décembre 2015 17:15

[Adam Sherman]

On Tue, Dec 1, 2015 at 11:35 AM, Philippe de Franclieu <philipped...@gmail.com> wrote:

What is the Code Page of your DBF file?

Set it to 850 International MS-DOS (with DBF Commander -Tools - Set Code Page) and give it another try.

I can't really do that, under Linux or Mac OS X. The documentation for this data specifies:

Le jeu de caractères sélectionné (le « code page ») est le Windows Latin-1 (windows 1252).

So I'm passing --encoding=cp1252 to ogr2osm.py, which, from reading the source, gets used when processing the strings (line 379). Maybe I need to try a different encoding?

[Adam Sherman]

I am getting somewhere… :)

For some reason, I believed that Python 2.7.x automatically perceived strings as utf8, but apparently not. Using u'Régionale' in my translation file, plus passing --encoding='cp1252' to ogr2osm.py, works.

A.

[Adam Sherman]

Yes! Proof of concept on a small file, all the way from AQreseau's data to my tablet running OSMAnd. Looks fantastic:

[Guoping Wu]

please tell me how to fix the CP 1252 compiling error