POI Search

[Emux]

From Mapsforge past we're happy to revive a noteworthy API (resided in prototypes branch).

After communication with Karsten Groll (who worked on that), we present to you an improved version of POI API.

Details can be found in our guide: https://github.com/mapsforge/mapsforge/blob/dev/docs/POI.md

In a nutshell we provide tools and libraries for POI offline Search.
You can compile your own POI with poi-writer and use a poi reader for the search.

Note: The library is still in alpha state and is very likely to change.


I'd like to mention chronologically the work since that era,
as can be seen in parent issue: https://github.com/mapsforge/mapsforge/issues/728

- Prototype work adapted in today's Mapsforge

- POI Writer many improvements: Osmosis plugin, parameters, etc.

- POI mapping / schema various improvements and fixes

- Add Samples example with POI search

- Detailed documentation

- Organize / cleanup code

- Restructure in modules per platform

- Fix UTF-8 encoding

- SQLite R-tree wrapper with native libraries (Android NDK)

- Implement AWT support

- Fix category loading endless loop

- Add preferred language option in poi-writer

- Add find POIs by name

Note: Multilingual POIs is an incubating feature.

We'll see if it's feasible to host also poi (besides maps) in our server.

--
Emux

[Christian Kernbeis]

Hi Emux,

These are real great news, _thats_ the feature Mapsforge was IMO missing so far.
The last week I worked on providing POIs for Openandromaps and I'm very happy that a Mapsforge standard will be provided for this.
I will immediately start to support this feature and provide Mapsforge POI-Databases for my Maps.

One important thing: Does the poi-writer support Pois bound to ways (Areas) and relations too?
Is the POI-Feature already included in the latest version of Atlas for testing?

Thanks for all your efforts and
Best regards,
Christian

[Emux]

Hi Christian,

On 30/11/2015 11:54 πμ, Christian Kernbeis wrote:

These are real great news, _thats_ the feature Mapsforge was IMO missing so far.

I totally agree on that. :)

The last week I worked on providing POIs for Openandromaps and I'm very happy that a Mapsforge standard will be provided for this.
I will immediately start to support this feature and provide Mapsforge POI-Databases for my Maps.

That's great news!

One important thing: Does the poi-writer support Pois bound to ways (Areas) and relations too?

Currently we parse the nodes based on the poi-mapping (please improve it if you want).

But the library is in its infancy, so we'll enhance it more along the way.
I added a relevant issue for that.

Is the POI-Feature already included in the latest version of Atlas for testing?

Atlas / Cruiser Beta will support POI in their next version (coming soon).

--
Emux

[Ludwig]

I am just checking it out a bit and POI search is definitely a great addition to mapsforge. 

Many thanks to Emux for taking up the old code and obviously also to Karsten Groll for implementing this for mapsforge and under the mapsforge license in the first place. It should have been integrated much earlier, but it was simply something that fell on the wayside during the the project's various transformations. Until now. 

It is a very good starting point with clean code and I think with most of the heavy work out of the way adding various refinements will not be difficult.

+10, really well done.

Ludwig

 

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/565C2501.2030302%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Christian Kernbeis]

Hi Emux,

I'v taken a look at the poi-mapping.xls.
A questions:

How sub-tags are handled eg. amenity=place_of_worship and religion/denomination subtag
or  amenity=shelter and  shelter_type=*, aso..

Usually these POIs are rendered according to their subtags and, if no sub_tag is present, according to the main tag.
Is it possible to handle this with the current Version of the poi-writer?


Concerning the node/way/relation-issue: POIs bound to ways would be important, relations would be fine - however, resolving relations is a performance nightmare so IMO its no that important.


Best regards, Christian

[Christian Kernbeis]

.. is a performance nightmare, so no real issue if resolving of relations is missing.

[charles galvez]

This is great improvement for Mapsforge!  

I knew this would come.  

And hoping to have routing. Soon. 

[Emux]

On 30/11/2015 02:29 μμ, charles galvez wrote:

And hoping to have routing. Soon.

Back then, there were also routing experiments, though now we have robust implementations, like GraphHopper.

--
Emux

[Emux]

On 30/11/2015 02:23 μμ, Christian Kernbeis wrote:

How sub-tags are handled eg. amenity=place_of_worship and religion/denomination subtag
or  amenity=shelter and  shelter_type=*, aso..

Usually these POIs are rendered according to their subtags and, if no sub_tag is present, according to the main tag.
Is it possible to handle this with the current Version of the poi-writer?

Good question Christian, and that's what we need for checking the implementation.

If an element is tagged as e.g. "amenity=shelter" and "shelter_type=basic_hut",
then it will be included as in poi-mapping we already declared only the main tag.

If in poi-mapping we want to declare also the sub-tags for more fine tuning during the search,
that's recommended to be done under the "amenity=shelter" category as sub-tree.

As seen in the poi mapping schema, a category can have both mappings (for its own, see here) and sub-categories.

Is this what you're looking for?

--
Emux

[openandromaps osm]

Hi Emux,

I suppose that I have'nt got the point whats the intended use of this POI database.

If the pois would have been used as an "overlay" over the main map the tags/subtags would have to be bound to icons and this would need a poi-theme similar to the map-theme to overcome the restrictions of the built in icons, using custom ones
.. or extending the map-theme so it could act for driving the behaviour of the POIs too (and maybe replacing the graphics-only Icons on the map by more intelligent POIs)
Icons would have popup-window on touch with links to a detail-window, a kontextmenu aso...
;-))
I know a day has only 24h - sorry, just a little bit of daydreaming :-)

OK, the writer is running with Osmosis-Latest flawless and I will wait till Atlas with poi-support is out.

One last request: I'v seen that only the "name" tag is stored in the DB (right?), it would be fine to add some more tags like "phone" "url" maybe "opening_hours".
Its not necessary to add all variants of these tags eg contact:#### , website:#### this could be unified with ease by tagtranform prior to poi/map_writing.
I dont want to pain for you - however implementing one/more tags is IMO a basic design issue is hard to change once released.

thanks and kind regards
Christian

[Emux]

On 30/11/2015 06:06 μμ, openandromaps osm wrote:

If the pois would have been used as an "overlay" over the main map the tags/subtags would have to be bound to icons and this would need a poi-theme similar to the map-theme to overcome the restrictions of the built in icons, using custom ones
.. or extending the map-theme so it could act for driving the behaviour of the POIs too (and maybe replacing the graphics-only Icons on the map by more intelligent POIs)
Icons would have popup-window on touch with links to a detail-window, a kontextmenu aso...

Initial idea of the POI API was to retrieve the (filtered) POIs for an area for "using" them.
"Use" can have many meanings, certainly the first coming in mind is as overlays on top of map.
That is not use with render themes, but overlaying them by code, implement popup, etc.

Your idea for POI + render themes is certainly very interesting. We'd need to think over its possibilities.

OK, the writer is running with Osmosis-Latest flawless and I will wait till Atlas with poi-support is out.

:)
Believe me I've made many tests to ensure that.

One last request: I'v seen that only the "name" tag is stored in the DB (right?), it would be fine to add some more tags like "phone" "url" maybe "opening_hours".
Its not necessary to add all variants of these tags eg contact:#### , website:#### this could be unified with ease by tagtranform prior to poi/map_writing.

Yes currently only the name (or name:X) is stored.
I intend to enhance it by making multilingual POI (like we have now the maps).

You are right about the extra tags (I created an issue).
And since it's a DB - we can be more flexible, just need to decide which ones.

I dont want to pain for you - however implementing one/more tags is IMO a basic design issue is hard to change once released.

Don't mention it, we try to specify the best implementation for all of us.
And thank you for that, your experience is valuable here!

--
Emux

[Ludwig]

In terms of rendertheme use: if someone would put the MapDataStore interface on top of the poi db, this could come almost automatically.

I am actually in favour of making the interface to the MapDataStore slightly more powerful, e.g. having options of only retrieving ways and for retrieving pois for multiple tiles in one go, so that different sources could be combined easier and that the label layer could make use of that. That would allow a smoother transition from the current MapFile (with pois) to a solution where pois are only in the poi db but are still rendered ok (so no double storage).

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/565C8975.6050503%40gmail.com.

[Emux]

On 01/12/2015 12:28 πμ, Ludwig wrote:

In terms of rendertheme use: if someone would put the MapDataStore interface on top of the poi db, this could come almost automatically.

I created an issue for that too.

For reference, POI API is a bit independent right now. It even has its own core model structures for doing its things. So a decision to make is to combine it more with rest Mapsforge model.

I am actually in favour of making the interface to the MapDataStore slightly more powerful, e.g. having options of only retrieving ways and for retrieving pois for multiple tiles in one go, so that different sources could be combined easier and that the label layer could make use of that. That would allow a smoother transition from the current MapFile (with pois) to a solution where pois are only in the poi db but are still rendered ok (so no double storage).

I split the MapDataStore enhancements from Reverse Geocoding as it involves additional features.

--
Emux

[Christian Kernbeis]

Hi Emux,

I posted a modified poi-mapping, the post have been deleted - are there limitation with google groups?

Best regards
Christian

[Emux]

On 03/12/2015 04:27 μμ, Christian Kernbeis wrote:

I posted a modified poi-mapping, the post have been deleted - are there limitation with google groups?

Sorry, it went into spam folder, I unmarked it.

--
Emux

[Emux]

On 03/12/2015 04:27 μμ, Christian Kernbeis wrote:

I posted a modified poi-mapping, the post have been deleted - are there limitation with google groups?

I received it now in the mail, though it does not appear in the forum.
Can you resend it to have it there too?

--
Emux

[Emux]

In case anyone started playing with the POI API, there is currently an error in our search algorithm.

It has been identified, I'm in the middle of tests and will push the fix soon.

--
Emux

[Christian Kernbeis]

Hi Emux,

I worked through the poi-mapping with existing, OSM oriented, category structure.
There ist still lots of room for enhancements left - its just a first step.

Basicly I removed lots of tags like vending machines, offices, barriers, cameras, cranes, aso..
Added places, natural (peaks, vulcanos), bicycle category,
My approach was to provide pois users are regular searching for - as said its far from perfect yet.
The resulting size of the db is about 90% the size for the original poi-mapping.

For the future I would like to add a category "Activity"
including sub-categorys like
- Cycling
- Hiking
- aso..

This would mean to have same tags under different categories.
Eg: camp_sites, water_wells, springs, shelters, aso.. will appear under more than one category.
Is this possible without bloating the database, is the tag writen multiple times to the db or only referenced to different categories?

Best regards
Christian

[Emux]

Hi Christian,

Thanks for working on poi-mapping.
I'll check the xml in detail and proceed with the changes.

Forgot to mention earlier that all categories need probably more descriptive names.

On 04/12/2015 02:40 μμ, Christian Kernbeis wrote:

This would mean to have same tags under different categories.
Eg: camp_sites, water_wells, springs, shelters, aso.. will appear under more than one category.
Is this possible without bloating the database, is the tag writen multiple times to the db or only referenced to different categories?

I'm afraid having same tags under different categories is not (currently) supported.
It was not specified in such way from the start when the API was conceived.

For instance it's not like the render theme styles, where we can put an overlay under more than one styles. And then select a style and enable/disable multiple overlays on map.

With POI API if someone wants to search e.g. restaurants, he can search them directly (amenity=restaurant) or via some category "above" of them (Food, Amenities).

Of course all these are under open discussion, to select the best approach.
(that's why the API's 'alpha' state)

--

Emux

[Emux]

BTW Christian in a previous post you wrote "The last week I worked on providing POIs for Openandromaps".

Can you explain how did you intend to accomplish that, is there something we can benefit here?

--
Emux

[Emux]

I pushed the fix for the POI Search algorithm.

Essentially using a new different implementation for the filtering.

--
Emux

[Christian Kernbeis]

BTW Christian in a previous post you wrote "The last week I worked on providing POIs for Openandromaps".

Can you explain how did you intend to accomplish that, is there something we can benefit here?

Hi Emux,
Just plain GPX-Pois.
There is a real fine Tool for extracting pois, it uses a poi-mapping-file too, resolves ways and relations and is real fast:
https://github.com/MorbZ/OsmPoisPbf
Output ist csv > from here on conversion with GPSBabel or own script.

However, importing > 50.000 Pois via GPX is a pain even on fast smartphones - for me thats no solution.
.. and GPX has limited support for custom fields.

So I stopped this.

Best regards,
Christian

[Emux]

On 05/12/2015 10:05 πμ, Christian Kernbeis wrote:

However, importing > 50.000 Pois via GPX is a pain even on fast smartphones - for me thats no solution.

+1
In the past I tried also with GeoJSON, and it was again not good for such large sizes.
They cannot beat a DB performance (or memory) on that.

A complete solution is probably SpatiaLite, but we don't need (currently) such an overhead - as we don't use spatial functions in our queries.
Though (as I have written in our guide) for anyone interested, I have successfully tested our implementation with it too.
For now the R-tree extension does the job well.

--
Emux

[Emux]

Christian I merged your improved poi-mapping in repository.

(I just updated its formatting and posted your annotation in the relevant issue)

Again thanks for your valuable help!
There is certainly much space for improvements on that.


Also I think we should probably sort the categories in it for better maintenance.

--
Emux

[Razvan Calugarasu]

Regarding POI feature, why don't use this project https://github.com/kiselev-dv/gazetteer that already extract not only poi's but also full geocoded buildins, full geocoded POIs, streets, cyties, administrative boundaries and also offers extracts for some countries.

[Emux]

Thanks for the info.

The goal here is not the extraction per se, that's the 1st part.
The 2nd and more serious part involves the <offline> POI Search in Android / Java.

I agree that OSM extraction is already explored by many and can be done in various ways (Christian mentioned another tool above).

Such tools usually output the results in structured data formats (json, xml, csv) which are fine for serial reading, but not for logical (even spatial) random access queries.

Nevertheless going forward we'll see what combination will be the best.

--
Emux

[Emux]

A question about POI writing procedure.

Right now we start the process by inserting in database all the categories found in poi-mapping, regardless if they have content in the osm file or not.

Then we proceed to parse the osm and writing in the DB the matched POIs.

With that way eventually in the DB we may have categories that are empty for the specific osm.

What do you think, should we instead insert each time only the full categories per osm?

--
Emux

[Ludwig]

We could also delete the ones that are empty afterwards.

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/56635D58.1%40gmail.com.

[Emux]

Yes that's the right time doing it, starting from the empty leafs and going upwards.

--
Emux

[Christian Kernbeis]

Poi.DB's for Openandromaps based on the poi-mapping are available here:

http://ftp5.gwdg.de/pub/misc/openstreetmap/openandromaps/pois/

During development of the POI-Feature these DB's will stay in this seperate tree on the server:
More will come (for all maps provided by OAM) step by step

If there is a new version of the poi-writer available I will update these DB's

Best regards
Christian

[Emux]

Thanks Christian,

What poi-mapping did you use for them?

From the dates it isn't probably the latest published (which has your improvements in it).

--
Emux

[Christian Kernbeis]

Hi Emux,

It's the poi-mapping I posted here in the Forum

[Emux]

I added on POI Writer a final stage of DB post process, i.e. removing the categories that:

- Aren't linked to any POI
- Don't have any child category

That way we produce POI DBs that have valid full information in all aspects.

--
Emux

[Emux]

I published new Atlas Beta / Cruiser Beta with integration of Mapsforge POI Search feature, for broad testing.

In both apps you'll can specify a ".poi" file (menu vs settings) and initiate the Search from the map (right click vs long press) in visible bounding box.

At the search dialog you can:
- search for a name
- search for categories
- combine both
- none of them returns all the POIs

You can see the found POIs list later from the menu.

With it we're able to again push Mapsforge to its limits, i.e. overlay some thousands of POIs without the engine to complain! :)

(Christian above published ready POI files)

--
Emux

[Stephan75]

I tried the new POI search with fresh Atlas beta,

and a POI file from http://ftp5.gwdg.de/pub/misc/openstreetmap/openandromaps/pois/

and like mentioned in the German speaking forum of http://www.openandromaps.org

it seems that POIs are only found when they are tagged as a single node in raw OSM data.

To find POIs that are tagged as ways , try http://overpass-turbo.eu ... 

and use its wizard mode by entering for example

"amenity=restaurant and type:way in XXXX" where XXXX can be a name of a place or city where we have valid boundary relations for in OSM data.

Can anyone confirm?

Or anyone is but yes able to find POIs tagged as ways in Atlas or Cruiser beta?

Greets, Stephan

[Emux]

Thanks for the feedback Stephan.

Currently during OSM parsing we process only the nodes (not ways or relations).

We have in mind (among many other things) to include more element types in parsing (#738).

So eventually the search plays with what the database has been built.

--
Emux

[Ludwig]

There used to be an area2point (or so) plugin for osmosis, that would create a node for every area (with the same tags) in the center (whatever that is for a complex polygon). 

This can be pipelined in osmosis before the mapsforge poi extraction. 

I will try to find it again. 

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/566883BF.80207%40gmail.com.

[Emux]

On 10/12/2015 02:05 πμ, Ludwig wrote:

There used to be an area2point (or so) plugin for osmosis, that would create a node for every area (with the same tags) in the center (whatever that is for a complex polygon). 

This can be pipelined in osmosis before the mapsforge poi extraction.

Yes that would be an easy way to feed the poi-writer with the appropriate data.

Or else we'd have to (re)implement it ourselves (e.g. map-writer).

BTW it's already asked to not limit with POI (i.e. strictly points), but to include also ways (i.e. not converted but full lines / polygons) in the search. We'll need to think about it..

--
Emux

[Ludwig]

Full lines can be real trouble as OSM not only divides a lot of areas into several segments that we would need to assemble, but also divides roads into segments at intersections or when anything changes (e.g. a road with a bridge in the middle will show up as three parts), even though from a human perspective nothing changes (it is still Oxford Street for example).

It does not even help to merge roads based on name and shared points as some roads do not share points even though they are the same road (take a divided highway, lanes running in parallel).

For my own stuff I do quite extensive preprocessing, that merges OSM ways by name and proximity into roads. 

I think for a lot of stuff we are reaching the limits of osmosis. I think osmium is a better choice as it has processing for creating areas etc built-in. Osmium is C++ based, which does not play nice with gradle last time I checked. 

Basically we are moving into a reimplementation of lots of stuff that has been done elsewhere. Danger.

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/56692FC7.1090306%40gmail.com.

[Emux]

I agree we should think twice our steps here, our resources are limited for opening just too many doors..

The main question I received is how to make address search with the new API.

Something that can be solved easily by transforming the road segments w/ house numbers to center points, but still thinking a better solution.

--
Emux

[Ludwig]

I think we should define an interface for the search and provide a sample implementation that can be pretty simple. This interface could either be a Java software interface or, a bit lower, a DB schema. 

If we defined a DB schema it would be up to implementers to implement their own scheme for filling the DB. That would decouple us from OSM data and also from a specific implementation language, so e.g. writing an OSM extractor with Osmium would be ok as long as it fills a conforming DB.

Defining a Java interface would give a bit more flexibility, e.g. allowing online search.

Maybe we should do both. 

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/566952EA.6050009%40gmail.com.

[Emux]

+1
Nice ideas (need to think over it first).

For start now that the POI API is more mature, I intend anyway to review the DB schema, in order to organize also our waiting features (multilingual, more tags, etc).

--
Emux

[Emux]

To better organize our work I added a new 'poi' label in our repository.

So now to view POI related issues you can search them with something like:
https://github.com/mapsforge/mapsforge/labels/poi

Tip: the rest (mainly map) issues can be filtered with -label:poi syntax:
https://github.com/mapsforge/mapsforge/issues?utf8=%E2%9C%93&q=is%3Aopen+-label%3Apoi+

--
Emux

[Christian Kernbeis]

Hi Ludwig,

There is an incredible fast tool for extracting pois (nodes/ways/relations), written in java and it accepts pbf as source.
The output is csv - however, the basic code may be usefull.
In my experience big files like germany.osm.pbf are no problem, the author claims that even the planet-latest is possible with 9GB RAM.

https://github.com/MorbZ/OsmPoisPbf

There is a further method of converting relations and ways to nodes: osmconvert with param --all-to-nodes
I use this method for calculation of landuses+natural areas and setting labels for these tags.
However, on windows there are some limitations (strict 32bit) and the whole process envolves osmfilter + osmconvert in a time consuming toolchain.


Best regards
Christian

[Emux]

Thanks Christian for the information.

I'm familiar with Osmconvert. Tools like that seem appropriate, as they can be used directly in conjunction with our current tool chain and help in POI production. We'll need to check them.

--
Emux

[womisa]

Hi

can you create a (simple) (public) database model, for example, this http://renenyffenegger.blogspot.de/2014/09/openstreetmap-convert-pbf-to-sqlite.html
or this https://alastaira.wordpress.com/2011/04/15/loading-open-street-map-data-in-sql-server-part-ii-ways/. ?

So that everyone has their own POIs can generate with his tools and store this in the (Mapsforge)-database with a public interface.....


Thanks
Achim

[Emux]

Thanks for the links.

To clarify a bit the things here. We don't mean to create a new OSM storage type or complex GIS solution.
Not wanting to describe an OSM file 'as is' in a DB - without spatial features.
And certainly not a server based DB (e.g. PostgreSQL/Postgis).

There is an ultimate goal, that is "offline search" (starting with POI).

For that we need a DB structure that can accept "spatial" queries.
We go with SQLite, as it's the most simple and server-less DB, and the 1st candidate was SpatiaLite.
We tried it, it works, but it's much more complex for simply area queries and hard to maintain.
So what follows is a R-tree implementation which actually works pretty well.

Our current model can be seen here - though we may improve it with careful steps.
As I have already mentioned it's in alpha state, we have much work ahead and organize.
https://github.com/mapsforge/mapsforge/labels/poi

In the near future I intend to push work in writer for extra tags, multilingual poi, metadata, and many others.
And let's not forget the ways processing.

--
Emux

[Emux]

For anyone playing with POI on Android, I've put SQLite wrapper in its own module, for better handling it via dependencies (and possibly swapping it in the future).

--
Emux

[Emux]

I pushed a new version of POI Writer introducing the bounding box option.
https://github.com/mapsforge/mapsforge/blob/dev/docs/POI.md#basic-options

That way we're able to specify explicitly which data to process from a large osm/pbf source.
(like we do with map-writer)

--
Emux

[Emux]

I pushed another batch of updates regarding POI API.

- Add 'comment' option in poi-writer.

- Introduce POI DB spec version (similar to map file).
That could become handy in the future with possible DB schema changes.

- Implement POI DB metadata via a new table.
e.g. [bounds, comment, date, language, version, writer]


Just published Beta versions of Atlas / Cruiser include all above + new 'POI info' tool.

You may need updated poi files, as validation checks also for 'metadata' existence.

--
Emux

[womisa]

Hi Emux,

thanks for the new version. I have generated a new poi database poi from Baden-Württemberg (Windows). This works for me with Atlas very well.

Question: Plan you the search possibilty for  example more tags and/or for the POI display?
...
amenity=Restaurant  or amenity=restaurant (no case sensitive)
name=*besen*  all names include "besen"

Thank for your work!
Achim


Am Sonntag, 29. November 2015 18:39:46 UTC+1 schrieb Emux:

From Mapsforge past we're happy to revive a noteworthy API (resided in prototypes branch).

After communication with Karsten Groll (who worked on that), we present to you an improved version of POI API.

Details can be found in our guide: https://github.com/mapsforge/mapsforge/blob/dev/docs/POI.md

In a nutshell we provide tools and libraries for POI offline Search.
You can compile your own POI with poi-writer and use a poi reader for the search.

Note: The library is still in alpha state and is very likely to change.


I'd like to mention chronologically the work since that era,
as can be seen in parent issue: https://github.com/mapsforge/mapsforge/issues/728

- Prototype work adapted in today's Mapsforge

- POI Writer many improvements: Osmosis plugin, parameters, etc.

- POI mapping / schema various improvements and fixes

- Add Samples example with POI search

- Detailed documentation

- Organize / cleanup code

- Restructure in modules per platform

- Fix UTF-8 encoding

- SQLite R-tree wrapper with native libraries (Android NDK)

- Implement AWT support

- Fix category loading endless loop

- Add preferred language option in poi-writer

- Add find POIs by name

Note: Multilingual POIs is an incubating feature.

We'll see if it's feasible to host also poi (besides maps) in our server.

--
Emux

[Emux]

Thanks for the feedback.

Actually yes, the extra tags #739 and multilingual POI #737 are the next candidates. :)

Then of course will follow the searching on any fields #759.

--
Emux

[Emux]

On 18/12/2015 08:56 μμ, womisa wrote:

amenity=Restaurant  or amenity=restaurant (no case sensitive)

Good remark, currently I think we parse poi-mapping 'as is'.
More flexibility wouldn't be bad.

name=*besen*  all names include "besen"

The POI API can accept a pattern in name search (via 'LIKE').
e.g. in your example you can pass "%besen%" in code as pattern

In Atlas / Cruiser I've set already that pattern, so it just needs the plain name "besen".

--
Emux

[womisa]

...  OK. I think provide query the database the power and flexibility would be a useful feature....

Best regards
Achim

[Emux]

On 18/12/2015 08:56 μμ, womisa wrote:

amenity=Restaurant  or amenity=restaurant (no case sensitive)

I pushed a new poi-writer which - among other minor improvements - has case insensitive parsing of category & POI tags.

--
Emux

[Emux]

I pushed a new POI writer which parses also ways, besides nodes.

A new option ways is introduced for that (requiring more RAM).

The procedure can be optimized more, for now please test the results.


(If we need in the future, we'll think also about relations)

--
Emux

[womisa]

Hi Emux,

i have tried your new version without success.

>cmd /K ..\_OSMOSIS\20151217\bin\osmosis --rb file=..\_OSM\bw.pbf --poi-writer file=bw.poi preferred-language=en tag-conf-file=poi-mapping.xml

>>osmosis set
...
SET EXEC="%JAVACMD%" %JAVACMD_OPTIONS% -Xmx9g -cp "%PLEXUS_CP%" -Dapp.home="%MYAPP_HOME%" -Dclassworlds.conf="%MYAPP_HOME%\config\plexus.conf" %MAINCLASS%  %OSMOSIS_OPTIONS% %*
.....


After four hours,


D:\PoiGenerator>cmd /K ..\_OSMOSIS\20151217\bin\osmosis --rb file=..\_OSM\bw.pbf --poi-writer file=bw.po
i preferred-language=en tag-conf-file=poi-mapping.xml
Dez 20, 2015 7:03:38 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Osmosis Version 0.44.1
Dez 20, 2015 7:03:39 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Preparing pipeline.
Dez 20, 2015 7:03:39 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: POI writer version: mapsforge-poi-writer-dev-SNAPSHOT
Dez 20, 2015 7:03:39 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: POI format specification version: 1
Dez 20, 2015 7:03:39 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Loading POI categories from XML...
Dez 20, 2015 7:03:39 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Adding tag mappings...
Dez 20, 2015 7:03:39 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Creating POI database...
Dez 20, 2015 7:03:39 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Launching pipeline execution.
Dez 20, 2015 7:03:39 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Pipeline executing, waiting for completion.

...no return...Endless Loop....?

The files
bw.poi              22.853 KB       ..i can not oben with the SQLITE Firefox Plugin
bw.poi-journal          93 KB

bw.pbf is the baden-wuerttemberg-latest.osm.pbf from Geofabrik

Greetings
Achim

[Emux]

I appreciate the quick feedback.

If you pass "ways=false" all work normal with nodes, right?
I tested your area with that option 'false' and it was completed in 1 minute.


But ways processing is currently hard on requirements, I'll review the implementation again (there are ideas to test).

BTW your unfinished DB is not workable because it's in a middle state of construction with open temp (journal) files.

--
Emux

[womisa]

Am Sonntag, 20. Dezember 2015 22:11:40 UTC+1 schrieb Emux:

I appreciate the quick feedback.

If you pass "ways=false" all work normal with nodes, right?
I tested your area with that option 'false' and it was completed in 1 minute

.That goes for me too

.

>cmd /K ..\_OSMOSIS\20151217\bin\osmosis --rb file=..\_OSM\bw.pbf --poi-writer ways=false file=bw.poi preferred-language=en tag-conf-file=poi-mapping.xml
 

Best regards
Achim
 

 

 

[Emux]

I published poi-writer with 'ways' option default = 'false' for user convenience.

We'll keep it that way until we improve further the ways processing for large areas.


Remember you can always use 'bbox' option for limiting the POI results.

--
Emux

[Emux]

On 20/12/2015 10:46 μμ, womisa wrote:

After four hours,

I pushed a new POI writer optimizing the Ways process by following a completely different implementation.

The writer performance is now vastly improved - stats can speak for themselves:

- The above region can be processed with Ways enabled in under 5' (from hours)
- With memory consumption only 40% more than with Nodes (from GBs)

If all are right we have made a significant progress, and it's just the start..

--
Emux

[womisa]

Hi Emux,

this new version works for me very fast! Thank you for tis version.

D:\VirtualBox\Shares\Internet\Downloads\OSM\ConverterTools\PoiGenerator>cmd
.\_OSMOSIS\20151217\bin\osmosis --rb file=..\_OSM\bw.pbf --poi-writer ways=
file=bw.poi preferred-language=en tag-conf-file=poi-mapping.xml
Dez 21, 2015 7:08:52 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Osmosis Version 0.44.1
Dez 21, 2015 7:08:52 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Preparing pipeline.
Dez 21, 2015 7:08:52 PM org.mapsforge.poi.writer.logging.LoggerWrapper log

INFORMATION: POI writer version: mapsforge-poi-writer-dev-SNAPSHOT

Dez 21, 2015 7:08:52 PM org.mapsforge.poi.writer.logging.LoggerWrapper log

INFORMATION: POI format specification version: 1

Dez 21, 2015 7:08:52 PM org.mapsforge.poi.writer.logging.LoggerWrapper log

INFORMATION: Loading POI categories from XML...

Dez 21, 2015 7:08:52 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Adding tag mappings...
Dez 21, 2015 7:08:52 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Creating POI database...
Dez 21, 2015 7:08:52 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Launching pipeline execution.
Dez 21, 2015 7:08:52 PM org.openstreetmap.osmosis.core.Osmosis run

INFORMATION: Pipeline executing, waiting for completion.

Dez 21, 2015 7:08:52 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Processing nodes...
Dez 21, 2015 7:10:04 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Processing ways...
Dez 21, 2015 7:11:30 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Committing...
Dez 21, 2015 7:11:31 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Finalizing database...
Dez 21, 2015 7:11:48 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Writing metadata...
Dez 21, 2015 7:11:48 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Cleaning database...
Dez 21, 2015 7:12:00 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Optimizing database...
Dez 21, 2015 7:12:03 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Added 449.972 POIs.
Dez 21, 2015 7:12:03 PM org.mapsforge.poi.writer.logging.LoggerWrapper log
INFORMATION: Estimated memory consumption: 213,09MB
Dez 21, 2015 7:12:03 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Pipeline complete.
Dez 21, 2015 7:12:03 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Total execution time: 191088 milliseconds.   <==================  Baden-Wuerttemberg.osm.pbf !!!!!!!!!!!!!!

Best regards
Achim

[Emux]

Glad to hear that :)

I can push the speed even further, but I preferred to stay a bit conservative for the sake of memory.

If there is need we can put a ram/hd option (like in map-writer) in the future.

Let's test it more first, we should see if we process correctly all needed Way POI.
e.g. double check with http://overpass-turbo.eu (loading the Map Call example)

--
Emux

[womisa]

Hi Emux,

I have another problem.....

The POI-Writer works now very fine, but

 if I add (copy) Mapwiter-Plugin into  in the ..osmosis Mapwriter / lib / default Direktory the Poi-Writer no longer works

D:\VirtualBox\Shares\Internet\Downloads\OSM\ConverterTools\PoiGenerator>cmd /K .
.\_OSMOSIS\20151217\bin\osmosis --rb file=..\_OSM\bw.pbf --poi-writer ways=true
bbox=48.8679,8.7386,49.0052,9.1111 file=bw.poi preferred-language=en tag-conf-fi
le=poi-mapping.xml
Dez 21, 2015 11:10:45 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Osmosis Version 0.44.1
Dez 21, 2015 11:10:45 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Preparing pipeline.
Dez 21, 2015 11:10:45 PM org.openstreetmap.osmosis.core.Osmosis main
SCHWERWIEGEND: Execution aborted.
java.lang.RuntimeException: POI format specification version is not an integer
        at org.mapsforge.poi.writer.osmosis.PoiWriterTask.<init>(PoiWriterTask.j
ava:65)
        at org.mapsforge.poi.writer.osmosis.PoiWriterFactory.createTaskManagerIm
pl(PoiWriterFactory.java:73)
        at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactory.cre
ateTaskManager(TaskManagerFactory.java:60)
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.buildTasks(Pi
peline.java:50)
        at org.openstreetmap.osmosis.core.pipeline.common.Pipeline.prepare(Pipel
ine.java:112)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:86)
        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Laun
cher.java:330)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
a:238)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
uncher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
356)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)
Caused by: java.lang.NumberFormatException: null
        at java.lang.Integer.parseInt(Unknown Source)
        at java.lang.Integer.parseInt(Unknown Source)
        at org.mapsforge.poi.writer.osmosis.PoiWriterTask.<init>(PoiWriterTask.j
ava:57)
        ... 15 more

D:\VirtualBox\Shares\Internet\Downloads\OSM\ConverterTools\PoiGenerator>

If I delete the Map-Writer-Plugin, the PoiWriter working again.....


What am I doing wrong again?

Best regards Achim

[Emux]

Thanks for the report.

I can confirm the issue and will look at it.

Note: the problem does not exist if we have the two jars in our profile (2nd instruction).

But we put them inside Osmosis libraries folder then the class loader confusion starts (due to common resources paths).

--
Emux

[Emux]

I pushed new versions of Map + POI writers fixing the aforementioned issue.

--
Emux

[womisa]

Thanks for your new versions. This works for me now very well!

I wait with joy for the upcoming new POI features.....

Thank you and Merry Christmas
Achim

[Christian Kernbeis]

Hi EMUX

I modified the map creation process of my maps and now its possible to create updated pois for all of the 281 maps at OAM within 24h.
Next update will be online on 24th of December (tomorrow) 8.00am MEZ

http://ftp5.gwdg.de/pub/misc/openstreetmap/openandromaps/pois/

Thanks for all your efforts
and
Marry Christmas

Christian
www.openandromaps.org

Am Dienstag, 22. Dezember 2015 10:10:35 UTC+1 schrieb Emux:

[Emux]

Thanks for the support Christian.

Wishing the best for all

--
Emux

[Emux]

I pushed the latest work on POI API:

- POI writer 'all-tags' boolean option #739
Export all tags. If false only the name will be exported (with 'preferred-language').

- Multilingual POI #737
Combined with above, we can handle multilingual POI via new PointOfInterest.getName methods.

I opted for code (similar to map) instead of DB implementation, more simple, more efficient.
If all tags are present, name retrieval is based on given language, fall back to base one and default name in that order.

- Search on any tag #759

As a bonus we can now search on the whole of the POI tags.

These features can be improved further, so let's start playing to see them in action.

I published also new Beta versions of Atlas / Cruiser, supporting the new POI API.
Since there aren't fixed POI languages (like in map), POI names are currently extracted based on device locale.

--
Emux

[womisa]

Hi Emux,

i have download the new version, but i have a execution error. What is wrong (Windows)?

Thanks
Achim

D:\VirtualBox\Shares\Internet\Downloads\OSM\ConverterTools\PoiGenerator>cmd /K .

.\_OSMOSIS\20151217\bin\osmosis --rb file=..\_OSM\bw.pbf --poi-writer all-tags=t
rue ways=true file=bw.poi preferred-language=en tag-conf-file=poi-mapping.xml
Dez 26, 2015 10:32:19 PM org.openstreetmap.osmosis.core.Osmosis run
INFORMATION: Osmosis Version 0.44.1
Dez 26, 2015 10:32:19 PM org.openstreetmap.osmosis.core.Osmosis main
SCHWERWIEGEND: Execution aborted.
org.openstreetmap.osmosis.core.OsmosisRuntimeException: Task type "pw" already e
xists.
        at org.openstreetmap.osmosis.core.pipeline.common.TaskManagerFactoryRegi
ster.register(TaskManagerFactoryRegister.java:44)
        at org.openstreetmap.osmosis.core.TaskRegistrar.loadPluginClass(TaskRegi
strar.java:357)
        at org.openstreetmap.osmosis.core.TaskRegistrar.loadPlugin(TaskRegistrar
.java:313)
        at org.openstreetmap.osmosis.core.TaskRegistrar.loadBuiltInPlugins(TaskR
egistrar.java:123)
        at org.openstreetmap.osmosis.core.TaskRegistrar.initialize(TaskRegistrar
.java:80)
        at org.openstreetmap.osmosis.core.Osmosis.run(Osmosis.java:81)

        at org.openstreetmap.osmosis.core.Osmosis.main(Osmosis.java:37)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.lang.reflect.Method.invoke(Unknown Source)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launchStandard(Laun
cher.java:330)
        at org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.jav
a:238)
        at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(La
uncher.java:415)
        at org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:
356)
        at org.codehaus.classworlds.Launcher.main(Launcher.java:47)

D:\VirtualBox\Shares\Internet\Downloads\OSM\ConverterTools\PoiGenerator>

[womisa]

Hi Emux,

sorry it was my fault (its to late)

Thanks for the new version
Achim

[Emux]

I improved POI Search example in our Samples, to show the multilingual names use.

- Changed POI category to something that usually exist in several languages.
(feel free to propose something better for the example)

- With tap on POIs, their name appears in device's locale (with usual fall back mechanism).

--
Emux

[vklein]

Build successfully the samples app from the dev branch.


Where I can find the germany.poi file?

Regards

Victor

[Emux]

We have not started yet to provide poi files for the world.

You can build it following the instructions in our guide:
https://github.com/mapsforge/mapsforge/blob/dev/docs/POI.md

Also Christian (OpenAndroMaps) provides regular poi builds here for testing.

Reminder: it's not necessary to have actual germany map or poi files, just any area renamed to that filenames for convenience.

--
Emux

[vklein]

Thanks!

Runs perfect with the berlin.map and berlin.poi

[openandromaps osm]

Hi Emux,

I have problems with the preferred-language option of the poi-writer.

I'v made two poi-db's for Mockba.map (Moskau), in the opprobriate langs the maps are:
Mokba  (preferred-language=en)
Mokba_N (no preferred lang parameter = map + poi in native language)

www.openandromaps.org/wp-content/files/pois/MOCKBA.zip (english)
www.openandromaps.org/wp-content/files/pois/MOCKBA_N.zip (natiive = no preferred-language)

Maps made with map-writer5
Pois with latest poi-writer-snapshot

Poi DB's :
www.openandromaps.org/wp-content/files/pois/MOCKBA.poi (english)
www.openandromaps.org/wp-content/files/pois/MOCKBA_N.poi (natiive = no preferred-language)

Both DB's are of exact same file-lenght, the header IMO shows the correct language.

However, when I open the pois in ATLAS and check in the poi categories "Airports" click on "all" the Name of the Monino Airfield
http://www.openstreetmap.org/node/1042058145#map=16/55.8364/38.1695
is always shown in english (name:en)





I'm pretty shure that I'm doing something wrong here......

Best regards,
Christian

[Emux]

Hi Christian,

(I mentioned something above, but probably should clarify better the documentation)

- 'all-tags=true' option (which you use) does not actually need the 'preferred-language' option, since essentially stores *all* the POI tags without filtering.

- 'preferred-language' option is only useful with 'all-tags=false', since it stores *only* the "name" at preferred language (with fallback mechanism).

- Since we don't have a fixed set of languages at POIs creation (as we do in multilingual maps), I opted for Samples and Atlas / Cruiser to show the name in device locale (with fallback again).

So in your example the name:en is shown because the Java finds you have english locale in your machine?

Note: all there are under research in order to find the best implementation.

--
Emux

[openandromaps osm]

Hi Emux,

Thanks for clearification!
pois on
http://ftp5.gwdg.de/pub/misc/openstreetmap/openandromaps/pois/
are updated every time I notice that a new poi-writer is out.

Here is a readme with version info, changelog:
http://ftp5.gwdg.de/pub/misc/openstreetmap/openandromaps/pois/README.txt

Best regards and a happy new year
Christian

[Emux]

Christian apparently you're using the oam-mapping.xml ?

If you have not made any changes since the last time I merged it in Mapsforge, could you use the official one?

The only changes I made back then were about formatting.
It will be far easier in the future to merge improvements if we have a common base.


Best wishes for the new year from me too!

--
Emux

[openandromaps osm]

Hi Emux,

Done, oam-mapping and poi-mapping are now the same.
I'currently updating the OAM-Maps (Europe/Germany) so new Version of pois will be online in 3 days.

Best regards, Christian

[womisa]

Hi Emux,

i have made a simple "hack" for this table model ==> http://renenyffenegger.blogspot.de/2014/09/openstreetmap-convert-pbf-to-sqlite.html

I have written a StaxParser with generate a text file with insert statments an then i include this with the following methode ==> http://erictheturtle.blogspot.be/2009/05/fastest-bulk-import-into-sqlite.html

My steps:
-download "baden-wuerttemberg-latest.osm" from the Geofabrik
-translate to SQL file >java -jar StaxTest.jar baden-wuerttemberg-latest.osm > baden-wuerttemberg-latest.sql
-create the database >sqlite3 baden-wuerttemberg-latest.sqlite < BW_command.txt > ausgabe.txt

duration: ca. 30 min database: 1,6 GByte

Now i cani test this with the the Firefox SQL Lite Manager Plugin ...search Streets, Addresses,BicyleRoutes....etc.

My goal is to extract for example bicycle routes, Ways ..etc. as gpx and show this selectable in a MapsforgeLayer......

I would also like this tables compatible with the MFG POI Model making (Views,? Your Tables ... etc.)

What do you think about it?

Thank you for your attention
Achim

Ps.: If unimportant simply ignore.....








Am Montag, 14. Dezember 2015 18:38:35 UTC+1 schrieb womisa:

Hi

can you create a (simple) (public) database model, for example, this http://renenyffenegger.blogspot.de/2014/09/openstreetmap-convert-pbf-to-sqlite.html
or this https://alastaira.wordpress.com/2011/04/15/loading-open-street-map-data-in-sql-server-part-ii-ways/. ?

So that everyone has their own POIs can generate with his tools and store this in the (Mapsforge)-database with a public interface.....


Thanks
Achim

[Emux]

I had examined that model and AFAIK it translates the whole osm file into a DB? Retaining its complete structure.

- I'm not sure if that's we need, the size requirements could be enormous, equivalent to the osm xml files (i.e. keeping even the geometries accuracy intact).
That could work nicely with server DBs, but with offline exchangeable files..?

- Also we need first of all spatial support in our structure. Simply a 1-1 storage for name string searching will not suffice. Advanced queries like: in a box, around a point, across a route, etc. are important as well.

So at the end, we need to define (I'm asking opinions here!) what is our ultimate goal with this API?
What do we intend to do with it?
Answering that will help us put exact specifications at the DB.


Note: like in map format, we need always to hunt the compact size and fast performance without neglecting the base which is the spatial use on maps.

--
Emux

[womisa]

I can shrink the database from unnessesary Infos in the Future with the SQL-Feature functionality...... But for now i play with the whole information...

This works for Baden-Württemberg (OSM) very well. My next Goal is the extraction from bicycle routes as GPX and load in to Atlas.....
Until now i can not the limit.

Thanks for your response
Achim

[Emux]

I integrated SpatiaLite (instead of custom SQLite wrapper) in Android POI API.

That will give us more flexibility (and compatibility) in the future.

For convenience we host the needed sources and native library files.

Anyone wishing to build them, the full package exists at its site.

--
Emux