Online vector tile server

[Bokrash]

Hello everybody,

I represent a start-up company and we are actually developing a project which uses vector maps. We’ve found Mapsforge maps very attractive, though there are some limitations regarding online vector tile sources.

I mean, for example, when the user wants to localize himself on the map, the application would request first the local cache (SQLite db); if not connected to the Internet (offline mode) it tries to request the offline map in the file, if one exists; if not found - it should request tiles from the online source.
Though coming from different sources, they all would have the same format as they are based on the same base map, Mapsforge in our case. Finally we would have an integral solution: several sources, adaptive approach to choose one of them, the same tile format. Plus, using the same format the cache would store the only tile for all these sources.

For raster maps, Mapnik for example, offers an excellent online tile server, really open without limits, but it does not offer offline maps; for vector maps, Mapsforge offers excellent offline maps, but it does not offer the online source. So why not to have both for vector tiles? 
By the way, most vector tile providers offer the limited access, - Mapbox for example (number of requests per year, as far as I know).

From your forum I could understand developer’s community is interested enough on having open online vector tile servers for Mapsforge maps, and there is no one for now. It is also said one of the most important causes is the encoder/decoder missing.

If I’m right, I’d like to discuss a common solution to help developers to fit their needs.

1. We’ve created the tile encoder/decoder pair for Mapsforge vector tiles for online use. For testing purposes we are using the VTM project and it works fine.

2. We'd like to set up the open online vector tile server (at least it is needed for our project), but we’d like to know in advance if the community will be interested in it.

Will you find this interesting and, if so, we’d like to discuss this further in details.

Best regards,
Bokrash

[Emux]

That seems interesting and certainly we can include in VTM or Mapsforge any new tile decoder which would offer more map source options to users.

Note that Mapsforge vector map file format was created with offline vector maps in mind.
Plus it was designed with microdegrees coordinate accuracy, mostly for outdoor usage.

For online vector map tiles, there are more suitable (open source) formats, like Mapbox MVT or GeoJSON vector tiles.
They are already well established in community and used by many map libraries and applications.
Also are supported by VTM and their tile decoders are available in the library.

Several providers exist for them, like Nextzen (after Mapzen) or OpenMapTiles, besides Mapbox.
And can find instructions online to build an own server providing such vector tiles.

Mapsforge remains the recommended choice for an offline vector map experience.

Raster maps cannot be offline - except small areas - as require huge capacity to store a whole country, also cannot change their map styles.

Regarding the cache, VTM is extremely fast, with Mapsforge maps usually don't even need a cache on rendering. :)

Any maps to appear the same, would need to include similar OSM tags and use a common theme schema.

--
Emux

[Bokrash]

Hi Emux,

Thank you for your reply and comments.

We have taken a look at VTM and Nextzen online tile services, - they are great.
Nevertheless, concerning tile storage optimization we could find useful to have the same tile format for online/offline maps in local cache, as it could be crucial for end users who don't have a good/fast/regular/low-cost Internet connection.

For now we have setup a local tile server for Mapsforge tile format encoding/decoding testing. It seems working fine. There are several results below... (up to zoom level 16).

Any comments and suggestions would be very appreciated!

Best regards,
Bokrash

[Emux]

If there is a pull request available for VTM (better in separate module) in order to review a new vector tile decoder together with a provider, all would be interested to test it. :)

--
Emux

[Bokrash]

Sure! We are carrying out some tests and solving some problems concerning tiles' loading. Next we'll migrate our database to the external server, it's a question of few days I hope...

Have a nice week-end.
Bokrash

[Emux]

BTW users would be interested also for the process that can produce those vector tiles which map library can decode.

--
Emux

[Bokrash]

Hi Emux,

I've set up an online server for Mapsforge vector tiles and uploaded a project with encoder/decoder to GitHub : https://github.com/ikroshlab/VTM/tree/master
For a moment there is only Basse-Normandie.map up to zoom level 17 for testing purpose.

Concerning map tiles I could notice some defects ( there are blue lines on the map from zoom level 9, render VtmThemes.TRONRENDER ), I suppose it is due to wrong encoding of "natural - water" in ways, but I'm not sure, so if you have any suggestions it would be helpful...

Any comments are welcome!

Best regards,
Bokrash

[Emux]

Thanks!

Hadn't the time to check anything yet, some quick notes:

Does the library module actually contain the whole VTM code in it and it has also mixed the encoder+decoder?
That way it's extremely difficult for anyone to study the new code changes.

It's expected to have 2 new separate and strictly Java modules:
- one for encoder (like Mapsforge map-writer)
- one for decoder (like VTM vtm-mvt).
They can have the VTM core or any other as Gradle dependencies. No need for Android libraries or aar!

Then the app can use only the light decoder module.
As the encoder module is expected to be heavier in dependencies and is mostly uneeded on the client.

--
Emux

[Bokrash]

We uploaded the project firstly for demo purpose to have a general look at the idea :)    Below there are some details.

The library module actually contains the original VTM library 0.5.9, plus org.oscim.android subpackage 0.5.9. We added a new subpackage org.oscim.tiling.source.mapsforgeonline which includes ForgeOnlineTileSource + TileDecoder classes. The last one is the purely isolated decoder for Mapsforge online tiles. Thus we can isolate this subpackage into the separate module of course.
Encoder part is now integrated into org.oscim.tiling.source.mapfile class for two reasons: 1. demo and test purpose 2. to make it possible to use local db cache for both online and offline Mapsforge tiles.  This encoder also exists as a separate package on the server side (pure Java SE), which can be uploaded to GitHub. We'll provide the source package with needed jars, so you could build a project from sources. (We used NetBeans IDE to build it).

So we'll upload both encoder and decoder separate parts to GitHub in few hours:)

Best regards,
Bokrash

[Bokrash]

Hi Emux,

We have just uploaded two separate packages (encoder + decoder) to GitHub.

There will be some issues to find a workaround for :
    - we worked with org.oscim version 0.5.9 and the GeometryBuffer class has a deprecated field "int pointLimit"
      and two methods were added: "getTypeInt()" and "setTypeInt(int)" for performance reason. The problem is that the MapElement class extends from this class
      and it's a part of ITileDataSink interface.
    - to use local DB cache effectively we had to change the ITileDecoder interface. If not needed, it can be removed but encoder/decoder parts must be revised.
    - for the same reason the ITileCache interface has also be changed.

Best regards,
Bokrash

[Emux]

Users wanting to try the process may find it difficult to follow or organize. :)

A recommended approach would be:

- Every code should be based on latest VTM as is under active development right now (in Mapsforge repository). Release 0.10 is ahead of us!

Of course rebase on latest snapshot if mean at some time to merge with upstream.

- Create regular Java (no Android) modules for any separate work (decoder or encoder) which is plain Java.
Note VTM is multi-platform library (Android, iOS, Desktop, Browser) with platform agnostic API.

- Use proper dependencies inside build.gradle files (like vtm core module), instead of copy whole VTM packages or classes. It will never work with upstream that way.

All these if mean to work together with current VTM library or meant build only an isolated "proof of concept" example?

I'll be happy to review any such pull request(s) which follow the common contribution guidelines and which should be easier to study without spending time searching for changes. :)

--
Emux

[Bokrash]

Hi Emux,

Thank you for your remarks!

We have just uploaded two projects to GitHub:  (https://github.com/ikroshlab/VTM )

   - VTMencoder: encoder package to encode map tiles from Mapsforge original map file to a byte array.
     This byte array can be put further into a database or so.
     As you suggested, this project keeps reference to VTM_0.9.2 as a Maven dependency, developed under NetBeans, pure Java SE.

   - VTMexample: this is the client part for the VTMencoder - it demonstrates how to get map tiles from online datasource using
     the ForgeOnlineTileSource.class and decode them.
     This project also keeps reference to VTM_0.9.2 as a Gradle dependency.
     This project is developed under Android Studio for now.

Nevertheless, there are some issues to fix before go further: (see attached screen-shots)
- after decoding, all the tiles look separated
- each tile has remarkable "mosaic" structure after zoom level 11.

We had all our best to fix these problems, but we failed :(
So I hope we could find a solution together. It is really important because without it there is no sense to go further. 
We'll surely try to follow your suggestions about the common approach of processing, but for now we have to focus on the above problems, IMHO.

Kind regards,
Bokrash

[Bokrash]

Hi Emux,

We had some problems with our online tile server during last days, it's ok for now... Sorry for possible troubles...

Best regards,
Bokrash

[Gustl22]

@Bokrash: this is a nice feature. Do you plan to start a pull request? Then Emux (or others) can review or help with mentioned issues. Or should it remain in its own repo? See post of Emux on 4th Mai.
Would be a pity about the done work.

[Bokrash]

Hi,

I started two pull requests - one for encoder in mapsforge/mapsforge repository, and the second one for decoder in mapsforge/vtm repository.
Many thanks to Emux and Gustl22 for clear explanations about how to do that! It is my first pull request, so I hope I've done it right way...

Kind regards,
Bokrash

[Emux]

Thanks for the pull requests!

I posted some quick remarks in both.
More concerning their needed remodeling so that anyone can study / test easily their code.

--
Emux

[Carlos Alberto Martínez Gadea]

Did this project go any further? I'd be keen to give it a try to Mapsforge vector tiles to use them as online source as well.

El divendres, 22 juny de 2018 8:30:07 UTC+2, Emux va escriure:

[Bokrash]

Hello,

unfortunately I had no time to advance the project, I hope I'll be back soon... :)

For now, as  I remember, all works fine, BUT after decoding - each tile has remarkable "mosaic" structure after zoom level 11. (see my comment above, with two screenshots).

Bokrash

[Carlos Alberto Martínez Gadea]

I saw your comment, just thought that since it was a while ago, there might have been progress with this.

Thank you anyway and looking forward to implement a 100% Mapsforge tile experience, both for online and offline tiles.

[Emux]

> looking forward to implement a 100% Mapsforge tile experience, both for online and offline tiles.

See also above, there are already available online vector tile formats optimized for that purpose and they're well established and tested from community working with all libraries / apps.

Offline vector maps are a different story, there is not replacement for Mapsforge (which is optimized for offline use), besides the offline MBTiles vector databases (also supported by VTM).

--
Emux

[Carlos Alberto Martínez Gadea]

There are definitely options out there that allow you to create an online server for vector tiles. I've been trying to do so by myself, but it seems quite hard to do it without a machine that has lots of RAM.

About the offline tiles, I've tried both MBTiles and Mapsforge tiles, and I'm happier with Mapsforge ones due to the size and the easyness of creating them, compared to MBTiles. Although I already create bounding boxed MBTiles. And they indeed work very nicely in VTM.