A New Map Format for Mapsforge?

[Ludwig]

What could a better format for map data look like?

Mapsforge currently uses a home-spun format for map data. Overall, this format has served mapsforge well, so any move away from it has to be considered carefully: it is always easy to think of benefits in the new and problems in the old, but redesigning the format requires work that could be spent on something else and will, at least for an initial period, introduce its own problems.

Overall Requirements (and Non-requirements)

• Mapsforge wants to render maps quickly from a compact storage format.
• Mapsforge does not want to be a general GIS system nor provide yet another general GIS storage format.

Deficits of the current format

• Flawed support for complex geometries, e.g. complex multi-polygons, and artifacts introduced through the tiling mechanism (area labels, area boundaries at tile labels).
• Not extensible at compile time: it is impossible to add custom data when creating a map file.
• Not extensible at run time: once created, it is impossible to add or remove data.
• Not searchable or routable, so data gets duplicated in user-specific formats.
• Support of non-OSM data only via faking OSM format.
• Complexity and lack of tool support. It is difficult understand the format and there is no way to read the data apart from displaying it as a map.

Requirements for a new format

• Support for arbitrary geometries.
• Compressed format fast for rendering retrieval.
• Extensibility at compile time: it must be possible to extend the core format to support custom data without breaking applications that do not know about the custom extensions.
• Extensibility at run time: it should be possible to add and update data at runtime. This includes support for loading additional geographic areas (e.g. a user panning to a new area could trigger a download operation where the new data area is retrieved and transparently added to the data store) as well as user additions (e.g. waypoints).
• Searchable/routable: I do not think that mapsforge should directly support routing (we should not be duplicating other project’s efforts), but the format should allow retrieval of information to assist routing (e.g. the road names, path information for route display).
• Support for generating map files from non-OSM data.
• Tool support for debugging and manipulating data.

Implementation Considerations

Geo-data format

• The GEOS WKB is not very compact and leads to very large map files.

• The current map file format is pretty compact, but it requires duplication for different zoom level ranges (different simplification factors).

• Google uses a start point + offset model similar to the current map file format, but with the option to encode simplification information (points only required at certain zoom levels). I am very much in favour of reusing tried and tested approaches and there is no shame in copying what Google has done (particularly as it comes as open source).

Database format

Using a standard database format gives a free retrieval engine, extensibility at compile time (add columns, tables), extensibility at run time (user can add data to DB), multi-threading support, support for search operations and a tried and tested code base on multiple platforms.

• Sqlite3: Ubiquitous, license-free format, supported everywhere, plenty of tool-support, light-weight, but of course without built-in geometry operations.
• Sqlite4:A modernized format of Sqlite3, which might be worth investigating as it could improve storage space required and provide better primary key retrieval. What about the number format?
• Spatialite: Sqlite3 + geometry, but with a very heavy footprint and somewhat complex to support on different platforms. The storage format for geo-data is not very compact.
• NoSQL Databases: I do not really have much experience with them, my impression is that they are good for single-purpose data storage, but are poor when it comes to support different usage scenarios and for distribution to heterogenous platforms. This model would probably require us to implement relational retrieval operations on top (complex, error-prone).

My personal preference at the moment is Sqlite3 (or 4, the difference is not that great from a development point) with a custom compressed geo-data representation.

Data “Pods”

I have in the past experimented with a format where the data for a tile was encapsulated in an SQlite3 table, the table name encoding the tile and geo-data within a table encoded so that only offsets from the tile origin need to be stored. (At that time I did not implement a geodata compression mechanism, so I ended up with very large map dababases). To retrieve data for a tile, it was simply required to calculate the table name, so retrieval operations were pretty fast as they operated on a geographically restricted data set. The encoding I had also allowed an geographic indexing based on a string prefix (so data for the same smaller area was encoded with the same string, neighbouring areas only varying at the end of the string).

Adding new geographic areas was then as easy as downloading a new tile-table and adding it to the DB. I thought that this would allow to implement a ‘vector-data tile server’.

Map Writer

For the support of OSM data we can continue to use Osmosis, but I think it would be much better to base the Map Writer on the Osmium framework, a C++ library that supports area construction (so we a proper 2D geometry out of OSM data), is fast and has a small memory footprint (Osmium can operate on an entire planet file!). (There is also a newer version, but it does not yet support relations, a big show-stopper). Once one understands the concept, writing code with Osmium is very fast.

Interface operations

The current interface for map data retrieval defined in the MapReader is primitive and should be more fine-grained, but we should, for ease of transition and testing, support the old interface on top of a new, more fine-grained access API. That would allow us to swap out the storage without disrupting the rest of Mapsforge.

I would also be in favour of making the format, a new writer and a new reader library available under a very liberal open source license to remove any uncertainties about licensing.

This is nothing I want to do in a hurry and certainly not if we cannot find a broad consensus. 

Ludwig

[Emux]

Thanks for the thorough summary Ludwig.

It's certainly an important case that needs careful handling.
Above all the requirement that whichever lib / tool should work universally in all OS.

--
Emux
Cruiser - Atlas

[Stephan75]

Hello Ludwig, thanks for your detailed first catalogue of requirements.

I have no detailed knowledge about the following, but what about the binary map formats used for example

by Osmand ... it has file extension *.obf ... with own sub databases for vector drawing, POIs, address data,

it is said to be based on Google's protobuf, or not?

and by Navit ... at least map drawing seems to be very fast from its offline data.

Both are opensource. Just to mention them in this thread.

Bye, Stephan

[Emux]

Thanks Stephan for the info.

I would like to note that though open source,
we must have in mind that GPL is different from LGPL or Apache License v2.0 or ... etc.

[Emux]

On 11/12/2014 12:54, Ludwig wrote:

I would also be in favour of making the format, a new writer and a new reader library available under a very liberal open source license to remove any uncertainties about licensing.

Ludwig can you explain more what you're meaning here?

[Ludwig]

I would also be in favour of making the format, a new writer and a new reader library available under a very liberal open source license to remove any uncertainties about licensing.

Ludwig can you explain more what you're meaning here?

 

If we were developing something entirely new I would not feel bound to keep the LGPL license.

I would be in favour of putting the format itself into the public domain, so the format is explicitly free to use for everyone without restriction, no need to ask a lawyer. It is something that can only aid adoption: the more writer and reader apps use the format the better. 

For the reader part I never felt that LGPL is entirely appropriate for the signed/DRMed mobile app format: a user simply cannot swap out a reader library, so license conformance is tricky. Maybe Apache would be better, but I am open to discussion.

The writer is a different beast: hidden in servers and not something that has a direct relation to users. Maybe here keeping LGPL or even something more restrictive could be appropriate, so that everyone benefits if someone improves the writer part (that would not stop map-makers having special data to set them apart).

To prior comments: yes multi-platform is essential. Maybe also encryption. 

Ludwig

[Emux]

Ok now I understand better, none objection from me.

[Andre Höpfner]

Hi,

 

This actually corresponds to what I find as well.

 

I do not wish to speak against it, but I think SQLite is too slow and not so compact. At least that my previous (unpublished) tests showed.

 

For the writer with Osmium I very much agree that I had set myself on the todo list for the writer, but had not yet mentioned in my post to the writer, because I had not even try.

 

I think just multithreading and speed (by Native C ++) is an important point for the writer.

 

+1, +1, +1000 For the statement {yes multi-platform is essential}

 

Greetings from Berlin, Andre

[Emux]

On 11/12/2014 20:53, Andre Höpfner wrote:

I think just multithreading and speed (by Native C ++) is an important point for the writer.

Just thinking aloud..
If go with C and be able to multiplatform for the writer,
 then we could do the same for the reader too, though here there is not ready solution.
(Or I push it too much?!)

[Andre Höpfner]

Or equal to Native code in Java.

 

In the OpenGL library used by me there is a sub-project, with which one native code in Java (as c ++) can be compiled so that one obtains a * .jar, which runs on Windows, Linux, Android, iOS and Mac.

 

https://github.com/libgdx/libgdx/wiki/jnigen

 

I had tried this once with Windows and was very surprised how well that works. The only hack if you want to create for Mac and iOS, you need a Mac compile system.

 

Now that I have a Mac, I have not tried it yet, but this would make up.

 

Greetings Andre

[Maki]

I can't comment on the deep technical aspects because, as usual, I'm not competent enough.

I'm very happy about the "search-ability" thing, as I really miss that opportunity.

Also "Compressed format fast for rendering retrieval" is interesting, zipped maps are significantly smaller than the original. If the map can be internally compressed we can deliver directly the .map file which would make downloading and installing on a phone much easier.

Another thing that I think could be useful is support for diff upgrades, if reasonably possible of course. Most maps we see around (not necessarily MF) are several hundreds MBs, and most of that data doesn't really change often (or at all). It's a pain to keep things up to date, especially for people that can only rely on mobile data.

I see no mention of raster data (or I missed it). I think the ability of mixing raster+vector would be very powerful, especially if coupled with display density independence, something I think only MF has right now. We examined hillshading recently but that's just one possible limited application.

Maki.

[fzk]

Hmm, a new map format ... that is something critical. What I miss so far is the long term vision for a new map format. What do you want to achieve within the next 5 years with it? Is the planned support for routing or searching really essential? Could this not better be done by online services? A lot of questions. But first there are answers needed for the three classic questions:

- Where did we come from?

- Where are we today?

- Where do we want?

Klaus

[Emux]

A bit philosophical I think, but the "offline" plays a major role here (along with custom vector rendering).
Or else we could just use an already established online service (OSM, Google, Here etc) for all our jobs, mapping, geocoding, routing.
But if we have already the information locally for viewing, it's always best to use it for other purposes too, like searching if we could.

[Richard Z]

On Thu, Dec 11, 2014 at 07:03:00PM +0100, Ludwig wrote:
> >
> > I would also be in favour of making the format, a new writer and a new
> > reader library available under a very liberal open source license to remove
> > any uncertainties about licensing.
> >
> >
> > Ludwig can you explain more what you're meaning here?
> >
>
> If we were developing something entirely new I would not feel bound to keep
> the LGPL license.
>
> I would be in favour of putting the format itself into the public domain,
> so the format is explicitly free to use for everyone without restriction,
> no need to ask a lawyer. It is something that can only aid adoption: the
> more writer and reader apps use the format the better.
>
> For the reader part I never felt that LGPL is entirely appropriate for the
> signed/DRMed mobile app format: a user simply cannot swap out a reader
> library, so license conformance is tricky. Maybe Apache would be better,
> but I am open to discussion.

the license conformance is indeed legally tricky because android apps can't use
dynamic linking without significant hassle. This can be easily solved by using
LGPL *plus* giving additional permissions (or clarifications), in this case the
explicit the permission to build signed android apps. The copyright holder is
always free to give additional permissions and in this case it can be assumed that
this is actually how most contributors expect LGPL to work so it is not nearly
as tricky as a licence chance.

The main advantage of LGPL vs Apache or BSD style licenses that I see is that
(at least in theory) all improvements that someone else makes to the library
while implementing his app must be available in under LGPL.

Richard

---
Name and OpenPGP keys available from pgp key servers

[Richard Z]

On Thu, Dec 11, 2014 at 08:10:37AM -0800, Stephan75 wrote:
> Hello Ludwig, thanks for your detailed first catalogue of requirements.
>
> I have no detailed knowledge about the following, but what about the binary
> map formats used for example
>

> by *Osmand *... it has file extension *.obf ... with own sub databases for

> vector drawing, POIs, address data,
> it is said to be based on Google's protobuf, or not?
>

> and by *Navit *... at least map drawing seems to be very fast from its
> offline data.

Navit has some real cool points, written in very portable c, compact data
format suitable for routing and search, seems quite a bit faster than
mapsforge.
It also seems to have even more trouble with multipolygons than mapsforge.
Would be nice if the two projects could be combined.

[Emux]

Yes the Android license case because of the packaging is a bit strange.
We have talk about it again in this forum as other have asked the same question.

For completeness here is the original announcement by Thilo for Mapsforge license change:
https://groups.google.com/forum/#!msg/mapsforge-dev/WWJVjZwc2OM/F9We-R5QgEwJ

[fzk]

"Video killed the Radio Star" was the first MTV video in 1981 and marks a media shift.
"Online killed the Offline Star" is imho the upcoming usage shift within the next year.

What does that mean concerning the map format?
Serverbased map data and clientside rendering could be the paradigm to combine the up-to-dateness of the data with the power of the mobile devices.
And the Mapsforge lib could act as the clue between server and client and ensures the best benefit-cost ratio for the user.

But that's much more as only a new map data format ...

Klaus

[Ludwig]

Thanks for all the contributions. This discussion is far from over. 

Multi-lingual support is one thing I forgot. 

On the online/offline front I think that offline functionality is absolutely essential despite all the online hype...by companies that make money with advertising and harvesting personal data. 

On the online front we will never be better than Google, but as a free and open source project we can and should provide in areas commercially uninteresting for the big players. Ability to work off-the-grid is essential for that. That does not preclude support for a mix of online-offline and I found the idea of map-diffs interesting. 

I was planning to revise the original email in the next few days to include the suggestions and put that out in our documentation. 

As I said, this discussion is far from over. Keep the ideas and comments coming. 

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/66e989c0-d19f-4afd-874a-18cd1135e079%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Emux]

On 18/12/2014 12:39, Ludwig wrote:

Multi-lingual support is one thing I forgot.

Yes that's essential feature for the future!

On the online/offline front I think that offline functionality is absolutely essential despite all the online hype...by companies that make money with advertising and harvesting personal data. 

On the online front we will never be better than Google, but as a free and open source project we can and should provide in areas commercially uninteresting for the big players. Ability to work off-the-grid is essential for that. That does not preclude support for a mix of online-offline and I found the idea of map-diffs interesting.

My view lately is that even big players promote their "offline" feature and people actually want that (e.g. Here).
And other *offline* map / nav solutions become free / open source after major acquisitions (e.g. Maps.Me) aiming for global audiences.

[André Miranda]

I'm glad to see mapsforge is evolving really fast and a new file format would be more than welcome.
I'm (trying) to use the writer as a library in a web server, but we're facing some issues(memory consumption and kludges) and probably we're going to back out and create a standalone converter(maybe using osmosis itself).
We have considered to write a custom map writer that could be as simple as possible for our constraints, but this would require a lot of effort and time.
As we are converting KML files to MAP and drawing it as a layer over an actual map, I had to tweak the style for ferry ways just to get lines drawn as desired. Custom elements(non-OSM) would be really useful in my case.

[Matthew Hall]

I have an application built using MapsForge that is distributed to employees to help people find buildings. A center with around 200 buildings spread out over a good size campus, hard to know where a building 123 is. Having offline maps has allowed the application to be very quick and work even when cell is slow and there is little to no wifi between buildings. Offline also helps those users with tablets and no cell modems to use the app with no wifi. For directions, I used a routing tool (forgot over the 2 year period) to pre-calculate all possible routes between each building, figure out the shared paths, and created a SQLite file that can find the route quicker than asking Google Directions. At the time, Graphhopper was too young and I couldn't find any other offline routing tool.

[Maki]

Hi to all and my best wishes for happy holidays (even if a bit late)

Re-elaborating some already raised points, and adding some new things.

I think in terms of user experience there are three points that need attention:

- Lack of interaction

- Speed

- Ease of management

I don't know how much support would be needed in the map file itself for the first point, but it should be investigated. I mean, maps are currently mostly static, apart from pan and zoom we can't do anything else. It would be interesting to trigger actions when touching "live" elements on the map. Getting a popup with the element's description, opening a webpage, things like that. It's probably too early for a complete implementation, but I think it will be more and more important in the future, so leaving the map format open to this could avoid another file format change in the future.

Speed: any idea where the bottleneck currently is? One would think it's drawing stuff, but to me it's quite slow even with simple maps/themes. So maybe data retrieval/parsing could be improved.

Another thing to evaluate is the map-style interaction. Changing map style currently requires a complete redraw of the map which is slow. Maybe things could be rearranged so that elements from the previous rendering/parsing can be reused.

Finally, ease of management is probably the worst offender for the end user. Files are big and monolithic, they generally come zipped, so must be uncompressed and moved around manually; plus you need external themes that are at least in part specific for every map supplier, but nobody warns you if you are using an OAM theme with a FZK map. Oruxmaps and Locus automate some of this work but it's still mostly a pain due to limited control about the download process.

What about making the file format open to future expansion and direct modification by the library? For future expansion think about how TIFF can store stuff that didn't exist in first place, like ICC color profiles, EXIF data or geocoding information. In our case it can be useful to store metadata about the map requirements (e.g. a map needs a specific theme) or to embed a default theme, or routing/indexing information, or whatever comes to mind in future without rewriting the whole thing or giving up on something useful because "it's hardcoded".

For direct modification by the library I'm referring to the idea Ludwig expressed about mixing different maps on the fly. That's great but it would complicate even more the things on the user side. The same concept could be used inside the single mapfile. I mean storing independent chunks of data that can be updated/replaced by the library on the client. For example one could set up an app that downloads only a user-chosen area from a tile server and assembles the map file on the fly. Could that be done in an efficient way? 

Maki

[Emux]

Hi Maki, my best wishes too!


On 30/12/2014 06:00 μμ, Maki wrote:

- Lack of interaction

I totally agree with you here.

- Speed

Probably we need to revisit the multi-thread rendering and eventually take advantage of the multi-core devices.

Finally, ease of management is probably the worst offender for the end user. Files are big and monolithic, they generally come zipped, so must be uncompressed and moved around manually;

That could be solved very easily with an automated unzipper.

For example one could set up an app that downloads only a user-chosen area from a tile server and assembles the map file on the fly. Could that be done in an efficient way?

I just mention the current situation on map writer. On big / complex areas we need very large memory and/or sdd hd cache (not suitable for the wide range of mobile devices).

After holidays I'd like with Ludwig to specify the future goals of Mapsforge.

IMHO I summarize them in two categories in order of importance for implementation.

- Fixes (adding new things without fixing the existing ones just does not seem right).
I have already expressed my concern regarding the gesture system (many have reported that too) which appears more sensitive now in Lollipop.
Or the hardware acceleration off directive.

- Additions (there are things we'd like to have in the library and things that we need to have).
Like proper map rotation - it's a major missing feature for a map library after so much evolution.
Very close and related is the float zoom levels.

[Ludwig]

Thanks for the comments, I agree that the static-ness of the map file is one of the biggest issues. That is why I personally believe that on the implementation front using a database format would be the best.

• data could be updated on the fly (e.g. a user adding points, or the map could be updated on the device by just transferring data newer than a timestamp, or just data for another map area (however small or large, so if you are just interested in a few km around you, that could be downloaded and not just an entire county)
• applications could add their own columns (e.g. additional information on a point, for pop-ups) but if we agree on some sort of standard the database could still be used by applications that are not aware of these add-ons.
• additional tables for whatever can be added, so yes we could be including renderthemes in it. 
• and by using a standard DB format we would not be reinventing the wheel and get lots of tools for free. 

I tried this with an sqlite DB last year using point encodings as an indexing trick and that seemed to work well, but obviously speed needs to be looked at. Issue that I did not address then was compressed encoding, so I ended up with very large map files, but emux sent an interesting link to something Google is using.

 

I might still have the code, but actually it falls more into the category 'Write one to throw away' (Fred Brooks, The Mythical Man Month).

I would love to hack out a prototype that incorporates the Google compression stuff, but for that I need a few days entirely free to really focus on it.

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/312007c9-382d-4644-9c0d-716a65aa3584%40googlegroups.com.

[Emux]

I vote for that too Ludwig, so far the database format appears to me also as the more appropriate / flexible for what features we have all discussed.

[Maki]

Il giorno martedì 30 dicembre 2014 18:24:08 UTC+1, Emux ha scritto:

Finally, ease of management is probably the worst offender for the end user. Files are big and monolithic, they generally come zipped, so must be uncompressed and moved around manually;

That could be solved very easily with an automated unzipper.

That's what Offline Map Downloader, Oruxmaps and (I believe) Locus already do. But you still have to download a very large file, and to unzip you need at least as much free space on the SD because at one point in time you'll have both the compressed and the uncompressed files around. And both download and unzip require a lot of time. That's not a problem for you and me, but for the average smartphone crowd it can.

 

For example one could set up an app that downloads only a user-chosen area from a tile server and assembles the map file on the fly. Could that be done in an efficient way?

I just mention the current situation on map writer. On big / complex areas we need very large memory and/or sdd hd cache (not suitable for the wide range of mobile devices).

I'm sorry, I haven't been very clear on that point, actually I knew I wasn't. The problem is that currently if you want, say, a Sicily map you need to download the whole Italy. My idea is that the client library could request only some tiles from the server and store those in the .map file. But just only store, no processing whatsoever. All the processing has to be done offline on a remote computer.

Similarly if the map can be split in virtual layers, data could be updated in a differential way. So, for example, a car navigator app could update frequently only highways without re-downloading each time buildings, land use and amenities. Of course there's a downside, because taking advantage from such an approach will require a dedicated server while a zip can be stored anywhere.

Maki

[mic...@vonglasow.com]

I came across this thread only today. Having done some work with Navit, I know its internals, including some of the upsides and downsides of Navit's binfile format [1].

One strength of the format are the nested tiles, which avoids clipping of objects at tile boundaries. The top level has one tile for the whole world, each tile is divided into 2×2 sub-tiles, with a total of 14 levels (though this could be extended). Each object is generally placed as low as possible in the hierarchy, and as high as necessary to not cross any tile boundaries. (For example, since the equator and null meridian are boundaries of the first sub-tile level, any objects that cross either of them would go in the top-level tile.) Some important objects (primarily roads) are never placed below a certain level, even if their geometry would fit into a lower level – thus search operations never need to go below a certain level.

The map file is essentially a zip file, with one file for each tile. File size is comparable to what Mapsforge currently uses.

This approach enables a cool use case which I would love to see in Mapsforge as well: a map download server [2] which lets users select an arbitrary rectangle on a map (or use a predefined one) and download it as a map file.

A downside is that, since Navit was started before OSM, its data types differ. For example, Navit currently supports only one single, generic ramp type, without the distinction between motorway_link, trunk_link etc. that OSM has. OTOH, the format is extensible and efforts are underway to introduce different ramp types. Conversely, Navit distinguishes between "city" and "land" for most road types – a distinction which can only be inferred indirectly from OSM data.

Another potential weak point is that map browsing is explicitly a non-requirement for Navit and the binfile format may not have been designed for it. OTOH, displaying a navigation map does have similar requirements, so it's quite likely that the data needed for map display is there.

Extensibility at run-time is somewhat limited by the fact that Navit requires tiles to be ordered within the zip file, and if tiles are "missing", a zero-length placeholder file must be included instead.

So, while I do not think that the Navit file format is suitable, it sure does have interesting concepts to look at.

Currently, there are three points that I'd love to see improved over the current situation: One is map downloads for Mapsforge with the same flexibility and great UX as the Navit project does – see above.

The other is that Mapsforce currently offers only dedicated regions, requiring me to switch maps whenever I cross a border. (I'd get a map holding all of Europe, but that's currently not available for download.)

The last one is that currently I need to keep two versions of the same map on my device – one for Navit, one for Mapsforge. That means I have to download twice the amount of data, wait twice as long and need twice as much space on my device. I'd love to see one common map format which both Navit and Mapsforge could use. (In Navit the map format is pluggable, thus different map formats can be supported, as long as they contain the necessary information – and data is arranged in a way that it can easily be searched.) I had in the past proposed adding to Navit a map driver for the Mapsforge format [3], though I'm not sure the current Mapsforge format is suitable for that.

I'd therefore add the following requirements:
* Sufficiently generic format to support both map rendering and full navigation, so that the format can be used by Mapsforge and navigation apps alike
* Support for splitting files (a map of e.g. Europe is likely beyond the file size limitations of some file systems) in a way that is transparent to the user (no manual switching of maps required as I cross a border)

Dynamic loading of map data – in some situations it would be great to have, so the map format should support it. OTOH, I still like to download my maps in advance whenever I can, so this use case should still be fully supported.

Michael

[1] http://wiki.navit-project.org/index.php/Binfile
[2] http://maps.navit-project.org/
[3] http://trac.navit-project.org/ticket/1235

[Emux]

Thanks Michael for the info.

In the past we have read many times this request from the users - besides map viewing to also have search feature.

And in other forums I have read requests for Mapsforge and GraphHopper to join forces in order to have one common format that has it all, vector maps + offline routing.

About the need to have the same map data multiple times, I'm afraid since the map developers put different things in their maps we still have to host them all in our devices.
i.e. Mapsforge default OSM, OpenAndroMaps with elevation, Freizeitkarte are only some of the different implementations

[mic...@vonglasow.com]

Hi Emux,

Glad to hear I'm not the only one with this request :-)

As for different things in the map – if I understand that correctly, Freizeitkarte and OpenAndroMaps use the Mapsforge format but have some extra objects/attributes which are not included in the maps available for download from Mapsforge. Which means that maps are interchangeable between apps, albeit with some limitations – for example, OpenAndroMaps with original Mapsforge maps would not display elevation because the map file doesn't have that data.

Or I could take this one step further and build my own map file, with a superset of all the data contained in Mapsforge default OSM, OpenAndroMaps and Freizeitkarte, and use it across all Mapsforge apps.

Therefore, first of all we should have a map file format that supports the features we need, with some of the data being optional. Which objects and attributes get included in the map file is then up to those who offer map files for download.

I'm taking my personal use case as an example, consisting of a navigation app and an app with a map viewer component. Let's say the maintainers of the navigation app also offer map downloads with all the data their app needs. That data would most likely work for the map viewer. It might be missing some data, such as elevation data and possibly certain types of POIs, which the map viewer app is theroretically capable of displaying – but since I don't really need them in the map viewer, I could easily use the navigation maps with both apps if there was a common file format.

Eventually, a common file format that's used by a number of apps/libraries out there allows for map downloads to be offered data providers who are separate from app developers. Geofabrik already has map downloads in various formats, so why wouldn't they add another format if it's sufficiently widespread? If the app data providers were to include all the data needed by the most common applications, we'd need extra maps only for apps with particularly exotic requirements. But a sufficiently generic file format is the prerequisite for all of this.

Michael

[Emux]

On 08/01/2015 06:26 μμ, mic...@vonglasow.com wrote:

As for different things in the map – if I understand that correctly, Freizeitkarte and OpenAndroMaps use the Mapsforge format but have some extra objects/attributes which are not included in the maps available for download from Mapsforge. Which means that maps are interchangeable between apps, albeit with some limitations – for example, OpenAndroMaps with original Mapsforge maps would not display elevation because the map file doesn't have that data.

Exactly, e.g. OpenAndroMaps have additional elevation contours, outdoor elements etc.
You could download some maps and compare - they are bigger than default ones and accompanied with their themes too.

Or I could take this one step further and build my own map file, with a superset of all the data contained in Mapsforge default OSM, OpenAndroMaps and Freizeitkarte, and use it across all Mapsforge apps.

You could do that too, or even go beyond that and build your own entirely custom map.
I often build many different maps from open or proprietary licensed data, both land and marine maps for various cases.

Geofabrik already has map downloads in various formats, so why wouldn't they add another format if it's sufficiently widespread?

Geofabrik provides vector data in some GIS formats like osm/pbf or shp.
These are suitable for direct processing inside GIS programs like QGIS, Osmosis etc.
More work is needed on top in order to transform them into properly optimized custom formats for use inside other readers like Mapsforge, GraphHopper, OsmAnd, OSRM etc.

[Michael von Glasow]

On 08/01/15 18:32, Emux wrote:

On 08/01/2015 06:26 μμ, mic...@vonglasow.com wrote:

As for different things in the map – if I understand that correctly, Freizeitkarte and OpenAndroMaps use the Mapsforge format but have some extra objects/attributes which are not included in the maps available for download from Mapsforge. Which means that maps are interchangeable between apps, albeit with some limitations – for example, OpenAndroMaps with original Mapsforge maps would not display elevation because the map file doesn't have that data.

Exactly, e.g. OpenAndroMaps have additional elevation contours, outdoor elements etc.
You could download some maps and compare - they are bigger than default ones and accompanied with their themes too.

Or I could take this one step further and build my own map file, with a superset of all the data contained in Mapsforge default OSM, OpenAndroMaps and Freizeitkarte, and use it across all Mapsforge apps.

You could do that too, or even go beyond that and build your own entirely custom map.
I often build many different maps from open or proprietary licensed data, both land and marine maps for various cases.

Exactly – and having a file format that is not limited to Mapsforge but can also be used by other applications and for purposes that go beyond the feature set of Mapsforge would take this flexibility to yet another dimension.

Geofabrik already has map downloads in various formats, so why wouldn't they add another format if it's sufficiently widespread?

Geofabrik provides vector data in some GIS formats like osm/pbf or shp.
These are suitable for direct processing inside GIS programs like QGIS, Osmosis etc.
More work is needed on top in order to transform them into properly optimized custom formats for use inside other readers like Mapsforge, GraphHopper, OsmAnd, OSRM etc.

Indeed, you are right – I thought I had seen them on the list of Navit map providers, but I probably confused that with Cloudmade (who seem to have retired that service since). There are, however, some other providers that offer Navit map data for download [1], along with processed OSM data in other formats – some of them might be interested in hosting downloads for a new, application-independent format.


[1] http://wiki.navit-project.org/index.php/OpenStreetMap#Pre-Processed_Maps

[Maki]

Il giorno giovedì 8 gennaio 2015 17:26:52 UTC+1, Michael von Glasow ha scritto:

As for different things in the map – if I understand that correctly, Freizeitkarte and OpenAndroMaps use the Mapsforge format but have some extra objects/attributes which are not included in the maps available for download from Mapsforge. Which means that maps are interchangeable between apps, albeit with some limitations – for example, OpenAndroMaps with original Mapsforge maps would not display elevation because the map file doesn't have that data.

It goes even a bit beyond that. It's not just a matter of objects/attributes being present or not. Things can be pre-processed/tagged in a different way. FZK has a deeper pre-processing of OSM data, while OAM is more true to the original. E.g. in OAM:

<rule e="way" k="tunnel" v="true|yes">

    <rule e="way" k="highway" v="primary">

        <stuff />

    </rule>

</rule>

becomes in FZK:

<rule e="way" k="*" v="primary_tunnel">

    <stuff />

</rule>

so you really need a very different theme. That's why I think there should be some sort of mechanism that verifies theme and map compatibility.

Maki

[Richard Z]

On Thu, Jan 08, 2015 at 07:32:34PM +0200, Emux wrote:
> On 08/01/2015 06:26 μμ, mic...@vonglasow.com wrote:
> >As for different things in the map – if I understand that correctly,
> >Freizeitkarte and OpenAndroMaps use the Mapsforge format but have some
> >extra objects/attributes which are not included in the maps available for
> >download from Mapsforge. Which means that maps are interchangeable between
> >apps, albeit with some limitations – for example, OpenAndroMaps with
> >original Mapsforge maps would not display elevation because the map file
> >doesn't have that data.
>

> Exactly, e.g. OpenAndroMaps <http://www.openandromaps.org/en/> have

> additional elevation contours, outdoor elements etc.
> You could download some maps and compare - they are bigger than default ones
> and accompanied with their themes too.

one little wish that should hopefully not be rocket science - would it be
possible to have some things - mainly elevation contours which tend
to change very rarely in an extra overlay map?
This would hopefully save map producers some KW of electricity, make the
downloads smaller and possibly allow combination of data which could otherwise
not be combined in a single map for license reasons.

I know it is possible in theory but apparently not good enough in practice?

[Emux]

On 09/01/2015 01:13 μμ, Richard Z wrote:

one little wish that should hopefully not be rocket science - would it be
possible to have some things - mainly elevation contours which tend 
to change very rarely in an extra overlay map?
This would hopefully save map producers some KW of electricity, make the
downloads smaller and possibly allow combination of data which could otherwise
not be combined in a single map for license reasons.

I know it is possible in theory but apparently not good enough in practice?

It's already possible in 0.5 release if you use two tile renderer layers (the higher transparent).
You can see an example of its usage in Samples.

In dev branch there is a more advanced less expensive method, you can read more about it here.

[Thilo Mühlberg]

Hello everybody,

as the main author of the currently used map file format and mapreader
software I would like to join the discussion and add my point of view.

I fully agree that the current map file format needs to be replaced. It
has too many limitations, most notably the lack of supporting arbitrary
(wildcard) tags and the clipping issues at tile borders. Also the map
file generation process is a problem. The Osmosis dependency makes it
hard to configure and run the software while limiting the possibilities
and performance of the mapwriter tool.

The file format and the software were invented five years ago when we
were focusing more on the map rendering part of the library. With all
the experience from the past years, the needed changes are now much
clearer as they used to be. From most to least important I would say:

- support an unlimited set of arbitrary OSM tags for every map item
- don't clip ways, store them only once in the file to reduce redundancy
- have a stand-alone, faster application to generate map files
- enable the mapwriter to easily import additional data such as SRTM
- improve query performance and reduce the used RAM/Java heap space
- offer command-line tools and Java APIs for mapreader and mapwriter
- higher test coverage and integration tests for mapreader and mapwriter
- all nodes and ways in the map file should have the original OSM ID
- read from multiple (adjacent) map files and merge the query results
- allow to create/update/delete map items, have a read/write file format

A single standard map file format across different OSM projects (e.g.
Navit) is an unrealistic goal, as different projects have also different
scopes, requirements and use cases. Having concurrent formats and
competing projects is a pretty normal and healthy thing. Let the users
decide which of them is better, that's called "evolution".

Storing additional resource files such as render themes, icon sets, etc.
in them map file might sound tempting. However, I would vote against
that as it mixes up different concerns and makes both, the map file
format and the software to read and write it more complex. I suggest to
use existing containers for that purpose, such as ZIP or TAR.

I doubt that embedded databases such as SQLite would meet all these
requirements. As general purpose tools they will neither be fast nor
compact or RAM efficient enough, given todays smart phone hardware.

I still think that the focus of the mapsforge project should be solely
on the offline capabilities. Ludwig is right, we will never be able to
compete with online services. A more user-friendly web service for
downloading arbitrary regions would of course be desirable. But this is
"nice to have", first the underlaying basics need to be right.

Best regards,
Thilo

[Emux]

Hi Thilo,

On 11/01/2015 03:13 μμ, Thilo Mühlberg wrote:

as the main author of the currently used map file format and mapreader
software I would like to join the discussion and add my point of view.

Thanks for the insight information, your experience is certainly valuable.

It has too many limitations, most notably the lack of supporting arbitrary
(wildcard) tags

+1

A single standard map file format across different OSM projects (e.g.
Navit) is an unrealistic goal, as different projects have also different
scopes, requirements and use cases. Having concurrent formats and
competing projects is a pretty normal and healthy thing. Let the users
decide which of them is better, that's called "evolution".

You are right here, we're not alone and in such cases also the other sides should be willing to agree with us.

Also we have to be careful with the licenses of each project,
we cannot mix LGPL / Apache 2.0 with GPL.

Storing additional resource files such as render themes, icon sets, etc.
in them map file might sound tempting. However, I would vote against
that as it mixes up different concerns and makes both, the map file
format and the software to read and write it more complex. I suggest to
use existing containers for that purpose, such as ZIP or TAR.

+1
That's the most versatile way keeping also the advantage of a map playing with multiple render themes / resources.
e.g. OpenAndroMaps has its maps in a zip, together with the proper render themes (separately) and they still can be used with other themes.

I doubt that embedded databases such as SQLite would meet all these
requirements. As general purpose tools they will neither be fast nor
compact or RAM efficient enough, given todays smart phone hardware.

Building SpatiaLite for Android I have seen performance without issues on older devices.
But yes eventually it depends on database structure, data format, indexing etc.

I still think that the focus of the mapsforge project should be solely
on the offline capabilities. Ludwig is right, we will never be able to
compete with online services. A more user-friendly web service for
downloading arbitrary regions would of course be desirable. But this is
"nice to have", first the underlaying basics need to be right.

My view also is that Mapsforge focus should be the offline without of course eliminating the online.
I already expressed my opinion for the future, there are many features which are "nice to have".
But we need revisit to some mechanisms for fixing and possible rewriting.
Also IMHO there are some features that are necessary to have a priori.
Then we could move to more advanced changes.

[fzk]

Am Sonntag, 11. Januar 2015 14:13:54 UTC+1 schrieb Thilo:

... A more user-friendly web service for downloading arbitrary regions would of course be desirable. But this is
"nice to have", first the underlaying basics need to be right. ...

 I do not agree with the statement "nice to have". The map and theme download and installation is a (very) critical part
and an high hurdle for a lot (or better to say the most) of the users. This part must be as simple as possible.

Klaus

[Ludwig]

It is because of the many requirements that we have that I gravitate towards a database as it allows changes at "compile time" (changes to schema, e.g. adding new tables) and "run time" (= changes to data) without requiring code changes. Using a database gives us an extensible format and code to access it for free, plus a great deal of tool support. 

We can of course always dream of a perfect format, but we need to take into account that we will spend a lot of work to even get close what an off-the-peg database like sqlite gives us (plus plenty of bugs etc). 

• Want to render only motorways? A simple SQL query gives you that without further coding. 
• Multi-language? Additional attributes? Add another column.
• A problem in the data somewhere: fire up a UI that reads the DB on the platform you want. 
• Building a download server? run a query against a server with a specific data request (for both type of data and extend) and the data can be added to the DB. 
  
• New developers coming onboard? Well, I assume most people know the relational model and then it is just a question of reading the schema. (Compare that to the number of people who understand the mapsforge mapfile format).
• Want to store renderthemes? Just add another table. Need metadata on themes: just add another column. (Zip-files are IMHO a very clunky way of doing things.) 

These things really count.  

Performance-wise sqlite is pretty bare-metal, after all that is what it is built for. I doubt we can do much better in general terms.

There are two issues that we should address however, but within the framework of a DB: indexing and coordinate compression.

• The way I had experimented with it was with plain sqlite, using encoded table names for faster lookup. Essentially one table = one tile, where the table name can be computed from the coordinates. That essentially provided a zero-cost indexing mechanism. 
• The problem I had was to compress the coordinate data, but Emux had shared a link to some Google code to do that efficiently. 

I have a little my doubts about spatialite, because with all it includes, it is a moving target while sqlite is stable as rock (both from a bug point of view as well as from ongoing code changes). spatialite is also very big. 

 

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/c7b67aa6-fd29-4ce1-a345-0177858f89bc%40googlegroups.com.

[Emux]

I find the SQLite path interesting, at least we need to test it.

The search would be ready from the ground.
Although it would be nice to have also spatial queries (something trivial in SpatiaLite).
e.g. find the restaurants inside the visible map bounds.

But there are also other methods for that, we'll see when we get there.

[Richard Z]

On Mon, Jan 12, 2015 at 10:01:37AM +0000, Ludwig wrote:

> Performance-wise sqlite is pretty bare-metal, after all that is what it is
> built for. I doubt we can do much better in general terms.

I am slightly worried that any SQL solution can develop unforseeable
bottlenecks. Transaction safety and the enormous flexibility of SQL
come at a cost.
Would you distribute your own copy of the sqlite with the project?

[Emux]

On 15/01/2015 05:19 μμ, Richard Z wrote:

Would you distribute your own copy of the sqlite with the project?

What do you mean?
Android has native support for SQLite (many wish to be spatial too),
and for Java we could use a known JDBC driver like Xerial.

[Ludwig]

Well, sqlite is special in this regard as it is implemented as an extremely lightweight SQL implementation. Read operations are not within transaction and not all of SQL is supported. 

The official statement is:

SQLite, on the other hand, strives to provide local data storage for individual applications and devices. SQLite emphasizes economy, efficiency, reliability, independence, and simplicity.
SQLite is not designed to compete with Oracle. SQLite is designed to compete with fopen().

 

(see https://www.sqlite.org/whentouse.html for more points in favour of sqlite)

It is always easy to unfavourably compare a solution that exists with an ideal solution: that ideal solution however has the real disadvantage that it does not exist and any implementation of it will fall well short of the ideal. (The current map file format is just one example of this fallacy.) 

The question IMHO is not if sqlite is perfect, the question is more: would we be able to do better? or stronger given limited developer time: would be be able to do better given the resources we have? 

I doubt it. 

Ludwig

 

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/20150115151926.GB14953%40rz.localhost.localdomain.

[André Miranda]

As the discussion is now towards SQLite, I'd like to mention that there are NoSQL alternatives out there, I've never used any of them, but I guess they should be considered.

Cheers,
André Miranda

[Emux]

On 15/01/2015 07:46 μμ, Ludwig wrote:

It is always easy to unfavourably compare a solution that exists with an ideal solution: that ideal solution however has the real disadvantage that it does not exist and any implementation of it will fall well short of the ideal. (The current map file format is just one example of this fallacy.) 

The question IMHO is not if sqlite is perfect, the question is more: would we be able to do better? or stronger given limited developer time: would be be able to do better given the resources we have?

My thoughts exactly..
If possible we want to avoid "reinvent the wheel", use existing solutions (if they perform fine) and focus our effort in additional features.

[Ludwig]

Re NoSQL my feeling is always that special purpose DBs are good when there is really only one purpose to support, but as soon as more flexibility is needed everyone starts to reinvent relational access methods on top of them. 

If someone has special experience with one of the NoSQL DBs and thinks it would be good to use that for our purposes, I would like to hear about it, though. 

That is probably another advantage of sqlite: not only we do not have to implement it, we also do not have to learn it as most of us are well familiar with the relational model.

As Emux said: better spend the time on additional features than on implementing base functionality.

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/54B7FEF3.4010003%40gmail.com.

[Thilo Mühlberg]

I agree with what Richard said, SQLite might be a problematic choice.
All the nice things that make it so much better than a "home-brew" map
format (e.g. the SQL interface, read/write support out of the box,
transactions, …) come at a cost which should not be ignored.

The question is not whether a well designed and implemented custom
solution is faster than SQLite (it certainly is) but whether the SQLite
performance is still "good enough". If the SQLite performance at some
point in time turns out to be a problem, there is very little one can
do. Besides rewriting the SQL query or adding more indexes, neither the
internal storage format nor the way the query is executed can be
influenced. A custom file format would not have this limitation.

The SQLite version 3.4.0 which is shipped even in the latest Android 5
is from 2007 and it doesn't support spatial indexes. Bundling a newer
SQLite version together with the application increases the size a lot
and reduces the portability. How would a mapreader work on a desktop PC?

The storage in typical Android smartphones has very limited read and
write performance (see [1] and [2]). Efficiently applying map changes on
the device (aka "incremental updates") is therefore not only difficult
but also very time consuming. Downloading a fresh version of the map
file from the internet might actually be faster in many cases.

Having a unified map database which supports map rendering, routing and
address searching would of course be nice. But it would also mean a lot
of work, as different indexes are required for each of these use cases.

I would personally recommend to follow the Unix philosophy: better just
do one thing, but do it well. Let others do the routing and searching
stuff. If we "only" offer a good map rendering solution, I am fine.

Best regards,
Thilo

[1] https://events.linuxfoundation.org/images/stories/pdf/lf_abs12_foy.pdf
[2] https://www.usenix.org/legacy/events/fast12/tech/full_papers/Kim.pdf

On 15.01.2015 19:00, Ludwig wrote:
> Re NoSQL my feeling is always that special purpose DBs are good when
> there is really only one purpose to support, but as soon as more
> flexibility is needed everyone starts to reinvent relational access
> methods on top of them.
>
> If someone has special experience with one of the NoSQL DBs and thinks
> it would be good to use that for our purposes, I would like to hear
> about it, though.
>
> That is probably another advantage of sqlite: not only we do not have to
> implement it, we also do not have to learn it as most of us are well
> familiar with the relational model.
>
> As Emux said: better spend the time on additional features than on
> implementing base functionality.
>
> Ludwig
>
>
>
> On 15 January 2015 at 18:54, Emux <deve...@gmail.com

> <mailto:deve...@gmail.com>> wrote:
>
> On 15/01/2015 07:46 μμ, Ludwig wrote:
>> It is always easy to unfavourably compare a solution that exists
>> with an ideal solution: that ideal solution however has the real
>> disadvantage that it does not exist and any implementation of it
>> will fall well short of the ideal. (The current map file format is
>> just one example of this fallacy.)
>>
>> The question IMHO is not if sqlite is perfect, the question is
>> more: would we be able to do better? or stronger given limited
>> developer time: would be be able to do better given the resources
>> we have?
>
> My thoughts exactly..
> If possible we want to avoid "reinvent the wheel", use existing
> solutions (if they perform fine) and focus our effort in additional
> features.
>
> --
> Emux

> Cruiser <http://wiki.openstreetmap.org/wiki/Cruiser> - Atlas
> <http://wiki.openstreetmap.org/wiki/Atlas_%28navigation_application%29>

>
> --
> You received this message because you are subscribed to the Google
> Groups "mapsforge-dev" group.
> To unsubscribe from this group and stop receiving emails from it,
> send an email to mapsforge-de...@googlegroups.com

> <mailto:mapsforge-de...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/mapsforge-dev/54B7FEF3.4010003%40gmail.com

> <https://groups.google.com/d/msgid/mapsforge-dev/54B7FEF3.4010003%40gmail.com?utm_medium=email&utm_source=footer>.

>
> For more options, visit https://groups.google.com/d/optout.
>
>

> --
> You received this message because you are subscribed to the Google
> Groups "mapsforge-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to mapsforge-de...@googlegroups.com

> <mailto:mapsforge-de...@googlegroups.com>.

> To view this discussion on the web visit

> https://groups.google.com/d/msgid/mapsforge-dev/CAAvpVPAw3odi4TcedkHyroWL0YGWm22kjPG%2Bzwzb7TheuXF4%3DA%40mail.gmail.com
> <https://groups.google.com/d/msgid/mapsforge-dev/CAAvpVPAw3odi4TcedkHyroWL0YGWm22kjPG%2Bzwzb7TheuXF4%3DA%40mail.gmail.com?utm_medium=email&utm_source=footer>.

[robvr]

Hi

I am developing a navigation application myself and have my own fileformat.

As a former DBA I have worked quite a lot with several DBMS systems, but I mst agree with Thilo,

I just can't see sqllite (Even if I never have used it), match performance of a dedicated fileformat.

You will need at least one index so space will be a problem too..

And on a smartdevice less space also means huge speed advantages, as IO is a bottleneck on thos devices.

Eg VBInts are faster than normal 4 byte ints, even if they need extra runtime processing. CPU's are faster than the IO system.

My comments on Thilos remarks are:

- support an unlimited set of arbitrary OSM tags for every map item 

  Store them not as strings but as numbers

- don't clip ways, store them only once in the file to reduce redundancy 

I use several files (roads separate from rest of ways, as I also use them for naviagation)

Tiles are in a file of their own and reference ways not by OSM id but directly by their offset in those files as VBInt

- have a stand-alone, faster application to generate map files 

Have that too, but in C#. Is slow compared with your filewriter (not C# but my implementation is the problem :)

- enable the mapwriter to easily import additional data such as SRTM 

?

- improve query performance and reduce the used RAM/Java heap space 

My app is significantly faster than mapsforge, but off less quality/detail. 

- offer command-line tools and Java APIs for mapreader and mapwriter 

good idea

- higher test coverage and integration tests for mapreader and mapwriter 

allwas good

- all nodes and ways in the map file should have the original OSM ID 

Why would you need those for other than for debugging. I replace them by their offset in their specific file andby theis 

even have references to them in eg the tiles file optimized: I use their offset in their seperate file as Id, and store that as VBInt

This saves a table lookup (runtime + storage advantage), but as they are created by the filewriter, debugging is more difficult. I write a special file that has OSM Id;s with the generated 'offset ID'  for debugging purposes,

but would recommend a special option to have the OSM ids in a debug version of the fileformat

Tile lookuptable needs fized pointersize, but size of pointer is determined by filewriter.

- read from multiple (adjacent) map files and merge the query results 

I would like to have that too :)

- allow to create/update/delete map items, have a read/write file format 

Would be great, but I have no tips on that.

Last week I updated my file format and took 9% off the size of the ways file.

Ways that are closed (eg have same start geopoint as last one) do not store that last position anymore but use a bit in an 
bitmask

Easily added it without need for changing existing maps. 

Grtz

Rob

Op donderdag 15 januari 2015 19:56:03 UTC+1 schreef Thilo:

>     <mailto:mapsforge-dev+unsub...@googlegroups.com>.

>     To view this discussion on the web visit
>     https://groups.google.com/d/msgid/mapsforge-dev/54B7FEF3.4010003%40gmail.com
>     <https://groups.google.com/d/msgid/mapsforge-dev/54B7FEF3.4010003%40gmail.com?utm_medium=email&utm_source=footer>.
>
>     For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google
> Groups "mapsforge-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to mapsforge-de...@googlegroups.com

> <mailto:mapsforge-dev+unsub...@googlegroups.com>.

[Maki]

Il giorno giovedì 15 gennaio 2015 19:56:03 UTC+1, Thilo ha scritto:

The question is not whether a well designed and implemented custom
solution is faster than SQLite (it certainly is) but whether the SQLite
performance is still "good enough".

That's a good question, the other one is: what do we gain in exchange for this slowness?

The optimal solution is the one that balances speed, size and richness. I'm more than willing to take a doubling in file size if in exchange I get the ability to download arbitrary areas, and/or filter the data at the source. Currently I'm constantly downloading 600-800MB maps just to get an area that could probably take under 50 MB. Double, or triple that, I'm still gaining a lot. I think that's a pretty common situation.

Talking about speed, I spent the best part (i.e. my free time) of the last two months building a few maps in Tilemill, that I exported in mbtiles format which is based on SQLite. Despite being much bigger than their MF counterpart they load in no time at all, while a complete screen redraw may take 15-20 seconds on my tablet when using Mapsforge. Of course, mbtiles is pre-rendered, but still shows that getting the data from the DB isn't currently the bottleneck, rendering is. Also Mapbox is apparently (I haven't tried) using now vector tiles in SQLite with their latest Mapbox Studio, so it's probably fast enough.

That said, as long as it works, to me anything is fine.

About storing themes in the map file I'd like to precise that I'm not suggesting this as the way to distribute themes, far from it. However, having played with maps from different sources I cannot not notice the problems that arise from the current situation. Nothing prevents the end user to use the wrong theme for the selected map. When an app fails to load the theme it falls back to the default Osmarender one, which actually only works with default maps. Try a FZK map with it and see. This is easily solved in two ways;

1) maps and themes should declare what dialect(s) they are using sot hat the app can at least warn the user if they mismatch

2) the maps should contain a basic fallback theme

Maki

[Ludwig]

but still shows that getting the data from the DB isn't currently the bottleneck, rendering is.

That is of course an important observation. Contrary to other comments here on this thread I am not convinced that sqlite will be much slower reading the data, even less if we consider that SQL access might reduce the amount of data we have to read in in the first place, e.g. if we only read the data that we actually want to render (e.g. if rendertheme does not render certain roads, we do not read the data at all). 

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/5dc3e2c5-056f-417c-9b85-6bcfc99b1bff%40googlegroups.com.

[Richard Z]

On Thu, Jan 15, 2015 at 07:55:56PM +0100, Thilo Mühlberg wrote:

> The SQLite version 3.4.0 which is shipped even in the latest Android 5
> is from 2007 and it doesn't support spatial indexes. Bundling a newer
> SQLite version together with the application increases the size a lot
> and reduces the portability. How would a mapreader work on a desktop PC?

this is my concern. For me a recent OruxMaps version with Mapsforge is
still usable even on a ridiculously old phone with Android 2.3.3.
No idea which SQLite version it has but I can hardly imagine you want
to deal with all the versions out there.

> The storage in typical Android smartphones has very limited read and
> write performance (see [1] and [2]). Efficiently applying map changes on
> the device (aka "incremental updates") is therefore not only difficult
> but also very time consuming. Downloading a fresh version of the map
> file from the internet might actually be faster in many cases.

if the updates/downloads are pretty rare it could be done by a separate
app downloading some kind of diffs and rebuilding the maps on-device?

Also - while at it - it would be good to encourage apps to share their
mapsforge maps as far as possible. Do not think this is much a technical
issue but documentinga reasonable default storage location, layout - and
perhaps adding this place as default path in some places.

[boh...@gmail.com]

From my experience, SQLite is quite fast, if you use it in the right way. Often faster than a custom data format.

And it is pretty lightweight too, I used it on devices ten times less powerful than the least powerful Android phone. Stuff like feature phones with 8MB of RAM or even less.

So yeah a custom data format may be faster if you do it right, but maybe not and if it's not you will have lost a lot of time for that. Though it might be worth it.

From someone who uses Mapsforge tile server to serve raster tiles on a webserver ;)

-- bohwaz

[robvr]

Hi

I would like to give some ideas for the mapwriter

1 make use of a dbms for intermediate storage
2 use its transaction mechanism to add the following features
Make the mapwriter restartable
Allow for distributed computing, eg let more than one computer perform subtasks on the mapfile creating, eg as background or null process
Either inside your own lan or even over the internet
eg let the mapsforge community make their processors be available and build/maintain the maps of the world together :)

This allows bigger maps to be created and reuse of the processed data for different maps or areas
Possibly a incremental update of mapdata could be implemented using a dbms


Grtz
Rob

[Ludwig]

I think our continuing memory problems give one more strong reason to go with a proper database: the built-in ability to read data selectively and through cursors, meaning that first of all only the elements that will be drawn are read and then in a piecemeal fashion.

Unfortunately that means that the whole rendering process needs to be redesigned to work the other way round: currently we read all data, and then render selectively. This has a lot of overhead for data that is not rendered.

IMHO, it would be much better if the process is driven by the rendertheme going through the layers, from the bottom up. For each layer, read only the data that is required. Read it through a cursor, rendering it onto a tile, then release data. Never a need to read (and decode) data that will never be rendered, never read all data at once.

This functionality comes free with a DB, for good reasons.

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/5f1995d2-055e-47f7-8b57-1d30b3f56756%40googlegroups.com.

[Emux]

I find nothing wrong with those thoughts.

For those not aware, multi threaded rendering taught us that currently it's very easy to break the memory barrier.

[Emux]

Something else in my mind for the future, is a potential render theme move from xml to json or at least support of.

e.g. Elevate complex themes are slow at parsing, even with the new xml parser and latest lib improvements.

[Charles Sweeney]

In the first place, the current file isn't bad.  I am very new to this but it was the only solution I found for an offlinemap and was attracted to it due to it's compact format, so small size is very important...mainly because users have to download it onto mobile devices.

Searchable could be interesting though I presently wouldn't use such functionality in the app I am writing as it uses British postcode data, which isn't available in the OSM data, so I keep the postcode data in the database along with the lat lon coordinates and simply pass them to the basic map viewer.

If you were to use a database I would be very happy with sqlite3.  Being new to this, it's the only app database I know but I love it for its sheer simplicity and performance.

Agree you shouldn't directly support routing, for the reasons you mentioned.  Creating a killer map engine is work enough!

On Thursday, 11 December 2014 10:54:20 UTC, Ludwig wrote:

What could a better format for map data look like?

Mapsforge currently uses a home-spun format for map data. Overall, this format has served mapsforge well, so any move away from it has to be considered carefully: it is always easy to think of benefits in the new and problems in the old, but redesigning the format requires work that could be spent on something else and will, at least for an initial period, introduce its own problems.

Overall Requirements (and Non-requirements)

• Mapsforge wants to render maps quickly from a compact storage format.
• Mapsforge does not want to be a general GIS system nor provide yet another general GIS storage format.

Deficits of the current format

• Flawed support for complex geometries, e.g. complex multi-polygons, and artifacts introduced through the tiling mechanism (area labels, area boundaries at tile labels).
• Not extensible at compile time: it is impossible to add custom data when creating a map file.
• Not extensible at run time: once created, it is impossible to add or remove data.
• Not searchable or routable, so data gets duplicated in user-specific formats.
• Support of non-OSM data only via faking OSM format.
• Complexity and lack of tool support. It is difficult understand the format and there is no way to read the data apart from displaying it as a map.

Requirements for a new format

• Support for arbitrary geometries.
• Compressed format fast for rendering retrieval.
• Extensibility at compile time: it must be possible to extend the core format to support custom data without breaking applications that do not know about the custom extensions.
• Extensibility at run time: it should be possible to add and update data at runtime. This includes support for loading additional geographic areas (e.g. a user panning to a new area could trigger a download operation where the new data area is retrieved and transparently added to the data store) as well as user additions (e.g. waypoints).
• Searchable/routable: I do not think that mapsforge should directly support routing (we should not be duplicating other project’s efforts), but the format should allow retrieval of information to assist routing (e.g. the road names, path information for route display).
• Support for generating map files from non-OSM data.
• Tool support for debugging and manipulating data.

Implementation Considerations

Geo-data format

• The GEOS WKB is not very compact and leads to very large map files.

• The current map file format is pretty compact, but it requires duplication for different zoom level ranges (different simplification factors).

• Google uses a start point + offset model similar to the current map file format, but with the option to encode simplification information (points only required at certain zoom levels). I am very much in favour of reusing tried and tested approaches and there is no shame in copying what Google has done (particularly as it comes as open source).

Database format

Using a standard database format gives a free retrieval engine, extensibility at compile time (add columns, tables), extensibility at run time (user can add data to DB), multi-threading support, support for search operations and a tried and tested code base on multiple platforms.

• Sqlite3: Ubiquitous, license-free format, supported everywhere, plenty of tool-support, light-weight, but of course without built-in geometry operations.
• Sqlite4:A modernized format of Sqlite3, which might be worth investigating as it could improve storage space required and provide better primary key retrieval. What about the number format?
• Spatialite: Sqlite3 + geometry, but with a very heavy footprint and somewhat complex to support on different platforms. The storage format for geo-data is not very compact.
• NoSQL Databases: I do not really have much experience with them, my impression is that they are good for single-purpose data storage, but are poor when it comes to support different usage scenarios and for distribution to heterogenous platforms. This model would probably require us to implement relational retrieval operations on top (complex, error-prone).

My personal preference at the moment is Sqlite3 (or 4, the difference is not that great from a development point) with a custom compressed geo-data representation.

Data “Pods”

I have in the past experimented with a format where the data for a tile was encapsulated in an SQlite3 table, the table name encoding the tile and geo-data within a table encoded so that only offsets from the tile origin need to be stored. (At that time I did not implement a geodata compression mechanism, so I ended up with very large map dababases). To retrieve data for a tile, it was simply required to calculate the table name, so retrieval operations were pretty fast as they operated on a geographically restricted data set. The encoding I had also allowed an geographic indexing based on a string prefix (so data for the same smaller area was encoded with the same string, neighbouring areas only varying at the end of the string).

Adding new geographic areas was then as easy as downloading a new tile-table and adding it to the DB. I thought that this would allow to implement a ‘vector-data tile server’.

Map Writer

For the support of OSM data we can continue to use Osmosis, but I think it would be much better to base the Map Writer on the Osmium framework, a C++ library that supports area construction (so we a proper 2D geometry out of OSM data), is fast and has a small memory footprint (Osmium can operate on an entire planet file!). (There is also a newer version, but it does not yet support relations, a big show-stopper). Once one understands the concept, writing code with Osmium is very fast.

Interface operations

The current interface for map data retrieval defined in the MapReader is primitive and should be more fine-grained, but we should, for ease of transition and testing, support the old interface on top of a new, more fine-grained access API. That would allow us to swap out the storage without disrupting the rest of Mapsforge.

I would also be in favour of making the format, a new writer and a new reader library available under a very liberal open source license to remove any uncertainties about licensing.

This is nothing I want to do in a hurry and certainly not if we cannot find a broad consensus. 

Ludwig

[Charles Sweeney]

The ability to use several map files and select the correct one based on a provided lat lon location.

The ability to pan between different maps.

Just now with mapsforge you have to hard-code the name of the map file to use.  I would like to use several smaller maps rather than one large map (I'm making a map of Britain so would prefer to use a seperate map for Scotland and a separate map for England and Wales), but to do this you would need to change the map file when panning between the two maps, you would need to know which one to show, based on the lat lon location.

Is there a way to do that now?  If not, I think it's very important to have this facility.

[Emux]

Yes it's already possible.
Check the MapDataStore wiki.

[Charles Sweeney]

Interesting...thank you!

[Emux]

From my recent research, if we ever move to SQLite we could use an object/relation mapping (ORM) tool (e.g. Android greenDAO).

[Kees van Reeuwijk]

A few thoughts of mine on the file format:

- It would be helpful if t here was room for some metadata. A map title, a creation date, a description, what tool created the map, etc. A simple key/value scheme similar to mp3, jpg, and many other file formats would probably be best.

- It would be helpful if the file format would explicitly allow multiple files to contribute to the same area, so that people could add data if and only if they are interested in it: think of ski tracks and lifts, elevation contours (although a different approach may be better), public transport info, naval information, and so on. The area covered by these special files may not be the same as for the `main' information.

- Putting everything in a database sounds like a plan, but it makes it very difficult for the user to manage memory usage. In particular, removing stale or no longer interesting data will be difficult. In other words: all that data about Japan was very helpful while the user was on holiday, but now that he's back home and won't go there for years, it is just taking up space, so how is he going to remove it from the database?

- Similarly: what happens if there is an update of a certain area?

Final and craziest thought:

- If the files are only intended to build a database anyway, wouldn't it make sense to just use the mainstream OSM file formats? You can still use specific tools to filter and rewrite the data, but I'm assuming the files will be read once and then thrown away, making the ultimate compactness less relevant.

[Murray Hughes]

One project that I find very interesting is https://github.com/conveyal/osm-lib

I've been using it to create my extracts before converting them to .map files and it's insanely fast compared to other processing methods (osmosis).

Could maybe the same MapDB be used as a backend for Mapsforge, it's something I want to play around with.

Murray

—
Sent from here

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/cdd55746-8367-4052-9051-809bea3d683d%40googlegroups.com.

[Karl]

Hi,

since I am currently also thinking about the file format, I might as well add my thoughts to this discussion.

What I have in mind, and what I am currently working on, is a navigational software based on (some parts of) Mapsforge and on Graphhopper. I do not intend to put this on a smartphone. In the end I might build my own device based on e.g. the Raspberry PI :) (It should not be that hard.)

(1) What I am missing, is some way to search for data, in particular for addresses based on city, street and house number. One may want to search for other things, but that is what I would need.

(2) File size is definitely an issue for me! For the navigational software I would need the Mapsforge data (currently 1.5 GB for Germany) for the map, the routing map for Graphhopper, and the data for the search (of addresses). Compare this with data for e.g. a Garmin device. It is routable, searchable and should be smaller than 1.5 GB (for Germany) -- for whatever reasons.

One can reduce the current file size by about 30% to 50% without any losses, see number (3).

(3) I do not understand the intention of the current format. As far as I understand, it seems to mix projected coordinates and not projected coordinates. If true, I would consider this a bad design. To be precise: It somehow depends on the Mercator projection, which is used through the whole of Mapsforge. On the other hand, points are saved with WGS84 coordinates, e.g. with latitude and longitude in micro-degrees.

I would try to use only one set of coordinates throughout. That means, the file format should be either completely independent of any projection and use WGS84 coordinates throughout, or use only projected coordinates.

I suppose one cannot use WGS84 throughout, due to the tile structure of the format. The tile structure must -- as I think -- know about the projection. So one might save everything in projected coordinates instead. Hence, the projection will only be present in the writer and it will be easy to support other projections (yes, there are other projections besides Mercator :) ). The map-drawing part does no longer need to know anything about any projection. It just draws a two-dimensional map. Only additional layers, if present at all, might need to know about the projection and might need to use it. Anyway, this seems to be an advantage to me.

One may reduce file size. In Germany, using micro-degrees for latitude and longitude takes four bytes for each latitude and between three and four bytes for each longitude. One may resort to delta coordinates with respect to each tile. One may save the integer part of 128 * x and 128 * y as delta coordinates (x and y are projected coordinates). For a tile size of 256 one needs only two bytes, instead of three to four bytes for micro-degrees. And one even has a higher precision than with micro-degrees :) (Germany is not particular in the respect.)

If you look at the file, only base zoom level 14 is what matters. The two other base zoom levels do not add much to the overall file size.

(4) I would not put the data into the database. I would expect this to require more code, more disk space and result in larger access times.

Although the access time with the current format is not an issue on my linux PC. I need between 300 and 400 msec to render one tile (using JavaFX :) ). About 37% of the time to render one tile is taken by the method processReadMapData. About 46% of the time is taken by the method fRenderTheme.scaleTextSize, which according to my understanding does nothing, since it always used a scaleFactor of one!? Less than 2% is taken to read the map data and about 10% is needed to render the ways (with JavaFX). Ok, this is an example, it depends on the tile, the zoom level, ...

Resorting to Java nio and memory mapped buffers one might improve access time to the file and memory requirements even more, but I don't know, if this is available for android (which doesn't interest me).

Karl

P.S. Yes, I am working on a port to javaFX and the part to render single tiles is almost ready. Only the line breaking algorithm in FXPointTextContainer and some severe testing should be missing. I am not interested in adapting the layer stuff to java FX, and I am not sure, whether it will readily fit into Mapsforge. So what I have will not be of much use to you, but in case you are nevertheless interested.....

[Emux]

Hi Karl,

Nice to hear that Mapsforge is tried on another platform.

(1) Mapsforge file format was created with main purpose the fast offline rendering of vector data. Unfortunately not with also search in mind.

(2) Well I use extensively GraphHopper too and I understand your concerns.
It would be good if we could have only one file format that would have them all (map + routing + search). But that's impossible unless we build something from the start.

i.e. we cannot mix Mapsforge + GraphHopper code together, they are two different projects managed independently.
Besides I don't know how Peter would feel about that :-)

(3) Web Mercator (EPSG:3857) projected coordinate system is the de facto standard for web mapping applications for visualizing earth on 2D space.
All mobile implementations like ours (not being 3D ellipsoids) are just expressions of it.
The underlying coordinates are defined using the WGS84 ellipsoidal datum as is the common rule.

(4) I would depend that in the implementation. And I'd expect access time to be DB like.

Text scale is configurable by the user. It maybe not have many uses on desktop (except 4K), but on mobile with the many screen densities we certainly can use it - along with the important variable tile size.

Mapsforge began on Android and along the way it was moved also on Java (due to common base).
BTW Android supports some classes of NIO packages and even them not fully.


We are currently two people (Ludwig and me) who manage Mapsforge.
Our resources are limited and there are a lot that we can add / improve.

In my list are always native map rotation, hardware acceleration, steplessly variable zoom levels, etc. Not particularly in that order.
At least I'm happy for the refactored Android gesture system in 0.5.2 - it was something that always bothered me.

--
Emux

[Karl]

Hi Emux,

Am Samstag, 22. August 2015 11:31:02 UTC+2 schrieb Emux:

[...]
 

(1) Mapsforge file format was created with main purpose the fast offline rendering of vector data. Unfortunately not with also search in mind.

Search would be nice, but if too complicated, one may also provide search data as an optional add-on. But maybe a new format could be build such that one may provide such an optional add-on :) I have not thought about whether this is possible for the current format.
 

(2) Well I use extensively GraphHopper too and I understand your concerns.
It would be good if we could have only one file format that would have them all (map + routing + search). But that's impossible unless we build something from the start.

i.e. we cannot mix Mapsforge + GraphHopper code together, they are two different projects managed independently.
Besides I don't know how Peter would feel about that :-)

Please don't get me wrong here, I don't want one format for everything (no "eierlegende Wollmilchsau") and I don't want to mix Mapsforge and GraphHopper etc.

I am no expert on the Garmin file format (it has been reverse engineered in parts) but it seems to me, that the Garmin format has different data structures for search, routing, the map etc. For what I have in mind, these data do not have to reside in one file. So there would be the map data from Mapsforge in one file, the routing map from GraphHopper in another file etc. This is fine. But compare the overall size of the data of this approach with the size of a Garmin file. I do not think these files are smaller because they have one format (data structure) for everything. I think, they need less space for the map data as Mapsforge etc. I don't know at what cost (less data, less accuracy, ...)

But anyway: I only want to say that reducing disk space for the Mapsforge data should be one design goal. And there is room for considerable improvement, as I showed.

(3) Web Mercator (EPSG:3857) projected coordinate system is the de facto standard for web mapping applications for visualizing earth on 2D space.
All mobile implementations like ours (not being 3D ellipsoids) are just expressions of it.
The underlying coordinates are defined using the WGS84 ellipsoidal datum as is the common rule.

I don't unterstand, what you want to say. The current file format mixes Mercator and WGS84. I do not think this is good design. One should use either only projected coordinates or WGS84 coordinates throughout. I think the tile structure will rule out WGS84, so use projected coordinates for everything. The user would not notice this. For one, it would improve the software, as the renderer does not need to care about the projection any more.

That you save points as WGS84 in the current format will not buy you anything. The current format requires, that the renderer converts that to 2D using Mercator. What for? Convert it in the writer and the renderer does no longer have to care. I think, this is better software design.

Maybe there are real arguments against it. The fact, that OSM uses WGS84, is no argument. OSM is a GEO-database, Mapsforge a tool to show a map. So one must project somewhere. My question is about when to do it. Currently you do it in two places, I suggest to do it in one place. The latter will provide some advantages.

(4) I would depend that in the implementation. And I'd expect access time to be DB like.

Text scale is configurable by the user. It maybe not have many uses on desktop (except 4K), but on mobile with the many screen densities we certainly can use it - along with the important variable tile size.

Ok, I didn't know that. But do you need to call this method for every tile rendered? Isn't it enough to call it once during initialization? Or once, when the user changes the scale? If possible, it would save a lot of time to render a tile.
 

Mapsforge began on Android and along the way it was moved also on Java (due to common base).
BTW Android supports some classes of NIO packages and even them not fully.

Yes, this is what I have read. But I don't know, whether android supports enough of nio for the use of memory mapped buffers. But who needs to improve something, that taked 2% of the time (well, on my linux PC at least) :)
 

We are currently two people (Ludwig and me) who manage Mapsforge.
Our resources are limited and there are a lot that we can add / improve.

Yes, I know what you mean :) I also have to earn my living and my project is progressing very slowly. But one needs to have a hobby :) And some day I will retire :)

And don't get me wrong here either. You (Ludwig) started this thread :) I added my thoughts for you to think about. They may help you or they may not. I do not "want" anything :)
 

In my list are always native map rotation, hardware acceleration, steplessly variable zoom levels, etc. Not particularly in that order.
At least I'm happy for the refactored Android gesture system in 0.5.2 - it was something that always bothered me.

Maybe javaFX could help you. It should make certain things easier. But I don't know, whether it is available on android. Oracle offered javaFX for the ARM processor, but removed it. OpenJDK wants to support the ARM, but I am not sure, whether it is available yet. I would need it for the Raspberry PI :)

Karl

[Ludwig]

Just to add to the discussion:

A blazing fast geo database with LevelDB, Go and Geohashes

Ludwig

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/bbd90fb4-7c07-4916-849d-92ea6c23ab77%40googlegroups.com.

[Karl]

Hi,

I would like to add some other information, which has primarily to do with writer, but may help the discussion on the new format as well.

I tried to create a map file on my Linux box for Germany. My main memory of 8 GiB is too small for the RAM based method, so I tried the HD based method. I broke it off after around 7 hours of maltreating my hard drive :(

Somehow I thought, there must be a better way do do things. After some experiments, I still think so. The key idea is to use a B+tree with in-memory internal (B+tree-)nodes. Consider Germany: there are almost 250,000,000 nodes in the corresponding pbf-file. Each node has at least an id, latitude and longitude, so the file has at least 5.6 GiB of uncompressed data for the nodes (which on has to store temporarily). One still has to add keys and values for the tags, but I guess most nodes should have no tags. So one could write the node-information sequentially onto a hard drive. It will form the leaf nodes of the B+tree. Currently osmosis provides these data sorted, so one may readily construct a corresponding B+tree afterwards using bulk loading.

I have not tried bulk loading yet, so the internal nodes of the B+tree are exactly 50% full -- because osmosis provides the data sorted. Nevertheless, I need less than 100 MiB for the internal nodes of the B+tree. I guess one could do even better if bulk loading is used.

Reading and writing the data using my in-memory B+tree is very fast. I have tested this with 250,000,000 nodes with id, latitude and longitude. (Well, it is not that easy. It depends on the details of how the data of the leaf nodes are laid out on the hard drive). Maybe I should add, that I used nio MemoryMappedBuffers. So I didn't buffer any data on the heap but let the OS decide about which pages to buffer in system memory.

I conclude the following from these experiments:

(1) It could indeed be possible to improve the performance of writer, when in-memory B+trees (with bulk loading) are used for nodes, ways and relations. (I have another idea on how to improve the sorting and merging ways for multipolygons, but I have not tested it yet.)

(2) If it is indeed an advantage to have ways and nodes of the mapfile in something like a DB: why not put them into the map file as leaf nodes and use a custom built implementation of an in-memory B+tree?

Database systems are built to provide good performance for all kinds of problems. I am convinced one should be able to do much better with a custom built solution. Currently a mapfile is fixed, so no insert and delete is necessary. Locking is not necessary, ...

Karl

[Ludwig]

For much faster processing of OSM data I would start from Jochen Topf's Osmium Tool http://osmcode.org/osmium/ . This has the added advantage of having solved lots of problems with relations in OSM.

Custom formats can be fine, but they need to be maintained. I think that is the strongest argument against them for a project with such little resources. 

:-(

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/48bddf20-7975-4a5a-9e65-d2afb69dc231%40googlegroups.com.

[Emux]

On 11/09/2015 05:01 μμ, Karl wrote:

I tried to create a map file on my Linux box for Germany. My main memory of 8 GiB is too small for the RAM based method, so I tried the HD based method. I broke it off after around 7 hours of maltreating my hard drive :(

Probably wrong forum for discussing map-writer issues, but I suppose you have tried all known Java memory tricks (64bit, Xmx, etc).

--
Emux

[Karl]

Hi Emux,

That was not the point of my post, but since you mentioned it: I don't know of any particular forum to discuss map-writer issues. Could you give me a link to the right forum?

Thanks in advance
Karl

[Emux]

On 11/09/2015 09:11 μμ, Karl wrote:

That was not the point of my post, but since you mentioned it: I don't know of any particular forum to discuss map-writer issues. Could you give me a link to the right forum?

I meant opening a different topic for map-writer issues. :)

--
Emux

[Karl]

Hi Ludwig,

Am Freitag, 11. September 2015 16:34:03 UTC+2 schrieb Ludwig:

For much faster processing of OSM data I would start from Jochen Topf's Osmium Tool http://osmcode.org/osmium/ . This has the added advantage of having solved lots of problems with relations in OSM.

Sounds interesting. Can you give me a link to the problems, that have been solved with osmium?

Osmosis itself is not slow and I didn't say so. It takes about 5 minutes on my box to parse the pbf-file for Germany (2.5 GiB) (the process event only counts the number of nodes, ways and relations but does nothing else). I think this is good enough. The Osmosis HD-datastores that map-writer uses seem to be one problem. I don't remember, but it seemed to have taken hours to write the data to disk (in the process event). As far as I understand map-writer, data are only written to disk in this event. Using my B+tree implementation, I can write 250,000,000 nodes (without tag information) in about another 5 minutes to my hard disk! Depending on the details of the disk layout, I can do a full read of the data in between 1(!) and 5 minutes.

I don't know how often map-writer reads the same data from disk, but I would expect much room for improvement.

I don't yet understand, how osmium can improve this situation. One can request the data for a given ID. But a pbf-file has no index, so how fast would that be? So you may not want to use the pfb-file itself as a datastore and request single data through osmium, or would you? What I have written above indicates, that parsing a pbf-file is not the problem, but building the HD-datastore and processing the data afterwards, in the complete-event. I don't see, how osmium can help here???
 

Custom formats can be fine, but they need to be maintained. I think that is the strongest argument against them for a project with such little resources. 

Maybe. But a B+tree is not that complicated. Compared with what I have seen of the complete mapsforge project this is really trivial!

Karl

[Ludwig]

Can you give me a link to the problems, that have been solved with osmium? 

The biggest advantage of osmium is that is able to construct areas from osm data. 

 

--

You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/7a3b144a-ca16-4a5b-96da-9dfd3c1a186c%40googlegroups.com.