Mapsforge Map Creator 2

[Emux]

I implemented a new automatic generation for Mapsforge maps and pois (based on our guide).

https://github.com/mapsforge/mapsforge-creator

The repository contains everything needed, you can see the readme for the requirements, configuration and usage.

map-creator takes 3 arguments per map:
  - continent/country[/region] (Geofabrik hierarchy)
  - ram or hd (for map-writer)
  - languages (optional)

v3 maps with default language:
./map-creator europe/germany/berlin ram

v4 multilingual maps:
./map-creator europe/germany/berlin ram en,de,fr,es

In v3 and v4 folders, I include batch scripts to build all countries for continents or all regions for some countries (regular or multilingual maps).

We'll try the new process also in Mapsforge server, as it's much faster and lighter on resources.

Any feedback or ideas / improvements are welcome!

--
Emux

[Ludwig]

In my opinion the move to utilize the extracts from geofabrik is the right approach. Extracting the data takes a long time, so this should speed the map generation process up considerably.

But I also think that this should be more flexible in the sense that other sources should be allowed more easily. The main rationale for this is to allow others to hook in their extracts (or custom generated data sources). This would not only be for their benefits, but also for ours: the more the scripts get used by others, the more input we will get in terms of bug fixes, improvements (maybe wishful thinking).

I think the new scripts need some more error handling. One of the main problems with the current setup has been that the many errors that occur are not handled correctly and result in corrupt files. 

When everything runs fine, our current scripts are fine (if slow), but if errors occur, we generally do not know until someone alerts us. The current setup produces too many logs in too many places. 

What we need is a) an alert if a generation for a region failed that is suitable for a cron alert, b) a hint to log files where more info is to be found. I think the new scripts will also produce so much logging noise so that the result is not very useful.

There also needs to be something that refreshes the coastline files. They do change a bit and should really be in sync as much as possible. Before I run a batch on our server, I delete the input files, forcing our current script to get the coastlines again.

Another point: If I read the scripts correctly, osmosis will now overwrite a previously generated file in situ (unless there will be other scripts that move the file after it has been generated). This will not only result in an invalid file while it is being generated (which then might be picked up by our mirrors), but also if any error occurs, it will leave an invalid file in place. 

Something else we put in place was to check the date of files when to regenerate something. This has been useful to rerun the whole script if errors occurred to speed the regeneration of files up that were corrupted. 

The size check before overwriting was also useful (but failed with the recent germany.map because the sizes were within the allowed range for a change).

I am really not a script programmer and while I can read the scripts, I often fail to see the subtle problems. I have the feeling there might be more stuff lurking.

Ludwig

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mapsfo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mapsforge-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/6e51f8c1-b56e-5a67-0e9f-bb7e7f9f51bc%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Emux]

Thanks for the feedback Ludwig!

This is an initial implementation to have a base to work on.

The main script is intended to ease the process for Geofabrik extracts.
We can add other sources in their own separate scripts if needed (could require TagTransform).

I posted the process in a public repository, so that community can test it (some have started), propose ideas and post pull requests with improvements.

--
Emux

[Robin Boldt]

Thanks Ludwig, there are a couple of good points mentioned that would be nice if they would be included in the map-creator. I opened a couple of issues for each of your ideas (I hope I didn't miss anything).

To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

[Emux]

Thanks for the issues creation Robin!

--
Emux

[Emux]

I pushed another set of improvements in new Mapsforge Creator.

Now it's more automated than ever and supports many advanced features, that existed in old creator and some new ones.

Like maps v5, automatic data / coastlines download & update, check maps date & size, TagTransform, and many others.

If you use the new creator and have ideas for further enhancements, please inform us.

--
Emux

[MERCURY EDITS]

can anyone help me to create .map file for this .osm file found in this link:::https://drive.google.com/file/d/1pHPbC9zdcP83aP8JqzV81WSPHBCjN3CV/view?usp=sharing

[Emux]

Can see the map creation guide with all the details.

--
Emux

[Carlos Ferreira]

Hi Emux,

I have an hard requirement to create a map application that works fully offline and displays an USA map. The map should allow multiple zoom levels and all the maps together should fill at most 30 MBs of disk space. In your opinion, is it possible to generate such a map? Even if we have a map with roads only? Right now, I am using .map files of all US states and they require around 2.5 GBs of disk space. The major constraint is bandwidth. That's why the client asked for an offline map solution and that's way they also want to download very small map files.

Best Regards,

Carlos

[Carlos Ferreira]

Hi again,

I forgot to tell you. I am using the project https://github.com/mapsforge/vtm.

Thanks,

Carlos

[Emux]

Well we have in server a world.map with less details of 3MB.

Can create small maps for any area, but need to decide which data to put in them to keep their size down.
And can also keep one or two of the zoom-intervals in map-writer options to have only simplified geometries.

--
Emux

[Carlos Ferreira]

I will try to create something with the mapsforge-creator and see what I can get.

Thank you for the very quick answer,

Carlos

[Emux]

mapsforge-creator is more for easy map creation with all data that the input osm / pbf files contain.

If need to filter the map data to create smaller maps, then can use the detailed map creation guide for help in the steps.

--
Emux

[Carlos Ferreira]

Thanks for pointing me in the right direction.

[Carlos Ferreira]

Hi Emux,

If I use a TileSource created with the OSciMap4TileSource builder, can I stop the downloading of tiles for certain zoom levels, e.g., between 5 and 10? If so, how can that be done?

Best Regards,

Carlos

quinta-feira, 28 de Junho de 2018 às 12:21:26 UTC+1, Emux escreveu:

[Carlos Ferreira]

My question was not clear enough. I know I can deny the downloading of tiles between a zoom level with zoomMin and zoomMax. What I want to do is to download tiles if i am close to POI and  to stop downloading tiles if I am too far away from one. It will depend on the lat, long and zoom levels.

Best Regards,

Carlos

[Emux]

This topic is for Mapsforge creator v2.

For questions related to other subjects please create new topics.

--
Emux

[Carlos Ferreira]

Ok, will do that.