Various problems with TIle rendering and the solution to them

[Dominik Pfennig]

Hi there,

I have developed a Mapsforge Tileserver in Java that not only serves the Tiles but also generates the data (.map Files) from an .osm.pbf source File.

While in development I encountert several problems that needed to be solved. I want to share them here to maybe help another person that encounters them.

The Tileserver Project is Opensource and can be visited at:https://gitlab.metager3.de/open-source/TileServer

The Tileserver can be seen in action at: https://maps.metager.de/map/10.417353764581387,50.977732997897476,9 (Be sure to have the Map centered on Europe as the Map currently is limited to that area)

The problems I encountered and solved for my instance are the following:

1. Directly at the coastlines were areas that seemed to be in the water (Streets, POIs, etc.)
2. Labels on the Tiles were rendered sometimes only in parts or not even shown at all. (Cut Labels)
3. In some areas of the Map the data seemed to be shifted along the X-Achsis
4. Failure in Mapfile Creation for some areas

Directly at the coastlines were areas that seemed to be in the water (Streets, POIs, etc.)

When you follow the Map creation guide be sure to use the Land Polygons mentioned (File size: ~500MB) and not the simplified version (File size ~ 23MB). I think at some point the latter one was linked in the guide. If you use the simplified one you won't get exact coastlines for larger zoom levels and thus floating streets in the water.

Labels on the Tiles were rendered sometimes only in parts or not even shown at all. (Cut Labels)

If you use the database renderer class for rendering the complete Tile including the labels Chances are that your Labels will get cut or not even shown at all. Instead you should setup the Databaserenderer to render everything but the Labels and after that add the data for the labels yourself (like Mapsforge does in the Labellayer)

this.databaseRenderer = new DatabaseRenderer(this.multiMapDataStore, GRAPHIC_FACTORY, tileCache, null,

false, false, null);

To do so create a Labelstore along with the Databaserenderer:

this.labelStore = new MapDataStoreLabelStore(multiMapDataStore, renderThemeFuture, 1f, displayModel,

GRAPHIC_FACTORY);

And then when you want to render a Tile you will use the Databaserenderer to create the Image without Labels but you keep the Data for the Image in Memory and add the custom Label Data afterwards.

Following creates the Processes to render the complete Tile:

if (!tileFile.exists()) {

// There is no generated Tile yet. We need to generate it.

BufferedImage image;

try (PipedOutputStream out = new PipedOutputStream();

PipedInputStream in = new PipedInputStream(out);) {

// Generate the Tile without labels

org.mapsforge.core.model.Tile tile = new org.mapsforge.core.model.Tile(x, y, (byte)z, 256);

Tile newTile = new Tile(tile, mapsforgeManager, out);

Future<?> tileFuture = tileExecutor.submit(newTile);

// Get the Labels

Future<List<MapElementContainer>> labelFetcher = tileExecutor.submit(new LabelFetcher(tile, mapsforgeManager));

// Read the Labels

List<MapElementContainer>labels = labelFetcher.get();

// Read in the Tile without Label

image = ImageIO.read(in);

tileFuture.get();

Canvas canvas = (Canvas)AwtGraphicFactory.createGraphicContext(image.getGraphics());

for(MapElementContainer element : labels) {

element.draw(canvas, tile.getOrigin(), mapsforgeManager.getGRAPHIC_FACTORY().createMatrix(), mapsforgeManager.getDisplayModel().getFilter());

}

}

if(z <= 13) {

// Save the File for Later Use

tileFile.getParentFile().mkdirs();

tileFile.createNewFile();

try(FileOutputStream out = new FileOutputStream(tileFile)){

ImageIO.write(image, "png", out);

}

}

ImageIO.write(image, "png", os);

}else {

try(FileInputStream in = new FileInputStream(tileFile)){

IOUtils.copy(in, os);

}

}

I use the PipedStreams to be able to use the AwtTileBitmap.compress() method to create an InputStream we can read from instead of an Outputstream to write to a File. And have the Image in Memory instead of being written to a File.

The Tile uses above mentioned Databaserenderer to render everything but the Labels:

public class Tile implements Runnable {

private org.mapsforge.core.model.Tile tile;

private PipedOutputStream out;

private MapsforgeManager mapsforgeManager;

public Tile(org.mapsforge.core.model.Tile tile, MapsforgeManager mapsforgeManager, PipedOutputStream out) {

this.tile = tile;

this.mapsforgeManager = mapsforgeManager;

this.out = out;

}

@Override

public void run() {

try {

mapsforgeManager.getRenderThemeFuture().incrementRefCount();

RendererJob rendererJob = new RendererJob(this.tile, mapsforgeManager.getMultiMapDataStore(), mapsforgeManager.getRenderThemeFuture(), mapsforgeManager.getDisplayModel(),

(float) 1, false, false);

AwtTileBitmap tileImage = (AwtTileBitmap) mapsforgeManager.getDatabaseRenderer().executeJob(rendererJob);

if(tileImage != null)

tileImage.compress(out);

this.mapsforgeManager.getRenderThemeFuture().decrementRefCount();

} catch (IOException e) {

e.printStackTrace();

}finally {

try {

out.flush();

} catch (IOException e) {

e.printStackTrace();

}

}

}

}

The LabelFetcher uses above mentioned LabelStore to get the Label data for not only the requested Tile but for all 8 Tiles around it, too. That way we will have all Labels even if the Labels cross the border of the requested Tile.

The Code looks the following:

public class LabelFetcher implements Callable<List<MapElementContainer>> {

private MapsforgeManager mapsforgeManager;

private Tile tile;

public LabelFetcher(Tile tile, MapsforgeManager mapsforgeManager) {

this.tile = tile;

this.mapsforgeManager = mapsforgeManager;

}

@Override

public List<MapElementContainer> call() throws Exception {

Tile tileTop = tile.getAboveLeft();

Tile tileBottom = tile.getBelowRight();

List<MapElementContainer> elements = null;

for(int xTile = tileTop.tileX; xTile <= tileBottom.tileX; xTile++) {

for(int yTile = tileTop.tileY; yTile <= tileBottom.tileY; yTile++) {

if(elements == null)

elements = mapsforgeManager.getLabelStore().getVisibleItems(new Tile(xTile,yTile,tile.zoomLevel,256), new Tile(xTile,yTile,tile.zoomLevel,256));

else

elements.addAll(mapsforgeManager.getLabelStore().getVisibleItems(new Tile(xTile,yTile,tile.zoomLevel,256), new Tile(xTile,yTile,tile.zoomLevel,256)));

}

}

elements = LayerUtil.collisionFreeOrdered(elements);

Collections.sort(elements);

return elements;

}

}

Notice how we do not specify the upperleft Tile and the Lower right Tile in the getVisibleItems() function but instead request the data for each Tile individually. The reason is that there is an error in the readLabels() function of the MultiMapDataStore. The function looks the following:

private MapReadResult readLabels(Tile upperLeft, Tile lowerRight, boolean deduplicate) {
	MapReadResult mapReadResult = new MapReadResult();
	for (MapDataStore mdb : mapDatabases) {
	if (mdb.supportsTile(upperLeft)) {
	MapReadResult result = mdb.readLabels(upperLeft, lowerRight);
	if (result == null) {
	continue;
	}
	boolean isWater = mapReadResult.isWater & result.isWater;
	mapReadResult.isWater = isWater;
	mapReadResult.add(result, deduplicate);
	}
	}
	return mapReadResult;
	}

This snippet occurs in many places of the current Mapsforge project but there is a bug for which I did not have the time yet to create a pull request to fix. The function checks if a mapfile supports the upper left Tile and reads the data in that case. But what if the lower right Tile or any Tile in between has data within another Mapfile and not the same Mapfile as the upper left Tile. In that case the Data for that Tile will not get read properly. And thats probably why some labels are either missing or are cut in half.

To solve this my Tileserver reads the Label Data for each of the 9 Tiles individually and will for sure receive all of the available Data in all Mapfiles.

We have split Europe into 2.000 individual Mapfiles which get connected in a MultiMapDataStore so the mentioned error occured quite often before this solution and is completely gone now.

In some areas of the Map the data seemed to be shifted along the X-Achsis

That's another error that occured. After each Update (we update the data every 7 days) the error occured randomly in a different place. For some zoom levels the rendered Tiles seemed to be shifted along the X-Achsis. 

I.e. the Tile (2136,1330,12) was in the position of Tile (2135,1330,12) and the Tile (2137,1330,12) was in the position of (2136,1330,12) etc.

The next row on the Y-Achsis had them shifted by 2 Tiles etc.

This seemed like a pretty random error as it always occured in different places and sometimes was gone. But the solution was a quite easy one.

I wasn't aware of the fact that the OSM Data for the Mapfiles should be sorted. Otherwise you would create that Strange behaviour.

The Tileserver would create the following warnings when trying to read from the unsorted Mapfiles:

LOGGER.warning("invalid current block pointer: " + currentBlockPointer);

To solve this issue you should take some time after merging the OSM data with the land polygons and water areas to read in the whole dataset again and sort it with the osmosis --sort pipeline. In my case I double checked and sorted the data again after splitting the osm file into 2.000 areas and reading each one in to convert it to a mapfile and sorted that data again:

ProcessBuilder pb = new ProcessBuilder();

pb.inheritIO();

System.out.println("bbox=" + bbox.minLatitude + "," + bbox.minLongitude + "," + bbox.maxLatitude + "," + bbox.maxLongitude);

pb.command("/bin/bash", osmosisExecutable.getAbsolutePath(), "--rb", "file=" + this.osmFile.getAbsolutePath(), "--sort", "--mapfile-writer", "file=" + mapFile.getAbsolutePath(),

"preferred-languages=de", "zoom-interval-conf=5,0,7,10,8,11,12,12,13,14,14,21", "type=ram", "bbox=" + bbox.minLatitude + "," + bbox.minLongitude + "," + bbox.maxLatitude + "," + bbox.maxLongitude);

Failure in Mapfile Creation for some areas

A lot of times I read of people that cannot create specific mapfiles because they have the osmosis process go OOM. Thats why I want to share my experience with memory consumption of the mapfile-writer process. I tried a lot of different configurations and they all worked for most of the mapfiles I generated but not for all. The Update process of above mentioned Tileserver generates around 2.000 Mapfiles for Europe every 7 days and my experience is that you get away with giving osmosis 2-4GB of memory for most of the Mapfiles (when they are as small areas as mine: around 10-20MB). But in some cases the process exeeds that limit by a lot. It doesn't matter if you specify type=hd or type=ram some areas simply consume a lot of memory and take a lot longer to generate than others.

I stopeed getting errors once I gave 12GB of ram to the osmosis process. This value depends heavily on the size of the OSM Data you want to convert.

Our server calls 10 of those osmosis processes at the same time and converts all 2.000 OSM Files to Mapfiles in around 15 hours without a single error.

Conclusion is quite simple:

If your process runs OOM when generating Mapfiles increase the memory for osmosis. If you cannot give more memory because of system limitations you will need to split down the desired area in smaller chunks and generate Mapfiles for those. You can combine them again in a MultiMapDataStore.

I don't think there is another way currently.

However I hope this list of problems and how to solve them will save someone some time in the future.

Happy Mapping

[Emux]

Thanks for sharing the project details!

1. Map creation guide states the "detailed" variant of land polygons that should use from OpenStreetMap Data website for optimal result.

The "simplified" land polygons are only mentioned in world map creation section, as that map is supposed to host less overall detail.

2. It has been explained many times in forum that if want to use Mapsforge for tiles creation, it was not its initial purpose and there are usually more convenient tools.
Like instead of generate / provide large bitmap tiles, to better use small vector tiles (mvt, geoson, ...) that are suited for modern clients, especially on mobile with required variable tile sizes.

BTW Mapsforge uses an internal label / symbol cache for adjacent tiles to cover such corner cases, either using in MapView a label layer or not.
If generate externally arbitrary bitmap tiles on demand, then need to solve that in external way too.

3. Strange we have not seen such case while building regular map files, it could depend on how everyone handles the input osm data before feeding them in map writer.

--
Emux

[Robin Boldt]

Hi Dominik,

thanks for sharing. This looks really impressive. I looked around the map and it feels extremely fast, good job.

I have seen some minor artefacts in high zoom levels like z>20, which are not a big issue anyway.

Is it possible to also server different formats than raster tiles, for example mvt or geojson?

Do you feel like sharing some more details about how the service works? Are the tiles rendered live, the URLs look like they are cached? Are they pregenerated? The README states that only z<13 are pregenerated. What specs are required to run the server, for example for Europe, as shown in your example.

Thanks in advance,

Robin

[Ludwig]

Thanks for sharing.

As the issues described here have been raised multiple times in the past, can I suggest to put an edited version of this on the mapsforge documentation website for easier reference?

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.
To post to this group, send email to mapsfo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mapsforge-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/6893e18f-61dc-4201-a6da-25cfbba9aef3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dominik Pfennig]

Hi Robin,

thank you.

I have seen some minor artefacts in high zoom levels like z>20, which are not a big issue anyway.

Can you give me an example? It might have to do with my render style. I didn't put too much effort in it yet.

 

Is it possible to also server different formats than raster tiles, for example mvt or geojson?

Currently the server only serves raster tiles. However it should be quite simple to generate other output formats. There will be an update to do so in the future because I want to support label rotation which is not possible with those raster tiles. 

 

Do you feel like sharing some more details about how the service works? Are the tiles rendered live, the URLs look like they are cached? Are they pregenerated? The README states that only z<13 are pregenerated. What specs are required to run the server, for example for Europe, as shown in your example.

I use two servers. One to generate updated mapfiles and one to use those mapfiles to render the tiles.

The Tileserver is a pretty standard root server. Mapsforge handles resources awesomely well with MultiMapDataStore.

The specs are:

~48GB Ram

Intel(R) Core(TM) i7 CPU         920  @ 2.67GHz with 8 cores

But I just use a Xmx of 12GB. The server is constantly at 0.x load and doesn't use 12GB so you probably can get away with less.

There is a small error in the readme: Only z<=13 are pregenerated. That's for Europe around 18 million Tiles with a size of around 20GB. All Tiles with z >= 14 are rendered live. They don't get cached either since rendering is fast enough on the higher zoom levels and I don't want to store much more tiles in the cache.

To make the update process even quicker and prepare for rendering the whole world I will not pregenerate tiles in future releases. Instead the Tileserver will save all Tiles with z <= 13 permanently to disk once generated by user request. Once there are newer mapfiles the TileServer will then send out the old cached Tile and trigger a process that updates the tile in the background for the next request.

The server that generates the mapfiles for europe is quite powerfull however you should get away with lower specs. You just  need to make sure to give enough RAM for processing the desired area. The more RAM you provide the faster the update process will be. I have tested the updater on my local machine with just 4GB Ram for a germany extract but I suggest using at least 12GB

Specs for the updater server:

256GB RAM

Intel(R) Xeon(R) CPU E5-1650 v3 @ 3.50GHz with 12 Cores

I start the Updater with 128GB RAM and it takes ~24h for processing the europe extract including mapfile generation and pregenerating the 18 Million tiles.

[Robin Boldt]

Hi Dominik,

thanks for the details. This sounds like a really great project!

> Can you give me an example? It might have to do with my render style. I didn't put too much effort in it yet.

I tried to find an example, but couldn't find one right now, maybe this was only a temporary issue or something like this... sorry for the troubles ;)

> There will be an update to do so in the future because I want to support label rotation which is not possible with those raster tiles. 

Nice, this sounds really interesting.

Please keep us updated about the process. I think it will be really interesting to see how the project evolves. 

Cheers,

Robin