How to filter out silly OSM relations/ways ?

[openandromaps osm]

Hi,

I have several maps taking ages (2-3days) to render - for no reason.
Now Great_Britain is among them.

One of the possible reasons for dramaticly increasing creation times with mapsforge are monster mp_relations.
So I wrote a script to count members for ways and relations and - got it.

There is a new relation ID 7681896 with 6543 members.

Tagged as

name	Ireland
name:de	Irland
name:en	Ireland
name:es	Irlanda
place	island
type	multipolygon

Its a simple collection of all coastlines, no need for this at all - I render land/sea/coast from preprozessed shp_files.

I'v found such relations with all maps needing much to much time for creation.
... Spain_Portugal, Arabian Peninsular,...

So I want to filter out these relations.
With TagTransform it would be possible to add a rule to skip a relation with the tag/values included above: island,ireland,multipolygon
IMO it would be more convenient to filter out single relations/ways by OSM_ID.

Does anyone know how to do this in map_creation process using the usual tools like osmconvert, osmosis and (preferable) TagTransform?


Best regards
Christian

[Emux]

Hi Christian,

My first thought would be TagTransform too.

Have you tried Osmfilter?
Its documentation mentions special tags like @id, are they for that purpose?

--
Emux

[openandromaps osm]

Hi Emux,

Osmfilter with --drop-relations="@id" works fine :-)
However it would mean to add one more step in the map creation prozess couse I dont use Osmfilter on the whole OSM-File in any step.

But your hint took me on the right way and I have found an easy solution (only tested on dummy data yet):
I made a csv file with 2 columns: type (way/rel) and the ID that have to be skipped during final merging of the verious preprozessed parts of the OSM data.
= cycleways, hikingways, land/sea, labels, aso, aso

Right bevor final merging a perl script (a very simple and short one) creates an OSM-File called "id_delete.osm" out of the csv file including blank relations/ways out of the csv.
So the monster relation 7681896 is written to "id_delete.osm" as

<?xml version='1.0' encoding='UTF-8'?>
<osm version="0.6" generator="csv2osm_kech61">
  <bounds minlon="-10.00000" minlat="49.00000" maxlon="2.50000" maxlat="61.25000" origin="osmfilter 1.2S"/>
  <relation id="7681896" version="1" timestamp="1970-01-01T00:59:59Z" changeset="1">
    <tag k="type" v="dummy"/>
  </relation>
</osm>

id_delete.osm is placed first in the final osmosis merge queue - so it has priority and all ids placed in the csv file are "deleted" = tagged as type=dummy or something like this.

osmosis.bat --rx id_delete.osm --sort-0.6 --rx base.osm --sort-0.6 --merge --wx result.osm

The advantage of this solution is the prozessing overhead is negligable and the base data is available during whole preprozessing - eg for prozessing node labels and their priority out of MPs.

best regards
Christian

[Emux]

Thanks for the solution description Christian, it seems simple and fast!

--
Emux

[freizeitkarte]

To detect and configure problematic relations will (in the long run) probably be very time consuming and error-prone.

Isn't it better to optimize the mapwriter performance?

What's the reason that mapwriter slows down in such a dramatically way?

[openandromaps osm]

.. and it depends on the tagmapping.


The Problem in my case is the area=yes in MP's cause place=island|islet|archipelago|peninsula is only included as NODE in my tagmapping and has no influece on prozessing tiimes.
( i preprozess these relations and set NODES for the labels so I dont need mapsforge writer for setting these labels ).
BUT - if anyone relys on mapsforge writer for prozessing the polygon-labes he/she has a serious problem.
EG. there is an island in Japan tagged as relation with > 18.000 members.

My way to identify such relations is to filter + count members. So for the island/MPs:

osmfilter planet.o5m --keep-relations="( place=island or place=islet or place=archipelago or place=peninsula ) and type=multipolygon and area=yes" --drop-nodes --drop-ways --ignore-dependencies -o=mp_isles.osm

then I count the members (take care if  relations are stacked)
 

   use strict;

    my $infile     = shift;    #mapname


    my $inextension   = "osm";
    my $outextension  = "csv";

    my $infilestring  = $infile . "." . $inextension;
    my $outfilestring = $infile . "_rels." . $outextension;


    print "\n", $infilestring, "\n";
    print $outfilestring, "\n";

    my $mem_count=0;
    my $cnt=0;
    my $id = 0;
    my $fh;
    my $line;
    my @speicher = ();
    my $t;
    my $member;


    open( fh, $infilestring ) or die $!;
    open( fh_out, ">", $outfilestring ) or die $!;

            while ( $line = <fh> ) {            
                if ( $line =~ /^\s*<relation id=["']([0-9]*)["']/ ) {
                    $id =  $1;
                    $cnt++;
                    #print fh_out $id,";";
                }
                elsif ( $line =~ /^*<member type="way"/ ) {
                    $mem_count++;
                }
                elsif ( $line =~ /^\s*<\/relation/ ) {
                    if ($mem_count > 100){
                    print fh_out $id,";";
                    print fh_out $mem_count,"\n";    
                    $member=(join ':',$mem_count,$id);
                    push (@speicher, $member );
                   
                    }
                $mem_count=0;    
                }
               

            }

    close(fh);
    close(fh_out);    
   

Result:

relID;members
4094778;200
5631846;1333
5685658;142
5695352;123
5729610;253
5868101;450
6038068;4674
6303382;425

Well, checking if these relations include other important tags is a good idea for shure ;-)

For relations like forest,wood,water (a BIG issue in Cannada) thats a complete different story.
I remember some years ago there were monster relations in Cannada, even the performant OSM tileserver were slowed down.
The only solution was that some mappers splitted these relations.

So further optimizing mapsforge writer for big relations would be fine - if this could be done...

Result for skipping the relations above for GreatBritain.map:
Drop of prozessing time from 3 days + (I stopped rendering after 3 days at 40% level 4)  down to 8 hours.

Best regards
Christian

[freizeitkarte]

It seems that multithreading is blocked in case of a mapwriter performance break down. But this is only an assumption.

[Emux]

Multithreading in map-writer is now configurable via threads option (default 1).

--
Emux

[freizeitkarte]

I run mapwriter with the option "threads=8" (because of 8 cores). But under some circumstances, perhaps because of the here described issue, mapwriter seems to use only one thread. Linux "top" shows only 100% instead of 400-700% cpu load.

[freizeitkarte]

I have run a test with the geofabrik GBR extract and mapwriter 0.8. The build performance is (very) slow. CPU load sticks at 100 percent. Expected is 400-700 percent. The here mentioned "monster relation" (id 7681896, the coastlines of IRL not GBR) has no influence on the performance. Conclusion: There must be something else that slows down the performance.

[Emux]

Prefer to use the writers master-SNAPSHOT version, to have always the latest improvements.

Also according to #930 the "label-position" algorithm (as option or in tag-mapping) can slow down significantly the process depending the region.
(that's why I removed it from default tag-mapping.xml)

--
Emux

[freizeitkarte]

"label-position" was not used for the test build of GBR. The problem with "label-position" is the full cpu consume and a never ending build. The problem with the GBR build is different. It seems that only one cpu (available are 8) is used.  

[Emux]

What command line for the map-writer did you use?

--
Emux

[freizeitkarte]

sh /home/kto/freizeitkarte-android-1712/tools/osmosis/bin/osmosis --read-pbf /home/kto/freizeitkarte-android-1712/work/Freizeitkarte_GBR/Freizeitkarte_GBR.transformed_plus_routes.osm.pbf --mapfile-writer file=/home/kto/freizeitkarte-android-1712/install/Freizeitkarte_GBR/Freizeitkarte_GBR.map bbox='48.76,-9.41,61.13,2.25' type=ram threads=8 debug-file=false map-start-position='51.5055,-0.1143' map-start-zoom=12 tag-conf-file=/home/kto/freizeitkarte-android-1712/theme/tag_mapping.xml simplification-factor=1.0 comment="(c) Map: FZK project (free for private use); Map data: OpenStreetMap contributors; Contour data: U.S. Geological Survey and J. de Ferranti"

[Emux]

Preferably common tests should be done with original Geofabrik extracts (cannot check custom data) and default writer settings / mapping first.

Also no need for quotes on map-writer options, don't know if they're even used correctly that way.

osmosis --rb file=great-britain-latest.osm.pbf --mw file=great-britain.map

with type=ram|hd and threads=8 if needed.

--
Emux

[Emux]

Also should set Osmosis with plenty of RAM via Java -Xmx option, e.g.

JAVACMD_OPTIONS="-Xmx8g"

--
Emux

[freizeitkarte]

The JAVACMD_OPTIONS parameter was set in the environment to "-Xmx56000" (56 GB).

[Emux]

Better use a unit in -Xmx as according to the documentation:

"If you do not add a unit, you will get the exact value you state; for example, 64 will be interpreted as 64 bytes, not 64 megabytes or 64 kilobytes."

--
Emux

[freizeitkarte]

Here the test results based on unmodified GBR data and the latest mapsforge tag-mapping.xml file (java heap was 56 GB):

kto@mercator:~/gbr-test$ sh /home/kto/freizeitkarte-android-1712/tools/osmosis/bin/osmosis --read-pbf great-britain-latest.osm.pbf --mapfile-writer file=Testmap_GBR.map bbox='48.76,-9.41,61.13,2.25' type=ram threads=8 debug-file=false map-start-position='51.5055,-0.1143' map-start-zoom=12 tag-conf-file=tag-mapping.xml simplification-factor=1.0 comment="Testmap to verify mapwriter performance"
Nov 08, 2017 5:13:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Osmosis Version 0.45
Nov 08, 2017 5:13:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Preparing pipeline.
Nov 08, 2017 5:13:06 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>
INFO: mapfile-writer version: mapsforge-map-writer-0.8.0
Nov 08, 2017 5:13:06 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask <init>
INFO: mapfile format specification version: 3
Nov 08, 2017 5:13:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Launching pipeline execution.
Nov 08, 2017 5:13:06 PM org.openstreetmap.osmosis.core.Osmosis run
INFO: Pipeline executing, waiting for completion.
Nov 08, 2017 5:13:06 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask process
INFO: start reading data...

Nov 08, 2017 5:16:32 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFO: completing read...
Nov 08, 2017 5:57:08 PM org.mapsforge.map.writer.osmosis.MapFileWriterTask complete
INFO: start writing file...
Nov 08, 2017 5:57:13 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 100% of sub file for zoom interval 1 (5,0,7)
Nov 08, 2017 5:57:13 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 10% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:13 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 20% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:14 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 30% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:16 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 40% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:20 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 50% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:22 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 60% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:22 PM org.mapsforge.map.writer.util.GeoUtils clipToTile
WARNING: invalid geometry prior to tile clipping, trying to repair 328880344
Nov 08, 2017 5:57:22 PM org.mapsforge.map.writer.util.GeoUtils clipToTile
WARNING: invalid geometry even after attempt to fix 328880344
Nov 08, 2017 5:57:22 PM org.mapsforge.map.writer.util.GeoUtils clipToTile
WARNING: clipped way is not valid, trying to repair it: 328880344
Nov 08, 2017 5:57:29 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 70% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:38 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 80% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:44 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 90% of sub file for zoom interval 2 (10,8,11)
Nov 08, 2017 5:57:50 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 10% of sub file for zoom interval 3 (14,12,21)
Nov 08, 2017 5:57:54 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 20% of sub file for zoom interval 3 (14,12,21)
Nov 08, 2017 7:37:26 PM org.mapsforge.map.writer.MapFileWriter writeSubfile
INFO: written 30% of sub file for zoom interval 3 (14,12,21)
...

Here some additional cpu load data from top:

INFO: completing read...

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                        
27498 kto       20   0 60.324g 0.030t  17860 S 100.3 48.6  19:17.78 java                                                          


INFO: written 20% of sub file for zoom interval 3 (14,12,21)

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                        
27498 kto       20   0 60.834g 0.033t  17860 S 100.3 54.3  64:43.28 java

CPU load sticks at 100% ... expected is 400-700%.

The step from 20% to 30% took 100 minutes. The build process will probably consume addition 600 minutes to finish.

Conclusion: The build process for GBR with an unmodified data source and the default mapsforge tag-mapping.xml is as slow as the build based on my data set.

[Emux]

Please try the simplest command (again avoid quotes) with snapshot version (and default options like simplification):

osmosis --rb file=great-britain-latest.osm.pbf --mw file=great-britain.map type=ram threads=8

BTW from the old days map-writer was never using full cpu load after the initial processing.

--
Emux

[Emux]

BTW changing the "threads" number has an apparent difference in creation time, so clearly plays a role in the whole process.

--
Emux

[freizeitkarte]

BTW from the old days map-writer was never using full cpu load after the initial processing.

Hmm ... what is your observation concerning CPU load? 

[freizeitkarte]

I can confirm the analysis / finding from Christian.

The reason for the performance breakdown of the GBR build is a huge relation: id=6038068 (with 4674 members)

The relation describes the land mass (coastline) of great britain and has the tags:

- type=multipolygon

- place=island

- area=yes

The resolving of the relation leads to an huge area. Nearly any other map object lays within this area.

The default tag-mapping.xml imports this area (closed way) via the tag mapping "area=yes".

Test with relation 6038068 included:

Processing: osmosis --read-pbf great-britain-latest.osm.pbf --mapfile-writer file=Testmap_GBR.map type=ram threads=8

Data: unmodified great britain extract from geofabrik

Environment: 56 GB heap, mapwriter 0.8, default mapsforge tag-mapping

Runtime: 116589 seconds (32.4 hours)

Test with relation 6038068 removed:

Processing: osmosis --read-pbf great-britain-latest-drop.osm.pbf --mapfile-writer file=Testmap_GBR-drop.map type=ram threads=8

Data: modified great britain extract from geofabrik (relation 6038068 removed)

Environment: 56 GB heap, mapwriter 0.8, default mapsforge tag-mapping

Runtime: 8717 seconds (2.4 hours)

Procedure to remove the relation from the data:

osmconvert great-britain-latest.osm.pbf --verbose -o=great-britain-latest.osm
osmfilter great-britain-latest.osm --verbose --drop-relations="@id=7681896 or @id=6038068" -o=great-britain-latest-drop.osm
grep --after-context=15 --before-context=15 "id=\"6038068\"" great-britain-latest-drop.osm
osmconvert great-britain-latest-drop.osm --verbose -o=great-britain-latest-drop.osm.pbf

With relation 6038068 the build process sticks at 100% cpu usage:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                        
27498 kto       20   0 60.834g 0.033t  17860 S 100.3 54.3  64:43.28 java

Without relation 6038068 the build process runs with much more cpu usage:

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                            

25876 kto       20   0 60.271g 0.050t  17708 S 737.2 82.4  49:17.34 java

What reduces the build performance of mapwriter in such a dramatically way?       

PS: The full output of both build processes is available and could be postet.          

[Emux]

Thanks for the tests.

Setting multiple threads in a process does not mean that all threads will be occupied "all the time", if there are no jobs to set on them.

So here would need a debugging in map-writer to see what happens with such large elements.
Probably that job continues while all other are finished (using one core), and new jobs will be processed after this one is finished too, as would need its output for them.

--
Emux

[openandromaps osm]

Its this one:

rel id 6038068
GreatBritain

place=island
type=multipolygon
area=yes

Skip it and everything runs fine for this region.
There are several relations like this one.

[Ludwig]

I think we should start collecting these relation ids and add another step to the mapgenerator process to filter those out automatically, to improve our map building process.

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-dev+unsubscribe@googlegroups.com.
To post to this group, send email to mapsfo...@googlegroups.com.
Visit this group at https://groups.google.com/group/mapsforge-dev.
To view this discussion on the web visit https://groups.google.com/d/msgid/mapsforge-dev/6d07a0bd-8be8-4d97-b18a-3bd94d2ffa70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Emux]

That would help in processes that handle official OSM data.

Mapsforge maps can be made from many different sources and custom data.

So need to be careful with ids fixed inside the code, unless we mean pre-process the osm data or optional map-writer steps.

--
Emux

[Christian Kernbeis]

The easiest way to solve this would be to modify the tagging in OSM.
There was such a relation = Spain Peninsular.

Frederic Ramm changed the tagging to
type=boundary
boundary=land_area
= problems gone

Now SainPortugal map reders fine
There is the same mess with relation arab_peninsular and some others, filter place + area + multipolygon - there lots of these rels.

AND it does not only affect mapsforge
- try to load these relations with overpass_turbo, its crashing
- try to load these relations in osm https://www.openstreetmap.org/relation/6038068 - if you zoom to GB browser is crashing
and I'm pretty shure that tileservers will run into problems with these rels.

I added one additional step to preprozessing, following the hint from EMUX = download planet_latest > to 05m > OSMFilter --drop-relations @id,@id....
Its the worst method to "solve" such problems but I'm tired to change tagging in OSM and start discussions ending in "we don't map for the renderers"

To add a fixed blacklist of rel-IDs to the mapsforge writer is a IMO a problem cause which realtion are cousing trouble depends on personal tag_mapping too.


Best regards
Christian
OAM.org

Am Donnerstag, 16. November 2017 11:07:35 UTC+1 schrieb Ludwig:

I think we should start collecting these relation ids and add another step to the mapgenerator process to filter those out automatically, to improve our map building process.

On 16 November 2017 at 15:27, openandromaps osm <openan...@gmail.com> wrote:

Its this one:

rel id 6038068
GreatBritain

place=island
type=multipolygon
area=yes

Skip it and everything runs fine for this region.
There are several relations like this one.




Am Mittwoch, 8. November 2017 06:34:44 UTC+1 schrieb freizeitkarte:

I have run a test with the geofabrik GBR extract and mapwriter 0.8. The build performance is (very) slow. CPU load sticks at 100 percent. Expected is 400-700 percent. The here mentioned "monster relation" (id 7681896, the coastlines of IRL not GBR) has no influence on the performance. Conclusion: There must be something else that slows down the performance.

--
You received this message because you are subscribed to the Google Groups "mapsforge-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to mapsforge-de...@googlegroups.com.

[Emux]

Certainly not fixed.

Probably add a new option in map-writer for an ids list to filter, something like:

filter-ids=1,2,3,...

--
Emux

[freizeitkarte]

It's very difficult to find huge problematic multipolygons. It would be very helpful if mapwriter prints out warnings for such objects.

[openandromaps osm]

I aggree with this.
MapsforgeWriter could (eg.) write a file with MP-IDs that have to be processed according to tagmapping with --mp-warininglevel > "N" Members

[Emux]

Huge multi-polygons discovery doesn't seem a work for map-writer.

After all when we get there, the write process has already begin.
What to do, run writer 2 times? Once for find the polygons and another to filter them?

Common OSM tools should provide such feature, don't they?

--
Emux

[Emux]

Or you both mean something different?

--
Emux

[Robin Boldt]

The big issue with OSM IDs is that they change. They are not static, writing lists of ids and suddenly they change is no solution.

I agree that a possible solution to the problem could be to simply ignore relations that exceed a certain amount of members.

> MapsforgeWriter could (eg.) write a file with MP-IDs that have to be processed according to tagmapping with --mp-warininglevel > "N" Members

Something like --skip-relations=1000000 (if the relation has more than 1 000 000 members - the number is completely made up BTW).

Probably the slowdown happens doing some "is in relation" check, even though these checks are fast (see ray casting algorithm), they scale linear with the number of members. Also running simplifications on huge geometries can slow down things, but since the slowdown happens for larger zoom areas I don't think that the reason is caused by simplification.

Cheers,

Robin

[Emux]

I created #1019 in repository to check it.

--
Emux