Multipolygon problems

[Jochen Topf]

Hi!

In the last days the OpenStreetMap Carto Style 4.0 is being deployed on
the OSMF tile servers. This new version of the style doesn't take
old-style multipolygons (where the tags are on the outer ways instead of
on the relation) into account any more. In a huge effort in the last
months we have converted all old-style multipolygons to the modern
tagging, so this is a good step!

Unfortunately, as a side-effect of this change, many multipolygon
relations now appear wrong on the map. This is the case for multipolygon
relations that have the same tags on the relation as well as on (some of
the) outer or inner ways. This is *wrong* tagging, and needs to be
fixed. (Note that this always was wrong tagging, even before we
deprecated old-style multipolygons, but the way the software worked with
old-style multipolygons, this problem was not visible on the map. But
now it is.)

Here is an example: http://www.openstreetmap.org/relation/1330741 . As
you can see (unless somebody fixes this :-) the clearing in the forest
that should just have grass, also has tree symbols on it. In many other
cases it is not this obvious, there are just islands in a river missing
or so.

There are about 50,000 cases like this worldwide, forests, waterways,
all sorts of areas. One of the largest problems is in New Zealand. There
are about 15,000 affected relations, most from the LINZ imports.

First, we have to make sure that there are no further imports of broken
data. I hope the people who have done those imports (and might still
continue) are here on this mailing list. If not please make them aware
of this issue and/or put me in touch with them. Second, somebody needs
to clean up the broken data, either automatically or manually. 99% of
the data has not been changed since the import, so it might be feasible
to do an automatic cleanup, but somebody has to do this. Otherwise we'll
have to do a manual cleanup, through tools such as Maproulette and the
OSM Inspector. I am currently in the process of creating Maproulette
challenges for other areas of the planet, but will not do this for
New Zealand at this time. Lets discuss this here first.

I can provide OSM data extracts, statistics, etc. if somebody wants to
look at the data.

All of this is part of a larger effort to fix areas in OSM. See
http://area.jochentopf.com/ for more information. There is also a thread
on the talk mailinglist at
https://lists.openstreetmap.org/pipermail/talk/2017-June/078203.html
and this issue
https://github.com/osmlab/fixing-polygons-in-osm/issues/36 .
News of the effort are posted regularly to
https://github.com/osmlab/fixing-polygons-in-osm/issues/15 .

Jochen
--
Jochen Topf joc...@topf.org https://www.jochentopf.com/ +49-351-31778688

[Jochen Topf]

Hi!

A week ago I wrote this email and nobody answered it yet. Does that
mean that nobody feels responsible for the import that created this data
and nobody here cares for this data?

I see three ways forward:
* We do nothing. The broken data stays in OSM. Not a good solution,
because every user of the data has to work around this or handle the
complaints.
* The New Zealand community steps up and fixes the data, automatically
or manually.
* We ask the Data Working Group to remove the broken import.

Jochen

On Thu, Jun 22, 2017 at 11:45:32AM +0200, Jochen Topf wrote:
> Date: Thu, 22 Jun 2017 11:45:32 +0200
> From: Jochen Topf <joc...@topf.org>
> To: nzop...@googlegroups.com
> Subject: [nzopengis] Multipolygon problems

> --
> You received this message because you are subscribed to the Google Groups "nzopengis" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nzopengis+...@googlegroups.com.
> To post to this group, send email to nzop...@googlegroups.com.
> Visit this group at https://groups.google.com/group/nzopengis.
> For more options, visit https://groups.google.com/d/optout.

[Rob Alley]

On Friday, June 30, 2017 at 8:30:31 PM UTC+12, Jochen wrote:

Hi!

A week ago I wrote this email and nobody answered it yet. Does that
mean that nobody feels responsible for the import that created this data
and nobody here cares for this data?

Jochen, this group seems pretty dead - your post has only had 4 views in the week it has been up.  I'm keen to help out with manual clean ups where possible.

Rob. 

[Robert Coup]

Hey Rob,

OT for the thread (am thinking of a reply to Jochen atm), but

On 30 June 2017 at 10:45, Rob Alley <alle...@gmail.com> wrote:

Jochen, this group seems pretty dead - your post has only had 4 views in the week it has been up.  I'm keen to help out with manual clean ups where possible.

I think most people read via email, which isn't tracked via Google's "views".

Cheers,

Rob :)

[Robert Coup]

Hey Jochen,

Can you summarise what actually needs to change for existing data from LINZ? (New stuff should be straightforward) -- the example feature you linked appears "fixed" to me now.

I think from reading there's several cleanups overlapping:

1. we've been pretty careful about only putting in valid polygon/multipolygons, so I don't think there's anything other than isolated cases of bad geometries?

2. the key one is change of tagging on multipolygons from being on outer/inner ways to solely on the relations? Should be straightforward to update the tools for new work. You said there's 15K but OSM Inspector shows 1x "old-style multipolygon" in New Zealand (a user-created one)... can you point to a specific LINZ-tagged feature that I can examine and then use to find others?

> Second, somebody needs to clean up the broken data, either automatically or manually. 99% of the data has not been changed since the import, so it might be feasible to do an automatic cleanup, but somebody has to do this

Are there tools for doing this that have been used elsewhere?

Cheers

Rob :)

--
You received this message because you are subscribed to the Google Groups "nzopengis" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nzopengis+unsubscribe@googlegroups.com.

[Jochen Topf]

On Fri, Jun 30, 2017 at 11:10:16AM +0100, Robert Coup wrote:
> Can you summarise what *actually* needs to change for existing data from

> LINZ? (New stuff should be straightforward) -- the example feature you
> linked appears "fixed" to me now.

Unfortunately it was fixed. But there are many more cases.

> I think from reading there's several cleanups overlapping:
>
> 1. we've been pretty careful about only putting in valid
> polygon/multipolygons, so I don't think there's anything other than
> isolated cases of bad geometries?

Yes, that is not the problem here.

> 2. the key one is change of tagging on multipolygons from being on
> outer/inner ways to solely on the relations? Should be straightforward to
> update the tools for new work. You said there's 15K but OSM Inspector shows
> 1x "old-style multipolygon" in New Zealand (a user-created one)... can you
> point to a specific LINZ-tagged feature that I can examine and then use to
> find others?

Sorry, the 15k was a copy-and-paste error. That's the number for Canada.
In NZ it was "only" about 8k, some of these seem to have been fixed in
the last days, so there are about 7k left.

You can download them from here:
https://tmp.jochentopf.com/954226a3acab882d28d8500ddef8203d/same-tags-nz.pbf

As I mentioned this is not old-style tagging (with the tags on the outer
ways and not the relation), these are all fixed. This is those cases
where the tags are on the outer way AND on the relation. That is neither
old-style nor new-style, just bad. These do not show up in the OSM
inspector.

To fix this you'd have to at least remove the tags from the ways for
everything that wasn't changed since the import which is almost
everything. Better would be to review each case. And at least those that
have been changed since the import need a manual review.

> > Second, somebody needs to clean up the broken data, either automatically
> or manually. 99% of the data has not been changed since the import, so it
> might be feasible to do an automatic cleanup, but somebody has to do this
>
> Are there tools for doing this that have been used elsewhere?

JOSM? There are no tools to do something like this automatically that I
am aware of.

Jochen
--
Jochen Topf joc...@remote.org https://www.jochentopf.com/ +49-351-31778688

[Jochen Topf]

Hi!

The problematic ways are now visible on the OSM Inspector. You have to
zoom in to at least zoom level 8 to see something:

http://tools.geofabrik.de/osmi/?view=areas&lon=176.14997&lat=-39.07976&zoom=8&overlays=same_tags_on_outer_ring

[Geoff Leyland]

> On 18/07/2017, at 2:29 AM, Jochen Topf <joc...@topf.org> wrote:
>
> Hi!
>
> The problematic ways are now visible on the OSM Inspector. You have to
> zoom in to at least zoom level 8 to see something:
>
> http://tools.geofabrik.de/osmi/?view=areas&lon=176.14997&lat=-39.07976&zoom=8&overlays=same_tags_on_outer_ring

So here [1] is one of the offending polygons in id. The tags to the outer way appear to be:

LINZ:dataset mainland
LINZ:layer native_poly
LINZ:source_version 2012-06-06
attribution http://wiki.osm.org/wiki/Attribution#LINZ
natural wood
source_ref http://www.linz.govt.nz/topography/topo-maps/


The tags on the relation appear to be:

LINZ:dataset mainland
LINZ:layer native_poly
LINZ:source_version 2012-06-06
attribution http://wiki.osm.org/wiki/Attribution#LINZ
natural wood
source_ref http://www.linz.govt.nz/topography/topo-maps/
type multipolygon


First, am I looking at the right things? If I am, could you please, very slowly, explain what the tags on the two of them should be? If it's the case that "natural=wood" should be removed from the outer polygon, then could that can be automated [2]?

Geoff


[1] https://www.openstreetmap.org/relation/3046186#map=18/-36.77315/174.48228
[2] I'm afraid I know nothing of *how* to automate it

[Jochen Topf]

On Tue, Jul 18, 2017 at 08:45:49AM +1200, Geoff Leyland wrote:
> > On 18/07/2017, at 2:29 AM, Jochen Topf <joc...@topf.org> wrote:
> >
> > Hi!
> >
> > The problematic ways are now visible on the OSM Inspector. You have to
> > zoom in to at least zoom level 8 to see something:
> >
> > http://tools.geofabrik.de/osmi/?view=areas&lon=176.14997&lat=-39.07976&zoom=8&overlays=same_tags_on_outer_ring
>
> So here [1] is one of the offending polygons in id. The tags to the outer way appear to be:
>
> LINZ:dataset mainland
> LINZ:layer native_poly
> LINZ:source_version 2012-06-06
> attribution http://wiki.osm.org/wiki/Attribution#LINZ
> natural wood
> source_ref http://www.linz.govt.nz/topography/topo-maps/
>
>
> The tags on the relation appear to be:
>
> LINZ:dataset mainland
> LINZ:layer native_poly
> LINZ:source_version 2012-06-06
> attribution http://wiki.osm.org/wiki/Attribution#LINZ
> natural wood
> source_ref http://www.linz.govt.nz/topography/topo-maps/
> type multipolygon
>
>
> First, am I looking at the right things? If I am, could you please, very slowly, explain what the tags on the two of them should be? If it's the case that "natural=wood" should be removed from the outer polygon, then could that can be automated [2]?

There should be no tags at all on the outer way. The outer way doesn't
exist as an object in its own right, it is only used to define the
geometry of the (multi)polygon.

Think of it this way: Each object in OSM (node, way, or relation) has
some tags that describe what this specific object is. In addition nodes
have a location and ways "inherit" this location information from their
member nodes to form a line (or, if closed, possibly a polygon
geometry). Multipolygon relations in turn "inherit" their geometry from
the member ways which form the rings of the multipolygon.

The way this is tagged currently, you have two objects with the same
tags, the way and the relation. What the way describes is a simple
polygon without any holes in it. What the relation describes is a
polygon with a hole. Both of these objects overlap. And that doesn't
make sense and leads to the wrong rendering.

Described in another way: Say you have a nature park. The nature park
consists of a forest with a like inside. The lake is in a "hole" in the
forest, there can't be a lake somewhere and a forest at the same time.
But the nature park contains both, the forest and the lake. Then you
would put the tags for the nature park on the outer way, the tags for
the forest on the relation and the tags for the lake on the inner way
of that relation. (Of course that's only the simplest case, you might
need several multipolygons, if more ways are involved.)

And yes, I believe this could be automated to a certain degree, because
this is a very specific problem that is relatively easy to detect.

[Andrew Davidson]

This should be relatively easy to fix with JOSM and Overpass.

Using this Overpass query:

[out:xml][timeout:250];
{{geocodeArea:New Zealand}}->.searchArea;
relation["landuse"="forest"](

area.searchArea);
way(r);
way._["landuse"="forest"];
(._;>;);
out meta;

will return all of the duplicate tagged ways. You can modify the way(r); line to check for ways without roles:

way(r:"");

and inner ways:

way(r:"inner");

to make sure there are not any other tagging problems before making changes.

I'd be happy to fix these myself but I think it would be better if someone from the NZ mapping community did.

[Jochen Topf]

Hi!

For your information: These problems have been mostly fixed in the rest
of the world now. The Canadian community even fixed their 15000 imported
objects (manually as far as I can see).

If somebody wants to fix the NZ data, too, you can start working from OSMI:
http://tools.geofabrik.de/osmi/?view=areas&lon=175.72888&lat=-39.14795&zoom=8&overlays=same_tags_on_outer_ring

[Rob Alley]

Jochen,

  I've been cleaning up a few of these problems in areas where I'm reasonably familiar.  Just to confirm, should we remove all tags (even attributions and source references) from the outer ring?

Rob.

[Jochen Topf]

On Tue, Aug 08, 2017 at 10:27:57AM -0700, Rob Alley wrote:
> I've been cleaning up a few of these problems in areas where I'm
> reasonably familiar. Just to confirm, should we remove all tags (even
> attributions and source references) from the outer ring?

Yes. You can think of it this way: The way doesn't exists really as its
own entity, it is just a "helper" object to define the geometry. So it
should have no tags at all.

[Robert Coup]

hey folks,

On 25 July 2017 at 03:27, Andrew Davidson <thes...@gmail.com> wrote:

This should be relatively easy to fix with JOSM and Overpass.

Using this Overpass query:

I'd be happy to fix these myself but I think it would be better if someone from the NZ mapping community did.

Perhaps you and Rob Alley could work together on this? I'm entirely swamped with work at the moment and unlikely to get my head above the parapet for another month.

Thanks,

Rob :)

[Rob Alley]

The clean up looks to be complete now, thanks to all who swarmed on this.