OSM routing and NZ address data

[Eliot]

OpenStreetMap now has routing on its main page map eg.

http://www.openstreetmap.org/directions?engine=osrm_car&route=-43.5616%2C172.6563%3B-43.5105%2C172.5834

https://blog.openstreetmap.org/2015/02/16/routing-on-openstreetmap-org/

This is very cool, but not all that useful in NZ because detailed address information is very sparse at the moment.
(e.g. in Christchurch)

Now, this data is available from LINZ e.g. https://data.linz.govt.nz/layer/779-nz-street-address-electoral/ with

CC BY 3.0 NZ licence.

So... (I guess this is mainly addressed to those cunning people who have managed importing other LINZ data)   what will it take to get this address data imported to OSM? 

[Glen Barnes]

It is probably worth checking out this post here:

http://groups.open.org.nz/groups/geomash/messages/topic/696sIg1rSLiumn8n0DcTiO

It looks as though some areas are a lot better than others and other areas will come on stream with updated addresses over the coming months. My preference would be to upload addresses on a per region basis as they get completed updated by LINZ. LINZ addresses in the past have not been amazing but the stuff coming from council and into LINZ should be pretty goood.

So the process I would think is:

• Download all current NZ addresses from OSM
• Process one region:
• Extract from the OSM DB a single region (say Rotorua)
• Dedup this with the LINZ data set
• Copy LINZ ID's across to existing addresses
• Upload existing address updates
• Upload new addresses
• Download all address data again and store OSM ID's against these ID's (for future work)
• Think about updates...

We also need to decide on a scheme for mapping addresses - http://wiki.osm.org/wiki/Addresses. There is a preceident for not merging address nodes with buildings "All Danish addresses are continuously checked by a bot and corrected against a service offered by the Danish government. Foreign cartographers mapping in Denmark should be wary that the Danish community does not approve merging of address nodes with buildings." I would be in favour of keeping the nodes as nodes.

I'm happy to help out with the uploading in batches but have no idea how to process everything else.

Glen

[kimo]

Well here we are in December 2015 and I have yet to see a single house address in NZ to be able to use Nominatim for finding an address.

I think we should simply load all the current LINZ address points ASAP.
I could not see any consensus on how road names and suburbs should be related to road centrelines. That would seem to be important to reduce duplication, make editing and maintenance easier.
Please don't use that dumb derived set of address-roadnames without thinking about it because the link to road centrelines is critical. We need the number and road name separated.
The locality should NOT be used, instead we must use the Fire Service primary suburb name which is the in-use names, not the ones made up by LINZ to disambiguate.

On Thursday, 19 February 2015 15:23:38 UTC+13, Eliot wrote:

[Hamish Campbell]

Well the big development since Feb is LINZ's AIMs project:

http://www.linz.govt.nz/news/2015-05/improved-system-for-managing-new-zealand%E2%80%99s-addresses

Obviously Kim, you're aware of this. When the full AIMs dataset is released that would be a good time to revisit this thread.

"...we must use the Fire Service primary suburb name which is the in-use names..."

Well I'd beg to differ on that ;)

"...not the ones made up by LINZ..."

Which ones are those? From what I understand, "Electoral Road Addresses" localities are (mostly) from NZFSC, as are the new AIMs localities.

Obviously "in-use" is highly contestable. OSM cheerfully displays the suburbs of Rosebank, Owairaka and St Lukes, so I suppose it depends on what the use is. Is an address only addressable if NZ Post will deliver there?

Regardless, it will be interesting to see how the AIMs release interplays with NZFSC's looooong standing position on licensing. If - somehow - locality information remains under the current license, we can't use that data for any OSM purpose.

[Eliot]

On Friday, 4 December 2015 15:49:55 UTC+13, Hamish Campbell wrote:

Regardless, it will be interesting to see how the AIMs release interplays with NZFSC's looooong standing position on licensing. If - somehow - locality information remains under the current license, we can't use that data for any OSM purpose.

Interesting. The following is a quote from "LINZ Data Service: Street Address Data Dictionary (Beta) v2.0"

suburb_locality varchar 80 Dannemora Suburb/Locality from the NZ Localities (NZ Fire Service owned dataset).
town_city varchar 80 Auckland Town/City from the NZ Localities (NZ Fire Service owned dataset).

But the data is already being published by LINZ as  Creative Commons Attribution 3.0 New Zealand
so anyone can use it without any contact with the nzfs

The NZFS can't own the name itself, so what does "NZ Fire Service owned dataset" even mean in this contex?

===============================================
On the general topic of address import:
I took a look at presence of addr:housenumber using https://overpass-turbo.eu , very roughly there are 40,000 of these in the major centres.
Compared to 1.9 million entries in the abovementioned database.

I asked this question about the Beta database:

Can you clarify whether the Beta status applies to the schema, the data or both?

Assuming no schema change, will the transition from Beta just mean some added/deleted/modified records that bring the data up to some quality standard, with most records remaining unchanged. Or a more wholesale re-release with new ids?

Trying to get an idea of what happens if I use this data now,...

And got this reply

Just to confirm, no schema change will be made. This layer will have its 'Beta' tag removed and over written by the latest data, therefore, more of a wholesale re-release.

Address_ids will change only if they have been updated since this layer has been up, as we are carrying over Landonline's IDs.

I'm still keen to see comprehensive address data imported into OSM, but don't know how to go about helping make it happen.

--
Eliot
(Python/C programmer/OSM contributor with time on his hands)

[Glen Barnes]

On Thursday, 27 October 2016 15:47:53 UTC+13, Eliot wrote:

On Friday, 4 December 2015 15:49:55 UTC+13, Hamish Campbell wrote:

Regardless, it will be interesting to see how the AIMs release interplays with NZFSC's looooong standing position on licensing. If - somehow - locality information remains under the current license, we can't use that data for any OSM purpose.

Interesting. The following is a quote from "LINZ Data Service: Street Address Data Dictionary (Beta) v2.0"

suburb_locality varchar 80 Dannemora Suburb/Locality from the NZ Localities (NZ Fire Service owned dataset).
town_city varchar 80 Auckland Town/City from the NZ Localities (NZ Fire Service owned dataset).

But the data is already being published by LINZ as  Creative Commons Attribution 3.0 New Zealand
so anyone can use it without any contact with the nzfs

The NZFS can't own the name itself, so what does "NZ Fire Service owned dataset" even mean in this contex?

So basically we can reverse engineer the NZFS dataset from the LINZ dataset and the parcel dataset (NZFS boundaries are snapped to parcel boundaries. 

So yeah - tons we can do but we need some people to make it happen. Some people who have geo skillz. I'm not sure that the existing community has the bandwidth at the moment? Would it be worth doing a one day OSM hack event to kick start some of this?

Cheers

Glen

 

[Geoff Leyland]

> On 28/10/2016, at 11:22 AM, Glen Barnes <barnacl...@gmail.com> wrote:
>
> So basically we can reverse engineer the NZFS dataset from the LINZ dataset and the parcel dataset (NZFS boundaries are snapped to parcel boundaries.

I imagine there's some tricky problem with all the parcels that don't have addresses (like road parcels). Doable, but plenty of fiddly detail?

> So yeah - tons we can do but we need some people to make it happen. Some people who have geo skillz. I'm not sure that the existing community has the bandwidth at the moment? Would it be worth doing a one day OSM hack event to kick start some of this?

I don't know about my skillz, but that could be fun.

[Eliot]

On Friday, 28 October 2016 11:22:51 UTC+13, Glen Barnes wrote:

 

So basically we can reverse engineer the NZFS dataset from the LINZ dataset and the parcel dataset (NZFS boundaries are snapped to parcel boundaries. 

 While this kind of wizardry would be nice, I don't see a priority in doing this, vs the value of importing the items from the (simplified) LINZ address data.

Is this the place to discuss this, or hash out the details over on the OSM LINZ wiki page?

I'll put forward a naive view (I'm expecting "its not that simple" to be a response...)
In many cases a direct mapping from  LINZ -> OSM will provide a lot of value. i.e. get me where I want to go using OSM routing apps.

geom ->  (location of the OSM point)
full_address_number ->  addr:housenumber
full_road_name -> addr:street
address_id -> LINZ:address_id

Even just these 2 would be useful.

I can see the trickiness coming in the next two, may be redundant depending on whether the node in question is already inside a defined suburb, hamlet, town or city...

suburb_locality -> addr:suburb OR ????
town_city -> addr:city OR ???

And of course there is the matter of dealing with the 40,000 duplicates that would be created by a blind import.

So yeah - tons we can do but we need some people to make it happen. Some people who have geo skillz. I'm not sure that the existing community has the bandwidth at the moment? Would it be worth doing a one day OSM hack event to kick start some of this?

FWIW I'm located in Christchurch

[Eliot]

I notice that the Beta designation has disappeared from the LINZ dataset at

https://data.linz.govt.nz/layer/3353-nz-street-address/

[Josh Gaby]

I'm new to the group and to mapping in general, I wanted to use NZ street numbers with OSM maps so I converted the LINZ street address data into a open data csv format and imported it into JOSM as individual nodes.

Not sure where to go from here though as it doesn't look like a decision has been made in regards to importing it to OSM.

Is anyone else working on this and if not is there a layer for addresses in linz2osm?

[Eliot]

Hi all,

another day not being able to navigate to an address using OSM.  Two years and 3 months since I started this thread. It is time for action.

Does anyone want to join me?

It would be great if existing linz2osm tools could help, but if not I'm sure there are other ways.

To reiterate my earlier email, I propose to import all the addresses from https://data.linz.govt.nz/layer/3353, and worry about merging/removing duplicates afterwards.

How about having a real-time chat on the RocketChat at

https://chat.nzoss.nz/channel/opengis

Drop in and leave a note about what time would suit for a 'meeting'

[Andin]

Probably be best for the data to go through the LINZ2OSM process so workslices can be checked out.

Not sure what the procedure is on data clean up before it can be merged. Some sort of community comments.

I do think however, that because there's probably not a lot of address data existing for NZ, most of this will be a straight upload rather than a tedious merge task.

Who can load this into the LINZ2OSM tool, and what are the barriers to making this happen?

-A

[Eliot Blennerhassett]

Hi Andin. 
You may be surprised.

last time I looked (see earlier in this very thread) there were approximately 40,000 NZ addresses in OSM. Some are poor quality (i.e. only contain addr:streetnumber).  There is a mix of polygons and points.  At least a few address interpolations (I know, I created some).

E.g this returns 16000 lines of XML, about 1000 addresses:
"wget -O sth_chc_address.osm  "http://www.overpass-api.de/api/xapi_meta?*[bbox=172.6,-43.58,172.7,-43.54][addr:housenumber=*]"

There are about 2 million points in the linz address database.

I think that there is no conflict between a [polygon (building) with an address and potentially other info e.g. business etc,] and bare address points that just associate an address with a position. In Christchurch for instance, buildings have been removed, but the addresses are still valid.

I.e. the first says "This building is a bicycle shop at 1 Abc Road", the other says "This spot is 1 Abc Road"

I suspect that linz2osm will need some extra processing to deal with the complexities of the linz address schema.  E.g there are columns suburb_locality, town_city. Locality addresses have empty town_city.

Will it even be necessary to include these in the OSM import, or will addr:housenumber and addr:street be sufficient.

There is the provision in the linz data for water_name, water_route name. I'd guess this class of addresses would need separate processing.

Etc.

--
You received this message because you are subscribed to the Google Groups "nzopengis" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nzopengis+...@googlegroups.com.
To post to this group, send email to nzop...@googlegroups.com.
Visit this group at https://groups.google.com/group/nzopengis.
For more options, visit https://groups.google.com/d/optout.

[Robert Coup]

Hey folks,

I don't have any spare cycles to develop on linz2osm at the moment, but I'm happy to point it to datasets or merge/deploy changes. (If I miss messages please poke me, I'm drowning in email a bit atm)

Rob :)

To unsubscribe from this group and stop receiving emails from it, send an email to nzopengis+unsubscribe@googlegroups.com.

To post to this group, send email to nzop...@googlegroups.com.
Visit this group at https://groups.google.com/group/nzopengis.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "nzopengis" group.

To unsubscribe from this group and stop receiving emails from it, send an email to nzopengis+unsubscribe@googlegroups.com.

[Glen Barnes]

You will have to follow this process to get the address import underway:

http://wiki.openstreetmap.org/wiki/Import/Guidelines

I just have 0 bandwidth for this at the moment sorry. Note that if you don't go through the whole process of getting the import approved it will be reverted. We got permission for our existing layers but the street addresses is that much different that you will need new permission. 

Cheers

Glen

[Eliot Blennerhassett]

On 30/05/17 10:56, Glen Barnes wrote:
> You will have to follow this process to get the address import underway:
>
> http://wiki.openstreetmap.org/wiki/Import/Guidelines
>
> I just have 0 bandwidth for this at the moment sorry. Note that if you
> don't go through the whole process of getting the import approved it
> will be reverted. We got permission for our existing layers but the
> street addresses is that much different that you will need new permission.

From whom? OSM community or LINZ or ?

[Hugh Barnes]

Just the permission people, duh ..

[Glen Barnes]

The whole process from start to finish. We have the license to use the data so that is good. And if we use LINZ2OSM them we have the backing of what we had done previously (although there will be some objection and we need to update what fields we put on the actual items).

But we will need to cover how we do merges with existing data, whether we add the address to the building (I would keep them as separate points, there is precedence for this in some counties). And a whole bunch of other stuff as it comes up.

[Eliot Blennerhassett]

On 30/05/17 18:02, Glen Barnes wrote:
> The whole process from start to finish. We have the license to use the
> data so that is good.

I.e. we don't need permission from LINZ

And if we use LINZ2OSM them we have the backing of
> what we had done previously


>(although there will be some objection

Can you expand on why there will be some objection? From who?

> we need to update what fields we put on the actual items.

?

> But we will need to cover how we do merges with existing data, whether
> we add the address to the building (I would keep them as separate

> points, there is precedence for this in some countries).

I have made a number of suggestions about some of these aspects in this
very thread - do you have any comments on those?

--
Eliot

[Eliot]

I made a start here

https://wiki.openstreetmap.org/wiki/Talk:LINZ/Howto

please enhance if you're able

[Glen Barnes]

>(although there will be some objection

Can you expand on why there will be some objection?  From who?

Robin Paulson. Just go look at the imports list and search the archive for the building import. You will also get feedback from people on the list once you propose the import. I  suggest subscribing to the list now so you get an idea of the questions - https://lists.openstreetmap.org/listinfo/imports

 

> we need to update what fields we put on the actual items.

?

The LINZ2OSM needs some work and we currently store the attribution on the nodes and a bunch of other info which is probably unnecessary. You will get pushback from the imports list if you propose as-is  

> But we will need to cover how we do merges with existing data, whether
> we add the address to the building (I would keep them as separate
> points, there is precedence for this in some countries).

I have made a number of suggestions about some of these aspects in this
very thread - do you have any comments on those?

Only that I think we should keep address points as separate points rather than merge them with the buildings but that is just my opinion which I think you agree with? Others may have other opinions. 

Looking forward to what you come up with.

Cheers

Glen 

[Aydin Maxfield]

I reckon leaving them as points is best.  Software client side should be able to link them to buildings if that was needed for navigation.

I deal with a lot of rural properties with multiple buildings (farms) so it makes sense not to pin it to a building in particular.

Just my 2 cents for my narrow view of the world.

--

[Eliot]

Well it wouldn't be GIS group without some maps.

Addresses in OSM:

LINZ addresses:

I rest my case.

[Glen Barnes]

There is no question that it is a good idea to get these addresses is. It is just the process that needs to be followed and refine.

I can tell you from personal experience that you have to tread _very_ carefully with the imports list. ;-)

Cheers

Glen

[Eliot]

On Friday, 9 June 2017 15:50:09 UTC+12, Glen Barnes wrote:

There is no question that it is a good idea to get these addresses is. It is just the process that needs to be followed and refine.

I can tell you from personal experience that you have to tread _very_ carefully with the imports list. ;-)

I posted a week ago,

https://lists.openstreetmap.org/pipermail/imports/2017-June/005014.html

and have had no negative response (or positive response, so zero responses ;)

[Eliot]

My code and some docs here. https://git.nzoss.org.nz/ewblen/osmlinzaddr/tree/master

I can generate osmchange files for each linz [town_city + suburb_locality].
3124 files.  Limited to 8K points.  There are a few suburbs with more that split over 2 files.
Alternatively, I can just generate sets of between some min and max number of nodes, which reduces the number of sets to a manageable size.

I tried upload to the osm dev server. That worked OK, but its strange because the database doesn't contain any NZ data?  So not a good test of merging...

I have uploaded 2 sets :
Abel Tasman National Park - 1 point
Acton - 82 points

I put the linz address id in tag "ref",  but thinking maybe ref:linz or LINZ:address_id.
Or leave it out altogether, and just use matching on the other attributes...

[Glen Barnes]

So from the feedback on the imports list:

Mapping a country's addresses is not impossible, it is just difficult
and a lengthy task. Urban areas are relatively fast mapped if the
number of mappers per 1000 inhabitants is similar to some well known
countries in Central Europe. The lengthy part is mapping rural areas
where the distances between the settlements are large. If you want to
strengthen your urban communities, you should only import addresses in
rural areas. (This does not mean that you have to wait until urban
areas are finished)

That works in Europe with greater population densities but not so much in NZ:

https://en.wikipedia.org/wiki/Area_and_population_of_European_countries

http://www.stats.govt.nz/browse_for_stats/government_finance/central_government/nz-in-the-oecd/population.aspx

In any case we will be very careful with our import. We could of course ignore that comment. Probably the best and keep it to the technical aspects. We don't want to derail the thread with this type of talk.The group is actually split on some of these issues so bets to leave it alone.

To anticipate some questions.
* The source data is CC-BY, and the attribution requirements are well
documented.

We already have specific approval from LINZ:

https://wiki.openstreetmap.org/wiki/LINZ#Official_Approval_Granted

Post that back to him.

- What's the purpose of adding LINZ:address_id=* and linz:change_id=*
to OSM?
- You propose to add a tag source:revision=*. Where will it be placed?
On the changeset or on the imported objects?

Let's just go for ref:address_id on the node. This is all we need and anything more will get pushback from the list. 

I would actually do a reverse lookup afterwards for the changeset and keep a local mapping of LINZ ID's to OSM ID's for future use.

 

- You write on the wiki page that you are not sure whether to use
addr:city=*, addr:place=* or addr:hamlet=* for the field "town_city" of
the LINZ data. Why are you not sure? Is town_city a mix of information
which should be stored in two distinct fields in an ideal dataset or
are you unsure about the definitions of the three OSM tags?

I'm a little confused here as well. Can you provide some examples?

Note: We may get pretty picky here. If you look at the this list way back in the past you will see we spent a lot of time discussing various tags. It may seem picky but its worth it if we're going to import 1m addresses.

- Have you assessed the quality of the data by checking it on the
ground on a few locations in urban and rural areas?

How are we going to answer this one? The interesting thing is that some of this data is great and some is being updated quite a bit. Some have Been sourced from local authorities and eventually all of them will be from the TAs. Some aren't though...

- The wiki page should be improved by adding more technical details of
the import, namely how duplicates will be detected and which
software/workflow will be used to upload the data. Please publish the
source code of the scripts if you use any (at a sufficient location for
source code).

You will need this. Chuck it up on GitHub. We can put it on the Open New Zealand GH org if you want?

- Please add a list of user accounts who will upload the data. Their
user profiles at openstreetmap.org should mention who they belong to.
One import account should only be used by one natural person.

You will need to do that.

Although you write that your import proposal is in an early stage, you
have already started uploading data. https://www.openstreetmap.org/chan
geset/49492831 Please wait at least one week in addition whether
further concerns are raised. Please read the Import Guideline
carefully! I doubt that you haven't read it as carefully as necessary
to do a good import. (Btw, it is a very good idea to read most emails
of the last six months in the archive of this mailing list, too)

Please. please, please don't upload anything. You will just wind up the imports list people and they will make your life harder and start reverting stiff. I know you are chomping at the bit but from personal experience just don't do it. I cannot reiterate enough how tough the imports group can be. CROSS YOUR T'S AND DOT YOUR LOWE CASE J'S.

I'm seriously trying to help here so please listen. I'm not being a dick, just telling you exactly how things will go. Been there, done that.

Cheers

Glen

[Steve Hennerley]

Hey Eliot... 

Just revisited reading this discussion after noticing it kicked into action again (and being deflated when I found it and no progress a year or more ago). I'm amazed at how difficult it is for you to get government sourced authoritative information into an "open" project.....  

I've been running an "on prem" private OSM and nominatim server for internal use for a while now, and have users screaming out for address searching.  LINZ themselves have this dataset as a layer in their OSM implementation, do you (or does anyone else here) have a guide or some direction on how to import this data into an on-prem private installation in lieu of this getting into the "official" data?   I apologise for not having the required level of GIS capability I marvel at at the posters on this group (long time lurker) - but would really love to improve the OSM implementation I use to the level it could easily be.  All I need is the ability to do a nominatim search for a LINZ address on my nominatim server and get a result on the map (just like the LINZ site!) 

Thanks

Steve

[Eliot]

Hi all,

I've posted a snapshot of my generated data (The whole country + a Christchurch subset)

https://drive.google.com/drive/folders/0B8ccScr8Ew74cmxlaGkxYnZuczQ?usp=sharing


The zip files contain .osc files (OSM change, suitable for loading into JOSM. 
The .josm_tags files are suitable for direct copy/paste into josm changeset tags. (Copy the file contents, then In JOSM, select "Tags of new Changest, then the button with 3 + symbols)

I have uploaded a couple of sets in my local area (where I can say I  have on the ground knowledge).

https://www.openstreetmap.org/changeset/51449289#map=14/-43.5632/172.6781

https://www.openstreetmap.org/changeset/51449289#map=14/-43.5632/172.6781

I deleted the suburb key from the addresses where there is already a suburb outlline in OSM.
--
Eliot

[aaron mcewan]

hi,

thought i would try help with this, loaded up catlins_forest_park.osc

some of these addresses the road is not listed on OSM, only in the linz deprecated "electoral" road map, eg Alison road.

what would be the correct procedure to deal with that?

thanks, aaron m

[Eliot Blennerhassett]

On 13/01/18 14:25, aaron mcewan wrote:
> hi,
>
> thought i would try help with this, loaded up catlins_forest_park.osc
>
> some of these addresses the road is not listed on OSM, only in the linz
> deprecated "electoral" road map, eg Alison road.

So, it looks like there is no road there (paper road shows up on parcel
boundaries layer)

> what would be the correct procedure to deal with that?

In my opinion ;) :

* You can delete it from your dataset before upload.

* Check on https://data.linz.govt.nz/layer/53353-nz-street-address/ that
the point is still there in the latest version. (In this case it is)

* If you think it is still error, you could email addr...@linz.govt.nz
(per comment on the above linz page) with the details.

I've no idea about the bureaucratic process required to get the relevant
Territorial Authority to remove the address and notify LINZ of that...

--
Eliot