Nationwide address maps for large countries

1,153 views
Skip to first unread message

Harry van der Wolf

unread,
Aug 16, 2013, 3:09:19 AM8/16/13
to osmand
Due to the fact that large countries like Germany, France, Italy,
Canada, Russia and British_isles are split up in regional maps, it is
sometimes hard to find an address if you don't know exactly which
region your city/address belongs to.

We now created address maps for the above mentioned countries. Note
that these address maps are not maps for navigation, but only for
searching addresses. It means that you still need your "normal"
regional maps for navigation. We already had a few earlier tries and
setups appearing and disappearing in the past weeks and with slightly
other names, but now they will be generated regularly.
You will find the maps inside Osmand in your downloadable maps as
"<country> addresses_nationwide <continent>", like "Germany
address-nationwide europe".
The address maps do not come in a separate category but are part of
the "regular maps". You will find the British-isles and Itally address
maps in the Europe subfolder, the Germany and France address maps in
their respective Germany and France subfolders, etc.


So how does it work?
- In Search -> Address (house icon) select e.g. the "Germany
addresses-nationwide europe" map (or one of the other address maps).
- Search your city/address or postcode.
- After finding use "Show on map" or "Set as destination".
- OsmAnd will automatically use the actual map of the region for this
display or navigation action instead of the address map.

So again: you now have a search option for entire Germany, France,
Italy, Canada, Russia and British_isles.

Some things to keep in mind:
- As the address data in the address maps is very big compared to the
address data in regional maps, your search can take quite long. Also
because addresses allow some fuzzy search (Victor added some time ago
the functionality to automatically expand the search for a street in a
neighborhood city, if the direct city-address combination did not give
a search result).
- If you search for a city/address with one of these address maps and
do a "Show on map", it will position the "cursor" at the correct
location (coordinates) but if you don't have the "real" map for that
region, it will display a grey background with "no offline vector map
present for the location. You can download ...(etc.)". You really
need the actual regional map if you want to display an address or
navigate to it.

Andre

unread,
Aug 16, 2013, 11:31:15 AM8/16/13
to osm...@googlegroups.com

This is great!

I would like to add a hint for people who did find an address in Germany, France or any big country with regional maps.  After you have found the address,  you may wonder which regional 'complete' map to download for display and routing.

For that you need the name of the region. It is nice to know that this name can be found using the online nominatum address search, the uttermost right bottom icon in the address search form. I plea for another icon though, it is not very clear what this icon [>] means. See pictures.

In case there are multiple options, they are shown all, like the village Mittelberg, which is found multiple times in Germany and one occurrence in Austria

I think this is a typical FAQ question, it would be nice if it is contributed to the wiki documentation.

Screenshot_2013-08-16-17-04-22-1.png
Screenshot_2013-08-16-17-01-56-1.png

Victor Shcherb

unread,
Aug 16, 2013, 12:36:07 PM8/16/13
to osmand
Very important feature.

Thanks Harry



--
You received this message because you are subscribed to the Google Groups "Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Christopher Head

unread,
Aug 16, 2013, 3:14:30 PM8/16/13
to osm...@googlegroups.com
There seems to be some data missing from these address maps, along
with some other strange behaviour. One example:

I do a search in the “Canada british-columbia” region, choose
“Vancouver [city]” as the city, and “Nelson Street (Yaletown)” as the
street (side note, Nelson is a long street which passes through many
more neighbourhoods than just Yaletown, so I have no idea why that
parenthetical is there). In Intersecting Street mode, I can then choose
from quite a long list of streets.

If I do the same but choose “Canada address” region instead, then there
are two Nelson Streets (“Yaletown” and “Coal Harbour”, which are
actually the same street in real life), then Denman Street is not
visible as an intersecting street to either one (despite being in the
list under the Canada british-columbia region, and intersecting in
real life). Also, in the Coal Harbour variant of Nelson Street, all the
intersecting streets are listed five times each.

Chris

On Fri, 16 Aug 2013 09:09:19 +0200
Harry van der Wolf <hvd...@gmail.com> wrote:

signature.asc

Harry van der Wolf

unread,
Aug 16, 2013, 5:26:17 PM8/16/13
to osmand
2013/8/16 Christopher Head <hea...@gmail.com>:

> If I do the same but choose “Canada address” region instead, then there
> are two Nelson Streets (“Yaletown” and “Coal Harbour”, which are
> actually the same street in real life), then Denman Street is not
> visible as an intersecting street to either one (despite being in the
> list under the Canada british-columbia region, and intersecting in
> real life). Also, in the Coal Harbour variant of Nelson Street, all the
> intersecting streets are listed five times each.


If you search for "nelson street, vancouver" in openstreetmap.org,
you'll find also multiple hits for the same street.
If you follow the link
(http://www.openstreetmap.org/#map=17/49.27450/-123.11547) you can
even see that there really are two nelson streets: the bigger
secondary street and the smaller, parallel, tertiary street. And
what's more: openstreetmap also separates per district giving two
nelson streets.

The way these address maps are generated is differently from the way
they would have been created directly from OsmAndMapCreator.
"normally" one would create an address map from the Canada "raw data"
map for example. This is an extremely CPU and memory intensive
process.

For the creation of the address map, an extraction tool is used to
extract "all streets and addresses" from the base Canada map into a
new "raw data" address map. This "raw data" address map is then
converted to an OsmAnd address map.
This "all streets and addresses" extraction probably causes this
behaviour of finding more streets that are actually the same, just
like in openstreetmap.org itself.

Harry

Christopher Head

unread,
Aug 16, 2013, 8:52:30 PM8/16/13
to osm...@googlegroups.com
On Fri, 16 Aug 2013 23:26:17 +0200
Harry van der Wolf <hvd...@gmail.com> wrote:

> 2013/8/16 Christopher Head <hea...@gmail.com>:
>
> > If I do the same but choose “Canada address” region instead, then
> > there are two Nelson Streets (“Yaletown” and “Coal Harbour”, which
> > are actually the same street in real life), then Denman Street is
> > not visible as an intersecting street to either one (despite being
> > in the list under the Canada british-columbia region, and
> > intersecting in real life). Also, in the Coal Harbour variant of
> > Nelson Street, all the intersecting streets are listed five times
> > each.
>
> If you search for "nelson street, vancouver" in openstreetmap.org,
> you'll find also multiple hits for the same street.
> If you follow the link
> (http://www.openstreetmap.org/#map=17/49.27450/-123.11547) you can
> even see that there really are two nelson streets: the bigger
> secondary street and the smaller, parallel, tertiary street. And
> what's more: openstreetmap also separates per district giving two
> nelson streets.

OK, fair enough on it being two streets. Nevertheless, I guess consider
this a bug report that (1) the *non*-address-map country is missing the
second street (Nelson Coal Harbour) in that case, (2) the intersection
with Denman is missing in both versions of the address-map Nelson
Street, despite being present in the non-address-map Canada, and (3)
the intersecting streets with Nelson Coal Harbour in address-map Canada
are listed five times. Considering you only just announced the address
map feature, I guess it’s pretty new, so it’s understandable that there
are bugs to iron out with the extraction tool, so here are a few
examples for you to look into!
signature.asc

Stephan75

unread,
Aug 18, 2013, 1:04:09 PM8/18/13
to osm...@googlegroups.com
Quite nice feature!!

But has anyone already tried to produce an output from such a nationwide obf file when processing it with inspector.bat -vaddress test.obf >output.csv ?

When doing this with the one obf for Germany, I get a very big CSV file, but I have the feeling that this result file is NOT complete.

There are only few tools than can display such big CSV files. Try Notepad++, or do a googling for "edit big CSV files" , there are some more tools.

I only get a file with 252.579 lines ... and only for "Cities" in Germany ... I assume many other places are missing!

Anyone to reproduce?

Harry van der Wolf

unread,
Aug 18, 2013, 4:07:26 PM8/18/13
to osmand
Hi,

2013/8/18 Stephan75 <der.steph...@googlemail.com>:
> Quite nice feature!!
>
> But has anyone already tried to produce an output from such a nationwide obf
> file when processing it with inspector.bat -vaddress test.obf >output.csv ?

Yes, I did. For several files after many trials with different settings.
We use a different route to create these maps compared to the normal
maps route (see below).
I created the Germany address map directly and via the "address map"
route (also see below) and compared these as well.
They were not the same.

>
> When doing this with the one obf for Germany, I get a very big CSV file, but
> I have the feeling that this result file is NOT complete.
>

That feeling is correct, but it's not that simple.
Yes, there is data missing compared to the "normal" address maps:
around 2-3%. On the other hand: There are also some streets in the
address maps that are not in the normal maps.
And there are also more equal streets in the address maps compared to
the normal maps (as mentioned already by Christopher).
It all has to do with the (frustrating) inconsistency in the OSM maps.
This is part due to the history and changes in formats,
backward-compatibility and so on.
I did many long, long runs trying all kind of combinations.

> There are only few tools than can display such big CSV files. Try Notepad++,
> or do a googling for "edit big CSV files" , there are some more tools.
>
Well. That's windows of course ;)


> I only get a file with 252.579 lines ... and only for "Cities" in Germany
> ... I assume many other places are missing!

What do you mean with "I only get a file with 252.579 lines ... and
only for "Cities" in Germany ".
In the CSV you have City [City], but also City [TOWN], down to City
[HAMLET]. Or do you mean something else?


>
> Anyone to reproduce?

yes. :)
As mentioned before: It is extremely hard to produce an address map
directly from OsmAndMapCreator from the Germany.osm.pbf or
France.osm.pbf map. It is very CPU and memory intensive. That's why
another route has been chosen to get the data.
I did the "normal" route by creating a Germany address map from the
full Germany-latest.osm.pbf and I had to boot my 8GB i7 laptop to a
command prompt (without X, without (G)UI) to be able to generate the
map and it took 3½ days.
The OsmAnd server is about 4 times as fast but it will still take a
lot of time, only for Germany.

The route to create these address maps is as follows:
==
osmconvert basemap.osm.pbf --out-o5m -o=basemap.o5m
osmfilter basemap.o5m --keep="boundary=administrative addr:* place=*
is_in=* highway=residential =unclassified =pedestrian =living_street
=service =road =unclassified =tertiary"
--keep-ways-relations="boundary=administrative" --keep-ways=
--keep-nodes= --keep-relations= --out-o5m > basemap_address.o5m
osmconvert basemap_address.o5m --out-pbf -o=basemap_address.osm.pbf
==
(Note that shell based variables are removed to simplify it a little)
So we use osmfilter to get an address-only osm.pbf which is fed into
OsmAndMapcreator to create the final address obf. During this
conversion things get lost, things get duplicated and a few get added.

To repeat the osmfilter command:
osmfilter basemap.o5m --keep="boundary=administrative addr:* place=*
is_in=* highway=residential =unclassified =pedestrian =living_street
=service =road =unclassified =tertiary"
--keep-ways-relations="boundary=administrative" --keep-ways=
--keep-nodes= --keep-relations= --out-o5m > basemap_address.o5m
The "is_in" parameter is the biggest issue causing double/triple
streets/addresses and it is a remnant from "the old days", but without
it you only get 50% or so (can't remember exactly) of the data.


If anyone has ideas to improve it, please share your knowledge.

Harry

Stephan75

unread,
Aug 19, 2013, 1:31:04 PM8/19/13
to osm...@googlegroups.com
Hello Harry,

many thanks for your explanations!

I also like playing around with osmfilter in many ways ... mainly for Quality Assurance purposes.

About inspector.bat I mean that places like [town], [village] or [hamlet] are NOT included in the output file.

Even looking at the last line of that output file, it is a street name beginning with the letter "s" and not "z" as I expected.

So it seems that inspector.bat stops working in the middle of his work by some error that is NOT displayed.

So anyone with inspector.bat skills: Try to output with -vaddress to a CSV textfile and view it ... is it complete for your country when inspecting the new nationwide address maps??

Stephan

Harry van der Wolf

unread,
Aug 19, 2013, 2:28:57 PM8/19/13
to osmand
Hi,

I'm on linux, so I use unix commands.

When I do the inspector.sh action to a CSV, I do get a 178MB csv file
containing 3.322.407 lines (+3 million). (cat
Germany_addresses-nationwide_europe.csv | wc -l).
I'm using the inspector java stuff on a 64bit system. Are you on a
32bit windows?


When I do a "cat Germany_addresses-nationwide_europe.csv | grep -i
"City\ \[CITY\]" | wc -l", I get 85 cities (>100.000 inhabitants)
When I do a "cat Germany_addresses-nationwide_europe.csv | grep -i
"City\ \[TOWN\]" | wc -l", I get 2406 towns (>10.000 inhabitants)
When I do a "cat Germany_addresses-nationwide_europe.csv | grep -i
"City\ \[VILLAGE\]" | wc -l", I get 37899 villages (<10.000
inhabitants)
When I do a "cat Germany_addresses-nationwide_europe.csv | grep -i
"City\ \[HAMLET\]" | wc -l", I get 37583 hamlets

And doing these options and simply displaying them, I definitely get
everything up to and including "z"

When you load a +3 million lines CSV into Excel you have a problem as
Excel (2007-2010) only excepts up to 1,048,576 rows (same as
OpenOffice and LibreOffice what I use), so you'll never get to the end
of the file. That's why I use vi.

So far the numbers :)

Harry


2013/8/19 Stephan75 <der.steph...@googlemail.com>:

Harry van der Wolf

unread,
Aug 19, 2013, 3:02:03 PM8/19/13
to osmand
And as an example I extracted Aachen (simply the first city) and
attached it to this mail.
I only copied the first 5 columns. Some streets contain over 200
columns, as every column can also represent a house number.

you can also check whether it is equal to the windows version created
with inspector.sh.

Harry



2013/8/19 Harry van der Wolf <hvd...@gmail.com>:
aachen.csv.zip

Stephan75

unread,
Aug 22, 2013, 3:50:16 PM8/22/13
to osm...@googlegroups.com
Hello Harry again!

Now I also succeeded with Windows7 in generating CSV output from Germany_addresses-nationwide_europe.obf (obf has 412 MB) ...

The issue was really to use Java in 64bit edition ... and edit inspector.bat to include -Xmx2600MB

output file has 183.953.377 Bytes ... and according to Notepad++ it has 3.327.487 lines !!! wow!

I even will NOT try to load it into LibreOffice.

Finally I managed to delete most of the columns that I don't need for my further investigations by some tools loke OpenRefine formerly known as GoogleRefine, and by some special CSV editors that can load and manipulate big CSV files ("Ron's editor").

More results will follow!

Harry van der Wolf

unread,
Aug 22, 2013, 4:47:55 PM8/22/13
to osmand
Great you got further.

I have not investigated it further yet, but I wanted to load the csv
into a database (and that could also be ms access if you're on
windows).
I myself was thinking of mysql, but if you use ms access or
libreoffice/openoffice base you can get to 4 billion rows or up to 2GB
filesize.

It would require some preparation before being able to load it, but
then you can create nice views.

Another reason is also the the postcode csv files mentioned in the
other thread. To be able to work on it you need to convert it into a
database. So either the postgresql route or into something I'm
familiar with (well postgres is an swl DB too, so maybe that's the
best route to go.

But as said: I've not looked into it yet, just thought about it.

Harry
Reply all
Reply to author
Forward
0 new messages