Still issues about streets belonging to a place in offline data, especially in Germany

[Stephan75]

Hi all,

(at first: yes, I have found the thread "Streets in the City" from
April 2011 in Osmand's Google Group.)

I have had further investigations and test cases about the "list" of
all places and the "list" of all streets belonging to each place when
you do an offline address search in Osmand.

So in the north of Germany there is a town with a population of about
70.000 called Lüneburg. It is contained in the Osmand offline data for
Germany -> Niedersachsen.

The area of this town is limited by a boundary relation with
admin_level=8.

However, when you are searching offline for addresses via latest
Osmand dev and map data in Germany -> Niedersachsen -> Lüneburg there
are definitevly a bunch of streets missing that actually do exist in
the OSM data and are included in the above mentioned boundary
relation.

Further analysing showed that all missing streets inside the boundary
relation are near to some nodes tagged as place=village. Referring to
the postal address of each missing streets, all of them belong to the
city of Lüneburg. So these three nodes with the place=village with
names Rettmer, Häcklingen and Ebensberg should be retagged as
place=suburb.

(All streets near to a node with place=suburb inside Lüneburg are
listed correct by Osmand)

But on the other hand:

When Osmand's map creator is collecting all items for the list of
streets belonging to a city that is defined via a boundary polygon,
why are streets being excluded that are near a node with
place=village?

Because at least in Germany there are many boundary relations with
admin_level=8 or =7 for areas on the countryside where the villages
have to be tagged as villages and NOT as suburbs.

To investigate this topic further, is it possible to describe the
process of map creation concerning the list of places (and maybe
postal code areas?) and the lists of associated streets in a step by
step text (fitting for people that have less programming but good OSM
skills) that can be published e.g. in Osmand's wiki?

Or at least: in what part of the source code are the important things
for this process for each country?

In this context I would also like to know:

Does Osmand Mapcreator pay attention to boundary relations (in
Germany!) with admin_level=7 when creating the list of all places and
all associated streets in it?

Does it ALSO pay attention to the same with admin_level=8 ? or =9 ?

How is the sollution for places where no boundary relation with level
7 or 8 is available? Does it process a circle around the place-node?

Any hints can be helpful, thanks in advance.

[Pavol Zibrita]

Hi Stephan!

   Yes, it is quite a problem currently. But recently, the map creator was changed (is not yet working fully),
to resolve this issues. The new thing is that the streets will be in each boundary they are found.
And streets with same name but in one city should be also visible, etc. etc. Currently are being uploaded
indexes that can have multiple streets with same name in a city. This is not resolving the issue you
are describing, however is a step forward also.

   So we are looking forward for new testers :-) as well as developers. You can watch the changes that
are around IndexMapCreator (starting point).

Regards,
Pavol

[Victor Shcherb]

Hi

We are working on that issue and you can really help us to sort it out. Please download http://download.osmand.net/latest-night-build/OsmAndMapCreator-development.zip and try to create index for Germany -> Niedersachsen 

http://download.geofabrik.de/osm/europe/germany/ . It can take 2-6 hours.

So you will be the first who can see that issue goes.

Best Regards,

Victor 

2011/9/20 Pavol Zibrita <pavol....@gmail.com>

--
Best Regards,

Victor

[Stephan75]

Sure I will keep on testing for you, with pleasure ;-)

https://github.com/osmandapp/Osmand/blob/master/DataExtractionOSM/src/net/osmand/data/preparation/IndexAddressCreator.java

is this more specific the starting point?

Stephan

[Victor Shcherb]

If you check that file you can see latest changes we provided.

2011/9/21 Stephan75 <der.steph...@googlemail.com>

--
Best Regards,

Victor

[Stephan75]

Hello all, Pavel and Victor,

with that latest OsmAndMapCreator I managed an OSM extract for a big
area around the City of Lüneburg comparable to admin_level=6 in size.

It worked well and took only about 20 minutes or less.

And: SUCCESS !!!!

All mentiones streets of Lüneburg that are definitively inside the
city boundary with admin_level=8 and next to the three nodes with
place=village can now be found by Osmand on my phone. Many many
thanks for that step!

But:
There are now some streets listes to belong to Lüneburg which do NOT
belong to it in reality, they are outside the city boundary in reality
and in the OSM data.

Maybe you are still tweaking at the mapcreators processing.

There is a possibility how users can give you nice feedback about all
these address search issues:
Can you implement in current Osmand's developers version a feature:

When the user has selected one town, city or place in address search,
that a text file is written to SD card with all street names listed
about this town.

Then the user can do two things:
Count all streets by number that Osmand can offer for one place to
compare it with streets in reality

or

in Germany the OSM community has "official" lists from the municipal
administratives about several towns with all streets belonging to
them. With these lists they do a diff whether streets in real life are
still missing in OSM.

So with these listst we can also do a diff then to find out the
completeness of Osmand's street lists for each town. Please, can you
provide such a feature??

Or can you add this feature to mapcreator as an option to generate
lists for all places that have been processed from data source with
all streets inside ... these lists could be saved to the path where
the obf files are saved on a desktop PC, maybe in an extra folder.

Thankful greetings from Stephan

[Pavol Zibrita]

Hi Stephan!

    We are still working on it. You can also test my last changes ( I hope I haven't made it worse :-/ ).

For the street list I already implement output from binary inspector. You can use:

inspector -v file.pbf

that will list also regions, cities and their streets.

Regards,
Pavol

[Stephan75]

Hello Pavol,

On 23 Sep., 14:10, Pavol Zibrita <pavol.zibr...@gmail.com> wrote:
> Hi Stephan!
>
>     We are still working on it. You can also test my last changes ( I hope I
> haven't made it worse :-/ ).

We need simply trial and error ... with test cases for each
countries ... but keep on your good doing!

>
> For the street list I already implement output from binary inspector. You
> can use:
>
> inspector -v file.pbf

On WinXP via commad line "-v" is NOT accepted by inspector.bat ... and
it has to be "file.obf" or am I wrong?

>
> that will list also regions, cities and their streets.

sorry, no street listing so far.
Only output of:

C:\Dokumente und Einstellungen\Kainer\Desktop\OsmAndMapCreator-
development>java.
exe -Djava.util.logging.config.file=logging.properties -Xms64M -
Xmx512M -cp "./O
smAndMapCreator.jar;./lib/*.jar" net.osmand.binary.BinaryInspector
lklg_20110921
_2.obf
Binary index lklg_20110921_2.obf version = 1
1. Map data Lklg_20110921_2 - 5.300.585 bytes
1.1 Map level minZoom = 15, maxZoom = 22, size = 2.674.774
bytes
Bounds (left top - right bottom) : 9.635, 53.553 NE -
11.506, 52
.724 NE
1.2 Map level minZoom = 13, maxZoom = 14, size = 1.254.802
bytes
Bounds (left top - right bottom) : 9.635, 53.553 NE -
11.506, 52
.724 NE
1.3 Map level minZoom = 12, maxZoom = 12, size = 844.837 bytes
Bounds (left top - right bottom) : 9.635, 53.553 NE -
11.506, 52
.902 NE
1.4 Map level minZoom = 11, maxZoom = 11, size = 253.217 bytes
Bounds (left top - right bottom) : 9.635, 53.553 NE -
11.506, 52
.902 NE
1.5 Map level minZoom = 10, maxZoom = 10, size = 168.140 bytes
Bounds (left top - right bottom) : 9.635, 53.553 NE -
11.506, 52
.902 NE
1.6 Map level minZoom = 9, maxZoom = 9, size = 57.917 bytes
Bounds (left top - right bottom) : 9.635, 53.553 NE -
11.506, 52
.902 NE
1.7 Map level minZoom = 8, maxZoom = 8, size = 26.867 bytes
Bounds (left top - right bottom) : 9.635, 53.553 NE -
11.506, 52
.902 NE
1.8 Map level minZoom = 7, maxZoom = 7, size = 6.764 bytes
Bounds (left top - right bottom) : 9.635, 53.553 NE -
11.506, 52
.902 NE
2. Address data Lklg_20110921_2 - 618.866 bytes
3. Transport data Lklg_20110921_2 - 37.042 bytes
Bounds (left top - right bottom) : 10.188, 53.369 NE -
10.593, 53.078 N
E
Binary OsmAnd index /home/victor/projects/OsmAnd/data/osm-gen/saved/
Belarus-newz
ooms-new-rt.obf was not found.

[Pavol Zibrita]

You probably have old version. If running the inspector.bat does not show that it supports the -v option,
you have old version.

inspector.bat -v file.obf

should work (yes, not pbf, sorry)

Regards,
Pavol

[Stephan75]

Indeed your posted link above was leading to an older version.

So I fetched a nightly build of MapCreator, and running the
inspector.bat -v file.obf is running fine. If you even append
something like ">streetlist.txt" you have all output in a file.

So in the next days I will compare the result of according streets to
a place produced by Osmand's Mapcreator with the list of some online
OSM street ist comparing service which we have established for
Germany.

Bye, Stephan

[Pavol Zibrita]

I'm looking forward to your results. Maybe you should also use the latest map creator again. There were some changes recently also regarding generating the streets. (We still have some todos in this area).

Regards,
Pavol

[Aldo Bleeker]

Hi,

I've just searched a map of the province Noord-Holland, The
Netherlands, which I had generated with OsmAndMapCreator 9/28 (I
think), because I had noticed maps generated with earlier creators
showed no streets at all when seaching for streets in the village
Velserbroek. I've checked earlier versions of creator whenever the
changes indicated the creator had been improved, but so far all
creators showed the same issue.

I had checked earlier topics mentioning issues with streets, but so
far they didn't provide any solutions. This very topic seemed quite
promising, so I downloaded the creator and tried generating Noord-
Holland again to see if this street issue had been fixed.

My first impression was that generating a map takes a LOT more time,
earlier maps took I think less than an hour, and this one seems to
take many hours (not sure how many, I left it running overnight). If
so, the entire map of The Netherlands may take some time as it used to
take about 8 hours witrh earlier creators. However, if the end result
is reliable, it will be worth it.

So I uploaded the map to my Galaxy Tab 7, and checked Velserbroek once
again. And I can say that for the first time with my self-generated
maps I could see all streets in Velserbroek! Well done!

Regards,

Aldo Bleeker

[Aldo Bleeker]

The map of Noord-Holland seems much improved, so that's great! It does
take longer to create a map, especially stage 70. All in all the map
took almost 3.5 hours. But the end result is much improved, so that's
no problem.

Not all maps have this issue. I used creator to generate a map of the
Canary Island La Palma where we went on holiday, and we had no problem
with that.

Anyway, I'm happy to report the offline maps generated with the
creator have improved immensely. Thanks people for a great map
program!

Regards,

Aldo Bleeker

[andre van atten]

Hi,

I use map of the Netherlands, downloaded 22-07, and I dont have the problems you reported. I can find all streets in Velserbroek. So I suppose problems with mapcreator started after that date.

Andre

Op 2 okt. 2011 12:46 schreef "Aldo Bleeker" <aldo.b...@gmail.com> het volgende:

[Aldo Bleeker]

Actually, the map of The Netherlands you download - if you mean from
within OsmAnd - has the streets in Velserbroek, it's when you generate
your own map using OsmAndMapCreator, you'll notice these streets are
missing. In fact, even the downloaded map isn't quite right, if I use
the map of The Netherlands I downloaded as recently as 9/20, I can't
find a street in Amsterdam named De Entree (which I happen to know
exists, it's located opposite the Arena). However, with the map I
created using a recent MapCreator, all streets in Velserbroek, and De
Entree in Amsterdam are listed.

Regards,

Aldo Bleeker

On Oct 2, 1:20 pm, andre van atten <andrevanat...@gmail.com> wrote:
> Hi,
>
> I use map of the Netherlands, downloaded 22-07, and I dont have the problems
> you reported. I can find all streets in Velserbroek. So I suppose problems
> with mapcreator started after that date.
>
> Andre

> Op 2 okt. 2011 12:46 schreef "Aldo Bleeker" <aldo.blee...@gmail.com> het

[Victor Shcherb]

Hi All,

Yes, previous (first) Netherlands address index was terrible and didn't contain many addresses. That was first attempt of addresses after that we successfully introduced adminCityLevel and it solved many issues! Actually for Netherlands you should specify adminCityLevel=10 (in the Settings of OsmAndMapCreator), right now it is changed! 

Because in every country adminCityLevel is different and people forget to put it but in most countries still existing boundaries we decided to match City Label and City Boundary simply by name! Regardless admin_level. And in our last attempt we also introduce indexing for many cities so right now one street can belong to many cities that is much better rather pointing it to wrong city :) Also district part is fixed!

So we really hope it will sove many problems with data! :) And we will insist that any city should have boundary with exact name matching or specific relation! Data processing will take probably 2 times more but who cares once you will have proper and exact data :) 

Right now address indexes is actually good (I hope) and I'm going to introduce street name index (among all cities!) and also interpolation for building numbers as I see it is very useful for Netherlands where most of houses are not separated and  are part of one build building.

Victor

2011/10/2 Aldo Bleeker <aldo.b...@gmail.com>

[Stephan75]

Hi Victor,

your explanation about map, city and street processing in MapCreator
is very important for users that will generate own obf files for
testing reasons or for normal use. Good job!.

So would it be possible to extend the existing information content on
Osmand's google-wiki-page about each parameter that is offered in the
GUI of MapCreator and that you can tweak there? or create a single new
wiki page about map creation and all its parameters, options and
methods to deal with raw OSM data?

That would be very nice, Thanks in andvance!

Stephan