How to add or delete one road from .obf map files, in research about map update

[Xuemei Liu]

Hi, everyone,

We are doing research about automatically updating maps based on
user GPS traces. We have already finished designing an algorithm that
takes in offline GPS trace data and generates map. At the moment we
want to build a real time system (server) that takes in
users' (client) real-time GPS trace data, and generates map update for
users.
We found Osmand very good and decided to develop the feature in
it. However, now I come across a problem of how to update the .obf map
files used in Osmand. This occurs when a user drives along a road that
does not exist in the .obf map file, and we want to generate one new
road for the user and add it into .obf map file. I look through the
code and cannot find any class that can handle the issue.

Can anyone give me some advice?

Xuemei Liu

[Anasx C]

.obf files are created in a separate application, OsmAndMapCreator

[sabas88]

2012/3/29 Xuemei Liu <lxuem...@gmail.com>

Can anyone give me some advice?

obf are processed files.

You should emulate the creation process (indexing) and find a way to put it into the obf at the correct position.

As said by Anasx the creation is made by an external application (there's the source code in the repository I assume).

I am guessing :)

 

Xuemei Liu

Regards,

Stefano 

[Xuemei Liu]

Thanks for fast response. Is .obf the abbreviation of .osm.pbf ? Yes, OsmAndMapCreator is the application that creates .obf files but the code does not exist in  http://code.google.com/p/osmand/source/browse/. 

[andre van atten]

Yes,

It is osmand binary (protobuf) file, but not the same as the pbf from geofabrik, because it is segmented and indexed in another way.
You have to study osmandMapCreator, and writing/inserting to it might be complex, because of relations, keys and index integrity. And it is binary, build on assumption of  preprocessing. So quitte a challenge. Would like to know about the progress.
Nice choice for osmand though. It is brute, fast, compact and flexible.

Andre

Op 29 mrt. 2012 23:44 schreef "Xuemei Liu" <lxuem...@gmail.com> het volgende:

[Xuemei Liu]

However, I cannot find the source code of  OsmAndMapCreator. It does not exist in the  http://code.google.com/p/osmand/source/browse/, although I can download OsmAndMapCreator-0.6.8.2-beta.zip.

[Pavol Zibrita]

Hi!

  http://code.google.com/p/osmand/source/browse/DataExtractionOSM/src/net/osmand/data/preparation/IndexCreator.java

  Try to start there. However, I can imagine modifying and obf might be not so easy. The obf indexes the
entities in R-trees, a tree structure for spatial data. This structures are loaded in memory I think. You will have
to add the new element there, balance the tree, pack, and save to the file. Probably it is doable, R-trees are
ment for dynamic updating.
  Also note, that these structures are currently undergoing small refactoring in newIndex branch.

obf is more like Osmand's pBF :-).

Other question for you is, if you really want to bother with the obf files? You want to modify them on android device,
or on some server?

Best regards,
Pavol.

[Xuemei Liu]

We do map update calculation on the server side, and I think we will based on .osm map (We want to modify the osm map file automatically so that the update can be submitted to OSM server directly). Then, we will send the update back to Osmand user to update their .obf map. I just have some experience of using R-trees, and I think it will be a tough work to modify the obf file according to your introduce. Another scheme is just adding a patch file (update.pat) for the obf file (original map) to store the map update. As we calculate the map update on the server side based on the osm file, we need to know the correlation between the road IDs in osm file and those in obf file, e.g., we add a new road 000 to connect road 111 and road 222 in osm file (some intersection points will be added in osm file, and road 111 and 222 will be modified as well ), the new update will be sent to Osmand user and store in the update.pat file, and we need to know the  IDs of road 111 and 222 in obf file to modify them, and add the new road into the Rtree as well. So in this scheme, I need to know the correlation. Can anyone give some advice of how to implement this? I will start by reading the  IndexCreator code.

Thanks,

Xuemei

[Victor Shcherb]

The correlation is very simple.

OSM-id = L

OBF-id = X

if(object is Node)

X = L << 1;

else if(object is Way) 

X = L << 1 + 1;

2012/3/30 Xuemei Liu <lxuem...@gmail.com>

[Xuemei Liu]

Hi, Victor,

Can you tell  me where is the code is? I think I spent some time on investigating the code, but have not found the correlation in the code yet. Thanks.

Xuemei

[Victor Shcherb]

In some of IndexCreator's (IndexVectorMapCreator/convertGeneratedIdToObfWrite)

2012/4/1 Xuemei Liu <lxuem...@gmail.com>

[Xuemei Liu]

Thanks

[Xuemei Liu]

Hi, 

I debug the code in IndexVectorMapCreator and Osmand, and found the relation provided by you is exactly right! Then, I tried to find where is the "node" and "Way" stored in Osmand so that I can add the "new roads in my patch file" into the corresponding data structure (Then the new road can be shown in the Osmand and used for navigation). Now I have two questions,

1. I found  "node" and "Way" seem be stored in "CodedInputStreamRAF codedIS" in BinaryMapIndexReader. Is this right? If yes, how can I add data into the object "codedIS", which seems to be only for input. Or did I miss anything? 

2. I don't know yet where is the code for showing the map. Definitely, I need to know where is the code for showing "CodedInputStreamRAF" data on the screen, because I need to call the code to update the shown map when "new roads" are added.

This is my first try of Android app and I just read some material about Android development and did some small practice. Sorry for so many questions.

Xuemei

On Sat, Mar 31, 2012 at 6:03 AM, Victor Shcherb <victor....@gmail.com> wrote:

[andre van atten]

The code for showing the map is obviously in the osmand Android app itself, and no code is provided in the MapCreator.

Andre

Op 2 apr. 2012 03:47 schreef "Xuemei Liu" <lxuem...@gmail.com> het volgende:

[Xuemei Liu]

1. The code of MapCreater is  IndexVectorMapCreator.java

2. Can you tell me the exact function (or class) which is used to show the map when the "Map" big button is clicked in the menu. 

3. The upper question is not solved yet...

Xuemei

[andre van atten]

Hi Xuemei,

ad 2. No, Sorry, I can't. I would have to search for a long time. 
ad 3. That's right. Then you have to wait until the coders themself have time for you to sort it out. Their time is very sparse too;-)

Andre

Op 2 april 2012 07:55 schreef Xuemei Liu <lxuem...@gmail.com> het volgende:

[Xuemei Liu]

In fact I have spent long time on reading and debugging the code, but still have the previous problems and give out the questions. I just think you are the developers of Osmand, some of you may be familiar with what I asked, and can give some notes to me. Of course I will keep debugging and reading the code. Thanks

Xuemei

[Pavol Zibrita]

Hi!

   Probably the IndexVectorMapCreator#insertBinaryMapRenderObjectIndex method? Well, I'm helping with developing, but I also don't know each inch of the code :-/. Most helpful would be Victor,
but I think he is quite busy.

Best regards,
Pavol

[Victor Shcherb]

To be honest i'm wondering how it can be implemented. The maps were designed to be read-only on the device! If you want to modify them before, modify the Osmand file. I think you will find a lot of problems supporting cross references in the file

[Xuemei Liu]

Hi, Victor and Pavol,

Thanks for your response. I will read material about Google Protocol Buffers first, try to understand it, then consider feasible solution.

Xuemei

[Victor Shcherb]

The problem not in Protobuf! The problem is file structure. There are cross references between messages (simple byte shifts) and length part of messages. So if you try update one message, you will need to update the length of that message, the length of map block, the lenght of map index and so on. I'm not sure it will be easy. 

Victor

2012/4/2 Xuemei Liu <lxuem...@gmail.com>

[Xuemei Liu]

Then can you tell me or give me some document of how the .obf file is organized? I am not clear yet.  I just saw you use  Protobuf to read the data in the obf file, and thought it might be related to it. 

Xuemei

[Pavol Zibrita]

Hi!

   You should probably think about some other format for storing, for example sqlitedb, and providing ResourceManager a new implementation how to get offline data
for particular region, and so you would have some updateable offline resource, not very compact, but useful for inserting/retrieveing. For sure, it would than be nice to handle its size and somehow..
I don't know, if it is too huge to convert it to obf or whatever.

  Just a though how it can be done also other way.

Best regards,
Pavol

[Xuemei Liu]

ok, i will consider that. Thanks for the advice.

[Xuemei Liu]

Hi, Victor, Pavol,

Thanks for your advice and help, I made much progress.

At present I decided not to modify the .obf file, but just load our .patch file for update. As map data will be only used in rendering and navigation, so I modify methods MapRenderRepositories.java.loadVectorData and BinaryRoutePlanner.java.loadRoutes. I add code in the two methods to load roads from our patch file after loading map from obf files. I encode coordinates of added roads from lon&lat to x&y using MapUtils.get31TileNumberY() and MapUtils.get31TileNumberX(). Is this encoding method right? If no, what should it be? If yes, I have the following problem.

The added roads appear after rendering, but cannot used for navigation. Then I spent the whole night to debug the code, and find the following problem of inconsistent between coordinates written to obf file and those read from obf file, which makes the navigation not work.

Code that related to the problem: 

1. Generate obf file from osm file. 

        In net.osmand.data.preparation.IndexVectorMapCreator.java.insertBinaryMapRenderObjectIndex(), MapUtils.get31TileNumberY and MapUtils.get31TileNumberX are used to convert from lon&lat to x&y, and write x&y to binary file.

2. Read map data from obf file.

        In net.osmand.binary.BinaryMapIndexReader.readMapDataObject(), "(codedIS.readSInt32() << SHIFT_COORDINATES) + px" and "(codedIS.readSInt32() << SHIFT_COORDINATES) + py" are used to read binary bits to x&y coordinates. Why not just direct read x&y, but have such complex convention? Did I miss anything?

Example of the problem (the example is got by tracking coordinates in one road):

1.  lat='37.4210738' lon='-122.1330728'   ==> x='34518966' y='832711323', and x & y are written to obf file.

2. after reading the obf, x='34518964' y='832711296', not as expected.

Can you tell me why? I need to know it then my added roads can be used for navigation.

Xuemei

[Xuemei Liu]

Hi,

Can anyone give some notes?

Xuemei

On Apr 5, 12:30 am, Xuemei Liu <lxuemei3...@gmail.com> wrote:
> Hi, Victor, Pavol,
>
> Thanks for your advice and help, I made much progress.
> At present I decided not to modify the .obf file, but just load our .patch
> file for update. As map data will be only used in rendering and navigation,
> so I modify methods MapRenderRepositories.java.loadVectorData and
> BinaryRoutePlanner.java.loadRoutes. I add code in the two methods to load
> roads from our patch file after loading map from obf files. I encode
> coordinates of added roads from lon&lat to x&y

> using MapUtils.get31TileNumberY() and MapUtils.get31TileNumberX(). *Is this
> encoding method right? *If no, what should it be? If yes, I have the
> following problem.
> *The added roads appear after rendering, but cannot used for navigation*.

> Then I spent the whole night to debug the code, and find the following
> problem of inconsistent between coordinates written to obf file and those
> read from obf file, which makes the navigation not work.
>
> Code that related to the problem:
> 1. Generate obf file from osm file.
>         In
> net.osmand.data.preparation.IndexVectorMapCreator.java.insertBinaryMapRende rObjectIndex(),
> MapUtils.get31TileNumberY
> and MapUtils.get31TileNumberX are used to convert from lon&lat to x&y, and
> write x&y to binary file.
> 2. Read map data from obf file.
>         In net.osmand.binary.BinaryMapIndexReader.readMapDataObject(),
> "(codedIS.readSInt32() << SHIFT_COORDINATES) + px" and
> "(codedIS.readSInt32() << SHIFT_COORDINATES) + py" are used to read binary

> bits to x&y coordinates. *Why not just direct read x&y, but have such
> complex convention? Did I miss anything?*

> Example of the problem (the example is got by tracking coordinates in one
> road):
> 1.  lat='37.4210738' lon='-122.1330728'   ==> x='34518966' y='832711323',
> and x & y are written to obf file.
> 2. after reading the obf, x='34518964' y='832711296', not as expected.
>
> Can you tell me why? I need to know it then my added roads can be used for
> navigation.
>
> Xuemei
>
>
>
>
>
>
>
> On Mon, Apr 2, 2012 at 5:21 PM, Xuemei Liu <lxuemei3...@gmail.com> wrote:
> > ok, i will consider that. Thanks for the advice.
>

> > On Mon, Apr 2, 2012 at 3:45 PM, Pavol Zibrita <pavol.zibr...@gmail.com>wrote:
>
> >> Hi!
>
> >>    You should probably think about some other format for storing, for
> >> example sqlitedb, and providing ResourceManager a new implementation how to
> >> get offline data
> >> for particular region, and so you would have some updateable offline
> >> resource, not very compact, but useful for inserting/retrieveing. For sure,
> >> it would than be nice to handle its size and somehow..
> >> I don't know, if it is too huge to convert it to obf or whatever.
>
> >>   Just a though how it can be done also other way.
>
> >> Best regards,
> >> Pavol
>

> >> On Mon, Apr 2, 2012 at 11:23 PM, Xuemei Liu <lxuemei3...@gmail.com>wrote:
>
> >>> Then can you tell me or give me some document of how the .obf file
> >>> is organized? I am not clear yet.  I just saw you use  Protobuf to read the
> >>> data in the obf file, and thought it might be related to it.
>
> >>> Xuemei
>

> >>> On Mon, Apr 2, 2012 at 1:57 PM, Victor Shcherb <victor.shch...@gmail.com

> >>> > wrote:
>
> >>>> The problem not in Protobuf! The problem is file structure. There are
> >>>> cross references between messages (simple byte shifts) and length part of
> >>>> messages. So if you try update one message, you will need to update the
> >>>> length of that message, the length of map block, the lenght of map index
> >>>> and so on. I'm not sure it will be easy.
>
> >>>> Victor
>

> >>>> 2012/4/2 Xuemei Liu <lxuemei3...@gmail.com>

>
> >>>>> Hi, Victor and Pavol,
> >>>>> Thanks for your response. I will read material about Google Protocol
> >>>>> Buffers first, try to understand it, then consider feasible solution.
>
> >>>>> Xuemei
>
> >>>>> On Mon, Apr 2, 2012 at 9:15 AM, Victor Shcherb <
> >>>>> victor.shch...@gmail.com> wrote:
>
> >>>>>> To be honest i'm wondering how it can be implemented. The maps were
> >>>>>> designed to be read-only on the device! If you want to modify them before,
> >>>>>> modify the Osmand file. I think you will find a lot of problems supporting
> >>>>>> cross references in the file

> >>>>>>  On Apr 2, 2012 10:35 AM, "Pavol Zibrita" <pavol.zibr...@gmail.com>

> >>>>>> wrote:
>
> >>>>>>> Hi!
>
> >>>>>>>    Probably the
> >>>>>>> IndexVectorMapCreator#insertBinaryMapRenderObjectIndex method? Well, I'm
> >>>>>>> helping with developing, but I also don't know each inch of the code :-/.
> >>>>>>> Most helpful would be Victor,
> >>>>>>> but I think he is quite busy.
>
> >>>>>>> Best regards,
> >>>>>>> Pavol
>

> >>>>>>> On Mon, Apr 2, 2012 at 8:27 AM, Xuemei Liu <lxuemei3...@gmail.com>wrote:
>
> >>>>>>>> In fact I have spent long time on reading and debugging the code,
> >>>>>>>> but still have the previous problems and give out the questions. I just
> >>>>>>>> think you are the developers of Osmand, some of you may be familiar with
> >>>>>>>> what I asked, and can give some notes to me. Of course I will keep
> >>>>>>>> debugging and reading the code. Thanks
>
> >>>>>>>> Xuemei
>
> >>>>>>>> On Sun, Apr 1, 2012 at 11:13 PM, andre van atten <
> >>>>>>>> andrevanat...@gmail.com> wrote:
>
> >>>>>>>>> Hi Xuemei,
>
> >>>>>>>>> ad 2. No, Sorry, I can't. I would have to search for a long time.
> >>>>>>>>> ad 3. That's right. Then you have to wait until the coders
> >>>>>>>>> themself have time for you to sort it out. Their time is very sparse too;-)
>
> >>>>>>>>> Andre
>

> >>>>>>>>> Op 2 april 2012 07:55 schreef Xuemei Liu <lxuemei3...@gmail.com>het volgende:

>
> >>>>>>>>> 1. The code of MapCreater is  IndexVectorMapCreator.java
> >>>>>>>>>> 2. Can you tell me the exact function (or class) which is used
> >>>>>>>>>> to show the map when the "Map" big button is clicked in the menu.
> >>>>>>>>>> 3. The upper question is not solved yet...
>
> >>>>>>>>>> Xuemei
>
> >>>>>>>>>> On Sun, Apr 1, 2012 at 10:37 PM, andre van atten <
> >>>>>>>>>> andrevanat...@gmail.com> wrote:
>
> >>>>>>>>>>> The code for showing the map is obviously in the osmand Android
> >>>>>>>>>>> app itself, and no code is provided in the MapCreator.
>
> >>>>>>>>>>> Andre

> >>>>>>>>>>> Op 2 apr. 2012 03:47 schreef "Xuemei Liu" <lxuemei3...@gmail.com>

> >>>>>>>>>>>>> 2012/3/30 Xuemei Liu <lxuemei3...@gmail.com>

>
> >>>>>>>>>>>>>> We do map update calculation on the server side, and I think
> >>>>>>>>>>>>>> we will based on .osm map (We want to modify the osm map file automatically
> >>>>>>>>>>>>>> so that the update can be submitted to OSM server directly). Then, we will
> >>>>>>>>>>>>>> send the update back to Osmand user to update their .obf map. I just have
> >>>>>>>>>>>>>> some experience of using R-trees, and I think it will be a tough work to
> >>>>>>>>>>>>>> modify the obf file according to your introduce. Another scheme is just
> >>>>>>>>>>>>>> adding a patch file (update.pat) for the obf file (original map) to store
> >>>>>>>>>>>>>> the map update. As we calculate the map update on the server side based on
> >>>>>>>>>>>>>> the osm file, we need to know the correlation between the road IDs in osm
> >>>>>>>>>>>>>> file and those in obf file, e.g., we add a new road 000 to connect road 111
> >>>>>>>>>>>>>> and road 222 in osm file (some intersection points will be added in osm
> >>>>>>>>>>>>>> file, and road 111 and 222 will be modified as well ), the new update will
> >>>>>>>>>>>>>> be sent to Osmand user and store in the update.pat file, and we need to
> >>>>>>>>>>>>>> know the  IDs of road 111 and 222 in obf file to modify them, and add the
> >>>>>>>>>>>>>> new road into the Rtree as well. So in this scheme, I need to know the
> >>>>>>>>>>>>>> correlation. Can anyone give some advice of how to implement this? I will

> >>>>>>>>>>>>>> start by reading the  IndexCreator<http://code.google.com/p/osmand/source/browse/DataExtractionOSM/src/n...>

> >>>>>>>>>>>>>>  code.
>
> >>>>>>>>>>>>>> Thanks,
> >>>>>>>>>>>>>> Xuemei
>
> >>>>>>>>>>>>>> On Fri, Mar 30, 2012 at 12:26 AM, Pavol Zibrita <

> >>>>>>>>>>>>>> pavol.zibr...@gmail.com> wrote:
>
> >>>>>>>>>>>>>>> Hi!
>
> ...
>
> read more »

[Victor Shcherb]

Coordinates are written in delta mode and using 24th zoom in order to  compress the data as much as possible.

Routing data is working on the same level.

2012/4/6 Xuemei Liu <lxuem...@gmail.com>

[Xuemei Liu]

Hi, Victor,

In fact, at present, what I want to get is an conversion from lon&lat to x&y that can be used in Osmand (for added roads, I want to convert them into BinaryMapDataObject objects directly). I tried  "MapUtils.get31TileNumberX and MapUtils.get31TileNumberY" but it does not produce accurate right coordinates (with mini error that makes the roads not applicable for navigation).

From  BinaryMapIndexReader, I can see that you select some base point( px&py ), and compute the delta of x&y between one point and the base point.

However, from IndexVectorMapCreator, "MapUtils.get31TileNumberX and MapUtils.get31TileNumberY" are used to convert lon&lat to x&y and write directly to binary file. Where is delta mode and 24th zoom used? 

Xuemei

[Victor Shcherb]

Somewhere, I don't know it by heart :) BinaryMapIndexWriter? 

There is a little bug that is visible on zooms 18-24 however it is related only to 2nd, 3rd ... and other points of the way.

Victor

2012/4/7 Xuemei Liu <lxuem...@gmail.com>

[Xuemei Liu]

--Somewhere, I don't know it by heart :) BinaryMapIndexWriter? 

In net.osmand.data.preparation.IndexVectorMapCreator.java.insertBinaryMapRenderObjectIndex(), MapUtils.get31TileNumberY and MapUtils.get31TileNumberX are used to convert from lon&lat to x&y, and write x&y to binary file. 

For new roads in my patch file, I use  MapUtils.get31TileNumberX | Y to convert from lon&lat to x&y..

-- There is a little bug that is visible on zooms 18-24 however it is related only to 2nd, 3rd ... and other points of the way. 

What does this mean? I think maybe this is related to my problem.

Xuemei

[Xuemei Liu]

And there is no  BinaryMapIndexWriter. In fact I just need one conversion from lon&lat to x&y that can be used in Osmand.

[Xuemei Liu]

Hi, Victor,

Can you bother to give me some notes where is the but is? I think it's related to my bottleneck now.

"-- There is a little bug that is visible on zooms 18-24 however it is related only to 2nd, 3rd ... and other points of the way. "

Xuemei

[Victor Shcherb]

https://github.com/osmandapp/Osmand/blob/master/DataExtractionOSM/src/net/osmand/data/preparation/BinaryMapIndexWriter.java 

# codeCoordinateDifference method

2012/4/9 Xuemei Liu <lxuem...@gmail.com>