What's up with map updates?

[Mark Begbie]

I'm getting increasingly frustrated with the schedule for map updates.  We're now on the 14th of June and the New Zealand map update to 1st of June hasn't been released yet.  I could understand if it were not for Osmand also offering daily and hourly updates for people that pay a higher fee.  If all maps, globally, can be updated hourly, how can it possibly be justifiable to take over two weeks to roll that update out to other users?

One of the great things about OSM is the crowd sourcing nature of the map.  If I'm travelling somewhere I typically look at the map and add areas of interest to it before I go.  That way everyone can benefit from the research I do before I go somewhere.  With now a 6 week plus lag between early month updates and the map being rolled out that doesn't really work.

[Harald R.]

It is pretty easy to calculate the OSMAND maps locally, see here https://wiki.openstreetmap.org/wiki/OsmAndMapCreator

The 200 MB from https://osm-internal.download.geofabrik.de/australia-oceania.html should be convertedin a reasonable time.

[Mark Begbie]

Thanks for that, I'll have a look.

[V S]

All maps are updated now. Please use OsmAnd Live subscription and you will get incremental map updates every hour.

Our full generation map update is becoming more complicated and this month it was delayed by 5 days because of hardware issues though it is expected that it could take 2-3 weeks and maps of 1st/XX/20YY will be produced at 20/XX/20YY

[Greg Troxel]

V S <victor....@gmail.com> writes:

> All maps are updated now. Please use OsmAnd Live subscription and you will
> get incremental map updates every hour.
> Our full generation map update is becoming more complicated and this month
> it was delayed by 5 days because of hardware issues though it is expected
> that it could take 2-3 weeks and maps of 1st/XX/20YY will be produced at
> 20/XX/20YY

The osmaand live system works great. I use it typically before going
out mapping, or after doing pre-mapping before going someplace. (I find
it most efficient to draw everything I can from imagery first, and then
to make notes of things that weren't obvious - having the big things
drawn makes being oriented for notettaking much easier.)

I find that if I click "update", I get an update package that typically
shows a last change time within a few hours. But often, that's because
nobody updated the map that hour. I have not tested rigorously, but
it seems that

edit and upload
wait 1 hour
live update

almost always results in getting my updates.


My one nit about the updates is that the configuration of "update daily,
night" seems not to happen. I don't know if osmand has a background
service, or if it's supposed to happen some other way. But if I open
up live update and click "update now", it works reliably and quickly.

[Mark Begbie]

I totally get that the map updates are getting more complex.  I'm trying to build New Zealand on a Win 10 Core i7 with 8GB of RAM and struggling.

What I do NOT get at all, is how on the one hand it takes almost a month to update the map for one group of users and yet for another group it's done hourly.  Something just smells really bad about that.

I get, too, that developers need to find ways to deliver recurrent revenue and one time subscriptions with forever map downloads doesn't deliver that.  Perhaps ending the one-time payment option would be a way to go?  I don't know.  Getting a little off topic now, but another thing that worries me is the mapper payments reference in the OSMAnd live information.  I spend a lot of time mapping, and I don't expect to be paid.  What worries me though, is if there's a precedent set for paying for content it could upset and derail the whole free input free output model that the OSM data set is built on.  Payments should relate to value add services on top of the data, which should be available to all.

[Victor Shcherb]

Not sure, I could understand, I think New Zealnd takes 2-3 hours max to generate the map. And all maps are generated 8-9 days.

Live maps are quick cause they are generated incrementally but they require 24/7 monitoring system to properly catch up with all changes.

Best Regards,

Victor

--
You received this message because you are subscribed to a topic in the Google Groups "Osmand" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osmand/B3HMYXY507E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to osmand+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osmand/aa45a01b-629b-449a-9c55-f377a5aec229%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Majka]

As the live update works with tiny files (just the diffs), the live update system won't need much processing power in comparison with the complete map. I can understand why the live updates are working well and the normal maps are struggling.

What I would prefer would be free live update for mappers instead of the current schema. I am active mapper myself, but the current method is promoting low-quality edits, IMHO. It is counting only the number of changesets, not the number and quality of changes. In theory, the most problematic and discussed mapper active in our country , who we had to stop several times, would get paid for the problems he brings into the map. He just didn't ask for the payment, as far as I can see.

[Martin Trautmann]

On 19-06-17 15:54, V S wrote:
> All maps are updated now. Please use OsmAnd Live subscription and you
> will get incremental map updates every hour.
> Our full generation map update is becoming more complicated and this
> month it was delayed by 5 days because of hardware issues though it is
> expected that it could take 2-3 weeks and maps of 1st/XX/20YY will be
> produced at 20/XX/20YY

Sorry, but I do feel that OSMAND Live is pretty much useless, to offer
updates "EVEN PER HOUR", if in fact the maps are delayed by weeks for
the regular monthly schedule.

Personally, I do not need regular updates for an area I traveled last
year only. I do a monthly update for my local area. As of today, the map
available is still from the month before.

Schönen Gruß
Martin

[Xavier]

On Wed, Jul 03, 2019 at 10:30:29PM -0700, Mark Begbie wrote:
>I totally get that the map updates are getting more complex. I'm
>trying to build New Zealand on a Win 10 Core i7 with 8GB of RAM and
>struggling.
>
>What I do NOT get at all, is how on the one hand it takes almost a
>month to update the map for one group of users and yet for another
>group it's done hourly. Something just smells really bad about that.

Not hard to understand at all:

OSM Data for Torrey UT at zoom level 14:
https://www.openstreetmap.org/#map=14/38.3046/-111.4188

OSM Data for a portion of Los Angeles CA also at zoom level 14:
https://www.openstreetmap.org/#map=14/34.0440/-118.2600

Note the significant difference in detail level.

Your i7 w/ 8GB RAM machine would build a map data file for Torrey
UT *much* more quickly than it would build a map data file for Los
Angeles CA. In fact, it likely might build the entire state of Utah
faster than it would build just the city of Los Angeles, much less the
whole state of CA.

So a huge contributor to the speed of map updates is the density of
data to be built (under the reasonable presumption that the OsmAnd
devs. have a single size, likely large, server that is used to build
the updated maps). Areas that are lower density can get built faster
than areas of extreme density for the same size server performing the
build.

As for the hourly builds, they sound like they are built as deltas
against the last monthly build file, so the hourly builds only have to
consume the changes that occurred since the last monthly file was
built, which is a much smaller dataset, no matter the starting area,
than building the whole area from scratch. And OSM itself provides a
service that delivers those changes as just the changes since the last
change file, so OsmAnd does not have to first determine what changed
before building the hourly files.

[Greg Troxel]

Martin Trautmann <tr...@gmx.de> writes:

> Sorry, but I do feel that OSMAND Live is pretty much useless, to offer
> updates "EVEN PER HOUR", if in fact the maps are delayed by weeks for
> the regular monthly schedule.

It would be good to understand how it works before declaring them
useless.

osmand live provides files that contain differences from published map
files. So with a fairly recent map (and it seems to not have to be the
most recent one), and the delta updates, you have up to date data.

I find that I can add something to OSM, wait an hour or sometimes two,
and then get the live update, and have the new things shown in osmand.

So from my viewpiont it works as intended.


I also don't understand why people are so upset about monthly updates
being 10-20 days behind the end of the month. Yes, the next day would
be better, but 1) it's so much faster than any non-OSM approach and 2)
with osmand live, it matters very little. Even without osmandlive,
having map data that's between 20 and 50 days old at any point isn't
much worse than having data that's between 5 and 35.

[Xavier]

On Thu, Jul 04, 2019 at 02:14:21PM -0400, Greg Troxel wrote:
>
>I also don't understand why people are so upset about monthly updates
>being 10-20 days behind the end of the month. Yes, the next day would
>be better, but 1) it's so much faster than any non-OSM approach

If I remember correctly, back when I had a TomTom, the map updates from
TomTom were available either on a 6 month cycle or yearly cycle.

OsmAnd monthly updates, even if they don't arrive until the 10th of the
month, are already either six or twelve times more frequent than what
TomTom provided.

I don't know what cycle TomTom is on now, as I switched to OsmAnd when
TomTom decided to drop support for my old navigator.

> and 2) with osmand live, it matters very little. Even without
>osmandlive, having map data that's between 20 and 50 days old at any
>point isn't much worse than having data that's between 5 and 35.

20-50 days old seems better than 180 or 365 days old (and of course
with TomTom one had to pay per update or 'subscribe' to a limited
number of updates).

[Helmut Jarausch]

Are there any ongoing efforts to rewrite the software to create the maps in C++?

I would expect a significant speedup on multicore processors. 

--
You received this message because you are subscribed to the Google Groups "Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/osmand/20190704185647.GN1375%40d820.dp100.com.

[Mark Begbie]

Ā key factor here is the nature of the data. The data that systems like Tom Tom are built on is notionally complete. That is to say all significant features of a type which are in existence are in the data set when it is created. This means the data files for systems like Tom Tom change relatively slowly. But, it is only feasible commercially for limited data sets; roads, rail, geographic data, postcodes etc.

The data set for OSM is quite different. For one, it is FAR richer and more diverse. Tom Tom will not provide info on cafes with associated opening times, wheelchair access and WiFi availability information. In this regard OSM data is also far less complete.

My point is that in terms of comparison to the functionality of commercial GPS I'd say the OSM data is complete nowadays. If you are using OSMAnd as an alternative to Tom Tom, then monthly updates which come out a month behind are fine.

But, if you use OSMAnd for the far richer data, it is not such a good situtation. Or at least, you have to accept you could be missing out on a LOT of detail created in the interim 8 weeks.

The problem for me, is that it is this richer data that I want for trips. When, as is often the case, there is a lot of useful data missing i will mai it a part of my trip research. That way everyone gets to benefit in the long run. But I don't plan every trip far enough ahead that all the research is complete 2 months in advance. Hence the delay between updating the database and receiving that information in OSMAnd can be a problem.

Now I've sussed the map creator I can compile and download a new New Zealand map in 90 minutes of so.

[Martin Trautmann]

On 19-07-04 20:14, Greg Troxel wrote:
> Martin Trautmann <tr...@gmx.de> writes:
>
>> Sorry, but I do feel that OSMAND Live is pretty much useless, to offer
>> updates "EVEN PER HOUR", if in fact the maps are delayed by weeks for
>> the regular monthly schedule.
>
> It would be good to understand how it works before declaring them
> useless.

It would be even better to get this info, other than buying this option
first.

What is it that you can learn from https://osmand.net/osm_live about
this kind of behavior?

> osmand live provides files that contain differences from published map
> files. So with a fairly recent map (and it seems to not have to be the
> most recent one), and the delta updates, you have up to date data.

If this mechanism would be that easy, why would it be so time consuming
to create new map updates for every month?

Schönen Gruß
Martin

[Xavier]

Likely because the base monthly maps are not generated from the delta
difference files, but by reprocessing the entire OSM dataset anew each
time (the planet osm datafile, when uncompressed, is about 1 terabyte).

And, anticipating a "why" question, one would process the data this way
to prevent a common problem when handling long chains of delta data.
If any piece of delta data in the middle of the chain is missed
everything after the missed data that depends from the missed data
turns into garbage.

So one would regenerate anew for some cycle (in OsmAnd's case, that is
monthly) then build differencing files for a month. So any errors in
the delta chain propagate for no more than one month before they get
cleaned up (by the "start anew" montly cycle processing).

[Harry van der Wolf]

Java has come a long way. Java is one-time compiled and then almost as fast as C/C++.

Java already supports multicore processors for years and also the OsmAndMapCreator can be started with the number of cores to use (or via the config)

Note also that for the bigger maps, the disk access is way more important than the difference in speed for C/C++ or java.

Using lots of memoriesand superfast solid state disks gives you the best performance by whole factors. And yes: switching to C++ might get you the last few percentages.

Harry

Op vr 5 jul. 2019 om 10:50 schreef Helmut Jarausch <hjar...@gmail.com>:

To view this discussion on the web visit https://groups.google.com/d/msgid/osmand/CA%2BCgGZRsKdoZwQrBth1okp1WxXsD4bqnTrDk72X5CUXQvXbmCQ%40mail.gmail.com.

[Mark Begbie]

That answers my original lack of understanding on why hourly updates can happen for Live, but monthly updates are delayed. Thanks.

[Martin Trautmann]

On 19-07-09 17:47, 'Xavier' via Osmand wrote:
> On Tue, Jul 09, 2019 at 02:44:08PM +0200, Martin Trautmann wrote:
>> On 19-07-04 20:14, Greg Troxel wrote:
>>> osmand live provides files that contain differences from published
>>> map files.  So with a fairly recent map (and it seems to not have to
>>> be the most recent one), and the delta updates, you have up to date
>>> data.
>>
>> If this mechanism would be that easy, why would it be so time
>> consuming to create new map updates for every month?
>
> Likely because the base monthly maps are not generated from the delta
> difference files, but by reprocessing the entire OSM dataset anew each
> time (the planet osm datafile, when uncompressed, is about 1 terabyte).
>
> And, anticipating a "why" question, one would process the data this way
> to prevent a common problem when handling long chains of delta data.  If
> any piece of delta data in the middle of the chain is missed everything
> after the missed data that depends from the missed data turns into garbage.

If the mechanism is reliable, it would be sufficient for that task.
You could spread the load by building e.g. two of three from the delta,
while the third is built from scratch.

But as you tell me, using Osmand Live would require BOTH delta updates,
as well as a monthly full (proper?) reload of the maps?

Schönen Gruß
Martin

[Majka]

On Tuesday, 9 July 2019 22:42:08 UTC+2, Martin Trautmann wrote:

But as you tell me, using Osmand Live would require BOTH delta updates,
as well as a monthly full (proper?) reload of the maps?

Schönen Gruß
Martin

No, you need ONE full download of the map. Then, depending on the frequency and size of the changes, you simply download the deltas (live update) as you want/need. Some time later (might be several months or even years), OsmAnd MIGHT be slowing down, because you'll work with the "normal" file + several separate delta files. Then, it is time to download the full map again. This will remove the deltas, and you can start again. But apart of the possible slow-down, there is nothing there needing the new full download, you can simply continue with the live updates only.

[Martin Trautmann]

On 19-07-10 12:35, Majka wrote:

> No, you need ONE full download of the map. Then, depending on the
> frequency and size of the changes, you simply download the deltas (live
> update) as you want/need. Some time later (might be several months or
> even years), OsmAnd MIGHT be slowing down, because you'll work with the
> "normal" file + several separate delta files. Then, it is time to
> download the full map again. This will remove the deltas, and you can
> start again. But apart of the possible slow-down, there is nothing there
> needing the new full download, you can simply continue with the live
> updates only.

So there are two different opinions.

If deltas were that reliable, it would be a much faster and simpler task
to derive monthly updates from them.


Schönen Gruß
Martin