Help by creating maps

[Manfred]

Hi!

On
https://code.google.com/p/osmand/
I've read the following:

Help to the product
3. You can prepare/update data for some regions & upload to the
downloads.

I've made a newer Austria_europe_1.obf.zip (7.12.2011)
How and where can I upload?

Regards
Manfred

[Victor Shcherb]

Unfortunately our  http://code.google.com/p/osmand/downloads/list server is unstable.

And now we can not upload any file.

So we are working on it.

Victor

2011/12/24 Manfred <alfman....@gmail.com>

[Pavol Zibrita]

Give me a link where I can download it.

Regards,
Pavol

[hbogner]

New maps for croatia:
http://data.osm-hr.org/osmand/
Generated today with 0.6.8.2-beta.
Can't upload to google server so it's on our server.
Will set up daily generation of new maps.
Never managed to get automatic uplad to google server before.

Regards,
Hrvoje

[Manfred]

Hi Pavol!


Here it is:
http://members.inode.at/397412/osmand/Austria_europe_1.obf.zip

The obf-file inside the zip has not the correct name, it is simple
"Austria".
Sorry for that.

Regards
Manfred

[Pavol Zibrita]

I see the listing, but download is restricted.

Regards,
Pavol

[hbogner]

Fixed!
Hrvoje

[Pavol Zibrita]

Should be there.

Regards,
Pavol

[Manfred]

Hi Pavel!

Where?

Regards
Manfred

[Pavol Zibrita]

Downloadable by osmand.

Regards,
Pavol

[Manfred]

Hi Pavel!

Thank you!

Found it there now, but not on:
https://code.google.com/p/osmand/downloads/list?can=2&q=Austria&colspec=Filename+Summary+Uploaded+ReleaseDate+Size+DownloadCount

Switzerland ODF I don't find find there, POI only.

Are there different download sources?

An other thing:
I use 5 map files:
Austria,
Switzerland, Liechtenstein
Liechtenstein,
Germany Baden Württemberg,
Germany Bayern,

and would like to offer you to create these files once a month (one a
week)

If that would be O.K.for you, I would send you a link to download, and
you reply after download.

Regards
Manfred














On 28 Dez., 11:54, Pavol Zibrita <pavol.zibr...@gmail.com> wrote:
> Downloadable by osmand.
>
> Regards,
> Pavol

[hbogner]

Testing the final things in a cron job but http://data.osm-hr.org/osmand/
will have daily updated croatia maps.
Is there a way to automatically zip maps without uploading?
When I set upload to false it doesn't zip them.
Any idea how to automatically zip them without upload.

[Victor Shcherb]

Hi All

That page supposed to show all indexes on local machine and googlecode but shows only googlecode.

http://download.osmand.net/indexes.php

Right now we are working on Jenkins solution for indexes http://vs1543.ams2.alvotech.de:8080.

As I see spreading Jenkins with ready for osmand generation job  would be the easiest way for people who wants to help to maintain indexes for Osmand.

Probably master-slave relation should be installed between Jenkins and probably something else.

Also we are looking for a new file hosting solution we need at least 100-150 Gb of space and at least 5Tb internet bandwidth per month.

The main problem is to track stable index generation and non-stable. That's pretty easy to see.

Let's mark one version of OsmAndMapCreator as stable. And let's have all indexes generated by this OsmAndMapCreator.

All the time we are generating indexes we are generating them with latest OsmAndMapCreator and latest map data. If we figure out that OsmAndMapCreator generates buggy data we need to revert all generated data to stable one. At some point of time (if the data was tested or we did not get any complains) we mark all current indexes as stable and generate new one.

Victor

среда, 28 декабря 2011 г. пользователь hbogner писал:

[Pavol Zibrita]

http://download.osmand.net/indexes.php
was fixed and shows all indexes now. also googlecode and also local ones.

So yes, there are currently two sources not only googlecode. download.osmant.net redirects the download
to right place.

I don't agree about everything with Victor. The jenkins idea is somehow ... crazy. I can image it can be done,
and that some people we can really use as slave machines. This would probably mean have for each map a
jenkins job. But, why not.

I agree about to use some stable version of map creator, probably, when the setup would be in jenkins, this
is something that would be our problem.

However, I think better practice would be to release the files as soon as generated, this would distribute the download,
where releasing the files at once would generate a huge download peak. Also, there might be situations where
people really need more frequent update, HOWEVER, the we have to think about the download limits.

This problems started because the googlecode is currently unusable, at least according uploading. Anyway, this problem
might arise in future, so, we should try to resolve it.

The hosting is true, the map data is not that huge, backup could be hosted somewhere else, but the trafic might be even more.


About zipping: How to zip the files, well, you can just zip them, I will rezip them. Or, there is a utility, currently no sh exists for it,
but, I will add it to map creator. For now, you can change the IndexBatching class to IndexZipper in the batch_indexes.sh and as
parameter give it a directory where the maps are stored.

Regards,
Pavol

[Pavol Zibrita]

Well, don't forget about the offer :-).
We will have to come up with something first, maybe the setup with jenkins.

Regards,
Pavol

[hbogner]

It would be nice to have one version marked as stable, like JOSM has.
So we all know which version to use if we want to generate maps for
others.

On Dec 28, 4:48 pm, Victor Shcherb <victor.shch...@gmail.com> wrote:
> The main problem is to track stable index generation and non-stable. That's
> pretty easy to see.

> Let's mark one version of OsmAndMapCreator as *stable*. And let's have all

> indexes generated by this OsmAndMapCreator.
> All the time we are generating indexes we are generating them with
> latest OsmAndMapCreator and latest map data. If we figure out
> that OsmAndMapCreator generates buggy data we need to revert all generated
> data to stable one. At some point of time (if the data was tested or we did
> not get any complains) we mark all current indexes as stable and generate
> new one.

I can set uplad from false to true and get them ziped automatically,
but that generates extra bandwith and needs more time.
If I set upload false they do not zip and I have to do it manually, it
would be nice to have an option zip_files="true/false" without the
need to upload them.
I'm having trouble with zip in cli. Don't know khow to zip multiple
files to multiple archives, bzip2 has it easy " bzip2 * " and it
works :D

On Dec 29, 1:15 pm, Pavol Zibrita <pavol.zibr...@gmail.com> wrote:
> About zipping: How to zip the files, well, you can just zip them, I will
> rezip them. Or, there is a utility, currently no sh exists for it,
> but, I will add it to map creator. For now, you can change the
> IndexBatching class to IndexZipper in the batch_indexes.sh and as
> parameter give it a directory where the maps are stored.

Here are some new maps: http://data.osm-hr.org/osmand/
I used bzip2.
Just started the whole world on few machines just to see how long it
takes, so when it's all finished I'll upload the rest.
If it doesn't take to long I'll set it to run weeky/montly.

Regards

[hbogner]

Arghhh, I forgot that the cron job setup deletes all old files in
osmand web directory, now changed.
Will upload again after it's all finished.

[Stephan75]

Do you still have a general server problem?

If yes, does it make sense to consult for example
http://wiki.openstreetmap.org/wiki/OSM_Servers_in_Rambler to ask
whether their machines can create Osmand's maps regularly and offer
the download?

Greets, Stephan

On 25 Dez. 2011, 00:33, Victor Shcherb <victor.shch...@gmail.com>
wrote:
> Unfortunately our  http://code.google.com/p/osmand/downloads/listserver is

[Victor Shcherb]

I'm working on that solution. Having distributed map build is primary goal for now.

For example I think Rambler would be most interesting in frequent update of Russia indexes however updating US indexes can take 1 week even on fast server. So providing general solution is primary goal and after that we can think what parts can be updated more frequently and on what machines.

Victor

2012/1/1 Stephan75 <der.steph...@googlemail.com>

[hbogner]

Asia, Africa, Australia, all Americas finished and uploaded, Europe
will be finished today, only Benelux left, but won't have net access
later today so won't be able to upload until tomorrow.

Regards

[hbogner]

Finally all finished and uploaded, whole world. I'll get some
statistics later.
Will read later about Jenkins.

On Jan 2, 12:01 am, Victor Shcherb <victor.shch...@gmail.com> wrote:

> I'm working on that solution. Having distributed map build is primary goal
> for now.
> For example I think Rambler would be most interesting in frequent update of
> Russia indexes however updating US indexes can take 1 week even on fast
> server. So providing general solution is primary goal and after that we can
> think what parts can be updated more frequently and on what machines.
>
> Victor
>

> 2012/1/1 Stephan75 <der.stephan.step...@googlemail.com>

[Victor Shcherb]

Hi hbogner,

Where you upload indexes? There is a special task just to upload generated indexes.

It is good if it goes through a verification server (that's pretty fast) .

The final list is http://download.osmand.net/rawindexes/?C=M;O=D

Victor

2012/1/3 hbogner <hbo...@gmail.com>

[hbogner]

To the same server where I placed croatia indexes, the link is few
posts before this one.
http://data.osm-hr.org/osmand/world/
12G of bz2 format, whole regions.xml file was processed.

I'll read later Jenkins.
This was started on 30.12.2011. and finished yesterday, but I was off-
line so couldn't upload before.
Old automatic upload never worked for me, never managed to configure
it properly.

Hrvoje

[Stefano s]

I asked in Italian talk-it ML if GFOSS association can provide a
similar service for Italy, we'll see what will happen :)
Stefano

[Victor Shcherb]

Hi Hrvoje

The old automatic update was very cumbersome and hard to configure. But new one is representing very tiny layer.

So what you need just unzip all obf files from you bz2 (it is not supported) or use zip.

And upload all of them to http://vs1543.ams2.alvotech.de:8080/job/GenerateIndexes/ws/indexes/ through ssh. In that case I will need your public key to to give you access.

It would be good if you can share your efforts with automatic download.osmand.net.

If you can also help with spreading indexes from your site that would be also good but that's a different story. 

In that article I described how to setup 2 jobs : 1) Generation 2) Upload (that you need now).

Best Regards,

Victor

2012/1/3 hbogner <hbo...@gmail.com>

[La Colle]

Hello,

In this server there is one file which seems wrong
(France_alsace_europe_1.obf.bz2) because osmand can not index it. is
it possible to regenerate it ?

Thanks a lot for your great job !

Bye.

On 3 jan, 19:19, hbogner <hbog...@gmail.com> wrote:
> To the same server where I placed croatia indexes, the link is few

> posts before this one.http://data.osm-hr.org/osmand/world/

[Victor Shcherb]

Hi All

From that link  http://data.osm-hr.org/osmand/world I found some files that were corrupted. And now all these files already migrated to new.osmand.net:8080. So all them will be available soon through regular download.osmand.net

Best Regards,

Victor

2012/1/4 La Colle <nicolas...@brume.org>

[hbogner]

Yes, that was only temporary upload place started before new upload
system, only index that will be daily generated on that server is
Croatia.
My goal was to update all those indexes, some were generated in
november, some even before that.
Yesterday all those indexes were uploaded to http://new.osmand.net:8080/
As it turns out some of those indexes were corrupt, and that was
filtered by the new system, and corrupt files never went to
production.
As Victor said use regular download.osmand.net and I'll generate new
indexes as times goes and upload them to http://new.osmand.net:8080/
where they will go thru validator.
The goal is to set up a cron job as a part of http://new.osmand.net:8080/
jenkins to update the whole planet regularly, but testing asks for
time which I don't have now.

Regards,
Hrvoje

On Jan 4, 3:33 pm, Victor Shcherb <victor.shch...@gmail.com> wrote:
> Hi All
>

> From that link  http://data.osm-hr.org/osmand/worldI found some files that

[Victor Shcherb]

We actually don't need testing from your side. That's why validator exists. 

I'm thinking to improve it to  tests the map data carefully but simply checks are already there.

Best Regards,

Victor

2012/1/4 hbogner <hbo...@gmail.com>

[hbogner]

Testing from my side would involve how to setup
http://code.google.com/p/osmand/wiki/OsmandMapUpdates
First thing is to figure out how to set it up, manual for dummies who
have no time to explore but have space cpu power would be a good
thing :D

Regards
Hrvoje