BTSync initiative

[Victor Shcherb]

Hello,

There is a demand to have maps downloaded as soon as they are released. Unfortunately it is hard to satisfy that demand because bandwidth limitation. But we can try to use Torrent to technology to help each other.

For now I would like to present BTSync technology http://labs.bittorrent.com/experiments/sync/get-started.html# and give it a try.

I'm publishing read-only keys to folders with latest maps : 

- Russia

R2NGOYG7TSJL7BT5QB5QFULORCH7TQ43X

- France

RYL64R2O3EKFBXKX3Z4C6NRT374WLMJD7

- GB

RW2VAHTU62NV5W5CLC5YS6XGXWHSEKTIA

- Germany

RAMO45FOHSUHZNCIPEJX4EMDPOW44HOT6

- Europe

R7MSXJBL7PDB74LI3A5HTIUQ7QDQE5ITU

- NorthAmerica

RJ6BUYMK4CDT64G3JZWVKAK2UPNGALXXI

Files in your folders will be automatically updated as soon as they are released. I hoped the server will maintain these connections (port for advanced configuration 5858 but it is optional). 

Drawbacks : 

- It is impossible to select single file from the folder. At least for now ;)

- There is no official client for Android, but they are working on it and there is one unofficial. 

-------------------------------

FYI : 

For now it doesn't give you possibility to update maps on your phone immediately and safe. But you could setup something like a chain of updates BTSync -> Folder -> Dropbox/SkyDrive... (Files selective) -> Sync on Android phone in folder backup.

Once you see everything is ok you activate obf file and it is ready for use. This is my imaginary setup please use at your own risk :

Victor

[Victor Shcherb]

For now it is possible to exclude files from sync, but not to include 

http://forum.bittorrent.com/topic/19262-whitelisting-in-syncignore-syncallow-directorytree-file/page__hl__select. 

Be aware all these folders > 1GB.

[john whelan]

Works quite nicely.

Thanks John

--
You received this message because you are subscribed to the Google Groups "Osmand" group.
To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Nick A]

Hi,

It works okay for me for GB & I see there are others also seeding.

I haven't bothered adding the secrets for Europe, Germany etc as at present I am not using those maps, but would it be of any benefit to the system if I added it and seeded at those times I am online?

Would there be any merit in reducing the download speed available from the main server, so that more of the load is shared by the other seeders (may be something to consider in the future if there are insufficient seeders at present).

Presumably also, it would be good to keep adding comments to this listing so that others get to read it, perhaps join in, and reduce the load on the OSMand servers.

Regards

Nick

[john whelan]

Looks like I'm beginning to seed now as well.  I suspect that your comments about reducing the feed from the main seed might be useful.  The full set of files is currently about 19 gigs by the way.

Cheerio John

[Victor Shcherb]

Hi,

Be aware of the traffic :) Maps are regenerated 3 times a month and binary diff can take around 10 GB. Basically it could cause redownload of 30GB a month.

Still this initiative is not to help for downloads (because downloads don't support peer to peer), but to always have fresh maps with minimal impact.

Unfortunately I can't generate key for every map, I just can't automate it :P And I'm not sure if 400-500 files could be fine supported by BTSync instance.

Don't get me wrong it is possible, if someone can share with me big BTSync config file & bash script to put all files into folders, I'm fine to go that way.

Victor

[john whelan]

Sounds like it could help with downloads as well.  I think the current set up is going to cause people to download a lot more data than they need.

My understanding of the process is once some one has the key for a folder then they can download the latest version.

So its a one off job to create the folders and needn't be on your machine but you'd need to link the key to a folder on your server.  Once done then its just a matter of when you generate a new file copy it into the appropriate folder.

I strongly suspect that 500 folders could be supported by BTsync.

Cheerio John

[john whelan]

Just to play with

Albania_europe_2
DTB6HSS23F4QI3FXNT2HMZPQRCIQTEPF

Andorra_europe_2
FYFSUR3FFV4VBBRT3GVMIMXBO6SBM4ZV

Austria_europe_2
ZY7WZFIJFQQA5I6G4ADLGY2PXICEWL6X

My machine isn't on all the time but the files are fairly small so its much faster than picking up all the European files.

Cheerio John

[Miguel D.]

Hi,

Since this topic was created BTSync has evolved a lot... there is an Android app, so it is possible to keep android devices up-to-date.

I've created a folder containing all obf maps. I have a script that periodically checks for updates and downloads new maps.

Use the following secret: BZVMBM27I7ASB4673MZDJYENGLTYXWJLA

This secret is read-only! Meaning that with that secret, users can only download files and cannot modify them or add.

For those that do not want to have 50gb of maps, you can create a text file named ".SyncIgnore" in the share folder. Each line should contain the file names that you do not want to download. For instance, the following file will ignore several continents except europe:

### START OF .SyncIgnore

*_asia_*
*_northamerica_*
*_centralamerica_*
*_southamerica_*
*_africa_*
### END OF .SyncIgnore

 

Note that my connection is not the best, but the share is online 24/7. If you already have some files you can place them in the share folder, and BTSync will index them.

Also, some files are still downloaded from osmand servers, and in the next couple of hours will be complete.

Miguel D.

[john whelan]

Looks sensible to me, I'll join in so that the feed speed will be a bit faster.

Thanks John

For more options, visit https://groups.google.com/d/optout.

[Miguel D.]

Currently the folder is around 38 GB.

I've appended a sample .SyncIgnore file (note the starting dot in the name).

You just have to edit it to your needs and place it in the share folder.

Some stats:

- *_europe_* : 19 GB

- *_asia_*: 8 GB

- *_northamerica_*: 9 GB

- *_centralamerica_*: 514 MB

- *_southamerica_*: 1050 MB

- *_africa_*: 816 MB

[EnSun]

Two questions:

1. Will I be syncing with all other Read-Only users or only with Miguel's server?
2. Does ind.cache file need to be deleted manually each time for re-indexing of the freshly downloaded maps?

Thanks in advance.

[Miguel D.]

Hi,

1. You will sync with all users.

2. OSMAnd should detect that the file has changed and re-index the new map. I think that there is no need to delete ind.cache manually, at least I've never do it.

Miguel.

[john whelan]

Basically the more users on line in the swarm the faster the download speed.  At the moment I'm seeing a rough max of 100 Kb/sec from NAS which I assume is Miguel, I'm currently uploading at 110 Kb/sec, I have seen 500 Kb/sec from one of the others in the swarm.  I'll probably come up XEON by the way.

Currently I have picked up 11 gigs of the 38 gigs available in roughly two days running 24 by 7, normally I don't have by machine on 24/7 but I will for the moment to get this one off the ground.  I suspect Miguel is running BTsync on a device rather than a conventional PC and I suspect I should look into getting a device dedicated to BTsync.

Cheerio John

[EnSun]

Thank you Miguel. What a great work this project is!

[EnSun]

Hello John,

Am I correct that you are downloading to a computer and not a mobile device (a phone)?

My intention is to sync to my phone directly and not to a computer. How feasible is this?

You are also saying "I should look into getting a device dedicated to BTsync". Does that mean a dedicated mobile device only for the purpose of BTSync or you mean using for example your mobile phone?

Thanks.

[john whelan]

BTSync can run on a variety of devices currently I'm running it on a Windows PC, I have a Nexus 7 which can also run it.  The Win 7 machine is a gaming one so it uses a lot more electricity than a NAS box or something like a Nexus 7 hence my comment of perhaps I should look at having a device on my network that is always on and dedicated to BTsync.  One day they may produce something for my ASUS router which already has the capability of having a USB stick plugged in and running Torrents.

If you run it on your phone or device the problem is the amount of bandwidth required which isn't a problem if you can limit it to wifi only.  I'd limit it to just the map you are interested in since most phones and small tablets such as the Nexus don't have enough memory/storage for 40 gigs of maps.

It should be perfectly feasible to btsync to the same folder as OSMAND uses for its maps but there is a small problem in that when a new version is available I assume that the old version is deleted before the new version is downloaded.  However it might take some time to download the new version.  My personal approach would be to use a tablet or PC for BTsync then just copy over the new version when it arrives.  You could use the data modified stamp but I suspect there should be a more sophisticated method which will unfold over time.

Cheerio John

[Miguel Domingues]

My current setup is the following:

- A dedicated machine that periodically checks for updates and keeps all obf files, and runs BTSync.

- An android tablet that syncs only through BTSync and where I use BTSync. Usually the tablet syncs in my home wifi at speeds like 8 MB/s.

If you want to run BTSync directly in the mobile phone, that is ok. Its up to you :)

Unfortunately my home connection upload can only send 100kb/s, but lets see how this evolves within the next month or two.

If the outcome is positive for most users. Also, if you have suggestions please send them :)

Miguel Domingues

---

From: john whelan john whelan
Reply: osm...@googlegroups.com osm...@googlegroups.com
Date: 24 May 2014 at 19:29:34
To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

You received this message because you are subscribed to a topic in the Google Groups "Osmand" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osmand/iZ012pkeBLw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to osmand+un...@googlegroups.com.

[john whelan]

I think once we get a few more machines in the swarm your upload speed won't be a bottle neck.

Cheerio John

[EnSun]

In my case, direct sync to the phone seems to be preferable for at least these reasons:

1. Storage capacity is not an issue due to the sample.SyncIgnore.txt file that Miguel has shared and the flexibility of using 64GB MicroSD.
2. If I still would need to copy the synced maps from a different computer folder (designated as BTSync Read-Only node) to my phone what am I gaining by doing so instead of just downloading from http://download.osmand.net/list.php (even though it might not be very fresh version but not sure how stale would it be considering the BTSyncing duration at this time!)

:-)

[EnSun]

Miguel - We would need to hope your tablet is healthy and fully operational, which begs another question:

Is there a plan (method) for us to be able to sync directly with the Osmand server to download fresh maps as your script does?

[Miguel Domingues]

My tablet is not the dedicated machine running BTSync. It's a Linux box that does all the work.

Miguel Domingues

[EnSun]

Thanks Miguel. I'm leaning toward a setup where I would sync with your machine using a PC and then have the phone/tablet syncing with the PC.  Not sure what the drawbacks would be though.

[Miguel Domingues]

That's my setup right now.
Drawbacks I don't see any but there's an advantage at least. Your phone will sync at home via wifi with your PC faster than it would via Internet.
My PC has the entire folder and my tablet has only some obf files (restricted via the sync ignore file).

I hope to see more people joining, it would increase the overall speed. It's one of those cases... The more the better.

Miguel Domingues

Em 25/05/2014 13:22, "EnSun" <asha...@gmail.com> escreveu:

Thanks Miguel. I'm leaning toward a setup where I would sync with your machine using a PC and then have the phone/tablet syncing with the PC.  Not sure what the drawbacks would be though.

[EnSun]

Thanks Miguel - So far the .Syncignore file is being ignored. I've placed it in the root of the read-only folder (Windows machine) with Africa for example excluded using your format in the sample file. But sure enough, Africa started to download!

[Miguel Domingues]

My bad.

In the sample file provided the ignore line is “*_africa_*”.

The file being downloaded is “Africa_2.obf” right?.

That name does not match the ignore rule, keep that file and add this one: “*Africa*” (note the up-case A).

Check if it works.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com
Date: 25 May 2014 at 18:57:42

To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

Thanks Miguel - So far the .Syncignore file is being ignored. I've placed it in the root of the read-only folder (Windows machine) with Africa for example excluded using your format in the sample file. But sure enough, Africa started to download!

[EnSun]

This is what I have:

### START OF .SyncIgnore
*_Asia_*
*_Centralamerica_*
*_Southamerica_*
*_Africa_*
### END OF .SyncIgnore

It starts to download Afreica and Asia!

[Miguel Domingues]

Note that there are several files for africa.

- Africa_2.obf

- Algeria_africa_2.obf

- Angola_africa_2.obf

- …

In the sync ignore file, the line:

*_africa_*

should ignore all country-specific files, and the line

Africa*

should ignore the continent file.

Try this file (please be carefull with the upper and lower case letters):

### START OF .SyncIgnore

*_asia_*

*_centralamerica_*

*_southamerica_*

*_africa_*

Africa*

### END OF .SyncIgnore

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 25 May 2014 at 19:31:15

[EnSun]

OK. Based on you have in your folder, and if I want to exclude Asia, Centralamerica, Southamerica, Africa and Australia, the correct file should contain:

### START OF .SyncIgnore

*_asia_*

*_centralamerica_*

*_southamerica_*

*_africa_*
*_australia_*
Asia*
Southamerica*
Centralamerica*
Africa*
Australia*

### END OF .SyncIgnore

Correct?

[Miguel Domingues]

Yes, that’s correct.

Is it working now?

Sometimes BTSync takes a while (a minute or two) to refresh the ignore list and take it into account.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 25 May 2014 at 19:53:49

To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

OK. Based on you have in your folder, and if I want to exclude Asia, Centralamerica, Southamerica, Africa and Australia, the correct file should contain:

[EnSun]

Yes, it has been downloading (minus the exclusions) about 1.2 GB in 3 hours with only few peers. Thanks for all the help.

[Miguel Domingues]

Nice to see this working.

The upload speed has also increased today (and during the weekdays) since I’m also using another connection to upload.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 25 May 2014 at 22:45:40

[EnSun]

I might face an issue with syncing between the read-only PC and my phone. The folder on PC has your read-only secret. If I give that secret to my phone I’m afraid it will start syncing with your machine as well (it’s the same secret after all).

[Miguel Domingues]

Yes, that’s correct.

I suggest to turn off the option “use mobile data” in the android device, avoiding the transfer of large amounts of data via your mobile operator.

I don’t see this as bad thing… your android device is synced where ever you are (and have a wifi connection).

Maybe in the future, BTSync android version will have some more options. For instance limit which peers to sync or restrict synching to a specific wifi network (your home network). 

You may also disable “auto-sync” in the android device for that specific folder, and sync the folder manually.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 26 May 2014 at 13:32:43

[EnSun]

Just curious, how "fresher" is your folder comparing to what's on download.osmand.net/list.php ? In other words, do you get your updated maps at the same time they are published on download.osmand.net/list.php or you grab them earlier?

Thanks.

[Miguel Domingues]

My folder is refreshed once a week, based on the files available in the download.osmand.net server.

I do NOT get them earlier, and not as soon as they are published, but at max there is a week diference between my files and the ones on the osmand server.

It’s just like if you were checking the osmand server every week and download the changed files.

Last update was 3 days ago, forced manually by me.

Scheduled updates occur on mondays in the afternoon.

If the number of peers increases I’ll try to update it more often, but for now I think that once a week is enough.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 26 May 2014 at 15:03:29

[EnSun]

I see only two nodes in swarm besides me. If they are syncing with the mobile devices, aren't they suppose to show on the list as well?

[john whelan]

Realistically we need slightly better publicity and more on line sharers and to save the load on the sharers / repeaters we probably should divide the world up so that not every one has to download and upload the entire database every week.

But I think its workable.

Cheerio John

On 26 May 2014 17:25, EnSun <asha...@gmail.com> wrote:

I see only two nodes in swarm besides me. If they are syncing with the mobile devices, aren't they suppose to show on the list as well?

--

You received this message because you are subscribed to the Google Groups "Osmand" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.

[EnSun]

I agree and a bit surprised as well that even with a high view count for this topic there are a very few interested users!

[Miguel Domingues]

@EnSun: yes mobile devices should appear in the list.

@john If we split the files into several secrets/shares we won't be able to set them to download to the same folder. This would make it impossible to sync directly to mobile devices. My intention is for this to be as simple as possible, so I believe this is the way to go. To avoid download undesired/unwanted maps users can use the .SyncIgnore file. 

Definitely we need more people online, having around 10 users always online would make this more viable. Right now I'm uploading from two different locations, with around 600 kb/s in total.

Miguel Domingues

Em 27/05/2014 01:13, "EnSun" <asha...@gmail.com> escreveu:

I agree and a bit surprised as well that even with a high view count for this topic there are a very few interested users!

--
You received this message because you are subscribed to a topic in the Google Groups "Osmand" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osmand/iZ012pkeBLw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to osmand+un...@googlegroups.com.

[john whelan]

I'm up to 27.5 gigs this morning after being connected for 5 days but eventually I'll probably use a .SyncIgnore file to limit myself to this side of the Atlantic ie North and South America but for the moment we've doing fine.  My download pipe is 1.5 Mbps and the upload is around a tenth of that.

Cheerio John

--

You received this message because you are subscribed to the Google Groups "Osmand" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.

[EnSun]

Miguel - I was talking about your mobile and John's unless you use a different method to get the data to your phones. Also in .SyncIgnore file, I had to change *_australia_* to just *australia* since there are  *_australia-* instances in your folder.

[Miguel Domingues]

I’ve synced my tablet last week, and haven’t connected BTSync again but later today I’ll sync it, so it will appear.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 27 May 2014 at 12:37:39

To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

Miguel - I was talking about your mobile and John's unless you use a different method to get the data to your phones. Also in .SyncIgnore file, I had to change *_australia_* to just *australia* since there are  *_australia-* instances in your folder.

[Miguel Domingues]

Let me start by thanking everyone that's contributing to this initiative.
Currently all connected devices are fully synced, and it it took less that a week to transfer all the data between everybody. That's impressive, considering we are talking of almost 40gb.

I hope to see more people joining this, and that those who have tried that continue online. It helps a lot in the dissemination of the maps between all of us. As some of might have noticed, during the week speeds were sometimes close to 1000 KB/S. I'll keep using that connection to speed up the whole process.

Just to bump the BTSync secret, here it is again: BZVMBM27I7ASB4673MZDJYENGLTYXWJLA
Also, remember that if you don't want all the maps, you can use the sync ignore file as described in the previous messages.

Thank you all again.
Miguel Domingues

[EnSun]

Thanks Miguel for your efforts. And, yes the speed has noticeably gone up and hopefully will stay that way going forward which makes the whole experience much more pleasant (not that much fun to download at 0.3KB/S where it was at the beginning of the week).

[EnSun]

Hello Miguel,

On download.osmand.net there is no map dated as 6/1/2014 at this time. And yet, two maps dated as 6/1/2014 were synced this weekend:

Czech-republic_europe_2.obf
Germany_baden-wuerttemberg_

europe_2.obf

What's going on?

[Miguel Domingues]

The listing from http://download.osmand.net/list.php is not updated immediately as maps are uploaded (I don’t know why…).

My script checks the following XML index http://download.osmand.net/indexes.xml , this is updated as soon as maps are available in the server.

Maybe someone from the osmand team may explain this.

If you search the XML file for “Czech-republic_europe_2.obf” you will find this:

<region type="map" containerSize="332948911" contentSize="532986119" timestamp="1401596692000" date="01.06.2014" size="317.5" targetsize="508.3" name="Czech-republic_europe_2.obf.zip" description="Map, Roads, POI, Transport, Address data for Czech-republic europe”/>

And if you check the size of that file on your computer you will see that it has the exact same size as contentSize.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 2 Jun 2014 at 13:04:01

[Ernst Went]

I'll join in with a machine, that's aways online. Europe only, maybe more later ...

Just for the record: the mobile btsync app syncs only files, that are selected in the app. Nothing will be downloaded by default - so there is no need for a SyncIgnore on a mobile device. The files you have selected are synced whenever the btsync app is running.

Ernest

[EnSun]

The mobile app behavior was a bit confusing to me at first. The "selective" downloading can only be accomplished if you have the "Automatic Sync" option unchecked. Otherwise all available maps (by default) will be downloaded unless you have .SyncIgnore specified. In other words, if only certain maps are to be downloaded "automatically" when the sync is running the user needs to check the "Automatic Sync" option and use the .SyncIgnore file, if needed, to exclude the unwanted maps.

[Miguel Domingues]

@Ernst thank for keeping a machine online 24/7, it will help the distribution process.

@EnSun didn’t know that, I have mine with auto-sync enabled and use the .SyncIgnore file

Don’t know what speeds you are getting, but with the number of devices connected has surely increased.

On my end, I’ll try to keep everything up-to-date, and upload new maps as fast as possible.

Just for curiosity, if someone can answer the following questions please do :)

- How fast are you getting new maps?

- Have you reached the fully synched state?

- From 1 (bad) to 10 (excellent), how do you rate this syncing method?

If you have more comments please send them, I’ll be glad to read them.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 5 Jun 2014 at 12:47:25

To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

The mobile app behavior was a bit confusing to me at first. The "selective" downloading can only be accomplished if you have the "Automatic Sync" option unchecked. Otherwise all available maps (by default) will be downloaded unless you have .SyncIgnore specified. In other words, if only certain maps are to be downloaded "automatically" when the sync is running the user needs to check the "Automatic Sync" option and use the .SyncIgnore file, if needed, to exclude the unwanted maps.

--

[john whelan]

My personal view

I'm not sure when the new maps are released but I think as long as the map is no more than a month out of date most real end users would be satisfied.  Print maps were used for years being released every two or so years.  There will always be a few who would like an instant update as soon as something on OSM has been edited.

At the moment BTSync tells me there are 9 devices available, yesterday I was seeing 1700kBs down the quoted maximum is 15Mbps and around 100 kBs up at the same time.  Currently I'm seeing nothing either way.  I think when I start to see a steadier stream of uploads four or five days after the new updates then that will indicate we have a few more "real" end users.

I have cut back to America and oceania so that means I'm down to 27 gigs in the folder and less transatlantic traffic.  Currently I'm on time of day for hydro (electricity) so the machine is more likely to be on at off peak times unless being used for something else.

Cheerio John

--

You received this message because you are subscribed to the Google Groups "Osmand" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.

[EnSun]

For me, the download speed is a way too slow comparing to my uploads. While I'm downloading in less than 100KB/s at best, my upload numbers show all the way to 1 MB/s. Considering the size of these files it takes more than a day to sync about 13 GB. BTSync is a "full" file sync, i.e., once a file is modified the entire file needs to sync, so the download speed, at least for me, is an issue even though not being a major hindrance. I would rate the BTSync, if it's used optimally, to refresh our maps about 8. The two points subtracted form a perfect 10 is due to the large file sizes and the low download speed - not the we can do much about them anyways.

Thanks Miguel.

[Miguel Domingues]

About the download speeds… Unfortunately I cannot use that high-speed connection (with speeds like 2 MB/s) al the time :(

However, at the moment I am not uploading at the max speed. In my interface I have peers who are not synced but I assume that they have the .SyncIgnore file.

Otherwise it’s strange that we are not transferring the maps with each other.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 5 Jun 2014 at 18:37:11

[john whelan]

The download speeds are more a function of how many others are in the group.  My highest download speeds have been when four others have been feeding me, the download speed was the total of the four that were uploading to me at the time and each had its own different rate.  If ever we reach two hundred syncing the download speed is very good.  LibreOffice torrent for example is a very fast download even though each user might only be uploading at 20Kbs.

Cheerio John

--

You received this message because you are subscribed to the Google Groups "Osmand" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.

[Ernst Went]

My download speeds vary between 800kB/s and 1M/s with seven clients actively feeding, when I connect my mobile.

My machine is fully synced (europe only) with around 18GB data. 

btsync is a very good choice for synchronization IMHO. The drawback - as already mentioned - is that it syncs only complete files. If the maps could be split into much smaller parts, then it would not be neccessary to sync data, that has not been changed. 

As OsmAnd searches addresses only in one obf-file (as far as I noticed), this is not feasible.

Or the files have to be put together on the client side (then splitting the obfs into map areas is a prerequisite. I don't know, if that is possible).

I am new to the osm community - don't know, whether map sizes have been discussed already. And as the osm data gets more complete, the sizes will still grow. The Netherlands are one big 1,3 GB lump of data.

Another solution would be the unix command rsync (https://en.wikipedia.org/wiki/Rsync), which is capable of delta encoding (https://en.wikipedia.org/wiki/Delta_encoding), but the handling is of course a bit more complicated. There are even rsync implementations for android as I just found out. But rsync distributes only, as this questions shows http://stackoverflow.com/questions/19929293/distributed-rsync-backup-and-parallel-restore

Nice WE to everybody

Ernst 

[Miguel Domingues]

As far as I know, BTSync also syncs files partially.

The problem is that when a OBF map changes, it size also changes, and the new data is spread across the entire file, so the file almost changes entirely.

It’s like you add a letter in every word in a text, the file is almost completely different.

I believe that splitting the files, and re-joining them on the client side is not an option.

My goal is to have my phone synced without any effort. Splitting files would require an extra step.

Also, it would not avoid the problem of downloading the entire map when a new version is available.

Another option would be distribute the zip files that are a bit smaller, but again it would require an extra step to unzip the files, and the devices connected would have the double amount of data (extracted obfs and zip files)

I’ve not tested but, I assume that the delta encoding of RSync would also see the new file as entire new file, instead of a file with only minor changes.

Once I’ve compared two consecutive versions of a map (over 500MB). Byte to byte comparison gave-me around 90% difference.

Miguel Domingues

---

From: Ernst Went Ernst Went
Reply: osm...@googlegroups.com osm...@googlegroups.com
Date: 7 Jun 2014 at 11:11:32

To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

My download speeds vary between 800kB/s and 1M/s with seven clients actively feeding, when I connect my mobile.

[EnSun]

BTsync syncs the entire file even if a single block of the map file is modified at best. It’s a file-level sync method (as it’s the case with others) since anything else would be block level sync which is a much more complex technology requiring change journaling (keeping track of changed blocks) – something not deployed in this sync software. Even if at the transport level the file “chunks” are used still all chunks need to be downloaded to reconstruct the file. Hence, if only a single byte of that 500MB OBF file is changed the entire 500MB needs to be downloaded for the purpose of syncing.

[Miguel Domingues]

According to this post from a BTSync team member, it transfers only the modified chunks.

http://forum.bittorrent.com/topic/17102-why-not-just-use-rsync/page-2#entry43532

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com
Date: 7 Jun 2014 at 15:54:54

[EnSun]

Thanks for the link, Miguel. So, the OBF file size change (even by one byte) is forcing the entire file to be downloaded by BTsync. From your observation, do you happen to know how often the OBF file sizes are NOT changed once the map updates are posted?

Thanks.

[Miguel Domingues]

I have no idea but from this experience the size always changes...

Miguel Domingues

[Vladimir Kotulskiy]

I'm publishing read-only keys to folders with latest maps : 

- Europe

R7MSXJBL7PDB74LI3A5HTIUQ7QDQE5ITU

I`ve added this folder few day ago and it is an empty still.
 

[Ernst Went]

Hi Vladimir,

this is a very old post. The keys are inactive.

Please read this post from May this year: https://groups.google.com/d/msg/osmand/iZ012pkeBLw/qxAbKXXQx-YJ

Ernst

[Vladimir Kotulskiy]

Why all files have the same suffix _2.obf



четверг, 12 июня 2014 г., 21:28:28 UTC+3 пользователь Ernst Went написал:

[Miguel Domingues]

OBF is the format used by OsmAnd to store its map data.

The “2” is version of that format I belive.

Miguel Domingues

---

From: Vladimir Kotulskiy Vladimir Kotulskiy
Reply: osm...@googlegroups.com osm...@googlegroups.com
Date: 19 Jun 2014 at 13:03:07

To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

Why all files have the same suffix _2.obf

--

[Volodymyr Kotulskyi]

When I download files from OsmAnd directly, all files have the suffix .obf so it is not very comfortable to use OsmAnd synchronization with BTSync at the same time (I installed BT Sync on my PC) because I have to rename files

--
Best wishes.
Vladimir

[Miguel Domingues]

Try this: backup all .obf files (the ones downloaded directly from osmand)

Then with BTSync sync all _2.obf files to the osmand folder (where the obf files were).

This way you will only have _2.obf files but that should be ok, since the old .obf files are also there as _2.obf.

Miguel Domingues

---

From: Volodymyr Kotulskyi Volodymyr Kotulskyi
Reply: Volodymyr Kotulskyi vladimir...@gmail.com
Date: 20 Jun 2014 at 10:34:38
To: Miguel Domingues miguelbraz...@gmail.com
Subject:  Re: BTSync initiative

[EnSun]

Running into an issue - The phone was upgraded today to Android 4.4.2 Kitkat. Now syncing to osmand folder has stopped since Kitkat won't grant permission to btsync to write anywhere on the external card, except "extcard/Android/data/com.bittorrent.sync".

[Miguel Domingues]

The best place to report bugs/problems with the BTSync app is the official forum: http://forum.bittorrent.com/forum/56-sync-general-discussion/

Create a topic there explaining the problem, someone from the dev team will hopefully fix the bug.

Miguel Domingues

---

From: EnSun EnSun
Reply: osm...@googlegroups.com osm...@googlegroups.com

Date: 23 Jun 2014 at 03:09:53

To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

Running into an issue - The phone was upgraded today to Android 4.4.2 Kitkat. Now syncing to osmand folder has stopped since Kitkat won't grant permission to btsync to write anywhere on the external card, except "extcard/Android/data/com.bittorrent.sync".

[EnSun]

This is not a bug. It's the Android 4.4 (Kitkat) requirement. Osmand is not the only app impacted by this. If you want to sync the maps directly to your Kitkat phone/tablet's external SD card, one workaround is to have the osmand folder placed at /storage/extSdCard/Android/data/com.bittorrent.sync.

[Leigh van der Merwe]

Is this initiative still going?  I will add my "considerable" (50-100kbs) bandwidth to the cause.

what is the read-only code?

Palitu

[Miguel Domingues]

Hi,

Yes, it is still going.

The read-only code allows you to receive all the files.

It’s just a “security” measure to prevent badly intended people from deleting the files and uploading garbage to the folder. 

Miguel Domingues

---

From: Leigh van der Merwe Leigh van der Merwe
Reply: osm...@googlegroups.com osm...@googlegroups.com
Date: 29 Jul 2014 at 04:43:52

To: osm...@googlegroups.com osm...@googlegroups.com
Subject:  Re: BTSync initiative

Is this initiative still going?  I will add my "considerable" (50-100kbs) bandwidth to the cause.

--

[Leigh van der Merwe]

I got it up last night, and so far have limited it to certain regions,  but i was getting 250-500kb/s (of a possible 1.1MBs), which is nothing to sneeze at.

May i make a suggestion?  Can you update the maps on a rolling basis, ie on a monday, do 1/7th, on a tuesday, do the next 7th.  This way we are not all download/uploading the 40gb every monday.

I also do not have a problem with syncing my OSMand directory on my 4.4.4 android.

Does anyone know the statistics of how many downloads each map gets in a given period?  should we prioritise different map regions?

[EnSun]

"I also do not have a problem with syncing my OSMand directory on my 4.4.4 android."

Are you syncing to your phone internal SD card or to the External SD card?

[Leigh van der Merwe]

Internal - i have the Nexus 4.

[EnSun]

That's why you have no problem.

[Leigh van der Merwe]

I have create a more exact .SyncIgnore file, which has a number of sections,

1. Continents
2. Large Countries
3. Other
4. Individual Maps

This allows the user to select individual continents to download, and further refine via either countries or individual files.  

There are simple instructions at the top for people that aren't familiar with how they work, and gives much greater control over what to sync.

[Harry van der Wolf]

I sent the main developer a mail about bitlet.
MapFactor uses bitlet inside the app to download maps via bittorrent starting the seed from their own servers (with an "advanced" option, which means well hidden away, to do a direct download). I know that the btsync initiative was started some time ago but that is again an "external" app.
With bitlet you can download from inside OsmAnd to a temporary download map inside the osmand folder "somewhere". You can stop and continue the download to your liking as it is bittorrent, and once downloaded the map is moved to the maps folder.

The bitlet java library can be found on https://github.com/bitletorg/bitlet

I did not get a response yet but it is of course holiday time.

Harry

--

You received this message because you are subscribed to the Google Groups "Osmand" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.

[Rodolfo]

Nice research Harry!
A built-in bittorrent solution would indeed be easier for the user and with a installed base of over a million, statistically there will always be many users online to download their maps. If they all allow uploading, the bandwidth issue would be solved right away.
Of course, users must opt to permit uploading their maps while they are downloading or connected by wifi.
The API examples of bitlet look very straightforward. If it works, this path looks very promising.
Thanks, Rodolfo

[Harry van der Wolf]

Victor also thinks it's promising, but it needs to be investigated first and that and follow up actions will take time of course.

Harry

[Alistair Grant]

On Wednesday, 30 July 2014 10:57:34 UTC+10, Leigh van der Merwe wrote:

I have create a more exact .SyncIgnore file, which has a number of sections,

1. Continents
2. Large Countries
3. Other
4. Individual Maps

This allows the user to select individual continents to download, and further refine via either countries or individual files.  

There are simple instructions at the top for people that aren't familiar with how they work, and gives much greater control over what to sync.

For the record, in BTSync 1.4 (and presumably later) the ignore list has been moved from .SyncIgnore to .sync/IgnoreList (see http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/).

Thanks for making this available!

Cheers,

Alistair

[Miguel D.]

Hi all,

Since BTSync 1.4 was released, instead of sharing through keys, now share links are available.

The folder share link (read only access) for the whole folder is: https://link.getsync.com/#f=obf&sz=46E9&s=PKQU7XDBN27PYWEAHURCLFTYJFINPAV4&i=COX3WFKI6MLRII33XLRXXAJ7AFU47WFD3&p=CBEVORLLMEZBR26XVPZXY4VFGFH4IVMN

Also, I've been able to add a new node with a faster connection (upload speed around 1 MB/s) that is online 24/7.

This should increase dramatically the sync speed when new maps are available.

Best,

Miguel

[john whelan]

If the new files could be released on Friday pm that would help as our local electricity is cheaper at the weekend all day as well as just over night.

Thanks John

--

You received this message because you are subscribed to the Google Groups "Osmand" group.

To unsubscribe from this group and stop receiving emails from it, send an email to osmand+un...@googlegroups.com.

[Miguel Domingues]

I don’t have any control when the files are released!

The only thing I do is download obf maps when released by the OSMAnd tem, and share them via BTSync.

Miguel Domingues

From: john whelan <jwhel...@gmail.com>
Reply: osm...@googlegroups.com <osm...@googlegroups.com>>
Date: 25 Sep 2014 at 23:44:41
To: osm...@googlegroups.com <osm...@googlegroups.com>>
Subject:  Re: BTSync initiative

You received this message because you are subscribed to a topic in the Google Groups "Osmand" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/osmand/iZ012pkeBLw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to osmand+un...@googlegroups.com.

[Max1234Ita]

On Friday, September 26, 2014 12:44:41 AM UTC+2, john whelan wrote:

If the new files could be released on Friday pm that would help as our local electricity is cheaper at the weekend all day as well as just over night.

Thanks John

As far as I know, maps are released at (almost) fixed time intervals of about 10 (ten) days: i.e., for Italy, new releases occur on day 1, 11 and 21 of the month.
I guess this time has been deliberately chosen, so that updates are guaranteed for all maps, as well as some "spare day" is available for routine maintenance on the servers.

Regards,
Max - Italy

[stf]

I stumbled on this thread the other day and it let me know that my ad hoc way of keeping my phone synchronized with the latest maps was lacking. :)

I've now installed BTSync on my laptop and am getting a directory synched with map files for North America (sorry not enough spare room on the machine for the whole set of maps). I've set BTSync to run on startup so my machine should be available for others that need or desire that subset of maps. And I'll be synchronizing a subset of the North America maps with my phone shortly, probably also using BTSync (though I might experiment with Syncthing for that instead).

Thanks for pointing out a nifty solution to always having up to date maps!

[Leigh van der Merwe]

You should post another link that doesn't require approvals, that way anyone can join without you having to manually approve them.

[Miguel Domingues]

That link shouldn't require approvals, at least it was generated that way.

Anyway here is a new one: https://link.getsync.com/#f=obf&sz=46E9&s=PKQU7XDBN27PYWEAHURCLFTYJFINPAV4&i=CPB6QB5APM66FIQQBCRWEZSDZQSHOCAQB&p=CBEVORLLMEZBR26XVPZXY4VFGFH4IVMN

And for those that prefer to add the key directly, the read-only key is: BZVMBM27I7ASB4673MZDJYENGLTYXWJLA

Miguel Domingues

--

[stf]

There are, I think, two issues with this thread. Not issues with your BTSync initiative, but with this thread:

1. It is hard to find the read-only key of BZVMBM27I7ASB4673MZDJYENGLTYXWJLA

2. It is hard to find the great prototype ignore file in, I think, post https://groups.google.com/d/msg/osmand/iZ012pkeBLw/6fen-IIOOSAJ (maybe there was a newer one but that is the one I found just now).

It would be great if there were some easy to find place that those two key pieces of information could be maintained together and maintained if they change in the future.

I don't recall reading this in the thread: The Bit Torrent Sync Android app appears to work the same under the covers as the desktop flavors in that it has a hidden .sync directory with a IgnoreList file in it. So you can set that up the same as you would for the desktop and only sync the files you need on the phone.

I've altered my IgnoreList file on my laptop to accept all your files. While I don't have it on 24x7, I do have it on for most of the day so it might be a help. It appears that both your "NAS" and "NAS Remote" boxes are not directly accessible to me and the download speed to my laptop is quite slow. Most of the other peers I see don't have the relay icon so presumably they could have a faster connection to my laptop when I have it on line.

Question: Have you changed your discovery script from what was described earlier in this thread? During the map update a few days ago it appeared that you discovered and grabbed the files from OsmAnd as they were being generated.

Thank you for your efforts on this!

[Miguel Domingues]

About the relay. NAS shouldn't use the relay server, you should be able to connect directly to that peer. NAS Remote connections are always through the relay server since in that connection I cannot forward the port in the routers. However, this second NAS is uploading sometimes at speeds high as 4 MB/sec. I guess it depends on the relay server load...

Both boxes are checking for updates in the OsmAnd server daily.
I've changed the frequency since NAS Remote was added, and with higher upload speeds peers get synced in less that a day.

Miguel Domingues

[Alistair Grant]

On 8 October 2014 19:31, Miguel Domingues <miguelbraz...@gmail.com> wrote:

About the relay. NAS shouldn't use the relay server, you should be able to connect directly to that peer. NAS Remote connections are always through the relay server since in that connection I cannot forward the port in the routers. However, this second NAS is uploading sometimes at speeds high as 4 MB/sec. I guess it depends on the relay server load...

Both boxes are checking for updates in the OsmAnd server daily.
I've changed the frequency since NAS Remote was added, and with higher upload speeds peers get synced in less that a day.

Miguel Domingues

Em 07/10/2014 14:12, "stf" <s.tod...@gmail.com> escreveu:

...

I've altered my IgnoreList file on my laptop to accept all your files. While I don't have it on 24x7, I do have it on for most of the day so it might be a help. It appears that both your "NAS" and "NAS Remote" boxes are not directly accessible to me and the download speed to my laptop is quite slow. Most of the other peers I see don't have the relay icon so presumably they could have a faster connection to my laptop when I have it on line.

Just as another point of data...  My server is showing a direct connection to NAS (no icon) and the use of a relay server for NAS Remote as Miguel expects.  The other 15 peers are also direct connections.

Thanks also for making this available.

Cheers,

Alistair

[Shorty66]

Are there any news regarding bitlet?
I would love to see that feature in Osmand.


I do not like BTSync very much as its closed source. Instead i do use "syncthing" which is open source and works pretty well.

[stf]

Is there an option on syncthing that I am missing? It looks like all nodes that share data must have keys exchanged and then for each directory that is shared the permissions must be set on what nodes are allowed to participate. This is great for a closed system and I am now using it for mirroring some directories between my computers. In that type of "everyone is known to and trusted by everyone else" environment, syncthing is great and seems to work well.

But it fails to scale properly in the use case that BTSync is filling on distributing OsmAnd maps as everyone would need to exchange keys and set permissions.

Also, the Android syncthing app seems to be really flakey on getting things setup and working. I don't trust it at this point to actually work all the time. The BTSync app for Android feels really solid to me.

For what it is worth, my BTSync peers display for obf now show NAS as having no relay while the "NAS Remote" still does. Maybe I was hallucinating yesterday.

[Shorty66]

No, you are not missing anything. Syncthing is intended for known environment. It does encrypt every single file transfer, though.

I didn't mean to propose syncthing as an alternative for BTSync in the Osmand usecase - i rather wanted to point out that btsync is closed source. The best alternative to btsync seems to be bitlet.

[john whelan]

Bitlet appears to require JAVA which is normally considered a security risk, for example the US Government recently recommended that JAVA should be removed from machines because of the security risk.  JAVA is from Oracle corporation which doesn't take security as seriously as some other companies, JAVA is certainly not open source.  It also appears to depend on URLs which again is suspect as they can be redirected to Malware sites and if you think it doesn't happen just remember more than 30% of the servers in the Internet backbone on a survey done in the US did not have the latest security patches installed and these are the servers that resolve the URLs for you.  The BTSync method can just use secrets or whatever they are called these days, it doesn't need URLs.

Generally speaking BTSync adds little load to the machine but Bitlet running in JAVA running in a browser also most certainly would require more machine cycles and eat up mare resources.  In the JAVA world you trade efficiency for the ability to run in a virtual machine.  ie one that is not optimised for any particular environment.

BTSync does something else that normal Torrent software does not do, it replicates a new version of the files in the folder when one appears in the master read only folder.  Version control is a problem in the torrent world for example LibreOffice is available in a torrent but keeping track of when to take the torrent down and connect to the new version is a manual effort.  The older versions are left floating around and with the map files this making sure the latest version gets out there is the reason for this initiative.

BTSync isn't perfect but its the best I've seen in a long time.

Cheerio John

[Zian Choy]

For a long time, BTSync seemed to keep up with new map data releases but I noticed that recently, it's fallen behind for the following files (and possibly others):

Us_california_northamerica_2.obf (last modified on 12/12/2014 at 8:41 PM)
Us_colorado_northamerica_2.obf (last modified on 12/12/2014 at 8:59 PM)
Us_connecticut_northamerica_2.obf (last modified on 12/12/2014 at 9:03 PM)

Is anyone else experiencing this problem?

I tried downloading the latest Us_california_northamerica file from http://download.osmand.net/list.php and the file is different so I'm guessing that there's been new data since 12/12/2014.

Sincerely,

Zian Choy

On Saturday, June 1, 2013 7:52:44 AM UTC-7, V S wrote:

Hello,

There is a demand to have maps downloaded as soon as they are released. Unfortunately it is hard to satisfy that demand because bandwidth limitation. But we can try to use Torrent to technology to help each other.

For now I would like to present BTSync technology http://labs.bittorrent.com/experiments/sync/get-started.html# and give it a try.

I'm publishing read-only keys to folders with latest maps : 

- Russia

R2NGOYG7TSJL7BT5QB5QFULORCH7TQ43X

- France

RYL64R2O3EKFBXKX3Z4C6NRT374WLMJD7

- GB

RW2VAHTU62NV5W5CLC5YS6XGXWHSEKTIA

- Germany

RAMO45FOHSUHZNCIPEJX4EMDPOW44HOT6

- Europe

R7MSXJBL7PDB74LI3A5HTIUQ7QDQE5ITU

- NorthAmerica

RJ6BUYMK4CDT64G3JZWVKAK2UPNGALXXI

Files in your folders will be automatically updated as soon as they are released. I hoped the server will maintain these connections (port for advanced configuration 5858 but it is optional). 

Drawbacks : 

- It is impossible to select single file from the folder. At least for now ;)

- There is no official client for Android, but they are working on it and there is one unofficial. 

-------------------------------

FYI : 

For now it doesn't give you possibility to update maps on your phone immediately and safe. But you could setup something like a chain of updates BTSync -> Folder -> Dropbox/SkyDrive... (Files selective) -> Sync on Android phone in folder backup.

Once you see everything is ok you activate obf file and it is ready for use. This is my imaginary setup please use at your own risk :

Victor

[john whelan]

I'd like to add Cameroon as I'm doing some HOT mapping there, can someone remind me what to change in the phone.SyncIgnore file.

Thanks John

[Eric Cloninger]

Wanted to jump in and say Thanks for making the Sync keys available. I was looking for Torrents of the OBF files for North America and western Europe for some travels this year. It took me about 30 minutes to get all the set up and figure out which keys to use, but 18 hours later I have all the files I need. Great work! I'll leave Sync running for a while and monitor the bandwidth on my router. I'm sure it doesn't hurt to have another peer. I showed 8 peers last night while I synced and now there are 9. Does that mean I'm now peering?

I'd been using a Nexus 5 and osmand+ was a little small for me given the detail of the OSM maps. I need to learn more about turning off features. I just got a Note 4 running Lollipop and it really pops. Fast redraws and the UI is responsive. I've tried to assemble custom tiles for Garmin handhelds from the OSM sources in the past, but the tools to get all that done were beyond my abilities. I give you guys credit for making it all work down to my device.

[john whelan]

If you're running the sync then you're peering.  I think we get a refresh about three times a month, if bandwidth is a concern you can use the ignore file to restrict which files you're syncing.  I suggest just using the ones you're interested in.

Cheerio John