Re: [go-nuts] Proposal: Open source Bittorrent Sync clone in Go

[Dario]

Amazing project. I love it!

2013/6/12 Mark Theunissen <mark.th...@gmail.com>

Is anyone interested in collaborating on an open-source clone of the Bittorrent Sync[1] application, or perhaps know of any existing efforts to do this? Go seems like an excellent language choice for this technology.

Probably you already found them but allow me to point some implementations:

More recent: https://github.com/jackpal/Taipei-Torrent

Older:

• https://github.com/royger/wgo
  
• https://github.com/jessta/gobit
  

 

With the recent revelations of PRISM and NSA spying, people will be looking for ways to share data without handing it over to third-parties or central servers. Bittorrent Sync uses the very resilient torrent protocol in a novel way, similar to Dropbox. It allows users to share entire folders both in read-write mode and read-only mode, by distributing keys that must be similar to magnet links in some way. They claim the application provides full privacy. However, since neither the application nor the protocol is open source, and there are no plans for open-sourcing it[2], we can never really be sure.  

We've hacked together a proof-of-concept distributed social network that uses JSON files and Bittorrent Sync as it's backend, called Vole[3]. It's written in Go + Ember.js, works great and is open source. The problem is that it doesn't actually integrate with the sync app, meaning that manual steps are required to use it. But it's a good demo of what may be possible when we turn the server model upside down: a really simple to use network that feels like all the existing web apps but is self-controlled and decentralized. I believe that a distributed network that uses the torrent protocol and only requires "dumb" tracker servers is a better solution than other approaches to this problem.

+1

 

Feedback or ideas welcome, I'll post here again if we decide to pursue an implementation of this.

I hope to be be able to contribute after summer.

PS: my own Vole ID :) HLL5XQRKDN4R5IJAY7BD3XMTUM7OKM3W

--

Dario Castañé

http://www.dario.im | http://twitter.com/im_dario

[Andrew Wilkins]

On Wednesday, 12 June 2013 21:43:05 UTC+8, Mark Theunissen wrote:

Is anyone interested in collaborating on an open-source clone of the Bittorrent Sync[1] application, or perhaps know of any existing efforts to do this? Go seems like an excellent language choice for this technology.

...

Feedback or ideas welcome, I'll post here again if we decide to pursue an implementation of this.

Camlistore [0] seems like it'd be a good fit here, if you're not wedded to Bittorrent Sync. It recently hit 0.1.
FWIW, Camlistore calls out decentralised social networking as a "lofty but persistent goal" of the project.

[0] http://camlistore.org

Cheers,
Andrew

[Ian Davis]

Hi,

This looks really interesting. Have you looked at bitmessage.org much? There is a Go implementation at https://github.com/ThePiachu/GoBit which may be of interest

Ian

On Wed, Jun 12, 2013 at 2:43 PM, Mark Theunissen <mark.th...@gmail.com> wrote:

Is anyone interested in collaborating on an open-source clone of the Bittorrent Sync[1] application, or perhaps know of any existing efforts to do this? Go seems like an excellent language choice for this technology.

With the recent revelations of PRISM and NSA spying, people will be looking for ways to share data without handing it over to third-parties or central servers. Bittorrent Sync uses the very resilient torrent protocol in a novel way, similar to Dropbox. It allows users to share entire folders both in read-write mode and read-only mode, by distributing keys that must be similar to magnet links in some way. They claim the application provides full privacy. However, since neither the application nor the protocol is open source, and there are no plans for open-sourcing it[2], we can never really be sure.  

We've hacked together a proof-of-concept distributed social network that uses JSON files and Bittorrent Sync as it's backend, called Vole[3]. It's written in Go + Ember.js, works great and is open source. The problem is that it doesn't actually integrate with the sync app, meaning that manual steps are required to use it. But it's a good demo of what may be possible when we turn the server model upside down: a really simple to use network that feels like all the existing web apps but is self-controlled and decentralized. I believe that a distributed network that uses the torrent protocol and only requires "dumb" tracker servers is a better solution than other approaches to this problem.

Feedback or ideas welcome, I'll post here again if we decide to pursue an implementation of this.

- Mark

[1]: http://labs.bittorrent.com/experiments/sync.html

[2]: http://forum.bittorrent.com/topic/17782-bittorrent-sync-faq-unofficial/

[3]: http://vole.cc

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Mark Theunissen]

On Wednesday, June 12, 2013 8:55:11 PM UTC-5, Andrew Wilkins wrote:

Camlistore [0] seems like it'd be a good fit here, if you're not wedded to Bittorrent Sync. It recently hit 0.1.
FWIW, Camlistore calls out decentralised social networking as a "lofty but persistent goal" of the project.

[0] http://camlistore.org

This looks like a great project. I think the biggest features we need that aren't implemented in Camlistore are: (1) Windows support and (2) all the tricks torrent clients use like signalling, tracking servers, DHT, peer discovery, and (3) syncronization/sharing semantics aren't quite equivalent. 

Although ultimately an implementation of Bittorrent Sync would have to use a similar hashing protocol I'd imagine.

- Mark

[Mark Theunissen]

On Thursday, June 13, 2013 4:39:51 AM UTC-5, Ian Davis wrote:

This looks really interesting. Have you looked at bitmessage.org much? There is a Go implementation at https://github.com/ThePiachu/GoBit which may be of interest

 I have seen bitmessage, another interesting project with slightly different goals. I think the deal-breakers with bitmessage are that there is a proof-of-work that takes 4 minutes to compute, and that all users receive all messages and attempt to decode them[1]. This doesn't scale well, both in terms of data size and CPU usage for users. It works well at private one-on-one communications though.

The Bittorrent advantage is that you do for social networking what Bittorrent did for file sharing. Distribute the seeding of large datasets, plus allow P2P connections through NAT and firewalls, and make it very hard to prosecute individuals for seeding content, as opposed to the ease at which Twitter can (and does) censor an account.

- Mark

[1]: https://bitmessage.org/bitmessage.pdf 

[hoefnix]

On topic, but not necessary related to Go. http://prism-break.org is a list of FOSS alternatives.

[Guilherme Lino]

Great idea! Also vole! We are heading for the future here! Open source is the way :D

[Guilherme Lino]

http://en.wikipedia.org/wiki/RetroShare
http://en.wikipedia.org/wiki/Private_peer-to-peer
http://en.wikipedia.org/wiki/Freenet
http://en.wikipedia.org/wiki/GNUnet

There are already so much why doesnt these things catch on?

Anyway i'm very interested in collaborating to this project.

[Mark Theunissen]

I think they don't catch on because they look like this: http://retroshare.sourceforge.net/pics/retrosharetransfers.png. People associate desktop apps with word processing and spreadsheets. The web browser is where sharing and networking happens, so it's important to uphold that expectation when trying to change the model.

- Mark

[jum...@gmail.com]

But the alternatives listed here all require self-hosting (or a cloud service), unless I saw it wrong? What OP wants is something akin to btsync, decentralized syncing without requiring a cloud/server. (Unless I understood the whole thing wrong?)

If so, it doesn't look like btsync have released the protocol either? Which is a shame..

Btw, imo btsync really could have been the big thing here, in terms of the purposes of http://prism-break.org (free software and anti-nsa and all that). They even said that don't expect to include any paid features into it (if I recall correctly), so I'm not really sure what their motives are in keeping it all gratis closed. I mean it can't be security through obscurity surely..

Anyways if this does eventuate into something (and of course, I really hope it does), then you'll probably need to consider mobile devices in addition to desktops since one of the main use cases for many would be across various devices involving tablets/smartphones whatnot these days (especially considering the prism leaks). Maybe this means some wrapper html5 web app that can scale across platforms (android, iOS, sailfish os, firefox os etc.)? This will be for later of course, but just putting it out there early just in case.

[Guillermo Estrada]

I would love to collaborate on this project in anyway I can. It is a great idea! Go was made for this kind of things, I'm just finishing an UDP torrent tracker with Go and its so simple and fun to code stuff like that. I'm going to read the info on the drafts to see if I can come up with some fresh ideas.

[Mark Theunissen]

Awesome. It seems like there is some interest in this, any ideas for a name?

Bitfolder?

SyncBit?

...

[Derek Tracy]

There is the obvious Gopher with the tie in to the old gopher protocol and the Go mascot, or GoShare

---------------------------------
Derek Tracy
tra...@gmail.com
---------------------------------

--

[Guilherme Lino]

GoBit
GoByte

Quarta-feira, 12 de Junho de 2013 14:43:05 UTC+1, Mark Theunissen escreveu:

[jum...@gmail.com]

Second GoShare, sounds good.

[Ruben Pollan]

I like the idea of creating an open source bittorrent sync-like program. And
it's great to do the first implementation on Go. But what I see important is to
define a protocol, independently of the language, and give it a name without
'go' in it. I think it should be as bittorrent, where the protocol is
independent of the implementation, and there is actually several implementations
of it.

If you want we can give to the implementation in go a name with 'go' in it.


To add some more dummy name proposals:
* bitshare
* shareme

Quoting jum...@gmail.com (2013-06-20 10:58:03)

--
Rubén Pollán | http://meskio.net/
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
Nos vamos a Croatan.

[jumpwah]

found this.

[Alex]

I see this thread has stalled. Is anyone doing anything with this idea?

[joshua....@gmail.com]

I was wondering this as well (whether anyone was doing anything with this), but I think maybe it is too soon to actually start an alternative. As far as I can tell from browsing various forums and discussions, nobody really knows what protocol btsync uses.  I think the best plan is to make an open source but fully protocol compatible program, but that won't be possible until more details are released by BitTorrent Inc. Otherwise, a lot of work will need to go into developing a new protocol that won't be compatible, hence not leveraging the existing btsync peer network. 

[syzot...@gmail.com]

I'm interested in this project, is this happening? I agree that we should define a protocol independent of the program itself and leave "go" out of the name, though.

We don't know as of yet (unless I'm wrong - if so correct me) if the protocol or btsync program will ever be released open source. Therefore we should define an open protocol and switch later if btsync's protocol is released (and if it's better). It's better to start asap rather than waiting and seeing if btsync will be opened, imo.

Anyone want to start a forum or irc channel for this, so we can collaborate more easily?

Btw I've only just recently started using go, but I have experience with a bunch of other languages so I think I should be fine.

[Mark Theunissen]

A replacement for BTSync is considered a high priority project by the FSF, and people are accumulating ideas here:

http://libreplanet.org/wiki/Group:SyncReplacement

Let us know how you get on. I'm only available to help from next year.

- Mark

[ner...@gmail.com]

Hi folks I'm not a programmer and can't help a lot in coding but have some idea for you, here is a new project that I recently discovered it's called Sharefest (https://www.sharefest.me/) it uses WebRTC protocol it is p2p and acts very similar to torrent but currently it's limited to 500mb probably by project it self and it's opensource! So my first thought was: it's probably the best btsync alternative if somebody can build desktop clients around it! Hope it will help

[Srinath]

I was also thinking of building such a tool. In case you guys are going to start off, let me also know. I would like to contribute. I have written quite a few repos in go. I have implemented sessions store for gorilla web toolkit for mysql, couchbase. I have also written a package to match user agent strings with the ones in WURFL DB and get device capabilities. I hope I can contribute to code.

http://about.me/srinathgs

I write code @ Alamut

Srinath G S

On Wed, Dec 4, 2013 at 11:14 PM, <ner...@gmail.com> wrote:

Hi folks I'm not a programmer and can't help a lot in coding but have some idea for you, here is a new project that I recently discovered it's called Sharefest (https://www.sharefest.me/) it uses WebRTC protocol it is p2p and acts very similar to torrent but currently it's limited to 500mb probably by project it self and it's opensource! So my first thought was: it's probably the best btsync alternative if somebody can build desktop clients around it! Hope it will help

--

[cmor...@gmail.com]

I hope you can find a team to start developing the project.Bittorrent sync is going to be popular.and demand for open source clone increasing.

good luck

[d...@jsonified.com]

On Wednesday, 12 June 2013 15:43:05 UTC+2, Mark Theunissen wrote:

Is anyone interested in collaborating on an open-source clone of the Bittorrent Sync[1] application, or perhaps know of any existing efforts to do this? Go seems like an excellent language choice for this technology.

With the recent revelations of PRISM and NSA spying, people will be looking for ways to share data without handing it over to third-parties or central servers. Bittorrent Sync uses the very resilient torrent protocol in a novel way, similar to Dropbox. It allows users to share entire folders both in read-write mode and read-only mode, by distributing keys that must be similar to magnet links in some way. They claim the application provides full privacy. However, since neither the application nor the protocol is open source, and there are no plans for open-sourcing it[2], we can never really be sure.  

We've hacked together a proof-of-concept distributed social network that uses JSON files and Bittorrent Sync as it's backend, called Vole[3]. It's written in Go + Ember.js, works great and is open source. The problem is that it doesn't actually integrate with the sync app, meaning that manual steps are required to use it. But it's a good demo of what may be possible when we turn the server model upside down: a really simple to use network that feels like all the existing web apps but is self-controlled and decentralized. I believe that a distributed network that uses the torrent protocol and only requires "dumb" tracker servers is a better solution than other approaches to this problem.

Feedback or ideas welcome, I'll post here again if we decide to pursue an implementation of this.

- Mark

Hi Mark,

That is very interesting. My project for 2014 is writing implementations of the PPSP streaming protocol; http://datatracker.ietf.org/doc/draft-ietf-ppsp-peer-protocol/  which has a number of interesting features, including ability to live stream content using a transient signed root hash.

I'm not a golang subscriber but I'd like to keep up to date on vole progress!

A+
Dave

[Jakob Borg]

Hi,

I actually started writing exactly this. At this point, it's a working
proof of concept (as in, it works for me) but is lacking some user
friendliness. I focussed primarily on building a useful, open,
documented protocol and one working implementation of it - if it seems
useful to others, I would hope for other compatible implementations in
other languages for other platforms in the future. There's more
information in the README and protocol spec;

https://github.com/calmh/syncthing

//jb

2013/12/25 <d...@jsonified.com>:

[Starfish]

I want to contribute.

On Wednesday, June 12, 2013 3:43:05 PM UTC+2, Mark Theunissen wrote:

Is anyone interested in collaborating on an open-source clone of the Bittorrent Sync[1] application, or perhaps know of any existing efforts to do this? Go seems like an excellent language choice for this technology.

With the recent revelations of PRISM and NSA spying, people will be looking for ways to share data without handing it over to third-parties or central servers. Bittorrent Sync uses the very resilient torrent protocol in a novel way, similar to Dropbox. It allows users to share entire folders both in read-write mode and read-only mode, by distributing keys that must be similar to magnet links in some way. They claim the application provides full privacy. However, since neither the application nor the protocol is open source, and there are no plans for open-sourcing it[2], we can never really be sure.  

We've hacked together a proof-of-concept distributed social network that uses JSON files and Bittorrent Sync as it's backend, called Vole[3]. It's written in Go + Ember.js, works great and is open source. The problem is that it doesn't actually integrate with the sync app, meaning that manual steps are required to use it. But it's a good demo of what may be possible when we turn the server model upside down: a really simple to use network that feels like all the existing web apps but is self-controlled and decentralized. I believe that a distributed network that uses the torrent protocol and only requires "dumb" tracker servers is a better solution than other approaches to this problem.

Feedback or ideas welcome, I'll post here again if we decide to pursue an implementation of this.

- Mark

[Anthony Wertz]

Yes, this is a very interesting concept. I would be interested in helping.

[Starfish]

I experienced something that made me believe Bittorrent is indeed working with the NSA.

A while back I posted a thread on their forum: http://forum.bittorrent.com/topic/25831-are-you-working-with-the-nsa/

There were many threads in the forum, and most received comments from Bittorrent staff. My thread was among the most read, but no answer.

After a some weeks I went back and gave the thread a bump. A while later the thread was deleted; none of my other threads were.

On Wednesday, June 12, 2013 3:43:05 PM UTC+2, Mark Theunissen wrote:

[jonathan...@gmail.com]

Did this end up going anywhere?

[Brian Ketelsen]

I think it's SyncThing now.