ANN: P2P mesh replication demo app for iOS

[Jens Alfke]

The P2P talk I co-presented at Couchbase Connect gave me the chance to finally sit down and write a P2P mesh replication demo for iOS. This is something people often ask about here, and I wave my hands and go “blah blah Listener blah blah Bonjour”, but it’s nontrivial to actually implement.

I built it as an extension of Grocery Sync, on a “bonjour-sync” branch, but the P2P code is modular enough that it can be dropped into other iOS or Mac apps. 

The user model is that you pop open a browser that lists the other devices on the LAN running the app, and can choose people to follow (i.e. pull from). That setting is saved persistently, and from then on any time one of those devices is online and has new changes, your app will pull the changes across to you.

As part of building this I found some limitations in Couchbase Lite; for example, SSL cert hostname validation makes it infeasible to use P2P SSL connections. I filed issues and I’ve started putting in some fixes (although not in time for version 1.1.)

My code is for iOS or Mac OS, but in the session I described the techniques in a cross-platform way. And Android supports Bonjour (under a different non-trademarked name) so the platforms should be able to interoperate.

—Jens

PS: The talk also featured Pasin’s photo-sharing app, which he’s announced previously  It’s also very cool, and shows off a completely different technique: using QR code generation & scanning to let two devices securely ‘pair’ and connect to each other.

[Brendan Duddridge]

Ooh... that's fantastic!

Is this robust enough for lots and lots of data syncing? That is, if I import 1000 records while offline, then plug into the mesh, will all the devices in the mesh get all the changes? And if I had made changes on the other devices will that also get merged into the new device that just joined the mesh?

Thanks!

Brendan

[Jens Alfke]

On Jun 13, 2015, at 2:09 PM, Brendan Duddridge <bren...@gmail.com> wrote:

Is this robust enough for lots and lots of data syncing? That is, if I import 1000 records while offline, then plug into the mesh, will all the devices in the mesh get all the changes? And if I had made changes on the other devices will that also get merged into the new device that just joined the mesh?

Yes, it has correct behavior. The underlying replication architecture (inherited from CouchDB) is multi-master and supports arbitrary topologies. It does the right thing even if there are redundant connections and/or devices leave and rejoin.

This kind of mesh isn’t super scalable, though. I go into that a bit during the talk. Everyone syncs with everyone else, which leads to O(n^2) traffic on the LAN. The implementation optimizes that a bit by not re-advertising changes that arrive from peers, so any document update should lead to only O(n) replications.

A more scalable approach would probably use something like a dynamic spanning tree, which creates the minimal number of connections that connect all the peers, but those are more difficult to implement.

—Jens

[Brendan Duddridge]

Hi Jens,

I watched the talk after I posted the question and you had a perfect example of what I would like to use P2P for. And that's the example of using my app in the field and then syncing with other devices when no Internet connection is available. I've heard from a bunch of customers who use may app to gather data in the field during their archeological excavations and one common thing they all have is there's no Internet connection where they are, so they have to use my backup and restore function and use iTunes File Sharing to transfer the backup from their iOS device to their Mac via a USB cable. Cloud sync is impossible where they are. So I'm glad that this should be able to work well for them with a limited number of devices in the mesh.

I've posted an issue on github that I was having trying to sync 3 devices (iPhone 6, iPhone 6+, and Simulator) where it wasn't always syncing properly between devices and they could very easily get out of sync until you performed a restart. You may have already seen my post.

Thanks!

Brendan

[Jens Alfke]

On Jun 13, 2015, at 5:43 PM, Brendan Duddridge <bren...@gmail.com> wrote:

I've posted an issue on github that I was having trying to sync 3 devices (iPhone 6, iPhone 6+, and Simulator) where it wasn't always syncing properly between devices and they could very easily get out of sync until you performed a restart. You may have already seen my post.

What’s the issue number/URL?

—Jens

[Yonah Forst]

Great talk Jens!! I'm about halfway through now and it's been SUPER informative. 

I'm at the part when you talk about putting the last sequence number in the bonjour metadta and initiating a pull from a peer when that number changes. Is that method significantly better than just establishing a continuous pull replication? (using didFindService/didRemoveService to start/remove active replications).

[Yonah Forst]

Another question: you mentioned replicating over infrastructure wifi as well as bt. does 'includesPeerToPeer' automatically enable both? I'm able to replicate over btle, but I don't know if there's anything special I need to do to enable infrastructure wifi. (infrastructure wifi means no router in the middle, right?) 

On Saturday, June 13, 2015 at 8:15:34 PM UTC+2, Jens Alfke wrote:

[Brendan Duddridge]

Hi Jens,

The issue was posted in the Grocery repo:

https://github.com/couchbaselabs/Grocery-Sync-iOS/issues/13

Thanks!

Brendan

[Victor Fernandez]

Hi Jens

When the listener have started a bonjour service must be published right? Or a NSNetServiceAdvertiser must be initialized by separate?  

[Jens Alfke]

On Jun 24, 2015, at 2:56 PM, Victor Fernandez <vfern...@txmglobal.com> wrote:

When the listener have started a bonjour service must be published right?

If you set the listener’s Bonjour name/type it’ll be published for you.

—Jens