P2P sync using phonegap

[Bart Arribe]

Hi everybody,

I would like to sync my devices in p2p using the phonegap plugin on Android devices. I shouldn't have more than 10 devices and the LAN should be stable.

1. I'm thinking to make one device as a master and, if possible, take the role of the sync gateway. 

What do you think about that ?

2. For the moment, I'm not using any discovery service but instead, just setting the IP addresses manually in the source code. If I'm using the architecture with one master device as a star, do my devices need to know only the IP address of my master device? Or the master device need as well the IP address of the other remote devices in order to replicate any change on the database?

3. I tried a really simple example doing a p2p replication. It's working quite well when I provide both IP addresses. When I provide the IP address of my device A to my device B, the device A can make any change and the data is replicated well but it's not working in the other way, so from the device B to A. 

I'm using this function to make the replication with the same URL, the device's A one to both devices (probably why it's not working...)

function triggerSync(cb, retryCount) {

if (!config.user) {

       return log("no user")

   }

 

console.log("window.config.site.syncUrl " + window.config.site.syncUrl);

var remote = {

url : window.config.site.syncUrl

},

push = {

source : appDbName,

target : remote,

continuous : true

}, pull = {

target : appDbName,

source : remote,

continuous : true

};

pushSync = syncManager(window.config.site.server, push);

pullSync = syncManager(window.config.site.server, pull);

...

}

The logs, when it is not working : 

04-16 11:31:25.023: I/CBLSQL(749): Acme.Utils.ThreadPool(0)-PooledThread: Acme.Serve.Serve$ServeConnection@4221c580 Begin transaction (level 1)

04-16 11:31:25.023: I/CBLSQL(749): Acme.Utils.ThreadPool(0)-PooledThread: Acme.Serve.Serve$ServeConnection@4221c580 Begin transaction (level 2)

04-16 11:31:25.033: I/CBLSQL(749): Acme.Utils.ThreadPool(0)-PooledThread: Acme.Serve.Serve$ServeConnection@4221c580 CANCEL transaction (level 2)

04-16 11:31:25.033: W/System.err(749): com.couchbase.lite.CouchbaseLiteException: No existing revision found with doc id: 4237ce67-922b-46a9-9d5d-54f0c6504cc5

04-16 11:31:25.033: W/System.err(749): at com.couchbase.lite.Database.putRevision(Database.java:2981)

04-16 11:31:25.033: W/System.err(749): at com.couchbase.lite.Database.putRevision(Database.java:2916)

04-16 11:31:25.043: W/System.err(749): at com.couchbase.lite.router.Router.update(Router.java:1453)

04-16 11:31:25.043: W/System.err(749): at com.couchbase.lite.router.Router.do_POST_Document_bulk_docs(Router.java:904)

04-16 11:31:25.043: W/System.err(749): at java.lang.reflect.Method.invokeNative(Native Method)

04-16 11:31:25.043: W/System.err(749): at java.lang.reflect.Method.invoke(Method.java:515)

04-16 11:31:25.043: W/System.err(749): at com.couchbase.lite.router.Router.start(Router.java:443)

04-16 11:31:25.043: W/System.err(749): at com.couchbase.lite.listener.LiteServlet.service(LiteServlet.java:102)

04-16 11:31:25.043: W/System.err(749): at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

04-16 11:31:25.043: W/System.err(749): at Acme.Serve.Serve$ServeConnection.runServlet(Serve.java:2251)

04-16 11:31:25.053: W/System.err(749): at Acme.Serve.Serve$ServeConnection.parseRequest(Serve.java:2176)

04-16 11:31:25.053: W/System.err(749): at Acme.Serve.Serve$ServeConnection.run(Serve.java:1988)

04-16 11:31:25.053: W/System.err(749): at Acme.Utils$ThreadPool$PooledThread.run(Utils.java:1223)

04-16 11:31:25.053: W/System.err(749): at java.lang.Thread.run(Thread.java:841)

04-16 11:31:25.053: E/Database(749): com.couchbase.lite.CouchbaseLiteException: No existing revision found with doc id: 4237ce67-922b-46a9-9d5d-54f0c6504cc5

3. Any clues why is acting like that and how to do it properly?

3. Using phonegap, if i want to discover the other peers, how would you do that? 

I was thinking to make a plugin doing that in native language on Android and iOS, using for instance Bonjour service.

Thank you in advance for your help,

Bart

[Jens Alfke]

On Apr 16, 2014, at 6:14 AM, Bart Arribe <bart...@gmail.com> wrote:

1. I'm thinking to make one device as a master and, if possible, take the role of the sync gateway. 

Sure, you’ll just need the Listener enabled on that device. (On iOS that’s in a separate framework; not sure about Android.)

Note that this doesn’t have the channel or account features of the real Sync Gateway.

If I'm using the architecture with one master device as a star, do my devices need to know only the IP address of my master device? Or the master device need as well the IP address of the other remote devices in order to replicate any change on the database?

Only the master’s IP address needs to be known; the master doesn’t need to know who the other clients are. The other devices can run push and pull replications with it.

3. Any clues why is acting like that and how to do it properly?

I don’t know enough about CBL on Android or JavaScript to be able to answer. Hopefully Traun and/or JChris can help.

3. Using phonegap, if i want to discover the other peers, how would you do that? 

I was thinking to make a plugin doing that in native language on Android and iOS, using for instance Bonjour service.

Yes, I think you’ll need native code. On iOS this is easily done using Bonjour (NSNetService); in fact the CBLListener already supports advertising itself as a service, so you just need some code to browse services on the client devices. I don’t know what the Android equivalent is.

—Jens

[J. Chris Anderson]

On Wednesday, April 16, 2014 6:14:14 AM UTC-7, Bart Arribe wrote:

3. I tried a really simple example doing a p2p replication. It's working quite well when I provide both IP addresses. When I provide the IP address of my device A to my device B, the device A can make any change and the data is replicated well but it's not working in the other way, so from the device B to A. 

... 

3. Any clues why is acting like that and how to do it properly?

This looks like an Android issue, but it could be with your LAN -- sometimes reachability can be tricky. Jens is correct that the only IP address that needs to be published is the hub in the hub and spoke arrangement.

 

3. Using phonegap, if i want to discover the other peers, how would you do that? 

I was thinking to make a plugin doing that in native language on Android and iOS, using for instance Bonjour service.

For PhoneGap you probably want to use a plugin like https://github.com/jarnoh/cordova-dnssd or https://github.com/rossmartin/BonjourPlugin

[Traun Leyden]

This looks like an Android issue, but it could be with your LAN -- sometimes reachability can be tricky. Jens is correct that the only IP address that needs to be published is the hub in the hub and spoke arrangement.

I agree, this does look like an CBL Android issue.

I filed a ticket here:

https://github.com/couchbase/couchbase-lite-java-core/issues/161

@Bart can you mention which version of CBL you are using on the ticket?

[Bart Arribe]

Thank you for your quick answers. I am going to try the different plugins you mentioned to me, JChris, in order to discover peers.

For the ticket, I've just done it.

Bart

[Bart Arribe]

Hi,

I tried to implement a device as a master and the other one as a "client" that would connect to the master.

I give the IP address of the master device to the client. But now, I can only replicate from the master device to the client, impossible to make a POST request to the master device according to the logs : 

05-02 14:10:50.611: V/Database(23718): *** com.couchbase.lite.replicator.Replication$1@42138898: BEGIN processInbox (1 sequences)

05-02 14:10:50.611: V/Database(23718): *** com.couchbase.lite.replicator.Replication$1@42138898: END processInbox (lastSequence=null

05-02 14:10:50.861: I/Database(23718): com.couchbase.lite.replicator.Pusher$2@422074c8: Sending 1 revisions

05-02 14:10:50.861: V/Database(23718): com.couchbase.lite.replicator.Pusher$2@422074c8: Sending [{e8bb1d57-60b3-4cfd-b35a-812917c268e4 #3-185f2150-5903-46ec-b388-ec2194de94ca}]

05-02 14:10:50.941: E/Database(23718): Got error 404

05-02 14:10:50.941: E/Database(23718): Request was for: org.apache.http.client.methods.HttpPost@42129288

05-02 14:10:50.941: E/Database(23718): Status reason: Not found

I don't have any idea about where comes from the problem and don't know where to look for. :s

Please, let me know if you have any clues to solve this problem.

Thanks

Bart

[Bart Arribe]

Hi,

Do you know if there is the equivalent for Android device? Or how would you do it?

Thanks

Bart

[J. Chris Anderson]

On Friday, May 2, 2014 6:42:31 AM UTC-7, Bart Arribe wrote:

For PhoneGap you probably want to use a plugin like https://github.com/jarnoh/cordova-dnssd or https://github.com/rossmartin/BonjourPlugin

Do you know if there is the equivalent for Android device? Or how would you do it?

Maybe this looks promising: https://github.com/vstirbu/ZeroConf 

[Bart Arribe]

Thanks for the quick answer J.Chris. I'm gonna take a look on that soon :)

Le vendredi 2 mai 2014 14:40:30 UTC+1, Bart Arribe a écrit :

Hi,

I tried to implement a device as a master and the other one as a "client" that would connect to the master.

I give the IP address of the master device to the client. But now, I can only replicate from the master device to the client, impossible to make a POST request to the master device according to the logs : 

05-02 14:10:50.611: V/Database(23718): *** com.couchbase.lite.replicator.Replication$1@42138898: BEGIN processInbox (1 sequences)

05-02 14:10:50.611: V/Database(23718): *** com.couchbase.lite.replicator.Replication$1@42138898: END processInbox (lastSequence=null

05-02 14:10:50.861: I/Database(23718): com.couchbase.lite.replicator.Pusher$2@422074c8: Sending 1 revisions

05-02 14:10:50.861: V/Database(23718): com.couchbase.lite.replicator.Pusher$2@422074c8: Sending [{e8bb1d57-60b3-4cfd-b35a-812917c268e4 #3-185f2150-5903-46ec-b388-ec2194de94ca}]

05-02 14:10:50.941: E/Database(23718): Got error 404

05-02 14:10:50.941: E/Database(23718): Request was for: org.apache.http.client.methods.HttpPost@42129288

05-02 14:10:50.941: E/Database(23718): Status reason: Not found

It seems the error comes from the master device, not finding any existing revision ... 

05-02 16:04:25.156: I/CBLSQL(4672): Acme.Utils.ThreadPool(5)-PooledThread: Acme.Serve.Serve$ServeConnection@412c2618 Begin transaction (level 1)

05-02 16:04:25.156: I/CBLSQL(4672): Acme.Utils.ThreadPool(5)-PooledThread: Acme.Serve.Serve$ServeConnection@412c2618 Begin transaction (level 2)

05-02 16:04:25.156: I/CBLSQL(4672): Acme.Utils.ThreadPool(5)-PooledThread: Acme.Serve.Serve$ServeConnection@412c2618 CANCEL transaction (level 2)

05-02 16:04:25.164: W/System.err(4672): com.couchbase.lite.CouchbaseLiteException: No existing revision found with doc id: ece849e1-03fa-4ca3-bae6-2710bc2a2fe2

05-02 16:04:25.164: W/System.err(4672): at com.couchbase.lite.Database.putRevision(Database.java:2981)

05-02 16:04:25.164: W/System.err(4672): at com.couchbase.lite.Database.putRevision(Database.java:2916)

05-02 16:04:25.164: W/System.err(4672): at com.couchbase.lite.router.Router.update(Router.java:1453)

05-02 16:04:25.164: W/System.err(4672): at com.couchbase.lite.router.Router.do_POST_Document_bulk_docs(Router.java:904)

05-02 16:04:25.164: W/System.err(4672): at java.lang.reflect.Method.invokeNative(Native Method)

05-02 16:04:25.172: W/System.err(4672): at java.lang.reflect.Method.invoke(Method.java:511)

05-02 16:04:25.172: W/System.err(4672): at com.couchbase.lite.router.Router.start(Router.java:443)

05-02 16:04:25.172: W/System.err(4672): at com.couchbase.lite.listener.LiteServlet.service(LiteServlet.java:102)

05-02 16:04:25.172: W/System.err(4672): at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)

05-02 16:04:25.172: W/System.err(4672): at Acme.Serve.Serve$ServeConnection.runServlet(Serve.java:2251)

05-02 16:04:25.172: W/System.err(4672): at Acme.Serve.Serve$ServeConnection.parseRequest(Serve.java:2176)

05-02 16:04:25.172: W/System.err(4672): at Acme.Serve.Serve$ServeConnection.run(Serve.java:1988)

05-02 16:04:25.180: W/System.err(4672): at Acme.Utils$ThreadPool$PooledThread.run(Utils.java:1223)

05-02 16:04:25.180: W/System.err(4672): at java.lang.Thread.run(Thread.java:856)

05-02 16:04:25.180: E/Database(4672): com.couchbase.lite.CouchbaseLiteException: No existing revision found with doc id: ece849e1-03fa-4ca3-bae6-2710bc2a2fe2

05-02 16:04:25.180: I/CBLSQL(4672): Acme.Utils.ThreadPool(5)-PooledThread: Acme.Serve.Serve$ServeConnection@412c2618 CANCEL transaction (level 1)

[Yaron Goland]

This question is primarily targeted at Java Couchbase Lite.

So today, in theory anyway, I should be able to set up two separate listeners on the same box (Android or otherwise) and have both listeners talk to the same database file without problem. SQLite certainly supports this behavior.

But I've heard Jens mentioning that Couchbase Lite might be changing their underlying database.

So I wanted to check how committed is Couchbase to continuing this behavior?

The reason I care is that it's looking very likely that I will need to support two different listeners running on the same database files. One listener would be listening on an Internet routable address and would have all sorts of fun security. The other listener would be listening on localhost and would use different security. In theory I could set up two different endpoints in TJWS but believe me, that isn't fun. It would be vastly simpler to just set up two different listeners who may both hit the same cblite file.

But when Jens new database comes in will that still work?

        Thanks,

                Yaron

[Jens Alfke]

On May 2, 2014, at 8:20 AM, Yaron Goland <yar...@microsoft.com> wrote:

So today, in theory anyway, I should be able to set up two separate listeners on the same box (Android or otherwise) and have both listeners talk to the same database file without problem. SQLite certainly supports this behavior.

This won’t quite work. The Document class wants to know when changes are made to the database so it can update its internal state like currentRevision. Internally, when multiple threads are using the same database file, they post notifications so one Database instance can discover that another one has changed documents. (This happens during replication, for example.) But those notifications are within the process, so if two processes have the same database open they won’t notify each other.

This wouldn’t be that hard to fix (assuming that Android has interprocess notifications?) but it’s just never come up as something that needed to be done. It’s a non-issue on iOS due to app sandboxing.

But I've heard Jens mentioning that Couchbase Lite might be changing their underlying database.
So I wanted to check how committed is Couchbase to continuing this behavior?

ForestDB fundamentally has good concurrency support. (Better than SQLite’s, in that a writer doesn’t block readers.) But I think at the moment it doesn’t have any way to arbitrate multiple processes trying to write to the database at the same time; they won’t corrupt the database, it’s just that the last one to commit would probably overwrite the other’s changes. I’ll ask the team if that’s so, and if they plan to add some kind of file-locking to guard against that. (If not, it’s something CBL could probably add on its own.)

—Jens

[Yaron Goland]

And that's why I send mails to the group. :) 

Normally I wouldn't care about the change listeners because I don't ever use them (we ONLY talk over HTTP so nobody gets to use the manager object directly) but I can see this being a problem for continuous replications which do need the change listeners.

I suspect the only sane approach is to take advantage of TJWS's dual listener support. It is able to listen to more than one port at once. Although I'll probably have to write a custom adapter to get it to do what I want. But that isn't brain surgery.

    Thanks,

                Yaron

---

From: mobile-c...@googlegroups.com <mobile-c...@googlegroups.com> on behalf of Jens Alfke <je...@couchbase.com>
Sent: Friday, May 02, 2014 8:36 AM
To: mobile-c...@googlegroups.com
Cc: Wayne Carter; Traun Leyden
Subject: Re: Does Couchbase lite want to support multiple listeners sharing the same underlying database file?

 

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/68E7AE8C-C607-4A42-A1B3-38D4F7F1D2E7%40couchbase.com.
For more options, visit https://groups.google.com/d/optout.

[Traun Leyden]

Yeah, that approach seems like it will help you keep a grip on your sanity.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/1399047661645.80137%40microsoft.com.

[Yaron Goland]

You say that now.... :)

Wait until I seriously start the thread on getting rid of TJWS. See https://github.com/NanoHttpd/nanohttpd/issues/63 for a taste. :)

        Yaron

---

From: mobile-c...@googlegroups.com <mobile-c...@googlegroups.com> on behalf of Traun Leyden <tle...@couchbase.com>
Sent: Friday, May 02, 2014 11:26 AM

To: mobile-c...@googlegroups.com
Cc: Wayne Carter

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/CACSSHCEqe1%2BizaZ-%3DqhVP%3Dns_jnWATx5s%3D6HtduRzyjOXTOVWw%40mail.gmail.com.