Documents Sent Out of Sync from Mobile to CouchDB (CBL Android)

[Rajagopal V]

Hi

We are working on a system that uses CBLite + CouchDB and are testing some offline capabilities. I was thinking that the sequence in which the documents are created in the mobile is the same sequence in which you will get it back when using the _changes feed. But that doesnt seem to happen. We are using CBLite Android 1.0.4.

Here is what happens:

1. Turned off the signal on the mobile 

2. Created a document (which was rev. 1020 below)

3. Created another document (which depends on the prev document) -- rev. 1021 below

4. Go online

5. Figured out that record 1021 reached a custom script that we wrote that listens to the _changes feed before record 1020.

Database detail:

We have a dependent set of documents that get stored like this in the cblite database. (Taken from the revs table)

1020|452|1-3b3330898fc6ffe01715e7087903c1e8||0|0|{"createdByType":"5","order":0,"descriptionText":"Gg1","createdById":"1","shouldProcess":true,"uuid":"3dab22c4-c0e4-4246-beb5-1688ebab8001","type":"Appliancegroups","deleted":false,"clientId":"124","createdondatetime":{"date":"2015-03-18 18:00:23"}}|0

 

1021|453|1-1f41352bcd7f9a84df690fde1515684f||1|0|{"createdByType":"5","order":0,"descriptionText":"Tt1","createdById":"1","shouldProcess":true,"uuid":"0c3a947b-b1b5-4209-b068-cec7ca5eac75","related_uuid":"3dab22c4-c0e4-4246-beb5-1688ebab8001","type":"Appliancetypes","deleted":false,"clientId":"124","createdondatetime":{"date":"2015-03-18 18:00:37"}}|0

 

1022|452|2-a102220e0e21e1ae2244d312e0b45faf|1020|1|0|{"id":30,"createdByType":"5","createdbytype":5,"audit_deletedrecordid":null,"order":30,"createdById":"1","descriptionText":"Gg1","uuid":"3dab22c4-c0e4-4246-beb5-1688ebab8001","type":"Appliancegroups","deleted":0,"createdondatetime":{"date":"2015-03-18 12:30:47.000000","timezone_type":3,"timezone":"UTC"},"clientId":"124","createdbyid":1}|1

Im assuming that 1020 should have stored before 1021 in couchdb, but when doing a _changes feed, this is what I get (Ran curl -XGET http://username:password@localhost:5984/<db>/_changes?since=9156

......

{"seq":9179,"id":"fe26f303-46ff-462b-b915-317cd0aaa144","changes":[{"rev":"10-0cc7628c0add763a816095ab9c45b5cc"}]},

{"seq":9180,"id":"0c3a947b-b1b5-4209-b068-cec7ca5eac75","changes":[{"rev":"1-1f41352bcd7f9a84df690fde1515684f"}]},

{"seq":9182,"id":"3dab22c4-c0e4-4246-beb5-1688ebab8001","changes":[{"rev":"2-a102220e0e21e1ae2244d312e0b45faf"}]},

......

It looks like the document with uuid "0c3a947b-b1b5-4209-b068-cec7ca5eac75" which was revid: 1021 seems to have been sent before "3dab22c4-c0e4-4246-beb5-1688ebab8001" which is rev:1020 (and rev: 1022). 

Is that possible? I have a python script that listens on the _changes feed to do some backend operations (since our actual backend is PHP), so switching the order causes an issue since the records it depends on havent been received yet by the backend system.

Not sure if its related, but I get this when the PusherInternal was initializing:

03-18 18:00:55.771  17261-30037/com.commusoft.v2 E/RemoteRequest﹕ RemoteRequestCompletionBlock throw Exception



    java.lang.NullPointerException

            at com.couchbase.lite.replicator.PusherInternal$4.onCompletion(PusherInternal.java:401)

            at com.couchbase.lite.support.RemoteRequestRetry$1.completed(RemoteRequestRetry.java:178)

            at com.couchbase.lite.support.RemoteRequestRetry$1.onCompletion(RemoteRequestRetry.java:189)

            at com.couchbase.lite.support.RemoteRequest.respondWithResult(RemoteRequest.java:310)

            at com.couchbase.lite.support.RemoteRequest.executeRequest(RemoteRequest.java:260)

            at com.couchbase.lite.support.RemoteRequest.run(RemoteRequest.java:104)

            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:422)

            at java.util.concurrent.FutureTask.run(FutureTask.java:237)

            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:152)

            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:265)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)

            at java.lang.Thread.run(Thread.java:841)

I have been trying to verify this on CBLite-iOS but dont have a device at the moment, but will check as soon as I do. 

Regards

Raja

[Jens Alfke]

On Mar 18, 2015, at 6:31 AM, Rajagopal V <raja...@gmail.com> wrote:

We are working on a system that uses CBLite + CouchDB and are testing some offline capabilities. I was thinking that the sequence in which the documents are created in the mobile is the same sequence in which you will get it back when using the _changes feed. But that doesnt seem to happen.

Nope. There are no guarantees about ordering, sorry. Couchbase Mobile is a fully distributed system and that’s one of the things you have to sacrifice.

(In general: The CBL replicator pushes documents in the order they were last changed, i.e. the least-recently-changed document is pushed first, the latest-changed last. And SG’s _changes feed lists documents in the order that the most recent change was uploaded. But documents are being pushed in parallel HTTP requests on multiple sockets, and those will be processed in SG on multiple threads. If there are errors, docs that failed to upload will be retried later. It gets even crazier if you consider topologies other than stars, where document updates might reach one peer from different sources.)

It looks like the document with uuid "0c3a947b-b1b5-4209-b068-cec7ca5eac75" which was revid: 1021 seems to have been sent before "3dab22c4-c0e4-4246-beb5-1688ebab8001" which is rev:1020 (and rev: 1022). 

Yup, you’re being bitten by the “order in which they were last changed” detail. (FYI, the numbers 1020…1022 here are sequences, not revision ids.) At the time of replication the second document has a current sequence number of 1022, which means it gets sent after the document with a current sequence of 1021.

—Jens

[Rajagopal V]

Thanks for the response Jens.

On Wednesday, March 18, 2015 at 9:01:10 PM UTC+5:30, Jens Alfke wrote:

(In general: The CBL replicator pushes documents in the order they were last changed, i.e. the least-recently-changed document is pushed first, the latest-changed last. And SG’s _changes feed lists documents in the order that the most recent change was uploaded. But documents are being pushed in parallel HTTP requests on multiple sockets, and those will be processed in SG on multiple threads. If there are errors, docs that failed to upload will be retried later. It gets even crazier if you consider topologies other than stars, where document updates might reach one peer from different sources.)

We arent using the SG, but for my understanding, do the sequences go out of order because of the parallel HTTP Requests from CBLite ? Would it be possible to adjust the number of threads in CBLite?

 

To Give a bit more context into our app, there are REST based webservices that are called by a python script based on the documents that come in. So, if i have a Parent record that got created offline and a child record that got created later offline too, a python script(which calls _changes on a continuous mode) assumes that the parent would have been created (or atleast errored in the server) before the child record comes through. In this case,  the REST Api looks like    POST  /api/parent/<parentid>/child and it fails because the parentid could not be located (The python script typically does a GET on the parentid and feeds it into the POST request). Since the changes feed is processed one at a time, it is hard to figure out when the related document comes through. It worked well so far for the online mode since all documents get pushed as soon as they get created but we just hit this problem. Ill have to figure out a way to wait for the related record to get through to CouchDB.

Thanks

Raja

[Jens Alfke]

On Mar 18, 2015, at 9:02 AM, Rajagopal V <raja...@gmail.com> wrote:

We arent using the SG, but for my understanding, do the sequences go out of order because of the parallel HTTP Requests from CBLite ? Would it be possible to adjust the number of threads in CBLite?

Not without reducing performance.

But in the example you gave, the ordering difference is simply because of the way CBL (and SG, and CouchDB) order sequences. If you update doc A, then B, then A again, the result is that A will end up with a greater sequence number than B, so the order in which the docs are pushed, or read from the _changes feed, will be (B, A).

But you can’t rely even on the ordering above. In a more complex scenario with network errors or with multiple connections between peers, the ordering can end up being unpredictable. That’s simply the reality of a highly available, partition-proof system: it sacrifices consistency. (It’s an AP system that lacks C, in terms of the CAP theorem.)

—Jens