Offline/online syncing and conflicting revisions

[jd04]

Hi,

We're in the middle of the development of a tool using Couchbase Server/Sync/Lite (Android/Phonegap), for the classic example of multiple devices replicating through the Sync Gateway. At the moment we're attempting to manage some online/offline revisions of documents and conflicts, but I've hit a wall.

Here's the use case (both Devices on Continuous replication through the Sync to the Server)

• Device A/B online
• Device A creates document 1
• Device A looses network connection
• Documents are replicated to Device B from Sync
• Device B makes new revision of document 1
• Device A, while offline, makes a new revision to document 1
• Device A regains network connection

What I am seeing is the following:

• Server shows three revisions to document 1 - the initial document created by Device A, and the revision created by Device B, and the revision created by A after regaining network (with revision by B "winning")
• Device A and B now have two documents (i.e. document 1) with the same ID. A GET on the document while on the devices returns the same as when on the Server.
• From the devices...

sqlite> select * from docs where docid like '74%';
select * from docs where docid like '74%';
doc_id  docid
109     74fc4797-6f93-4fd3-b9af-3701784f3596


sqlite> select * from revs where doc_id = 109;
select * from revs where doc_id = 109;
sequence        doc_id  revid   parent  current deleted json
224     109     1-17adec98-d34e-458a-b975-59526bd6c986          0       0       {"test":"123"}
225     109     2-74af32aa-769c-4a6d-b8c2-06a425aafb55  224     1       0       {"test":"999"}
226     109     2-a11653df-6049-4eb0-a00f-9b5abc2111c8  224     1       0       {"test":"707070"}


sqlite> select * from maps;
select * from maps;
view_id sequence        key     value
1       225     74fc4797-6f93-4fd3-b9af-3701784f3596   {"test":"999"}
1       226     74fc4797-6f93-4fd3-b9af-3701784f3596   {"test":"707070"}

Map function

function(doc) {
    emit(doc._id, {
      test: doc.test
    });
)

Changes feed:

{"seq":"public:365","id":"74fc4797-6f93-4fd3-b9af-3701784f3596","changes":[{"rev":"2-74af32aa-769c-4a6d-b8c2-06a425aafb55"}]} ,{"seq":"public:366","id":"74fc4797-6f93-4fd3-b9af-3701784f3596","changes":[{"rev":"2-a11653df-6049-4eb0-a00f-9b5abc2111c8"}]}

If I run that view on the CB Server through the UI - I am just returned the single document, but on the devices, it returns both.

Can anyone explain to me what is happening here? I have the feeling I may be missing or confusing something fundamental with either the Sync Gateway, the Changes feed sequences perhaps, or even the View map function.

Would I be correct in saying that this is might be expected functionality? And that I need a Map function on the devices to get these conflicts and let the device user determine how to resolve?

(Lastly, I'd like to thank the Couchbase guys involved with the development of the product and support provided through here, both are really excellent. )

Thanks.

[Jens Alfke]

On Jan 7, 2014, at 10:01 AM, jd04 <jorden...@gmail.com> wrote:

• Server shows three revisions to document 1 - the initial document created by Device A, and the revision created by Device B, and the revision created by A after regaining network (with revision by B "winning")
• Device A and B now have two documents (i.e. document 1) with the same ID. A GET on the document while on the devices returns the same as when on the Server.

Yup, this is the expected behavior with doc conflicts (although it’ll be indeterminate which of the conflicting revisions “wins”.)

Changes feed:

{"seq":"public:365","id":"74fc4797-6f93-4fd3-b9af-3701784f3596","changes":[{"rev":"2-74af32aa-769c-4a6d-b8c2-06a425aafb55"}]} ,{"seq":"public:366","id":"74fc4797-6f93-4fd3-b9af-3701784f3596","changes":[{"rev":"2-a11653df-6049-4eb0-a00f-9b5abc2111c8"}]}

By default, /db/_changes will show only the winning revisions. If you want to see conflicts in the feed (as the replicator does) add “?conflicts=true” to the URL. The “changes” array for the conflicted doc will then show both revision IDs.

Can anyone explain to me what is happening here? I have the feeling I may be missing or confusing something fundamental with either the Sync Gateway, the Changes feed sequences perhaps, or even the View map function.

This is the way conflict handling works in Couchbase Lite, as in CouchDB. Resolving conflicts is not automatic; it’s left up to the application. The Conflict Management chapter of the CouchDB book might be helpful. In a nutshell: A document actually has a revision tree, not just a linear history. If conflicting histories are merged together during replication, the history branches. The leaf nodes of the tree (except for deletion markers) are the conflicting versions, and the “winner” is the deepest one (with ASCII comparison of the revision ID to break ties.) This implies that to resolve a conflict you should delete the revision you don’t want, and either leave the correct one alone or add a new revision to it (containing the result of a merge, probably.)

In the new native Android API (as on iOS), CBLDocument and CBLQuery both have some methods to detect conflicts, and you’d use CBLRevision to do the necessary deleting/updating.

(We need an example/tutorial of this, but don’t have one yet.)

—Jens

[Jens Alfke]

There’s also a Conflict Resolution section in our Couchbase Lite Concepts documentation, and a more detailed look in the iOS guide. (But in the latter, ignore the advice to create a view that looks for a “_conflicts” property; that’s obsolete.)

—Jens

[Sören Havemester]

Hello!

Can somebody give me a hint on how to delete a revision - I guess by some REST command - but I can't get any REST working via the TouchDBListener framework as indicated in the couchbase lite wiki.

Thanks a lot in advance!

Søren

[Jens Alfke]

On Jan 7, 2014, at 1:32 PM, Sören Havemester <res...@gmail.com> wrote:

> Can somebody give me a hint on how to delete a revision - I guess by some REST command - but I can't get any REST working via the TouchDBListener framework as indicated in the couchbase lite wiki.

What platform/API are you using?

In the beta 2 API you call -[CBLDocument deleteDocument:] (iOS) or Document.delete (Android).

—Jens

PS: Also, please don’t add an unrelated reply to an existing thread. Start a new thread instead.

[Sören Havemester]

Hi Jens,

I am actually looking for a solution on deleting one or more conflicting revisions of a document with online/offline replication conflicts. Those conflicts happen sometime in my App (MacOS) and it's easy to find them. But I've no clue how to delete those revisions. Could you give me a hint.

Cheers,

Sören 

[Jens Alfke]

On Jan 7, 2014, at 4:45 PM, Sören Havemester <res...@gmail.com> wrote:

I am actually looking for a solution on deleting one or more conflicting revisions of a document with online/offline replication conflicts. Those conflicts happen sometime in my App (MacOS) and it's easy to find them. But I've no clue how to delete those revisions. Could you give me a hint.

-[CBLSavedRevision deleteDocument:] —

/** Deletes the document by creating a new deletion-marker revision. */

- (CBLSavedRevision*) deleteDocument: (NSError**)outError;

—Jens

[jd04]

Thanks for the info Jens, much appreciated.

The explanations and docs make a little more sense to me now.

One issue I'm having is that checking the changes feed with "conflicts=true", on the sync gateway, doesn't give me all revisions in the changes array. It still shows the duplicated docs like below. Is there anything I could be missing here?

URL: server:4984/db/_changes?conflicts=true

On the device, the changes feed has just the winning revision.

{"seq":"public:447","id":"1581fe27-87cd-436c-b950-902a4bc34302","changes":[{"rev":"6-e2a8a32e-ab77-401e-9d3d-c2e9f09d2375"}]},{"seq":"public:448","id":"1581fe27-87cd-436c-b950-902a4bc34302","changes":[{"rev":"6-5871e0e5-eec9-4b65-be7d-8ad6163d1750"}]}

The approach we'll take on the devices is something like the following:

• On a list of documents; use a map function (emit(doc._id, 1)) to get all. then a reduce function (_sum) to only display one row for each doc to the user. 
• If they attempt to update/delete, alert them to the fact there is a a conflict, ask them to make a decision on resolution, and prevent any more changes to the doc until they do so.

Would you think this is a valid approach?

Thanks.

[jd04]

And just a further note to the above.

"In the new native Android API (as on iOS), CBLDocument and CBLQuery both have some methods to detect conflicts, and you’d use CBLRevision to do the necessary deleting/updating.

We're going with Phonegap, so I guess all will done through REST calls.

[Jens Alfke]

On Jan 10, 2014, at 4:46 AM, jd04 <jorden...@gmail.com> wrote:

One issue I'm having is that checking the changes feed with "conflicts=true", on the sync gateway, doesn't give me all revisions in the changes array. It still shows the duplicated docs like below. Is there anything I could be missing here?

URL: server:4984/db/_changes?conflicts=true

On the device, the changes feed has just the winning revision.

{"seq":"public:447","id":"1581fe27-87cd-436c-b950-902a4bc34302","changes":[{"rev":"6-e2a8a32e-ab77-401e-9d3d-c2e9f09d2375"}]},{"seq":"public:448","id":"1581fe27-87cd-436c-b950-902a4bc34302","changes":[{"rev":"6-5871e0e5-eec9-4b65-be7d-8ad6163d1750"}]}

Oh! I hadn’t noticed before that these are two entries with the same doc ID. That’s … not good. They should appear as a single entry with two elements in the “changes” array. This way will confuse the Couchbase Lite replicator and it won’t pull both conflicting revisions. So this appears to be a bug in the gateway. :(

Any idea how this came about? Is it easy to reproduce?

—Jens

[jd04]

On Friday, January 10, 2014 5:51:08 PM UTC, Jens Alfke wrote:

Oh! I hadn’t noticed before that these are two entries with the same doc ID. That’s … not good. They should appear as a single entry with two elements in the “changes” array. This way will confuse the Couchbase Lite replicator and it won’t pull both conflicting revisions. So this appears to be a bug in the gateway. :(

Any idea how this came about? Is it easy to reproduce?

—Jens

Hi Jens,

The steps to reproduce the two entries with the same ID in the Sync feed are the steps I outlined previously:

• Device A/B online
• Device A creates document 1
• Device A looses network connection
• Documents are replicated to Device B from Sync
• Device B makes new revision of document 1
• Device A, while offline, makes a new revision to document 1
• Device A regains network connection

I guess this is a pretty standard use case, so if there's anything else you need me to provide, please let me know. 

[Jens Alfke]

I looked at this further, and while the way the gateway is sending the revisions isn’t entirely correct according to the CouchDB API, it would be difficult to change it to send both conflicting revision IDs in one entry. And Couchbase Lite should be able to handle this form correctly — I’ve been looking through the code to figure out why it would lose one of the revisions.

Could you file an issue on Github, please, especially including the steps to reproduce? Thanks!

—Jens

[vi...@violetpath.in]

Can anyone plz tell me how to know total number of conflicted documents in database via REST ? thanks in advance 

[Jens Alfke]

On Mar 7, 2016, at 1:25 AM, vi...@violetpath.in wrote:

Can anyone plz tell me how to know total number of conflicted documents in database via REST ? thanks in advance 

Query _all_docs?only_conflicts=true and check the number of results.

—Jens

PS: Please start a new thread for new questions; don’t reply to an unrelated old thread.