sync_gateway not honoring revs_limit?

[Jeremy Kelley]

Current environment
- ubuntu 14.x and 12.x
- sync_gateway built from master on github about 10 minutes ago

I have some documents with many thousands of revisions... During the initial sync to CBLite on iOS, I'm seeing 2500 documents take sometimes 45 minutes or longer or just never finish at all. Using -LogSync and -LogSyncVerbose (awesome logging btw!), I've found it seems the hang up comes during the insertion phase trying to deal with all of the revisions.

========8<============
2015-04-29 12:29:29.930 lvlrios[4115:1035684] SyncVerbose: CBL_Puller[https://EXAMPLE.com/_sync_] inserting fa4ab2befc59471eaf900c7d25633188 ["2716-22fc2110693bf7545c2d86ebedb3c9e4", "2715-30d236c458218561e63ee2694da2c596", "2714-832ac666baf79fee62f352dd35126506",

ON AND ON AND ON AND ON
========>8============

During some intense googling action, I came across the new db config option of revs_limit in sync_gateway and gave that a whirl.

====8<=======
"databases":{
"lvlrsync":{
"revs_limit":100,
"username": "XXX",
"password": "totally secret d00d",
"server":"http://db.example.com",
...
====>8=======

Unfortunately, I am still seeing the MANY revs coming down the pipe and hanging during the initial sync.

Admittedly, i haven't dug into the syncgw code yet to see if revs_limit only applies to documents going forward or also to existing documents.

Any help appreciated. I have some users that can't sync at the moment.

-j

[Jeremy Kelley]

Just to follow up, I did some digging and realized this pruning with sync_gateway based on revs_limit doesn't take place until after a document update.  So I updated a document.  I am still seeing no change in the number of revs coming down during the intial sync.

-j

[Jens Alfke]

On Apr 29, 2015, at 11:50 AM, Jeremy Kelley <jer...@33ad.org> wrote:

Just to follow up, I did some digging and realized this pruning with sync_gateway based on revs_limit doesn't take place until after a document update.  So I updated a document.  I am still seeing no change in the number of revs coming down during the intial sync.

Does the document have a lot of (resolved or unresolved) conflicts? SG’s current rev-pruning algorithm has the problem that it won’t prune revisions that are recent ancestors of any leaf revision, even a deletion tombstone. That means that if you have a document with a zillion revisions, but there’s a conflict branch at least every revs_limit generations, nothing ever gets pruned.

We just fixed this for SG 1.1, and we have confirmation from a customer that it fixed an issue like yours. You could try building your own SG from the ‘release/1.1.0’ branch and see if it helps you. Or wait a week or two* for the 1.1 developer preview release.

—Jens

* timing not guaranteed

[Jeremy Kelley]

On Wed, Apr 29, 2015 at 5:13 PM Jens Alfke <je...@couchbase.com> wrote:

Does the document have a lot of (resolved or unresolved) conflicts? SG’s current rev-pruning algorithm has the problem that it won’t prune revisions that are recent ancestors of any leaf revision, even a deletion tombstone. That means that if you have a document with a zillion revisions, but there’s a conflict branch at least every revs_limit generations, nothing ever gets pruned.

 

So my question - is there a way I can reconcile conflicts in the sync gateway via API?  I did a quick search of the python api lib and didn't see anything related to conflicts.

If this can only be done on device, that's a bit of plumbing that concerns me.  I'd rather be able to handle these on the server for my users.  For me, conflict resolution is quite simple.  I just take the one with the latest timestamp and that will suffice.

 

We just fixed this for SG 1.1, and we have confirmation from a customer that it fixed an issue like yours. You could try building your own SG from the ‘release/1.1.0’ branch and see if it helps you. Or wait a week or two* for the 1.1 developer preview release.

This branch definitely helped!  I'm running it right now and syncs that were taking 15+ mins are now down to about 90 seconds or less.  Thanks!!

-j

 

[Jens Alfke]

On Apr 29, 2015, at 7:41 PM, Jeremy Kelley <jer...@33ad.org> wrote:

So my question - is there a way I can reconcile conflicts in the sync gateway via API?  I did a quick search of the python api lib and didn't see anything related to conflicts.

Not sure what Python library you’re talking about (a CouchDB client?), but the SG REST API lets you do can do anything a client can. Resolving conflicts is just creating new revisions using PUT and DELETE.

—Jens

[Jeremy Kelley]

http://developer.couchbase.com/mobile/develop/guides/couchbase-lite/native-api/document/index.html

The "resolving conflicts" section here is what I'm talking about.

-j

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/A78D629E-14D0-43EC-A1D6-89DF978972A6%40couchbase.com.
For more options, visit https://groups.google.com/d/optout.

[Jens Alfke]

On Apr 30, 2015, at 4:49 AM, Jeremy Kelley <jer...@33ad.org> wrote:

http://developer.couchbase.com/mobile/develop/guides/couchbase-lite/native-api/document/index.html

The "resolving conflicts" section here is what I'm talking about. 

Resolving a conflict simply consists of deleting the conflicting revisions you don't want, and either keeping the remaining one or replacing it with a new revision containing the merged data. So in the REST API it's just some DELETE and PUT calls. The CouchDB documentation describes how to do it (all of this should be applicable to SG.)

—Jens