Slow replication with Cloudant

[Jordan Wood]

I have some databases with quite a lot of small documents.  The one I'm using to test my replication from Cloudant has 37,311 documents with a total of 74,581 revisions.   Pulling the entire database takes over 7 minutes.  Does this seem unusually slow?  I'm seeing many messages like the following in the sync log:

22:35:21.432‖ Sync: CBL_Puller[https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239] inserted 89 revs in 0.031 sec (2857.8/sec)

22:35:21.432‖ Sync: CBL_Puller[https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239] Progress: 185 / 303

22:35:21.433‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 313, err: (null)

22:35:21.433‖ Sync: CBL_Puller[https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239] queued 10 remote revisions from seq=13934-g1AAAAOreJy10j0OgjAUB_BGjXoLHZwJhUJhkptoPzBAEBKCs95E7-GgMXHyBjqoJ8HSYogbDl1ek5eX90vevykAYBz1OZhxyvIiDDi1jDKKCx4XeWaa0GBpvuEkK40sLFMx3SOATqqqSqI-BWB4W4veSAyGyMEd1_zn0amodN6SF0kSbHkmg3rIoCYXLXmVJISQYOjpIZc1uW3JoyRD10HYQlrIbCAq2IlHqPuGvUuWEYwJ5xrZg2JPDftQmTILQVvPgRV7Vuzz5_eufF_ESjSyL8V-s31L1mbI5L7bcVvyAQ8yLGM (0 in bulk, 10 individually)

22:35:21.434‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 323, err: (null)

22:35:21.434‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 333, err: (null)

22:35:21.434‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 343, err: (null)

22:35:21.434‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 353, err: (null)

22:35:21.435‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 364, err: (null)

22:35:21.435‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 374, err: (null)

22:35:21.436‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 384, err: (null)

22:35:21.436‖ Sync: CBLReplication[from https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239]: active, progress = 185 / 394, err: (null)

22:35:21.935‖ Sync: CBL_Puller[https://couchClient:*****@couchdev.browzine.com:443/user-7fa8f310-c473-11e5-92f1-17c9cca90239] queued 81 remote revisions from seq=13944-g1AAAAOreJy10UsOgjAQBuBGjXoLXbgmFAqFldxE-8AAQUgIrvUmehOMiSs9gvFxEiwthi0uuplJJpP5kvlTAMA0GnKw4JTlRRhwahllFBc8LvLMNKHB0nzHSVYaWVimYntAAJ3VdZ1EQwrA-L4Vs4lYDJGDe575z6NzUemyIy-SJNjyTAb1kEFDrjryKkkIIcHQ00OuG3LfkZUkQ9dB2EJayGwkKjiIJtRjyz4kywjGhHON7EmxVcs-VabMQtDW82DFnhX7atmbZDe-L2IlGtm3Yn_ZfiRrM2Ry3-15LfkCIX0sbQ (0 in bulk, 81 individually)

Is the "0 in bulk, 81 individually" at all relevant?  Is there anything I can do to tune this process?

Thanks for the input!

Jordan 

[Jens Alfke]

On Feb 1, 2016, at 8:43 PM, Jordan Wood <wood.p...@gmail.com> wrote:

Is the "0 in bulk, 81 individually" at all relevant?  Is there anything I can do to tune this process?

The Couchbase Lite replication protocol doesn’t provide a way to pull documents in bulk, unfortunately. (There is _all_docs, but it doesn’t include the revision history of the docs, so it’s not useful for replication purposes.) This slows down replication, since every doc has to be pulled using a separate GET request.

(I’m not sure why this problem wasn’t recognized in CouchDB. My best guess is that it doesn’t show up if you have tons of net bandwidth and CPU, as most servers tend to, so it’s not a painful problem in CouchDB deployments. It is for mobile, though.)

We invented a nonstandard _bulk_get operation for Sync Gateway that solves this problem. Couchbase Lite will use it, if it sees (via the Server: response header) that it’s talking to Sync Gateway. It’s been documented for a while, but I don’t know if CouchDB or Cloudant have implemented it. If they have, we could modify the compatibility check in CBL.

—Jens

[Mike Rhodes]

The _bulk_get end point should be in CouchDB 2.0, and hopefully rolled out to Cloudant soon via the merge of https://github.com/apache/couchdb-chttpd/pull/33 a while back. I'm not 100% certain this is compatible with the sync-gateway version, but I _think_ it is.

I can file bugs once we roll it out into production with the CBL libraries if you want Jens?

Mike.

[Will Holley]

CouchDB 2.0 has a _bulk_get endpoint (see https://issues.apache.org/jira/browse/COUCHDB-2310) that was merged from rcouch. I don't know whether anybody has tested it for CBL compatibility but it works with PouchDB and Cloudant Sync. Cloudant has this deployed for internal testing and it will likely land on public clusters within weeks. If you need a Cloudant account moved to a cluster that supports it now (for testing only), let me know privately and I'll see what I can do.

Will

On Tuesday, 2 February 2016 06:32:15 UTC, Jens Alfke wrote:

[Jens Alfke]

On Feb 2, 2016, at 6:33 AM, Mike Rhodes <mike....@gmail.com> wrote:

The _bulk_get end point should be in CouchDB 2.0, and hopefully rolled out to Cloudant soon via the merge of https://github.com/apache/couchdb-chttpd/pull/33 a while back. I'm not 100% certain this is compatible with the sync-gateway version, but I _think_ it is.

Oh, cool! We should update CBL to use it, then. It would be best to rip out the rather crude Server: header check and go with something more extensible. The best approach for the replicator might be EAFP — always try _bulk_get first, then if that fails with a 404, fall back to individual GETs.

—Jens