CBLSocketChangeTracker can't handle too many requested documents in named replication

[David Quon]

Our replication has recently changed to a named replication by document ID similar to this.

        self.pullReplication.documentIDs = JSON;

        [self.pullReplication start];

Where JSON is 1356 document ID's.  Once we start the pull replication request we get the following error.

17:54:03.260‖ WARNING: CBLSocketChangeTracker[0x7af42800 database_name_here]: Can't connect, giving up: Error Domain=CBLHTTP Code=414 "414 requested URL too long" UserInfo={LOTS OF DOC ID's HERE NSLocalizedFailureReason=requested URL too long, NSLocalizedDescription=414 requested URL too long}

While the reason for the error is obvious is there anything we can do to configure CBL to handle numerous document ID's or have CBL do the replication above in batches?  If CBL cannot be configured to handle this many named document replication items is our best solution to make a batching system for requesting the named replication or are there any better solutions?

Thanks for the help as always.

[David Quon]

Forgot to mention we're using CBL for iOS version 1.0.4.

[Jens Alfke]

On Apr 14, 2016, at 1:55 PM, David Quon <dav...@amcoonline.net> wrote:

Forgot to mention we're using CBL for iOS version 1.0.4.

Couchbase Lite supports sending the _changes request as a POST, for that reason. What type of server are you using?

—Jens

[David Quon]

CouchDB.  I don't remember the exact version but let me know if that's important.

[Jens Alfke]

On Apr 14, 2016, at 6:25 PM, David Quon <dav...@amcoonline.net> wrote:

CouchDB.  I don't remember the exact version but let me know if that's important.

Ah, that explains it. Older versions of CouchDB didn’t support POST to _changes, so older versions of Couchbase Lite sent it a GET instead. In 1.2 we switched to sending POST to CouchDB. Unfortunately there was a slight protocol incompatibility that made this not work (issue #1139); it’s recently been fixed on the master branch, and there’s also a (unofficial) `release/1.2.2` branch containing the fix.

If you don’t want to upgrade all the way from 1.0.4 to 1.2-plus-unreleased-fixes ;) then you could edit the 1.0.4 sources so that CBLChangeTracker.usePOST is always set to YES in CBLPuller.m. At that point I think you’ll have the same problem as 1.2, so look at the patch that fixes #1139 and make a similar change (it’s minor; basically the _changes params should go into the URL as well as the POST body.)

—Jens

[David Quon]

Thanks for the explanation Jens.  So we tested with `release/1.2.2` (unofficial) and we're noticing that the replication is pulling everything (or it seems) and not just the `documentIDs` requested.  Also the replication does replicate a lot of documents (807 when we only requested 693) before just hanging and replication never finished.  Are these known issues?

We'll investigate the porting alternatives you suggested.  Thanks for your help as always.

[David Quon]

I tried back porting the changes specified here onto the `1.0.4` release but I'm getting the same behavior as specified earlier with `release/1.2.2` (unofficial) in that I'm requesting a finite number of documentIDs in the pull replication and it looks like I'm receiving the whole database.  Additionally on the device the replication still hangs and never finishes.  The device is an iPad 2 is running iOS 9.3.1.

One discrepancy using the iOS 9.3 simulator is that the replication does finish although it is receiving the whole database instead of the named replication.  I'm not sure if I have any devices with iOS 9.3 anymore but I can try to test to see if part of the problem is between iOS versions.

Let me know if you have any other suggestions.  Thanks for your help as always Jens.

[David Quon]

A couple of other notes replicating one fixed document ID using POST and CouchDB 1.6.1 returns all the documents.  If we go back to the default `1.0.4` without the POST the named replication by document ID works as expected.  Also on CBL 1.2.2 with CouchDB 1.6.1 it seems like replication is completed but CBL doesn't notify the replication completion.

[David Quon]

Also here's is our forked branch with the ported changes that was used for testing.

https://github.com/amco/couchbase-lite-ios/tree/release/1.0.4

[Jens Alfke]

On Apr 15, 2016, at 6:02 PM, David Quon <dav...@amcoonline.net> wrote:

Thanks for the explanation Jens.  So we tested with `release/1.2.2` (unofficial) and we're noticing that the replication is pulling everything (or it seems) and not just the `documentIDs` requested.  Also the replication does replicate a lot of documents (807 when we only requested 693) before just hanging and replication never finished.  Are these known issues?

That’s weird because the developer who reported the bug verified that the patch fixed it…

…Oh, I think I see. The initial report wasn’t about replicating a set of documentIDs, just regular replication. There still seems to be an incompatibility with replicating documentIDs: the CouchDB docs show the ‘filter’ property stays in the URL, and only the  ‘docIDs’ parameter goes in the POST body, but we still put ‘filter’ only in the body. (I swear this isn’t our fault; we implemented POST to _changes before CouchDB did, and documented the API. My guess is they implemented it without looking at our docs.)

I’m working on a patch … should have it pushed to that branch in 10 minutes or so. Let me know if it works for you.

—Jens

[David Quon]

Hi Jens.  Looks like we found the same problem as you.  We fixed it on our fork https://github.com/amco/couchbase-lite-ios/commits/release/1.0.4 and everything now works as expected for us.  I'd be happy to verify your fix if it differs from ours.  Would your fix be able to make it into `1.2.2` or future builds?  Thanks for your help as always as it's greatly appreciated.

[Jens Alfke]

Yes, I pushed my fix to release/1.2.2 yesterday.

—Jens 

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/5ae97dab-e728-45a1-ae48-8bb848f0d07f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Quon]

Thanks Jens for your prompt help with this problem as it's helping us ship our beta this week.  I ported your fixes into our forked copy of `1.0.4` (https://github.com/amco/couchbase-lite-ios/tree/release/1.0.4) and everything works as expected.  In case anyone else needs this fix they can refer to the fork or can use this zip (https://github.com/amco/couchbase-lite-ios/raw/release/1.0.4/couchbase-lite-ios-1.0.4.2.zip).  Thanks for your help as always.