Filtered replication

[dyowee]

Good day Jens,

We have this requirement to limit data being replicated, so we tried to do a filtered replicated by marking old documents with archived=true. But then we encountered the problem that CBL cannot sync anymore, saying 'WARNING Truncated changes feed', so we had to completely restore the CouchDB databases.

Is there a better way to do filtered replication? Probably by date ? Or would that make replication much slower? Is there a better way to limit data being replicated to the device without deleting records from the server?

Regards,

Joey

[Jens Alfke]

On Sep 14, 2014, at 11:57 AM, dyowee <csharpen...@gmail.com> wrote:

We have this requirement to limit data being replicated, so we tried to do a filtered replicated by marking old documents with archived=true. But then we encountered the problem that CBL cannot sync anymore, saying 'WARNING Truncated changes feed'

Well, that sounds like a bug somewhere, not a limitation of filtered replication.

That warning occurs if the JSON parser hits a syntax error reading the received _changes feed. It almost always means that the socket got closed unexpectedly. And that in turn usually means an intermediate gateway decided the socket's been open too long. (Some telco gateways will do this, and apparently also AWS load balancers.) The 'heartbeat' option on the _changes request is supposed to prevent this by telling the server to periodically send a no-op CRLF so the socket doesn't look idle, though.

In any case, this isn't something that's related to filtering, so it's weird that it started happening after you added that. Do you have any more clues?

we had to completely restore the CouchDB databases

What does that mean?

Is there a better way to do filtered replication?

Again, there's nothing wrong with what you're doing. It just seems to have triggered some kind of bug, either in CBL or CouchDB. Let's try to diagnose that.

—Jens

[dyowee]

Thank you very much.
I create backup of the CouchDB databases before I run a tool that marks old documents with the 'archived' flag, just in case an issue like this comes up. Unfortunately it did. I was hoping to use the same approach, but I'm not sure how I can replicate the bug.

[dyowee]

Also, we host our own CouchDB server and use nginx as a reverse proxy. We already did the same purge approach before with TouchDB, but it is hard to tell if the issue is with CBL or CouchDB.

On Sunday, September 14, 2014 1:06:21 PM UTC-7, Jens Alfke wrote:

[dyowee]

The nginx log says something like this:

[error] 4048#3452: *4439215 upstream timed out (10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond) while reading upstream, request: "GET /mydatabase/_changes?feed=normal&heartbeat=300000&style=all_docs&since=358688&filter=replddoc/myfilter HTTP/1.1", upstream: "http://xxx.x.x.x/mydatabase/_changes?feed=normal&heartbeat=300000&style=all_docs&since=358688&filter=replddoc/myfilter"

On Sunday, September 14, 2014 1:06:21 PM UTC-7, Jens Alfke wrote:

[Zack Gramana]

Jose,

Are you using the Xam component or the NuGet package?

If the component, do you mind trying the NuGet package instead? It has a more recent build in it.

Cheers,

Zack

Sent from my iPhone

--
You received this message because you are subscribed to the Google Groups "Couchbase Mobile" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/7c13098e-42d2-40fa-91e7-457c1f1622db%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Traun Leyden]

Zack,

I think you accidentally hijacked this thread .. this doesn't seem relevant.  You might want to make sure this message makes it to it's original target thread.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/79DB5050-A967-43C0-9F37-A562C4277C0C%40couchbase.com.

[dyowee]

Seems that changing the proxy_read_timeout fixed the issue :) Thanks!

On Sunday, September 14, 2014 1:06:21 PM UTC-7, Jens Alfke wrote:

[Zack Gramana]

There’s a known issue in the Couchbase Lite for .NET Beta 4 Xamarin component release that generates the exact same error message. I fixed it and updated the release within hours, but wanted to make sure this wasn’t compounding the problem due to a bad lib version.

To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/CACSSHCFJEyFDGCrMJiQW%2B4DRvL0i8TWps7QTDRvTk34PXSd-JA%40mail.gmail.com.