Server Filters for Pull Replication

[James Norman]

Is it possible to define a Filter on the sync_gateway, then specify the Filter in the PullReplication?  I see some references to this in the code and documentation, but can't find how to implement it or any examples.  In the java code for Replication.java the filterName docs say:

>> Name of an optional filter function to run on the source server. Only documents for which the function returns true are replicated. For a pull replication, the name looks like "designdocname/filtername".

However I can't find how to implement a filter, only the full Sync Function.

Thanks for any advice -james

[Jens Alfke]

On Oct 30, 2014, at 12:50 PM, James Norman <james....@gmail.com> wrote:

Is it possible to define a Filter on the sync_gateway, then specify the Filter in the PullReplication?

CouchDB supports server-side filters, but the Sync Gateway uses channels instead, which are more efficient.

—Jens

[James Norman]

Thanks, good to know.  I was looking for a workaround for the slow performance issue with large databases.  The idea was to have multiple databases on the client, then specify a filter for the "type" of document in the pull replication, which would reduce the number of documents in the database.  Rather than having 1 large database with all the documents, there would be 5 or databases that have a specific type of document, which would help with the performance of the updates and deletes as they are much faster on smaller databases.

I'm not sure how I could achieve this using channels unless I had multiple users.  Maybe I'll spend some time looking into the update/delete performance issue then there would be no need for a workaround.

-james

[Jens Alfke]

On Oct 30, 2014, at 1:26 PM, James Norman <james....@gmail.com> wrote:

Rather than having 1 large database with all the documents, there would be 5 or databases that have a specific type of document, which would help with the performance of the updates and deletes as they are much faster on smaller databases.

How many documents are there in all? And what platform/device is this? Performance shouldn't degrade that much with size; if it does, we should fix it.

—Jens

[James Norman]

This is the issue I was looking to workaround: https://github.com/couchbase/couchbase-lite-android/issues/444

The performance to update or delete documents degrades quickly with larger databases.  Here is the time to delete 200 documents at different database sizes:

DB with 10000 documents, 14 seconds to delete 200 documents

DB with 50000 documents, 81 seconds to delete 200 documents

DB with 70000 documents, 114 seconds to delete 200 documents

This is without replication running as well.  Our data will generally be 40-70K documents on the tablet, and we have a few use cases where we need to delete or update 200 documents at a time, so I was looking to balance that over multiple databases.

-james

[Jens Alfke]

On Oct 30, 2014, at 3:16 PM, James Norman <james....@gmail.com> wrote:

This is the issue I was looking to workaround: https://github.com/couchbase/couchbase-lite-android/issues/444

From the discussion it sounds like the cause is known, and it can be fixed by updating one of the SQLite indexes. I'd say it's better to wait a few days for a fix than to rework your app's data model to work around it.

—Jens