CBL 1.4 Background Database & View Indexing

[Scott Ahten]

I'm starting to run into performance issues on a iOS 1.4 CBL app with a very large number of documents (+30k). It seems that indexing is blocking queries and, in turn, blocking specific tasks. My strategy is as follows... 

First, drastically reduce the number of views to reduce the amount of indexing work. Second, try and keep indexes up to date by setting up a live queries, returning only one row, that reference the remaining views. Third, attempt to move indexing work into the background by associating these live queries with a second database connection using a background queue. 

When setting up a second connection, is it necessary to define map/reduce blocks for each database connection that points to the same file? 

Also, what determines where indexing work is performed? For example, if I save a model on the original database connection, will it be indexed on that database's thread? Are external changes processed by the replicator indexed on the CBL / replicator thread in the background?

Thanks, 

- Scott

[wbi...@attic.nl]

On Wednesday, 29 November 2017 18:06:58 UTC+1, Scott Ahten wrote:

I'm starting to run into performance issues on a iOS 1.4 CBL app with a very large number of documents (+30k). It seems that indexing is blocking queries and, in turn, blocking specific tasks. My strategy is as follows... 

Have you tried 1.4.1?

https://github.com/couchbase/couchbase-lite-ios/releases/tag/1.4.1 

with among others: "#1737 View indexing is very slow, due to mis-optimized SQLite query"

(I stumbled on this release myself, it was not announced here - very happy with it)

[Jens Alfke]

> On Nov 29, 2017, at 9:06 AM, Scott Ahten <lightand...@gmail.com> wrote:
>
> I'm starting to run into performance issues on a iOS 1.4 CBL app with a very large number of documents (+30k). It seems that indexing is blocking queries and, in turn, blocking specific tasks.

Hopefully 1.4.1 helps — there’s a SQLite query optimizer bug that we had to work around.

> First, drastically reduce the number of views to reduce the amount of indexing work.

Also, if you have related views (that will be queried at similar times, like by the same controller) give them names starting with a common prefix and a slash, like “detail/byName” and “detail/byDate”. The indexer uses that as a clue that it should index those views at the same time, i.e. if it’s indexing one it will index the other(s) too.

And try to optimize the code in your map block. It gets called many times, so small time savings add up. Don’t pull the same property out of the document dictionary twice; keep it in a local variable. Try to avoid parsing dates. Stuff like that.

It’s also possible to query a view without updating the index, if you’re OK with getting stale data.

> Second, try and keep indexes up to date by setting up a live queries, returning only one row, that reference the remaining views.

You can also directly update the index in the background by calling -[CBLView updateIndexAsync:]. This will run on the background thread and db connection that are used by the replicator.

> When setting up a second connection, is it necessary to define map/reduce blocks for each database connection that points to the same file?

Yes; any CBLDatabases on the same file need to have the same map functions set up.

> Also, what determines where indexing work is performed?

Views are usually indexed lazily, when queried. If you query a view and its index isn’t up-to-date, the indexer will run and scan the revisions that have been added since it was last indexed. But it’s also possible to index a view manually as I showed above.

—Jens

[Scott Ahten]

Thanks Jens. 

I've been able to merge all existing views down to three. 1.4.1 seems to have helped significantly. 

[Scott Ahten]

On Wednesday, December 6, 2017 at 12:30:07 PM UTC-5, Scott Ahten wrote:

Thanks Jens. 

I've been able to merge all existing views down to three. 1.4.1 seems to have helped significantly. 

After merging nearly all of my views, I compared initial sync and indexing times from a previous release running CBL 1.4.1. Syncing the same number of documents (51k) takes over twice as long (11:02 vs 5:56). To rule out my changes while merging views, I only swapped out 1.4.0 with 1.4.1 from our previous release branch. The sync times were virtually the same. 

One thing I noticed is that during replication under 1.4.0 completedChangesCount grows in small increments while changesCount jumps in very large increments. However, on 1.4.1, both completedChangesCount and changesCount grow in small increments, with completedChangesCount growing significantly slower (hence the slower sync time.) 

Was there change in strategy while processing the change feed in 1.4.1?

[Jens Alfke]

On Dec 7, 2017, at 9:53 AM, Scott Ahten <lightand...@gmail.com> wrote:

Was there change in strategy while processing the change feed in 1.4.1?

To be honest, I can’t remember. You can check the release notes, which should include a list of all Github issues that were fixed.

—Jens

Jens Alfke — Mobile Architect — Couchbase, Inc.

[pa...@couchbase.com]

One thing I noticed is that during replication under 1.4.0 completedChangesCount grows in small increments while changesCount jumps in very large increments. However, on 1.4.1, both completedChangesCount and changesCount grow in small increments, with completedChangesCount growing significantly slower (hence the slower sync time.) 

This is caused by the fix to reduce memory usage during the pull replicator. The fix applies the pause logic to gradually load changes instead of loading all at once. It is possible that this could cause the performance degraded. I have created https://github.com/couchbase/couchbase-lite-ios/issues/1980 to track the issue.