Indexing documents as soon as they are changed?

[Martin Rybak]

Why are view indexes only updated incrementally on demand, after a view is queried? It seems that the best time to index documents is as soon as they are changed. That way when a view is queried all it has to do is rerun the query, not index all changed documents, which can be slow.

I know this can be done with LiveQueries, but they come with re-querying and memory costs. Basically I am looking for another CBLIndexUpdateMode such as kCBLUpdateIndexContinuous.

I'm wondering if this is technically possible or what that the design decision against it was. Thanks,

Martin

[Jens Alfke]

On Dec 18, 2015, at 8:46 AM, Martin Rybak <martin...@gmail.com> wrote:

Why are view indexes only updated incrementally on demand, after a view is queried? It seems that the best time to index documents is as soon as they are changed.

This is a design decision that originates with CouchDB. I could ask Damien what his reasons were, but I think the answers are that

• In an append-only data store it’s more efficient to batch together multiple updates in one transaction, because each time the store is updated a lot of the internal b-tree nodes get written out.
• Not all views get queried right away, and some are used less than others. By deferring the indexing you don’t pay the cost of having a view until the next time the view is used, which can be a benefit with rarely-used views.

It’s worth revisiting this design, though. I know that Couchbase Server already updates views more eagerly.

That way when a view is queried all it has to do is rerun the query, not index all changed documents, which can be slow.

Well, for the most part you’re just pushing the slowness around. If views are updated when documents are changed, it makes all insertions & updates slower (as in typical relational database.) But I know it can make a difference in the user experience to not have the delay happen all at once when the user is waiting for a response.

—Jens

[Martin Rybak]

Thanks! Should I submit a feature request on Github?

Martin

[Jens Alfke]

On Dec 18, 2015, at 12:43 PM, Martin Rybak <martin...@gmail.com> wrote:

Thanks! Should I submit a feature request on Github?

I thought we already had one, but I guess not. I filed it just now as #1010. (But it’s a cross-platform feature so it will really apply to all 3 of our implementations.)

As a workaround, any time you’d like a view to be updated, you can issue an asynchronous query on that view. For best performance try to minimize the result set; for example you can set the limit to 1, or the endKey to a JSON null (i.e. [NSNull null]).

—Jens

[Martin Rybak]

Thanks. My workaround was to create a category on CBLView that creates and runs an async query with a limit of 1. For those views that I need to be indexed immediately, I invoke this category method in response to a kCBLDatabaseChangedNotification. Does this seem like the right approach?

On Friday, December 18, 2015 at 11:47:53 AM UTC-5, Martin Rybak wrote: