Pre-Loading queries and optimization.

[Frederic Yesid Peña Sánchez]

Hi.

We are currently using 1.0.1 release CBL, but we are facing 2 issues:

1. We are attempting a "pre-loading" of views, so the user logs-in, the app syncs all data and query all the default views (may be not to pretty, but allows to improve speed on the entire app).

¿Is there a way to "clean" memory after each query?. We are receaving a lot of memory warnings

2. When we sync the first time, an iPad mini (first gen)  goes up in memory towards 240 MB before the app is being killed by "memory pressure", but this does not happen with iPad Mini Retina or iPad 3rd+ Gen (65MB top).

How could we improve memory handling on these scenarios

[Jens Alfke]

On Aug 19, 2014, at 10:14 AM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

1. We are attempting a "pre-loading" of views, so the user logs-in, the app syncs all data and query all the default views (may be not to pretty, but allows to improve speed on the entire app).
¿Is there a way to "clean" memory after each query?. We are receaving a lot of memory warnings

Sounds like you're issuing queries that return a lot of data. If all you want to do is update the view's index, and you don't need the query results, you can just use a query with limit=1 so that almost no data will be returned.

2. When we sync the first time, an iPad mini (first gen)  goes up in memory towards 240 MB before the app is being killed by "memory pressure", but this does not happen with iPad Mini Retina or iPad 3rd+ Gen (65MB top).

Syncing itself doesn't use a lot of memory. But if you have live queries active, they'll keep triggering over and over during the initial sync as the pull replicator adds documents. And if these queries are returning a large number of rows, memory usage could get high.

It'd be best to limit the number of rows to something reasonable. If you're driving a table view that can only show maybe 20 rows, you don't need to populate it with all 10,000 rows, for example. This does get more complex to code since you have to use paging.

Another option is to not create the live queries on the first launch until the initial sync has completed.

—Jens

[Frederic Yesid Peña Sánchez]

El martes, 19 de agosto de 2014 12:24:32 UTC-5, Jens Alfke escribió:

On Aug 19, 2014, at 10:14 AM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

1. We are attempting a "pre-loading" of views, so the user logs-in, the app syncs all data and query all the default views (may be not to pretty, but allows to improve speed on the entire app).
¿Is there a way to "clean" memory after each query?. We are receaving a lot of memory warnings

Sounds like you're issuing queries that return a lot of data. If all you want to do is update the view's index, and you don't need the query results, you can just use a query with limit=1 so that almost no data will be returned.

I've limited al "preloading" queries to 1, but the memory keeps high and memory pressure kills app

2. When we sync the first time, an iPad mini (first gen)  goes up in memory towards 240 MB before the app is being killed by "memory pressure", but this does not happen with iPad Mini Retina or iPad 3rd+ Gen (65MB top).

Syncing itself doesn't use a lot of memory. But if you have live queries active, they'll keep triggering over and over during the initial sync as the pull replicator adds documents. And if these queries are returning a large number of rows, memory usage could get high.

It'd be best to limit the number of rows to something reasonable. If you're driving a table view that can only show maybe 20 rows, you don't need to populate it with all 10,000 rows, for example. This does get more complex to code since you have to use paging.

Another option is to not create the live queries on the first launch until the initial sync has completed.

I don't enable LiveQueries until initial Sync, (in fact they only go live on each page they are needed) so there is nothing to do with Live queries forcing several updates.

As i told you, the same app on a iPad Mini Retina ran perfect and under 70M memory consumption...

[Frederic Yesid Peña Sánchez]

Does "runAsync" respect "limit=1" ??

[Jens Alfke]

On Aug 19, 2014, at 10:54 AM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

As i told you, the same app on a iPad Mini Retina ran perfect and under 70M memory consumption… 

Same OS version? There's nothing Couchbase Lite would be doing differently on different hardware. (Except for 32-bit vs 64-bit code; is the Mini Retina a 64-bit CPU? But I'd think that 64-bit mode would use more memory, not less.)

You may need to troubleshoot using Instruments to find out what that memory bloat consists of and who's allocating it.

Does "runAsync" respect "limit=1" ??

Yes, it returns the same results as running the query synchronously.

—Jens

[Frederic Yesid Peña Sánchez]

So a good approach would be (On log-in):

1. Sync all data (replicator ON)
2. Run al "preload" queries (replicator OFF)
3. Enable replicator again.

This assuming replications with massive queries (100+) would affect concurrency on SQLite backend.

Also note that the app runs well without connecting to Xcode, but only Retina (mini) will load attached to xcode (and with less footprint -80MB- than mini 1st gen -250MB+)

The "limit=1" didn't made any impact in memory consumption.

[Jens Alfke]

On Aug 20, 2014, at 9:35 AM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

1. Sync all data (replicator ON)
2. Run al "preload" queries (replicator OFF)
3. Enable replicator again.

Sounds reasonable.

This assuming replications with massive queries (100+) would affect concurrency on SQLite backend.

Yes, this should only be necessary if you're fetching very large numbers of documents.

—Jens

[Frederic Yesid Peña Sánchez]

Yes in fact we are handling about 10k+ documents for any user, most of them with shared channels (Cities or Employees for example), other records are assigned by user and channel.

Good news (bad for me, my foul), the memory issue was "zombies" enabled in "run" schema.

If i whish to pause and resume replication, which would be the correct approach?

I've tried "_myPull.continuous=NO;" but the logs shows errors like:

12:34:01.706‖ Replication: CBL_Puller[http://kratos.xserv.com:4984/xserv-mobile2] took 90.197 sec; error=(null)
12:34:02.510‖ Replication: CBL_Pusher[http://kratos.xserv.com:4984/xserv-mobile2] took 90.989 sec; error=(null)
12:34:03.819‖ CBLRemoteJSONRequest[PUT http://kratos.xserv.com:4984/xserv-mobile2/_local/3b9848f6a2f4f3cbb1c5fa62fc5a22bdda788d14]: Got error Error Domain=CBLHTTP Code=404 "404 not_found" UserInfo=0x178696f0 {NSURL=http://kratos.xserv.com:4984/xserv-mobile2/_local/3b9848f6a2f4f3cbb1c5fa62fc5a22bdda788d14, NSLocalizedFailureReason=not_found, NSLocalizedDescription=404 not_found}
12:34:03.819‖ WARNING*** : CBL_Puller[http://kratos.xserv.com:4984/xserv-mobile2]: Unable to save remote checkpoint: Error Domain=CBLHTTP Code=404 "404 not_found" UserInfo=0x178696f0 {NSURL=http://kratos.xserv.com:4984/xserv-mobile2/_local/3b9848f6a2f4f3cbb1c5fa62fc5a22bdda788d14, NSLocalizedFailureReason=not_found, NSLocalizedDescription=404 not_found}
12:34:35.559‖ Replication: CBL_Pusher[http://kratos.xserv.com:4984/xserv-mobile2] took 33.037 sec; error=(null)

So i think i'm doing wrong..

[Jens Alfke]

On Aug 20, 2014, at 10:39 AM, Frederic Yesid Peña Sánchez <freder...@gmail.com> wrote:

Good news (bad for me, my foul), the memory issue was "zombies" enabled in "run" schema.

I had a feeling it'd be something like that :) I've done that kind of thing before too.

If i whish to pause and resume replication, which would be the correct approach?

Call -stop on the CBLReplication, then -start to restart it. Although I don't think stopping the replicator is important, if it's already caught up with what's in the remote db.

I've tried "_myPull.continuous=NO;" but the logs shows errors like: 

Huh. Did you do this while the replicator was already running?

—Jens

[Frederic Yesid Peña Sánchez]

Tests before and after "limit=1":

Before: ~40.4 MB
After: ~39.4 MB

So it's not that noticieable memory saving.. anyway keeping limit on my initialization queries.