How to turn on auto indexing to occur at specific interval of time from phongegap plugin ?

[Vinay Bhinde]

I use couchbase lite phonegap plugin in iOS and android hybrid app.

I have a view as follows which i initiate from my JavaScript code as follows 

function setupViews(db, cb) {
   //DESIGN DOCUMENT FOR ACCOUNT
   var design = "_design/account";
   db.put(design, {
     views: {
       ByType: {
         map: function(doc) {
           if (doc.type && doc.type == "account") {
             emit(doc.cat, doc);
           }
         }.toString()
       }
},function() {
     window.config.accountViews = db([design, "_view"]);
     generateCatDesginDoc(db, cb);
     //cb(false, db([design, "_view"]));
   });
}

I have read that it recommended that one should not emit whole document but in this specific use case i need to emit the whole document. Now i have about 100 documents added to couchbase and when i query this view for the very first time it goes and indexes it and takes about 5~6 seconds to complete and give me a result. This is a show stopper for me as there are two such view query's where the views are beign indexed for the first time and my app takes awfully long to complete the process(10 ~12 secs). I can't expect users to wait for such a long time.

In a normal couchbase server we can change the settings of couchbase server and instruct it to auto update the view after each specific time interval on its own regardless of the query. We do a post request as below and change the setting

http://192.168.1.100:8091/settings/viewUpdateDaemon

I also read that in native API there is some kind of parameter named "indexUpdateMode" you can use while querying a view which can be used to trigger index update asynchronously in background. What i want is that my view indexes should be automatically updated after specified time interval rather then being updated when queried. I can't find a way to do via REST API for cblite. 

Any help would be really helpful.

[Jens Alfke]

On Feb 4, 2015, at 5:50 AM, Vinay Bhinde <vin...@gmail.com> wrote:

I have read that it recommended that one should not emit whole document but in this specific use case i need to emit the whole document.

You really shouldn't emit the whole document. Why do you need to in this case?

Now i have about 100 documents added to couchbase and when i query this view for the very first time it goes and indexes it and takes about 5~6 seconds to complete and give me a result.

That's a very long time; it should not take that log. (I think I said that before.) Which platform does this occur on, and how big are your documents? Also, what version of CBL?

I'm guessing that your documents are pretty big, so emitting the entire document is slow.

I also read that in native API there is some kind of parameter named "indexUpdateMode" you can use while querying a view which can be used to trigger index update asynchronously in background. What i want is that my view indexes should be automatically updated after specified time interval rather then being updated when queried. I can't find a way to do via REST API for cblite. 

The CouchDB REST API for views has a "stale" parameter for this, but I just looked at CBL code and we haven't implemented it. We should, though; I've filed an issue.

—Jens

[Vinay Bhinde]

On Thursday, 5 February 2015 02:24:02 UTC+5:30, Jens Alfke wrote:

You really shouldn't emit the whole document. Why do you need to in this case?

Below is my sample document from 100 documents that i have in db

 

account: "Charity"
cat: "Expense"
channels: "vinay"
createdAt: "Thu Feb 05 2015 00:15:21 GMT-0500 (EST)"
initialBal: "0"
initialBalDate: "Thu Feb 05 2015 00:15:21 GMT-0500 (EST)"
note: ""
subCat: "Activities"
type: "account"

Now from from above 9 properties i need about 7 properties in my app from each document. So rather then emitting 7 different properties i emit the doc itself. 

Another way is that i emit : null from the view and then in a separate GET request get the document by passing the id i recieve in view query. But in that case when i need to get all these 100 docs i will have to first query my view which gives me 100 document id's and then i need to fire 100 ajax GET requests to get the complete doc itself. Hence the use of doc in emit of view. 

That's a very long time; it should not take that log. (I think I said that before.) Which platform does this occur on, and how big are your documents? Also, what version of CBL?

I profiled the app by watching the android logcat logs to find what happens when i query the view for the first time. All it shows is that there is re-indexing of views going on and as there are 100 docs it goes and re-indexes them and that process takes about 5~6 secs. The platform is Android (4.4.4). Also i use this on iOS too and time taken on iOS is little bit less then android.

I am using cblite phonegap plugin version 1.0.2 which i pulled from github master branch. (Also somebody previosuly said that the plugin version on master branch is actually 1.0.3 but at least the config files and cordova CLI show me 1.0.2)

 

I'm guessing that your documents are pretty big, so emitting the entire document is slow.

You can see my sample doc. I am guessing 9 properties document shouldn't be that too big. 

The CouchDB REST API for views has a "stale" parameter for this, but I just looked at CBL code and we haven't implemented it. We should, though; I've filed an issue.

—Jens

Thanks for filing the issue. What i exactly need right now is my specific view index being auto updated regardless of they are queried or not so that when i query them the index is always updated and the response time is quick.

[Jens Alfke]

On Feb 4, 2015, at 9:30 PM, Vinay Bhinde <vin...@gmail.com> wrote:

But in that case when i need to get all these 100 docs i will have to first query my view which gives me 100 document id's and then i need to fire 100 ajax GET requests to get the complete doc itself. Hence the use of doc in emit of view. 

You can add "?include_docs=true" to the query request and it will include the doc body in each row.

I profiled the app by watching the android logcat logs to find what happens when i query the view for the first time. All it shows is that there is re-indexing of views going on and as there are 100 docs it goes and re-indexes them and that process takes about 5~6 secs. The platform is Android (4.4.4). Also i use this on iOS too and time taken on iOS is little bit less then android.

I don't understand why it's so slow. I'm actually doing benchmarking and optimization on iOS right now, so I have numbers handy. CBL 1.0.3 on my iPhone 6 indexes about 1500 documents per second. (These are docs taken from my iTunes library metadata, they have about as many properties as yours do.) The speed you're quoting is about two orders of magnitude slower!

The only difference is Objective-C vs JavaScript for the map function itself, and JS will definitely be slower but it shouldn't be that much slower. I also don't think other PhoneGap developers have run into this in their apps.

—Jens

[Vinay Bhinde]

I don't understand why it's so slow. I'm actually doing benchmarking and optimization on iOS right now, so I have numbers handy. CBL 1.0.3 on my iPhone 6 indexes about 1500 documents per second. (These are docs taken from my iTunes library metadata, they have about as many properties as yours do.) The speed you're quoting is about two orders of magnitude slower!

The only difference is Objective-C vs JavaScript for the map function itself, and JS will definitely be slower but it shouldn't be that much slower. I also don't think other PhoneGap developers have run into this in their apps.

—Jens

By what you are saying i guess there is something i might be doing wrong. But currently the only clue i have is that i should not emit entire doc. Other then that i don't have much clue what to do to find the cause as a simple ajax call takes 5 secs to get a response the first time i query the view. I will use the include_docs parameter as u said and find out if there is something wrong in my map function. Just for you to see below is the design document which has the views for the sample doc i posted above. 

var design = "_design/account";
   db.put(design, {
     views: {
       ByType: {
         map: function(doc) {
           if (doc.type && doc.type == "account") {
             emit(doc.cat, doc);
           }
         }.toString()
       },

       BySubCat: {

         map: function(doc) {
           if (doc.type && doc.type == "account") {

             emit(doc.subCat, doc);
           }
         }.toString()
       }
     }
   }, function() {

     window.config.accountViews = db([design, "_view"]);

     //generateCatDesginDoc(db, cb);

     //cb(false, db([design, "_view"]));
   });

Other then emitting doc itself the map function is pretty simple.

[Vinay Bhinde]

Also i tested my above view query times without emitting the entire doc itself and emitting null instead. Still it takes 3-4 seconds on average for the first query.

[Jens Alfke]

On Feb 4, 2015, at 10:37 PM, Vinay Bhinde <vin...@gmail.com> wrote:

Also i tested my above view query times without emitting the entire doc itself and emitting null instead. Still it takes 3-4 seconds on average for the first query.

If I were debugging this [on iOS] I'd start by using Instruments and the Time Profiler to profile the app and see exactly where the time is going. It's pretty easy to start this up: in Xcode just press Cmd-I, then choose Time Profiler from the list of instruments, then when the window opens up press the record button (red circle).

Another approach is to turn on logging of the view indexing; use the logging flag "ViewVerbose".

—Jens