Querying any view in cblite for the first time after they are created takes very long (cordova)

[Vinay Bhinde]

hello i have a hybrid app using couchbase lite with cordova.

The issue i am facing is that when i query my couchbase views for the very first time they take very long ( >6 seconds) to send the data back (doesn't matter if there are 1 or 50 rows in response). Why is that so ? Is there anything we can do to decrease this time.

Once the views have been queried for the first time, the subsequent calls are very fast and not a problem. Any pointers ?

If you need any code samples of my views or how do i query them then i will happily provide them.

[Ron Williams]

Are you using the Cordova/phonegap plugin or are you using your own java class for your app?

[Jens Alfke]

On Feb 2, 2015, at 5:44 AM, Vinay Bhinde <vin...@gmail.com> wrote:

The issue i am facing is that when i query my couchbase views for the very first time they take very long ( >6 seconds) to send the data back (doesn't matter if there are 1 or 50 rows in response). Why is that so ? Is there anything we can do to decrease this time.

The first query will build the view index first, so it sounds like your indexing is slow. Take a look at the map function and see if it could be optimized. Also, what platform is this running on, and how many documents are in your database?

—Jens

[Vinay Bhinde]

On Monday, 2 February 2015 21:00:35 UTC+5:30, Ron Williams wrote:

Are you using the Cordova/phonegap plugin or are you using your own java class for your app?

Yes i am using phonegap plugin. 

[Vinay Bhinde]

On Monday, 2 February 2015 23:24:12 UTC+5:30, Jens Alfke wrote:

The first query will build the view index first, so it sounds like your indexing is slow. Take a look at the map function and see if it could be optimized. Also, what platform is this running on, and how many documents are in your database?

—Jens

I am running this on android device (4.4.4) and there are about 100 odd documents in cblite when first query is executed. 

[Jens Alfke]

On Feb 2, 2015, at 8:53 PM, Vinay Bhinde <vin...@gmail.com> wrote:

I am running this on android device (4.4.4) and there are about 100 odd documents in cblite when first query is executed. 

6 seconds is a very long time for that few docs. Try profiling the app during that activity and see what's keeping it busy.

—Jens

[Vinay Bhinde]

On Tuesday, 3 February 2015 12:42:58 UTC+5:30, Jens Alfke wrote:

6 seconds is a very long time for that few docs. Try profiling the app during that activity and see what's keeping it busy.

—Jens

Thanks for the reply. By profiling the app do you mean watching the logcat logs for android ? (I am using phonegap cblite plugin in my hybrid app)

 

[Jens Alfke]

On Feb 2, 2015, at 11:26 PM, Vinay Bhinde <vin...@gmail.com> wrote:

Thanks for the reply. By profiling the app do you mean watching the logcat logs for android ? (I am using phonegap cblite plugin in my hybrid app)

No, I mean using some sort of CPU profiler tool that will show you how much time is being spent in what methods. I have no Android development experience (and haven't used Java seriously in a decade) so I don't know what tools are available for this, but there must be some. Profiling is an essential tool.

—Jens

[Vinay Bhinde]

Hello,

I have some more data and hence i decided to post it here and take the discussion forward.

What i did was that i downloaded the Todo - Couchbase Lite Phonegap example from Github and modified its code a little bit. I simple entered 150 docs in cblite and then by using one of the  default view that is provided in it i query it and it takes around 7 seconds on MOTO E running android 4.4.4. This is just a simple test case to show that it is taking very long to index just 150 docs. Subsequent calls to the view are very fast and the time is in milliseconds. But again if there are more additions to the db and new docs are added then again indexing takes 7 seconds. I hope you guys can reproduce the issue with Todo Lite example.

Other query

The other query that i have is that in couchbase server whenever a query is made to a view in a design document then all the views in that design document are indexed. This same behavior is not replicated in CBLITE. Each view is updated only when a query is made to that view explicitly.

Due to this the response times for ajax calls to cblite from my hybrid app take too long and they render the app useless. Same thing runs lightning fast in iOS and i can understand objective c is faster. But the view indexing simply takes too much time to get the data.

 Any pointers ?

[Hideki Itakura]

Hi Vinay,

Thank you very much for investigating CBL PhoneGap Plugin for Android with ToDo Lite Sample.

Current official build of CBL PhoneGap Plugin contains CBL Android v1.0.4. CBL Android v1.0.4 or earlier version has bottleneck for View.updateIndex(). One of root causes is our customer collator implementation for CBL Android/Java. By fixing the ticket of https://github.com/couchbase/couchbase-lite-java-core/issues/502, master branch version of CBL Android improved View.updateIndex() significantly.  If you have time, please try master branch version of CBL PhoneGap Plugin for Android?

Download link:

http://cbfs-ext.hq.couchbase.com/builds/Couchbase-Lite-PhoneGap-Plugin_1.0.4-47-63.zip

How to update plugin:

```

phonegap plugin rm com.couchbase.lite.phonegap

phonegap plugin add ~/Downloads/Couchbase-Lite-PhoneGap-Plugin_1.0.4-47-63

phonegap platform rm android

phonegap platform add android

phonegap run android

phonegap run android --verbose

```

There is open ticket for CBL PhoneGap Plugin for Android View.updateIndex() performance issue. https://github.com/couchbase/couchbase-lite-java-core/issues/478

Thanks!

Hideki

[Vinay Bhinde]

Updating the plugin to the build in the download link DID THE TRICK !!

Significant performance improvement for indexing time. It came down to milliseconds from 7 seconds for 150 docs !!

Thank you for the help..it is really appreciated.

One thing i would like to suggest you guys that there has to be some kind of documentation or blog or atleast some note in the github repository page which will tell users that such kind of issues are to solved by updating the plugin version. I am sure there are lot of other developers who are using cblite phonegap plugin in thier hybrid apps and are facing the issue but have no idea that they need to update the plugin version. Please have some kind of a central place to keep developers updated when anything new comes out or there is a major bug fix.

Thanks for the help.

[Hideki Itakura]

Hi Vinay,

Thank you for sharing test results. I am honestly glad to hear the great result. 

We are preparing for v1.1.0 release which includes many bug fixes and improvements.

Thanks a lot!

Hideki

--
You received this message because you are subscribed to a topic in the Google Groups "Couchbase Mobile" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/mobile-couchbase/urIqcOocmq4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to mobile-couchba...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mobile-couchbase/84bdefe6-05e1-4231-83f7-0120cacc3975%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Thanks,

Hideki Itakura

Website: http://smilogram.net/

[Vinay Bhinde]

Hello Hideki,

Few days back you had given me a updated build of cblite phonegap plugin which improved the view performance for android. I reported that it took around 7 seconds initially to index about 150 docs but then it dropped to milliseconds. 

Seems like i judged it very early and with only a particular type of docs. I changed my document and added a few more fields to make it little big. I added around 100 docs on a click of button to cblite and then performed a view query. It took around 2.2 - 2.5 seconds for android to index that view first and then return the data. This time interval is very long for the users of my application. Every time i add 100 docs and then query the view, it takes about that much time to return the data. 

Now my application involves modules wherein i need to call 2-3 different views after adding about 12 or more docs. Now it takes 400 ms for android to index the view after adding 10 docs (the view which i am talking about here is a simple view with just a emit, no reduce or heavy emit's. If  view has a reduce function or some complex emit structure then the above times mentioned are even more higher). So when all my view are called one after another the time adds up and it takes about 5 seconds for my views with reduce function to return data from cblite.

Also there are many other screens which involve calling multiple views and whenever and significant change is made to db, the first query thereafter to all these views take time and when u add up the time it is in seconds.

Speaking from a user perspective i cannot make the user wait for those many seconds every time they create new records and hence the app becomes unusable on android.

Same app on ios indexes views in milliseconds. Even for 100 docs inserted , a view query takes anywhere between 150 ~ 200 milliseconds to index and return data. 2200 ms and 200ms. Thats a difference of 2000 ms between ios and android. You can check the same in your todo lite phonegap app which i have modified to show my stats. I am attaching the source code dropbox folder link here. You can download the app and run tests on it. 

I know the performance you get on iOS will not be matched on android due to fundamental differences but what i need to find out that is there any way possible to make performance on android relatively close to iOS if not match it. Can we do something to bring the view indexing time down. Are we doing anything fundamentally wrong while working with cblite ? maybe you can look at the ToDoLite code and help us out somewhere.

I am letting my environment know under which i got the view indexing times

FOR ANDROID

Device  - Moto G 1.2 Ghz quad core running android 5.0.2

Ram - 1 gb

view index time for 100 docs - around 2.2 secs

view index time for 10 docs - around 400 ms

FOR IOS

Device  - Iphone 5s with ios 7

view index time for 100 docs - around 150ms

view index time for 10 docs - around 20 ms

The cblite plugin used for above test was taken from the below link which u gave earlier.

http://cbfs-ext.hq.couchbase.com/builds/Couchbase-Lite-PhoneGap-Plugin_1.0.4-47-63.zip

Todo Lite Source code link

https://www.dropbox.com/s/xmiip8fbugh6682/ToDoLite%20%28copy%29.tar.gz?dl=0

Awaiting your feedback !

Thanks !

[Hideki Itakura]

Hi Vinary,

Thank you very much for your updated report.

The performance of View.updateIndex() on PhoneGap Android has two bottle-necks. One is Native JSON Collator for SQLite (https://github.com/couchbase/couchbase-lite-android/tree/master/src/main/jni/JsonCollator) andMap/Reduce JavaScript engine for Java (https://github.com/couchbase/couchbase-lite-java-javascript).  

- JSON Collator:

By addressing the ticket (https://github.com/couchbase/couchbase-lite-java-core/issues/502), JSON collator became fast if JSON is only contains ASCII characters. I believe your first simple test contained only ASCII ranged characters in JSON data. And your test result was good enough. However, I believe your second real test contains non-ascii characters (International characters) in JSON data. And I assume that performance was not good enough because of this. To solve this issue, we need to write native unicode collator. We already filed ticket for this (https://github.com/couchbase/couchbase-lite-java-core/issues/504). 

- couchbase-lite-java-javascript:

Currently we don't have good performance test tool for JavaScript engine. First we should write performance test, and measure the performance. Then we could try to improve the performance.

> Todo Lite Source code link

> https://www.dropbox.com/s/xmiip8fbugh6682/ToDoLite%20%28copy%29.tar.gz?dl=0

Thank you for source code. I will check performance here.

And I filed the ticket for PhoneGap View.updateIndex() performance issue on our github repo. https://github.com/couchbase/couchbase-lite-java-core/issues/630

Thanks,

Hideki

[Vinay Bhinde]

> Todo Lite Source code link

> https://www.dropbox.com/s/xmiip8fbugh6682/ToDoLite%20%28copy%29.tar.gz?dl=0

Thank you for source code. I will check performance here.

I have two buttons in the ToDoLite app named "100"  and "Query". When you will tap 100 button it will simply insert 100 in a bulk_docs request into cblite. I have also added a label which shows the time required for bulk_docs operation.

Now when you tap "Query" button then it will call a view simply without any parameters. The time required for the view query call will also be displayed. This way you check the time cblite is taking for indexing.

[Hideki Itakura]

Hi Vinay,

I confirmed the updateIndex() performance problem with your test code. Thanks.

From now, can we track this issue with Github Issue Tracker? https://github.com/couchbase/couchbase-lite-java-core/issues/630

Thanks!

Hideki