Indexed DB Open Cursor with multiple values for single key

[Deni Spasovski]

I would like to know if it's possible to open cursor in indexed db with multiple values for single key
In the specification there is only key range search which opens the cursor from value a to value b, however what i need is that cursor to be opened with specific values

current implementation:
var keyRange = IDBKeyRange.bound("a", "b", false, true);
var cursorRequest = index.openCursor(keyRange);

desired result set:
var keyRange = IDBKeyRange.bound(["a", "b", "c"]);
var cursorRequest = index.openCursor(keyRange);

[Joshua Bell]

Unless I'm misunderstanding, no. You need to do something like:

// if you want all the values:

["a", "b", "c"].forEach(function(key) {

  var request = index.get(key);

  request.onsuccess = function () {

    var value = request.result;

    if (value !== undefined) {

      // do something with key, value

    }

  };

});

// if you want to be able to stop early:

var keys = ["a", "b", "c"];

(function doNext() {

  if (!keys.length)

    return;

  var key = keys.shift();

  var request = index.get(key);

  request.onsuccess = function () {

    var value = request.result;

    if (value !== undefined) {

      // do something with key, value

      if (condition) 

        doNext();

    }

  };

}());

Is there a reason one of these approaches wouldn't be appropriate for your use case?

[Deni Spasovski]

Hi Joshua thanks for the quick response.

I already have custom implementation for this feature, however it's not so fast and I was wondering if such feature is possible with the native indexeddb api?

[Joshua Bell]

Chromium currently optimizes the basic case of a cursor walking over a store/index and pre-fetches the data from the backing store, which is why you may see that being faster than a sequence of get() calls. We're working on further optimizations, which should make both parallel and sequential fetches (my first and second examples) faster.

If you have a particular case where the performance is not what you'd expect can you upload a sample as a bug (go to http://new.crbug.com) and we'll do some profiling and see where the time is going.

--
You received this message because you are subscribed to the Google Groups "Chromium HTML5" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msg/chromium-html5/-/z3oLXc7tUDsJ.

To post to this group, send email to chromiu...@chromium.org.
To unsubscribe from this group, send email to chromium-html...@chromium.org.
For more options, visit this group at http://groups.google.com/a/chromium.org/group/chromium-html5/?hl=en.

[Deni Spasovski]

Hi Joshua,

Here is the procedure that I use for filtering the items, please run the test and let me know if there is a part which can be further optimized.
As you might notice i have a caching functionality which uses the advance function from indexed db and i skip the long lasting iteration process by storing the number of iterated items when once iterated.

As for the data i have 100K items with 200 properties  each.

If you need additional details please contact me back.

[Kyaw]

Hi!

I have run your js, but what I see is you have large loop on put method, which is async. As a rule, you don't loop async call, because none of them are invoked before finishing the loop. I think it will get problem for large scale.  

Perhaps, you could open database outside of PopulateOneStore.  

[Joshua Bell]

On Tue, Dec 4, 2012 at 6:45 AM, Kyaw <kya...@yathit.com> wrote:

Hi!

I have run your js, but what I see is you have large loop on put method, which is async. As a rule, you don't loop async call, because none of them are invoked before finishing the loop. I think it will get problem for large scale.  

Perhaps, you could open database outside of PopulateOneStore.  

On Wednesday, November 14, 2012 7:25:17 AM UTC+8, Deni Spasovski wrote:

Hi Joshua,

Here is the procedure that I use for filtering the items, please run the test and let me know if there is a part which can be further optimized.
As you might notice i have a caching functionality which uses the advance function from indexed db and i skip the long lasting iteration process by storing the number of iterated items when once iterated.

As for the data i have 100K items with 200 properties  each.

If you need additional details please contact me back.

Huh, I never saw this reply, sorry. Must have been eaten by the spam filter. :(  

Thread archive is here and it has Deni's attachments: https://groups.google.com/a/chromium.org/forum/?fromgroups=#!topic/chromium-html5/41OmIFYT784

The example is still very complicated so it takes some time to figure out what's going on. Please try and reduce your examples to the minimum necessary to demonstrate the issue.

So far as I can tell, the biggest problem is that you're doing a linear search for the values, via a combination of cursor.advance() (with your cached count) and cursor.continue(). Either way involves a linear walk over the table, which is going to be slow. I'm still not entirely clear on what the goal is, but I suspect that by using multiple cursors and using cursor.continue(key) you'd avoid the linear search.

 

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msg/chromium-html5/-/QmVFJSuCxjgJ.

[Joshua Bell]

On Tue, Dec 4, 2012 at 9:29 AM, Joshua Bell <jsb...@chromium.org> wrote:

On Tue, Dec 4, 2012 at 6:45 AM, Kyaw <kya...@yathit.com> wrote:

Hi!

I have run your js, but what I see is you have large loop on put method, which is async. As a rule, you don't loop async call, because none of them are invoked before finishing the loop. I think it will get problem for large scale.  

In Chromium, the only issue here is that the transaction builds up in memory until it commits. At some point it may put the process into swap. Otherwise, there's no issue with doing a tight loop of put() calls to populate the database, and recent performance optimizations should show a clear benefit to doing this once M25 goes to stable.

Blocking the UI thread on a tight loop in script isn't a good idea, but you're already using Workers for this which is a great solution. For interested readers, other approaches would be to populate in batches - either yielding by waiting on e.g. an onsuccess of the last put in a batch before starting the next batch, or using a separate transaction per batch.

Perhaps, you could open database outside of PopulateOneStore.  

Also, the population is being done using multiple Workers but all writing to the same database/object store. These transactions can't be run in parallel so there's no benefit to using multiple workers here.

[Deni Spasovski]

Hi Joshua and Kyaw,

I uploaded the whole example so that it would be easier for you to debug it.
I was only interested in profiling results of method "objectSearch". I use this procedure for populating data grid with dynamic paging and searching. So making it parallel is not possible. The search test search queries i've provided are the typical test cases for which the method is invoked.

So far with the advance function i've gained a lot of performance on consecutive searches, however i think there is still place for further optimizations, could you take another look at the search method.

Thanks,
Deni

[Kyaw]

On Wednesday, December 5, 2012 2:21:53 AM UTC+8, Deni Spasovski wrote:

Hi Joshua and Kyaw,

I uploaded the whole example so that it would be easier for you to debug it.

Where is it? I see only preivous attachment.

 

I was only interested in profiling results of method "objectSearch". I use this procedure for populating data grid with dynamic paging and searching. So making it parallel is not possible. The search test search queries i've provided are the typical test cases for which the method is invoked.

Interesting.

 

So far with the advance function i've gained a lot of performance on consecutive searches, however i think there is still place for further optimizations, could you take another look at the search method.

Thanks,
Deni

I started a IDB query demo http://dev.yathit.com/demo/ydn-db/nosql/nosql-demo.html I will add performance analysis there. 

[Deni Spasovski]

Hi Kyaw,

Could you give me more details about the demo you've posted?
As I can see from the code the search queries here based on indexes and multi column indexes, in my scenario this is not possible cause I can have more than 100 properties per item and creating indexes per multiple column would mean that database should have 10K+ indexes.
And from testing i've discovered that data insert suffer when there are more than 20 indexes on the database.
That's why i've developed the search procedure by opening one index and then iterating through the rest of the objects. The only thing that could drastically improve performance is choosing the index which will return the smallest data set and thus reduce the number of items that should be filtered by iteration.

[Kyaw]

Hi

Discussion is here: http://dev.yathit.com/ydn-db/nosql-query.html

Kyaw