IndexedDB - Performance test cases

[Parashuram N]

I wanted to understand the performance characteristics of various operations for IndexedDB for my jquery-indexedDB plugin and wrote up a small test suite for it here - http://axemclion.github.io/IndexedDB/perf/

Thought that this group may be interested in looking at the results and hence sharing with the group. Trying to compare different approaches to do similar things on IndexedDB  and tried to make the use cases resemble the code path most applications using IndexedDB would use.

Some interesting findings -

• Firefox, IE and Chrome use different underlying stores and the difference between using LevelDB and SQLite is well pronounced in the results
• It is sometimes faster to batch up read operations in sets of transactions
• cursor.advance is faster than cursor.continue
  
• (more in the test suits)

I also added a comment section at the bottom of the page - please do run the tests on your browser and leave your comments about interesting results that you notice.

Link - http://axemclion.github.io/IndexedDB/perf/

p.s: this is not a comparison on IndexedDB vs WebSql vs LocalStorage, or comparing performance between different browsers.

[Alec Flett]

I can't tell from the graphs if smaller == faster or smaller == slower - what are the units?

Alec

--
You received this message because you are subscribed to the Google Groups "Chromium HTML5" group.
To unsubscribe from this group and stop receiving emails from it, send an email to chromium-html...@chromium.org.
To post to this group, send email to chromiu...@chromium.org.
Visit this group at http://groups.google.com/a/chromium.org/group/chromium-html5/?hl=en.
For more options, visit https://groups.google.com/a/chromium.org/groups/opt_out.
 
 

[Parashuram N]

Longer graph === faster. It is basically operations per second * 1000.

I used Benchmark.js (used in JSPerf.com) for the numbers to make them statistically significant.

[Joshua Bell]

On Wed, Apr 17, 2013 at 3:07 PM, Parashuram N <n.para...@gmail.com> wrote:

I wanted to understand the performance characteristics of various operations for IndexedDB for my jquery-indexedDB plugin and wrote up a small test suite for it here - http://axemclion.github.io/IndexedDB/perf/

Thought that this group may be interested in looking at the results and hence sharing with the group. Trying to compare different approaches to do similar things on IndexedDB  and tried to make the use cases resemble the code path most applications using IndexedDB would use.

Cool!

FYI we have some IDB performance tests that run in an automated fashion as part of the Chromium build system. The code is here: http://src.chromium.org/viewvc/chrome/trunk/src/chrome/test/data/indexeddb/

The win7 results are here: http://build.chromium.org/f/chromium/perf/chromium-rel-win7-dual/idb_perf/report.html?history=150&rev=-1

The overall numbers aren't terribly useful since we keep adding tests which bumps the time. Drilling in to a particular test and expanding the range shows some of the performance work we've done over time: http://build.chromium.org/f/chromium/perf/chromium-rel-win7-dual/idb_perf/report.html?history=15000&rev=-1&graph=testRandomReadsAndWrites_10000_500_0_5_false

... and catches the occasional regression like this cursor test http://build.chromium.org/f/chromium/perf/chromium-rel-win7-dual/idb_perf/report.html?history=15000&rev=-1&graph=testCursorReadsAndRandomWrites_false_true_false_false (turned out to be a v8 glitch that was fixed quickly).

Just bringing this up because I hope we can share/learn from each other's benchmarks.

 

Some interesting findings -

• Firefox, IE and Chrome use different underlying stores and the difference between using LevelDB and SQLite is well pronounced in the results

I believe IE is using a custom store, but I don't know any details. From my testing it seems to be wicked-fast at some operations, e.g. initial opens and range deletes. Your benchmarks show it kicking butt on cursor operations too - cool!

 

• It is sometimes faster to batch up read operations in sets of transactions

Yep - there's overhead in creating/sequencing transactions. 

 

• cursor.advance is faster than cursor.continue
  

FYI we recently landed a performance fix for cursor.continue(key) (should be in M27), but that's unrelated to this benchmark which just exercises continue() and advance(1)

Per the later clarification in the thread c/o Alec, is sounds like this is backwards - cursor.continue() gets more ops/second?

 

• (more in the test suits)

I also added a comment section at the bottom of the page - please do run the tests on your browser and leave your comments about interesting results that you notice.

Link - http://axemclion.github.io/IndexedDB/perf/

Thanks again. I'm doing a happy dance here seeing that our IDB performance work over the past year has paid off.

 

p.s: this is not a comparison on IndexedDB vs WebSql vs LocalStorage, or comparing performance between different browsers.

--

[Parashuram N]

At this time, I am tempted to write more tests that compare IndexedDB and WebSQL, but will stick to just IndexedDB for the moment. :)

@Joshua Bell, those are some interesting test cases.
Pardon my naive question here, but I was looking at your test cases and was did not understand the test harness that you were running it with. I found an automation object, but is there any other framework that ensures that these tests are run multiple frameworks so that the data collected ignores other factors that may impact it ?
The biggest problem that I was having was about making this data statistically significant. Though I use Benchmarkjs, the blocking nature of IndexedDB make the standard deviations pretty large - large enough that minor differences are lesser than standard deviation, to make the test look accurate.


And here is a compile of my test results for the curious.

• Comparing keys is pretty much comparing different objects - numbers being the fastest and nested arrays being the slowest. [LINK]
• Interestingly in Firefox, specifying a version in indexedDB.open() is faster than not specifying a version. I guess they look up the database meta-data when it is not specified. [LINK]
• The presence (or absence) or keypath and auto-increment does not change the speed on add operation. This is interesting as I always thought that auto-inrement, or keypath would slow down the opreations as additional computation would be required. [LINK]
• Adding more stores to a transaction scope does slow down read operations. However, since reads do not block each other, should adding more stores into a read transaction really matter ? [link]
  
• Adding more stores to a write transaction does slow it down. However, in case of firefox, all writes in one tranasction is actually faster !! [link]
• In chrome, calling put is always faster than calling Add. On other browsers, Add is faster !! [link] No idea why.
  
• Grouping all read operations in a single transaction is faster than having multiple transaction. However in IE, grouping transaction is definitely faster - is this not supposed to be the general case given that read transactions are non-blocking. [link]
  
• However, multiple write transactions do slow down things as expected - due to contention issues [link]
• When using Cursors, instead of reading or writing in a single cursor, opening multiple cursors is way faster. Even in case of write, waiting for a cursor to sequentially write is slower than multiple cursors waiting and then writing [link]
• Adding Indexes does not seem to slow the read. What about multi-entry indexes where you would have to fill the index table - should that not be slower ? [link]
• Iterating using the cursors on primary key, or an indexes almost equally fast. [link]
• Getting just the keyCursor on index is faster that getting the entire objectCursor. [link]
  

I realized that discussing all the nuances here may be hard, so added a comments section on each test to talk about the specifics. Please feel free to send pull request for more scenarios that make sense (or tell me such scenarios and I would be happy to codify them).

[Joshua Bell]

On Tue, Apr 30, 2013 at 2:46 PM, Parashuram N <n.para...@gmail.com> wrote:

At this time, I am tempted to write more tests that compare IndexedDB and WebSQL, but will stick to just IndexedDB for the moment. :)

@Joshua Bell, those are some interesting test cases.
Pardon my naive question here, but I was looking at your test cases and was did not understand the test harness that you were running it with. I found an automation object, but is there any other framework that ensures that these tests are run multiple frameworks so that the data collected ignores other factors that may impact it ?
The biggest problem that I was having was about making this data statistically significant. Though I use Benchmarkjs, the blocking nature of IndexedDB make the standard deviations pretty large - large enough that minor differences are lesser than standard deviation, to make the test look accurate.

They're run as part of overall performance regression tests. Due to factors such as contention with other work on the machine and disk seeks, etc, the results of an individual run are subject to significant variation. The intent of the tests is to notice changes over time (e.g. a particular checkin set back the performance of a scenario by some amount). We're not to the point of chasing microbenchmarks - yet! :)

 

And here is a compile of my test results for the curious.

Nifty! Here's some semi-armchair commentary...

 

• Comparing keys is pretty much comparing different objects - numbers being the fastest and nested arrays being the slowest. [LINK]

Yep. In Chrome, this does a conversion from JS object to an internal key format, then a key compare that walks the keys. Arrays require the most work. Numbers should be the least.

We have some thoughts on speeding up key compares to just be a memcmp, but the conversion from JS is likely to be the most expensive part anyway.

• Interestingly in Firefox, specifying a version in indexedDB.open() is faster than not specifying a version. I guess they look up the database meta-data when it is not specified. [LINK]

The open() steps are complicated enough that I'll believe any voodoo could happen here. :) They trounce Chrome and IE so badly on this benchmark I bet they won't care, though.

• The presence (or absence) or keypath and auto-increment does not change the speed on add operation. This is interesting as I always thought that auto-inrement, or keypath would slow down the opreations as additional computation would be required. [LINK]

Compared to the IPC and event loop overhead, the bookkeeping for autoincrement is minimal. We also lazily update the with generated keys, which is probably why Chrome is so much faster here. (*josh does a happy dance*)

• Adding more stores to a transaction scope does slow down read operations. However, since reads do not block each other, should adding more stores into a read transaction really matter ? [link]
  

In that test it looks like you're comparing two transactions vs. one transaction with two stores. The overhead of managing a second transaction is probably contributing to the time. Have you tried comparing a transaction with one store vs. a transaction with two stores? I'd expect that to be nearly even.

• Adding more stores to a write transaction does slow it down. However, in case of firefox, all writes in one tranasction is actually faster !! [link]
  

Again, I think you're seeing more about transaction overhead than # of stores.

 

• In chrome, calling put is always faster than calling Add. On other browsers, Add is faster !! [link] No idea why. 
  

Weird - put is conceptually simpler since it avoids a read. So Chrome's result makes more sense to me. 

• Grouping all read operations in a single transaction is faster than having multiple transaction. However in IE, grouping transaction is definitely faster - is this not supposed to be the general case given that read transactions are non-blocking. [link]
  
• However, multiple write transactions do slow down things as expected - due to contention issues [link]
• When using Cursors, instead of reading or writing in a single cursor, opening multiple cursors is way faster. Even in case of write, waiting for a cursor to sequentially write is slower than multiple cursors waiting and then writing [link]

With multiple cursors you're probably ensuring that the event loop doesn't slow you down, and can keep the database thread busy.

• Adding Indexes does not seem to slow the read. What about multi-entry indexes where you would have to fill the index table - should that not be slower ? [link]

That does look suspicious. Double check the test logic? 

• Iterating using the cursors on primary key, or an indexes almost equally fast. [link]

Expected - under the hood they should be practically identical operations.

 

• Getting just the keyCursor on index is faster that getting the entire objectCursor. [link]
  

Expected - it avoids fetching/decoding the object. (The decode could be done lazily, though.)