IndexedDB API wrapper library

[Kyaw]

Hi All,

I would like to share my open source IndexedDB API wrapper library (https://bitbucket.org/ytkyaw/ydn-db). 

Javascript database module for Indexeddb, Web SQL and localStorage storage mechanisms supporting version migration, advanced query and transaction.

Goal

Beautiful database API for secure, robust, high-performance, maintainable, large-scale javascript web app.

Roadmap

1. Automatic schema (store, index, keyPath) generation.
2. Unit testing on compiled js.
3. ydn.db.Cursor object and db.open for more powerful readwrite usage, cursor resume
4. Parent-child relationship via parent schema
5. ydn.db.Query will compile to ydn.db.Cursor for IndexedDB and native SQL for WebSQL
6. Query join through through cursor zigzag
7. Full-text search indexing.
8. Storage installable as EventTarget and postMessager allowing interesting party to listen record update.

Features

• Support IndexedDB, Web SQL and localStorage storage mechanisms.
• Support all features of IndexedDB.
• Well tested closure library module.
• Support version migration, encryption, high level query and advance transaction workflow.
• Each method call is an atomic transaction. All methods are asynchronous.
• We adopt strict javascript coding pattern: no global, no eval, no error globbing, parameterized query, all public methods and constructors are strongly type, this is this, coding error throw error.
• JQuery plugin available (see download section).

Supported browser

Support all modern browsers on mobile, tablet and desktop.

Basic usage

Import lastest minified JS script (see download section) to your HTML files. This will create single object in the global scope, call ydn.db.Storage.

var db = new ydn.db.Storage('db name');

db.setItem('x', 'some value')

db.getItem('x').success(function(value) {
  console.log('x = ' + value);
}

Query

Calculate average by using query

q = db.query('customer').average('age');
avg = q.fetch()

Key-range query

q = db.query('customer', 'age', 18, 25).where('sex', '=', 'FEMALE').select('full_name')
young_girl_names = q.fetch()

Transaction

Example for updating an entity with a new property value relative to its current value.

db.transaction(function(tdb) {
   tdb.get('player', 1).success(function(p1_obj) {
        p1_obj.health += 10;
        tdb.put('player', p123_obj);
   });
}, ['player'], 'readwrite');

Encryption

String value data can be optionally encrypted using SHA-1 cipher.

db = new ydn.db.Store('store name')
db.setSecret(passphase); // generally send from server side upon login
db.setItem(key, value, 3600*1000); // data expire on one hour
db.getItem(key); // data will be decrypted using the provided passphase

License

Licensed under the Apache License, Version 2.0

[Ido]

Very cool!

What was the main reasons not to support the indexedDB 'syntax' and go with your own? simplicity?

[Kyaw]

The main reasons are robustness and ease of use. But if you see, the syntax is very similar to IndexedDB. 

Robustness come from reflection of IndexedDB database schema. Most common error will throw before opening a transaction.

It is simple request layer is hided. All database methods are overloaded (get, put methods accept array of records from multiple object stores). Transaction is very easy.

Best of all, there is fail back to WebSQL.    

[Parashuram Narasimhan]

Talking about IndexedDB API wrappers - here is a jquery IndexedDB plugin that I was working on - http://axemclion.github.com/jquery-indexeddb

[Kyaw]

Rarashuram! 

Wow!, so simple!

[Alec Flett]

Wow, this looks really nice! IDB needs more wrappers like this (the raw IDB API is too low-level for typical use IMO)

 I really like the way you simplified some of the querying (i.e. .where() and the like) and how it adapts to the presence of indexes or not. It would be really interesting to figure out when/if you could create indexes on the fly if it looks like the user is querying the same keypaths a lot.

(Looking further) wow there really is a lot there! The map/reduce stuff is really nice too - it would be great if IDB had a less-serial mechanism for cursoring through results, so that you could truly parallelize the mappers... but I suppose you could just kick off a cursor, and pass off the results as you encounter them, to a worker pool to process them.

It's sort of sad that you'd need to support WebSQL at all.. - I know nparashuram was working on a polyfill to adapt the IDB API - if it was robust enough maybe you could drop WebSQL support entirely :)

Alec

--
You received this message because you are subscribed to the Google Groups "Chromium HTML5" group.
To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msg/chromium-html5/-/l4P3WQ6GLQAJ.
To post to this group, send email to chromiu...@chromium.org.
To unsubscribe from this group, send email to chromium-html...@chromium.org.
For more options, visit this group at http://groups.google.com/a/chromium.org/group/chromium-html5/?hl=en.

[Parashuram Narasimhan]

About the IndexedDB - polyfill, I was able to get it working on a bunch of libraries (including pouch, jquery, LINQ, db.js and Ido's recent IndexedDB example ) and I would love to help you add the polyfill to ydn-db.

http://axemclion.github.com/IndexedDBShim

For what it is worth, it works of Cordova based apps (iOS/Android) too :)

[Kyaw]

I am not clear the indexing implement in  https://github.com/axemclion/IndexedDBShim/blob/gh-pages/src/IDBIndex.js is efficient.

I think CREATE INDEX is required.  http://sqlite.org/lang_createindex.html perhaps it might automatically create index, but still need data type to be specify to work correctly. 

[David Grogan]

I haven't seen anyone mention this polyfill yet: https://github.com/facebook/IndexedDB-polyfill

Apparently fb implemented idb on top of websql also.  A cursory glance suggests that it is independent of axemclion's but I'm not positive. I haven't played with it at all.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msg/chromium-html5/-/xLfMTzVi7qwJ.

[Kyaw]

Thanks David,

Wonderful. It implements indexing well, including multiEntry. Immitating IndexedDB API, implementing without data type in column defination is very very interesting. That in only leaking abstraction, I had. So urgly. Now seems workable. But yet to check collation are working properly and performance issue.

Thanks again.

Kyaw 

[Kyaw]

Thanks Alec. :-)

yes, implementing auto indexing (and object store creating) is in progress. also declearative query statement. 

So using indexedDB will be just like GAE datastore. super easy and never fail with InvalidStateError. ;-)

I also believe using the wrapper library will have higher performance than using directly assuming most js developer won't have time to think about (reading w3c doc is not enough) how to use properly. 

Kyaw  

[Parashuram Narasimhan]

About the efficiencies, we are looking at 2 options, as in this pull request - https://github.com/axemclion/IndexedDBShim/pull/18 and https://github.com/axemclion/IndexedDBShim/issues/17

Right now this library does not support multi-entry for performance. To support multi-entry, the indexes would have to be moved to a separate table, like the mozilla implementation of IndexedDB Indexes.

[Parashuram Narasimhan]

I have looked at that polyfill too, and its pretty stable. I tried sending a pull request and was hoping to unify both the implementations into one, but have not heard back anything from them.

[Kyaw]

Great idea! I also think web worker is the way to go for query. Currently my plan is to support installable postMessaging service. Developer have to put my library in the web worker and use the library through cross domain message. a set of database operation commands will be defined. Basically it is the same. for example the database operation

 db.get('store name', id)

will be defined as in JSON:

{

  'op': 'get',

  'store': 'store name',

  'id': id

}

post it to worker frame, and received the result on postMessage.  

So that is also solution for those don't like promise pattern but insist global eventing. 

[Alec Flett]

On Wed, Oct 10, 2012 at 6:23 PM, Kyaw <kya...@yathit.com> wrote:

> Great idea! I also think web worker is the way to go for query.

I'm curious why you think a worker is the way to go? Have you had
performance problems on the main thread?

I suspect that, at least currently, most performance issues, even with the
queries, are going to be I/O related, not CPU related, which means running
off the main thread won't buy you much... plus in chrome's implementation,
much of the heavy leveldb work is going to happen in a separate background
thread anyway...

..and if anything you're now paying the overhead of shuttling data between
the main thread and the worker through postMessage, which at least in
Chrome isn't very fast... but I'm willing to be proven wrong!

Alec

[Parashuram Narasimhan]

I was working on a LINQ (Microsoft's DB query language) library for playing with data, and wanted querying capabilities that go beyond indexes - typically something like
\[http://linq2indexeddb.codeplex.com/SourceControl/changeset/view/69753#1167101]

- Queries involving 2 fields in the data
- Queries across objectStores.

I remember that there were discussions in W3C IndexedDB lists about supporting multiple indexes (indexes on indexes), but that never happened.
Executing these queries is typically getting the data back and then using the workers to iterate over data and get the final result.

The hope was, if workers supported Sync calls, IndexedDB data can be retrieved and processes in the worker itself - till then we need to pass the input to the worker and get the output back.

[Kyaw]

On Friday, October 12, 2012 3:00:14 AM UTC+8, Alec Flett wrote:

On Wed, Oct 10, 2012 at 6:23 PM, Kyaw <kya...@yathit.com> wrote:

> Great idea! I also think web worker is the way to go for query.


I'm curious why you think a worker is the way to go? Have you had
performance problems on the main thread?

People want to do crazy thing during cursor iteration like, running regexp of large text (blog post content, etc). As you see, there is no effective way to index the long text.  

 

I suspect that, at least currently, most performance issues, even with the
queries, are going to be I/O related, not CPU related, which means running
off the main thread won't buy you much... plus in chrome's implementation,
much of the heavy leveldb work is going to happen in a separate background
thread anyway...

I see. thanks. it should be. 

 

..and if anything you're now paying the overhead of shuttling data between
the main thread and the worker through postMessage, which at least in
Chrome isn't very fast... but I'm willing to be proven wrong!

put our hope in ArrayBuffer :-D

postMessaging may not be bottom-neck. Suppose map/reduce operation for whole object store for query aggregate property, data transfer involve only the last operation.  

[Kyaw Tun]

On Fri, Oct 12, 2012 at 4:00 AM, Parashuram Narasimhan <n.para...@gmail.com> wrote:

I was working on a LINQ (Microsoft's DB query language) library for playing with data, and wanted querying capabilities that go beyond indexes - typically something like
\[http://linq2indexeddb.codeplex.com/SourceControl/changeset/view/69753#1167101]

- Queries involving 2 fields in the data
- Queries across objectStores.

I remember that there were discussions in W3C IndexedDB lists about supporting multiple indexes (indexes on indexes), but that never happened.
Executing these queries is typically getting the data back and then using the workers to iterate over data and get the final result.

The hope was, if workers supported Sync calls, IndexedDB data can be retrieved and processes in the worker itself - till then we need to pass the input to the worker and get the output back.

sync API is a terrible trap for developer. on implementation side, sync call will be just faking IO wait and no benefit over async.

it is incorrect to assume, async for worker thread and sync for main thread. 

running async API in worker thread has real benefit of multi-tasking IO and CPU.  

 

--
You received this message because you are subscribed to the Google Groups "Chromium HTML5" group.

To view this discussion on the web visit https://groups.google.com/a/chromium.org/d/msg/chromium-html5/-/1qRFxVdauTUJ.

[Arun Puri]

Is there a way to acess indexdb data in server side with C#

[PhistucK]

No. IndexedDB is a client side only storage. The client will have to serialize the data somehow and send it to the server.

☆PhistucK

To unsubscribe from this group and stop receiving emails from it, send an email to chromium-html5+unsubscribe@chromium.org.

To post to this group, send email to chromiu...@chromium.org.

Visit this group at https://groups.google.com/a/chromium.org/group/chromium-html5/.
For more options, visit https://groups.google.com/a/chromium.org/d/optout.