embedded database options for Nodejs

[Angelo Chen]

Hi,

What are the options for embedded database? I use redis and mongodb for now, but sometimes you made some small apps, and does not want to mix data with existing redis db or mongodb. it should be easier to install, now I'm looking at nosql, https://npmjs.org/package/nosql,   also ejdb, https://npmjs.org/package/ejdb, but seems you can not have your own data file for a individual app. sqllite is another, but it's not json based, any suggestions? Thanks,

Angelo

[Trygve Lie]

https://github.com/rvagg/node-levelup is one option.

Trygve

> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines:
> https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to nodejs+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Pedro Teixeira]

Check out levelDB: https://npmjs.org/package/levelup

-- 

Pedro

--

[Michael Nisi]

Consider LevelDB https://github.com/rvagg/node-levelup

--

[Angelo Chen]

looks like level db is a good option with all the recommendations, is it easier to install? it seems depends on some C libs.

[Michael Nisi]

npm install levelup

[Kevin Swiber]

Hey Angelo,

I'm working on an a persistent key-value store in Node right now.  It's called Medea, and it's currently 100% JavaScript.

https://github.com/argo/medea

It's pre-1.0 at this time, but I think the API is pretty stable.

Julian Gruber maintains a benchmark of databases/libraries called from Node.  You can find that here: https://github.com/juliangruber/multilevel-bench

                      Medea (10.000x)
          12,324 op/s ⨠ set small
          12,313 op/s ⨠ set medium
          12,248 op/s ⨠ set large
          40,566 op/s ⨠ get large
          44,246 op/s ⨠ get medium
          45,174 op/s ⨠ get small

                      levelUP (10.000x)
          38,374 op/s ⨠ set small
          33,019 op/s ⨠ set medium
          23,348 op/s ⨠ set large
          30,622 op/s ⨠ get large
          36,191 op/s ⨠ get medium
          38,326 op/s ⨠ get small




This is using an earlier version of Medea.  The "set" numbers for Medea are a lot better now than they were at that time, though I believe levelUP is still doing faster writes.

LevelUP (and LevelDB) have a few differences.  In Medea, a range query would have to be built from the calling code.  You'd have to filter listKeys() and then iterate over those keys doing a get on each one.

Medea keeps a copy of all keys  and value file IDs/offsets in-memory (to make a maximum of 1 disk seek per get), so your key set has to fit in memory.

It's working pretty good for me, and I know at least one other person who is using it for their project.

Medea is pretty stable now, but I expect stability issues to be raised and squelched as we approach 1.0.

That said, levelUP and LevelDB are great projects, and you can't go wrong with those options, either.  Medea being all-JavaScript is a pro for me at this time.

Cheers,

-- 

Kevin Swiber

@kevinswiber

https://github.com/kevinswiber

[Mike Parsons]

https://github.com/developmentseed/node-sqlite3

I've used this quite extensively on linux/windows/mac and even Raspberry PI

[Ben Taber]

There's dirty for super simple storage: https://github.com/felixge/node-dirty

[Angelo Chen]

Hi Ben,

this is interesting, I can find good use case for that, the repository
is of felixge, but the author is you, does that men felixge is no
longer in this project?

Thanks,

Angelo

On 4月19日, 下午10时07分, Ben Taber <ben.ta...@gmail.com> wrote:
> There's dirty for super simple
> storage:https://github.com/felixge/node-dirty
>
>
>
>
>
>
>
> On Friday, April 19, 2013 2:59:28 AM UTC-6, Angelo Chen wrote:
>
> > Hi,
>
> > What are the options for embedded database? I use redis and mongodb for
> > now, but sometimes you made some small apps, and does not want to mix data
> > with existing redis db or mongodb. it should be easier to install, now I'm

> > looking at nosql,https://npmjs.org/package/nosql,   also ejdb,

[Angelo Chen]

Hi Kevin,

I took a look, the 100% js codebase is one plus factor, one thing, is
there any utility to browse the database?

Thanks,

Angelo

On 4月19日, 下午8时03分, Kevin Swiber <kswi...@gmail.com> wrote:

> On Apr 19, 2013, at 4:59 AM, Angelo Chen <angelochen...@gmail.com> wrote:
>
> > Hi,
>

> > What are the options for embedded database? I use redis and mongodb for now, but sometimes you made some small apps, and does not want to mix data with existing redis db or mongodb. it should be easier to install, now I'm looking at nosql,https://npmjs.org/package/nosql,   also ejdb,https://npmjs.org/package/ejdb, but seems you can not have your own data file for a individual app. sqllite is another, but it's not json based, any suggestions? Thanks,

[Ben Taber]

Felix is still the owner and wrote most of the code, I've just been helping out lately.

--
--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en

---

You received this message because you are subscribed to a topic in the Google Groups "nodejs" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/nodejs/EwFbBLuKX28/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to nodejs+un...@googlegroups.com.

[Kevin Swiber]

Angelo,

Not at this time. Implementing a browser shouldn't be too difficult, though. Medea stores both keys and values as Buffer objects. You could iterate through the key-value store and stringify key and value Buffers prior to display.

There are a number of Web front-end data grids you could use for the UI.

--
Kevin Swiber
@kevinswiber
https://github.com/kevinswiber

> --
> --
> Job Board: http://jobs.nodejs.org/
> Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
> You received this message because you are subscribed to the Google
> Groups "nodejs" group.
> To post to this group, send email to nod...@googlegroups.com
> To unsubscribe from this group, send email to
> nodejs+un...@googlegroups.com
> For more options, visit this group at
> http://groups.google.com/group/nodejs?hl=en?hl=en
>
> ---

> You received this message because you are subscribed to the Google Groups "nodejs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.

[Angelo Chen]

Hi Kevin,

just tried, got a question, is there a way to open() at beginning,
then use it later? the sample is, you have to put the code in a
callback, this does not work:

ar Medea = require('medea');

var medea = new Medea();
medea.open()

medea.get('hello', function(err, val) {
console.log(val.toString());
medea.close();
});

On 4月19日, 下午11时10分, Kevin Swiber <kswi...@gmail.com> wrote:
> Angelo,
>
> Not at this time.  Implementing a browser shouldn't be too difficult, though.  Medea stores both keys and values as Buffer objects.  You could iterate through the key-value store and stringify key and value Buffers prior to display.
>
> There are a number of Web front-end data grids you could use for the UI.
>
> --
> Kevin Swiber

> @kevinswiberhttps://github.com/kevinswiber

[Angelo Chen]

Hi Ben,

looks like, you have to put access code in db.on('load', ...)

how to just open the database once at beginning, then use later? say
all the database code will be in mydao.js, exporting some CRUD
methods, so I can call from another js, dao.append({id:1, data:'d'})

in this append method, I have to wrap the function in a db.on('load',
function(rec){}) ?

Angelo


On 4月19日, 下午10时07分, Ben Taber <ben.ta...@gmail.com> wrote:

> There's dirty for super simple
> storage:https://github.com/felixge/node-dirty
>
>
>
>
>
>
>
> On Friday, April 19, 2013 2:59:28 AM UTC-6, Angelo Chen wrote:
>
> > Hi,
>
> > What are the options for embedded database? I use redis and mongodb for
> > now, but sometimes you made some small apps, and does not want to mix data
> > with existing redis db or mongodb. it should be easier to install, now I'm

> > looking at nosql,https://npmjs.org/package/nosql,   also ejdb,

[Kevin Swiber]

There's no synchronous open at this time. This has been an issue a couple of times when porting apps to Medea as most Node database libraries use synchronous database initialization by default.

Adding db.openSync is more time-consuming work than you may think. There's a lot going on when you open a new Medea database. It's the most expensive operation, because it:

1) checks for a directory (creates if necessary)
2) opens a new data file for writing
3) opens a new hint file for writing
4) creates a lock file
5) writes the pid to the lock file
6) scans existing files
7) loads key to file ID and offset mappings from each file into memory

To create a sync version of all that means creating *Sync methods all over the place. I don't think this will be something we introduce in v1.

For servers, I usually….

db.open(function() {
server.listen(process.env.PORT || 3000);
});

--
Kevin Swiber
@kevinswiber
https://github.com/kevinswiber

[Ben Taber]

No guarantees that you can read a record before .load has been called, but you can write before load.

https://github.com/felixge/node-dirty#dirty-event-load-length

To just open the file once on app init, wrap it in its own module, and then require that into other modules.  Something like below would work.

-- db.js

var db = require('dirty')('./data.db');

var isLoaded;

db.on('load', function() {

  isLoaded = true;

});

db.ready = function(cb) {

  if (isLoaded) {

    return cb();

  }

  db.on('load', cb);

}

module.exports = db;

-- otherFile.js

var db = require('./db.js');

db.ready(function() {

  db.get();

  db.set();

  // etc

})

[John Fitzgerald]

When using these embedded databases, like node-levelup, how do you handle cluster based apps? I'd love to use an embedded database, but also need to run our express app on all 8 cores of the machine.

I know I can process.send to children & master, but that seems like a clunky way of interfacing. Creating a "db" process separate that responds over HTTP and have the workers do requests against that is possible, but not sure if there's a more common / clean way of implementing. 

--

--
Job Board: http://jobs.nodejs.org/
Posting guidelines: https://github.com/joyent/node/wiki/Mailing-List-Posting-Guidelines
You received this message because you are subscribed to the Google
Groups "nodejs" group.
To post to this group, send email to nod...@googlegroups.com
To unsubscribe from this group, send email to
nodejs+un...@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/nodejs?hl=en?hl=en
 
---
You received this message because you are subscribed to the Google Groups "nodejs" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nodejs+un...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
John R. Fitzgerald

[Kevin Swiber]

Hey John,

I created medea-clusterify[1] to handle this use case.  It's using worker.send and process.send to do exactly as you describe.  This is not ideal, but it works.  The master process holds the Medea instance, and the workers communicate with it.  This means the master process can bottleneck.  However, you can still handle a lot more connections concurrently.

Doing a key-space partition is another way to handle this.  To do that, you have to route traffic between processes based on owned key-space.  There has been some discussion about this in a GitHub issue[2].  As discussed in this issue, cross-platform memory mapping would be a good way to not duplicate in-memory data structures per process, but that won't come until at least v2.

LevelDB (and levelUP) restricts usage to a single process, as well, though you could build some multi-process support around it.

[1] https://github.com/argo/medea-clusterify

[2] https://github.com/argo/medea/issues/10

-- 

Kevin Swiber

@kevinswiber

https://github.com/kevinswiber

[Gabriel Farrell]

For those interested in leveldb this recent NodeUp is worth a listen: http://nodeup.com/fortyone.

[Patrick Maina]

Try unqlite which has a binding for node.js https://npmjs.org/package/unqlite