Tokyo Cabinet is 14 times faster than MySQL

[David Beckwith]

I just did a simple benchmark of inserting a million rows into a Tokyo
Cabinet .fdb file versus a MySQL table. The Tokyo Cabinet insertions
were 14 times faster.

Time.now.to_f:
-------------------------
Start Tokyo: 1224676065.98518
End Tokyo: 1224676080.02634
Server version: 5.0.37
Start MySQL: 1224676080.07674
End MySQL 1224676246.60687

Granted, that's not a realistic scenario, but it shows what might be
possible.

[Julian Leviston]

It'd be nice if it had a DataMapper object (DO).

That'd be bloody wicked.

Julian.

[Kyle Drake]

Tokyo Cabinet is a cool power tool, but it's not a relational
database. It only does key/value store. I'm not sure there would be
much of a point to making a DM interface, since you wouldn't be able
to use most of the methods in it. I think it would be much easier to
just write a set of model methods for getting/setting the data.

Kyle Drake
Net Brew Design
http://www.netbrewdesign.com

[Jarkko Laine]

On 23.10.2008, at 3.16, Kyle Drake wrote:

>
> Tokyo Cabinet is a cool power tool, but it's not a relational
> database. It only does key/value store. I'm not sure there would be
> much of a point to making a DM interface, since you wouldn't be able
> to use most of the methods in it. I think it would be much easier to
> just write a set of model methods for getting/setting the data.

DataMapper is not just for relational databases: http://merbist.com/2008/09/29/write-your-own-custom-datamapper-adapter/

//jarkko

--
Jarkko Laine
http://jlaine.net
http://dotherightthing.com
http://odesign.fi

Check out my latest book, Unobtrusive Prototype, fresh off the
Peepcode oven:
http://peepcode.com/products/unobtrusive-prototype-js

[Daniel N]

On Thu, Oct 23, 2008 at 3:33 PM, Jarkko Laine <jar...@jlaine.net> wrote:

On 23.10.2008, at 3.16, Kyle Drake wrote:

>
> Tokyo Cabinet is a cool power tool, but it's not a relational
> database. It only does key/value store. I'm not sure there would be
> much of a point to making a DM interface, since you wouldn't be able
> to use most of the methods in it. I think it would be much easier to
> just write a set of model methods for getting/setting the data.

DataMapper is not just for relational databases: http://merbist.com/2008/09/29/write-your-own-custom-datamapper-adapter/

//jarkko

--
Jarkko Laine
http://jlaine.net
http://dotherightthing.com
http://odesign.fi

There's only a handful of methods you need to use to setup for a DataMapper adapter as defined in the AbstractAdapter.  Not sure how you'd go with a model id though although you can use composite keys etc... doesn't seem to sit...

~Daniel

[Matt Aimonetti]

hehe since my DM adapter was mentioned I guess I should give my opinion.

I actually agree with Kyle and I would probably not use an adapter unless Tokyo Cabinet can handle conditional statements.

A good example would be couchDB. I don't think the adapter makes a lot of sense. Don't get me wrong, it's great to have one and help people getting started, but DM is very much RDBMS oriented, with a concept of clean/dirty objects, collections, relationships.  I feel, Couchrest (by Chris Anderson) is a better fit since it's a lighter layer between your models and your data.

-Matt

[Julian Leviston]

C'mon guys,

If you can store data in a thing, DM should be able to wrap it...

I mean, if you can store key value pairs, we could *easily* write something the yields relationship-type mappings, right?

I'm also keen to start using a proper object database like gemstone/s with merb... (maglev hm? :))... because then we can write proper ruby objects for our model with proper relationships and not use this cobled SQL crap anymore.

The thing then, though, is that we'd need some legacy-backuppy-database support thing... for when we have clients that require the database to be able to be backed up and restored to some sql type thing.

I personally detest SQL backed databases.

Jules

[David Beckwith]

> DataMapper is not just for relational databases: http://merbist.com/2008/09/29/write-your-own-custom-datamapper-adapter/

Pretty awesome! Thanks!

On Oct 22, 9:33 pm, Jarkko Laine <jar...@jlaine.net> wrote:
> On 23.10.2008, at 3.16, Kyle Drake wrote:
>
>
>
> > Tokyo Cabinet is a cool power tool, but it's not a relational
> > database. It only does key/value store. I'm not sure there would be
> > much of a point to making a DM interface, since you wouldn't be able
> > to use most of the methods in it. I think it would be much easier to
> > just write a set of model methods for getting/setting the data.
>
> DataMapper is not just for relational databases:http://merbist.com/2008/09/29/write-your-own-custom-datamapper-adapter/
>
> //jarkko
>
> --

> Jarkko Lainehttp://jlaine.nethttp://dotherightthing.comhttp://odesign.fi

[David Beckwith]

I want to do some more tests to see if Tokyo Cabinet would really
offer much of a performance advantage -- and in what cases.

Here are some test cases (Tokyo Cabinet vs. MySQL) I'm thinking about
implementing (while using INNO DB and turning off ACID for MySQL):
* joins (say user has many posts, say 100 posts each)
* graph traversal with depth 11 and doing a read and write to each
node over a graph with like 3 edges per vertex, with depth 11.

It may be the case that Tokyo Cabinet doesn't significantly improve
over MySQL to warrant creating an ORM for it. That's why I want to do
the preliminary tests to see if they are at least encouraging.

[Dan Kubb (dkubb)]

On Oct 22, 10:56 pm, "Matt Aimonetti" <mattaimone...@gmail.com> wrote:
> hehe since my DM adapter was mentioned I guess I should give my opinion.
>
> I actually agree with Kyle and I would probably not use an adapter unless
> Tokyo Cabinet can handle conditional statements.
>
> A good example would be couchDB. I don't think the adapter makes a lot of
> sense. Don't get me wrong, it's great to have one and help people getting
> started, but DM is very much RDBMS oriented, with a concept of clean/dirty
> objects, collections, relationships. I feel, Couchrest (by Chris Anderson)
> is a better fit since it's a lighter layer between your models and your
> data.

I think DataMapper seems RDBMS oriented mostly because we were
pragmatic about the design and decided to base some of the abstraction
around something we understood well enough to be successful with.
That's not to say we don't want to become more agnostic towards
different storage engines though. In order for that to happen people
need to start writing adapters that bend and stress DM's API. The
adapter API is relatively stable right now, but that's not to say if a
new abstraction can be found that better supports all the storage
engines we won't consider it.. we just need real world cases to build
on first.

I'm beginning to see three separate use cases that DM can handle, but
that not all underlying storage engines support:

- Read/Write by Key
- Read/Write by Search
- Aggregate Reporting

I should start by saying that "Aggregate Reporting" is something DM
can handle with plugins like dm-aggregates, but there will never be
native support for that built into dm-core. If you're doing alot of
reporting or anything that requires the equivalent of GROUP BY, COUNT,
SUM, AVG, etc and you're not using an RDBMS based adapter, then DM
might not be your best option.

However, the other two cases should be handled by DM pretty well. The
main limitation is that some engines like Tokyo Cabinet can only
lookup an object by it's key blazingly fast, but not perform searches.
Then there are engines like Sphinx that can handle searches extremely
well. The engines that can do both, like the RDBMS based ones,
typically perform nowhere near the specialized engines.

DM makes it possible to define a single model, but use different
repositories and engines for different purposes. For example you
could use one engine for key lookups, one for searching and one for
aggregate reports. You could use hooks, message queues and/or
background processes to keep the data in sync between the various
engines. If you're at the level where you've outgrown generalized
engines like the RDBMS ones, then its probably worth considering.

Dan Kubb
(dkubb)

[Kyle Drake]

> I want to do some more tests to see if Tokyo Cabinet would really
> offer much of a performance advantage -- and in what cases.

If you're doing simple get/put of data, it's going to blow any SQL
server out of the water. But doing things like tables, foreign keys
and joins is just not possible. I don't have any specific benchmarks,
but I've seen tasks that take days on MySQL take seconds on DBMs like
Tokyo Cabinet. The difference for simple data store can be
substantial.

You can certainly write a DM adapter. But you wouldn't be able to use
most of the functionality. Even doing things like setting properties
would be pointless. DM is an Object Relational Mapper, but since the
database is not relational, the pieces don't fit. It's not to bash
Tokyo Cabinet or DM, they're both great tools. They're just designed
for fundamentally different things.

[sonicpond]

On Oct 23, 3:29 pm, "Kyle Drake" <kyledr...@gmail.com> wrote:
> DM is an Object Relational Mapper, but since the
> database is not relational, the pieces don't fit.

you make a really good point, as obvious as it might seem, and it
raises a question for me.

i'm thinking about data-persistence in general. assuming that the use
of a dbm like tokyo cabinet is mandated (let's just assume that it
is), is it unreasonable (or illogical) to use it from within an mvc
framework?

i'm really new to merb. everything is so astonishingly abstracted
that i'm having a hard time visualizing how to do "manual" labor (like
retrieving a marshalled object out of a dbm and doing something with
it).

[Justin Reagor]

i'm really new to merb.  everything is so astonishingly abstracted
that i'm having a hard time visualizing how to do "manual" labor (like
retrieving a marshalled object out of a dbm and doing something with
it).

Sorry that I'm jumping into the middle of this conversation, but I just saw that there is a Tokyo Cabinet DM adapter on Github. Unless this is what you guys are talking about already... ;)

:: Justin Reagor

:: just...@gmail.com