What should Ashikawa be?

[Lucas Dohmen]

Dear Ashikawa Community,

Since the first meeting about the Ruby Adapter for AvocadoDB, there was a lot of discussion about how we should implement it: Should it be a standalone project or a driver for an existing ORM/ODM? If it should be standalone, should it be a a fork of an existing project or an entirely new one? In this thread I want to discuss these questions.

I will start with describing the goals of the project as I see them:
If you want to build an ODM in the Ruby Community, it is very important that it integrates well with Rails and Sinatra, because those are the most important Web Frameworks in the community. This should be the most important goal for Ashikawa.
The other goal should be to leverage the flexibility that a Document Store provides in comparison to a relational database and allow to access the capabilities of AvocadoDB as a Key-Value-Store and Graph-Database.

When implementing AvocadoDB, I see the following options:

1. Implement it as a driver for DataMapper
2. Implement it as a driver for ActiveRecord
3. Implement it as a driver for Mongoid (if that is possible)
4. Implement it as a fork of Mongoid
5. Implement it from scratch

1 + 2:
In the Ruby Community there are a lot of people with strong opinions. Between the DataMapper and ActiveRecord Camp there are lot of discussions. Dan Kubb, the maintainer of DM (DataMapper), contacted Frank via Twitter. He pointed us to the high number of non-relational systems supported by DM. I found on their [Wiki](https://github.com/datamapper/dm-core/wiki/Adapters) that there is third-party support for MongoDB, Github (yep), Riak... and many, many more. I think that the [MongoDB adapter](https://github.com/solnic/dm-mongo-adapter/) is a good example for what is possible. But, as Dan said on Twitter: "It can work with document stores, but ones where the properties are known ahead of time." I have not further explored the possibilities of ActiveRecord in that regard, but I think it is the same there.
In my opinion that is a sacrifice of flexibility: A driver like that should not be the only option. But maybe it is a good idea to support it as one possibility. Why? Because we could provide developers using a framework like that to switch to AvocadoDB from another database with very little effort.

3+4:
Mongoid is an ODM for -- surprise, surprise -- MongoDB. If you look at their [Github Repos](https://github.com/mongoid) they seem to change some things right now. They created Origin, which is a DSL for building queries for MongoDB and Moped, which is a "MongoDB Driver for Ruby". In the current version I don't see an option to provide a different driver to Mongoid, but maybe Moped is the effort to allow that in the future.

5:
Implementing the functionality from scratch allows us to leverage the strength of AvocadoDB. The problem is that we would make it a higher effort for people to switch to AvocadoDB with an existing application and the effort to make it compatible with Rails and Sinatra (and the effort in general) would be higher for us.

Therefore I propose a modular system that consists of multiple gems. One should be the core that provides a wrapper around the REST interface of AvocadoDB that will be used by all other gems. I would suggest "ashikawa-core" as the name for this project. It should provide everything that the REST interface offers to the Ruby developers. Nothing more, nothing less. Don't provide a DSL to create queries here (because that may very between the "higher-level" gems), only the possibility to send a query to the database as a string.

We should also create a gem that creates AQL-queries with a nice DSL. I started this with [brazil](https://github.com/moonglum/brazil). The project is currently on hold, because AQL is currently redesigned. I will continue with the development as soon as "AQL 2" solidifies. This should also be usable by "higher-level" gems in my opinion as it is an equivalent to Arel (from ActiveRecord) and Origin (MongoDB).

I would love to hear what you think about this topic. I will start with ashikawa-core sometime very soon, because we will need something like that anyway. As soon as there is something to see in that area, I will of course inform this mailing list.

Best Wishes,
Lucas

[Masahiro Nakagawa]

Hi Lucas,

Because Document Store is different from RDB and KVS,

I like 3, 4 or 5 approach.

I implement each handler for Document Store, RDB and KVS in product.

In this case, Document Store(or AvocadoDB) specific library is very useful.

But I am not a web application developer,

so I don't know the merit of 1 and 2 approach for web framework.

I can't imagine the same approach using different paradigm databases.

My friend said "Use Mongo instead of ORM!(or write SQL directly)" in Mongo meetup.

Finally, I will create fluent-plugin-avocado with ashikawa-core :)

Masahiro

[Frank Celler]

[OFFTOPIC]

That reminds me, that we need something like "capped collections" in Mongo for AvocadoDB. I assume that could be quite useful for storing Log-Files.

[Lucas Dohmen]

Hey Everyone,

@Masahiro: Thank you for your feedback :) The fluent-plugin sounds great! :)

@Frank: Yep, that sounds useful :)

I have some more thoughts to share with you ;) I emailed with Dan Kubb (Project lead of DataMapper and Ruby Hero of 2009) – he gave me some feedback on my thoughts as well :)

AvocadoDB is a combination of a Document Store, Key-Value Store and Graphdatabase. In my opinion, every part should get its own Ruby Library (all based upon ashikawa-core, of course ;) ). Dan suggested [moneta](https://github.com/wycats/moneta) for the Key-Value Store access. It's a very simple and straightforward library – implementing our "adapter" for it should be really easy when using ashikawa-core. This also provides us with a benchmark tool that will compare the performance of AvocadoDB with Redis, Memcached, MongoDB and others. This is very useful in my opinion.

The Graphdatabase should also not be accessed with a classical ODM tool. In my bachelor thesis I will compare the results of AvocadoDB with the results of Neo4j. Maybe I will built a extensible OGM (Object Graph Mapper ^^) that supports both at the start. A benchmark suite as in Moneta would be nice. I have some thoughts on how to implement that and will post them to this mailing list as soon as they get more precise.

The third part is the Document Store which should be the focus of this discussion. Of course we discussed DataMapper as one solution. I asked him, how he would handle "embedding" versus "referencing" another document. This is in my opinion an absolute must-have for an ODM, because of the possibility to store more complex datastructures in the database. With DM1 this is not possible, and in my opinion it's not possible with ActiveRecord as well. But it will work in DataMapper 2. Here's a direct quote from the mail Dan send me:

"Ahh, good question. What we were going to be doing in DM2 was have two classes; one called "model", which is it's own entity (I guess it's own document in ODM terms), and then have an "embedded model", which has identical DSL syntax to the model. The object instances would be one of those two different types. The relationship declarations (has and belongs_to) would be able to have both as "targets", and then internally DM would know how to map that to the backend.

The advantage of this is that something could start off as an embedded model, and then get "promoted" to a full model just by changing one line of code at the top of the class. It would also be possible to change from a model to an embedded model too. Details still need to be worked out how to handle data migrations of course, but that was the general idea.

So in concrete terms, we could have:

class Article

extend DataMapper::Model

property :title, String, :key => true

has n, :comments

end

class Comment

extend DataMapper::EmbeddedModel

property :author, String

property :text, Text

belongs_to :article

end

The only difference between having Comment as an embedded model and a full model would be to change the extend statement to:

extend DataMapper::Model"

I think this is an excellent feature. But it's in DM2, and DM2 is not there yet ;)

Best Wishes,

Lucas