GSoC 2015: Elixir - NoSQL adapters for Ecto

[Michał Muskała]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hi,

My name is Michał, and I'm a computer science student at the Silesian
University of Technology in Gliwice, Poland.
As far as programming goes I've got experience in Ruby (I've worked
for a year as a Ruby developer), C/C++ and Lisp (as an Emacs user). I
have also some experience with Elixir: I've made some minor
contributions to Elixir itself (my Github handle is michalmuskala),
and build a toy project with Phoenix.

I'd like to help with what is listed as Idea #3 - NoSQL adapters for
Ecto. I think it would greatly help in expanding the Elixir community
by providing easy access to more databases.
I'm not sure how doable it is to create an abstract NoSQL adapter,
given the variety of NoSQL databases. What I think would be most
beneficial is to provide support for PostgreSQL's jsonb columns (it's
a really great feature I've already used). As far as other databases
goes I think it would be great to provide support for mnesia database,
because of it's low overhead in use, furthermore support for MongoDB
would be also beneficial, as I think it's one of the most popular
NoSQL databases (there is an erlang module for MongoDB support,
but I'm not sure how useful it would be).

I think I'm well prepared for this project experience-wise (am I?),
but I'm not entirely sure about the goals I've presented above. Am I
thinking in the right direction? Are those databases I proposed
good choices or are there some better ones? Are there maybe some
desired ones I've missed?
I'd appreciate any tips.

Regards,
Michał Muskała.
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iQIcBAEBCAAGBQJVDYufAAoJEO8XySJy0L8ZkQEP/jSjC6xQaka8rbMYcagS1wdH
lLLpSgdjyZ6aMfl1IKeUT8cHxFrUGQC4JlVDjBmX21ZkGBCpF6ovvV5TGm+PtWYP
OZ5/uuU/RhSf5Atyk02SdOO9+ADXh/CQS2Qwv55wrCbSLXAmqGpYTHcnnU3FWAfS
J2lKVSfp67/KzM8Xwn0KunAZYLjt4Ri0m0BJuDC9qDoEaOjf+MbwsIXIlYy/kbtN
M3KigSC4tiUu/prcDFYukCuSSw3sjm9O5UpiBse3+jr2T8Cb65oGU+iFYccBGCdv
fkd9I3ZspqS55ntSeVNfZ7VofUVMjaSgxe4HgNvMNikPzr2UKB2dMm5jp6PB7rfI
FlNe2eGTp66acOOYdKWDlwa7AmIbCgpcwb+nSsa8LHyK+L6oe6SHTpI2qRIai2UI
FRTaEmaqdFrVuxwVaB/iCRGCqW2N4zn4RGH5wZj/wpZaltdjxa0/o+HzgIFC98cQ
A8pLW0pSzGeuzKhA0748f7cDOkZfX3cMu53UowsGEZoIwBFUDV3HV2se8OA70xBk
04imi4WmjTm7ltGys2n+qdrrim1Y80OCoBmGTdyvvxVJMkG79797jZFZAHwST147
1Y3v+gqkZ9dxJmoJzVomGAbdagoi+1qAYDIrxFsjrZJtFACkP3PHQWsVacmEK1Pw
js1qOBXzLJWMrC4rpltd
=OlL4
-----END PGP SIGNATURE-----

[José Valim]

Cześć Michał!

Welcome to the Beam Community!

Based on your comments, I believe you are on the right track. Indeed, we are not sure how doable a NoSQL adapter is, but that is exactly one of the goals of the project. :) I was personally planning to support MongoDB, although I am not a user, because it seems to be the closest semantically to what we have in Ecto today. Then, as we develop the MongoDB adapter, we will need to provide new abstractions like embedding which, at some point, we should consider porting those to existing adapters (most likely PostgreSQL in this particular example).

Then, if there is time (I think it is unlikely), we can experiment with other adapters, for example, a mnesia one.

Regarding the Mongo driver, I think we can use the Erlang driver, at least as starting point, and evaluate if we should roll our own or not when we have a better idea of the pros and cons of taking such decision.

If you want some reference, I believe you can look into the MySQL adapter as an example:

https://github.com/elixir-lang/ecto/pull/461

It took about 1 month of work from a developer relative new to both Elixir and Ecto to write the pull request above for the MySQL adapter using an existing MySQL adapter.

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer

[Ringo De Smet]

Michal, José,

On Saturday, 21 March 2015 17:25:35 UTC+1, José Valim wrote:

Cześć Michał!

Welcome to the Beam Community!

Based on your comments, I believe you are on the right track. Indeed, we are not sure how doable a NoSQL adapter is, but that is exactly one of the goals of the project. :) I was personally planning to support MongoDB, although I am not a user, because it seems to be the closest semantically to what we have in Ecto today. Then, as we develop the MongoDB adapter, we will need to provide new abstractions like embedding which, at some point, we should consider porting those to existing adapters (most likely PostgreSQL in this particular example).

Then, if there is time (I think it is unlikely), we can experiment with other adapters, for example, a mnesia one.

Regarding the Mongo driver, I think we can use the Erlang driver, at least as starting point, and evaluate if we should roll our own or not when we have a better idea of the pros and cons of taking such decision.

If you want some reference, I believe you can look into the MySQL adapter as an example:

https://github.com/elixir-lang/ecto/pull/461

It took about 1 month of work from a developer relative new to both Elixir and Ecto to write the pull request above for the MySQL adapter using an existing MySQL adapter.

I'm not a student, so not an applicant for GSOC. But as a startup, I'm looking for support of Riak & AWS DynamoDB in ecto. This nosql support would be used in production in our upcoming product. We are looking at Riak for our on-premise deployments and DynamoDB for our SaaS based hosted offering. This means that the adapter(s) would get enough runtime quite fast. How do you see such contribution/support going?

Ringo 

[José Valim]

For some context, there are 4 main types of NoSQL databases: column-based (cassandra, hbase), key-value (riak, redis), graph-based (neo4j) and document-based (mongo).

Although I am not really a fan of Mongo, I believe it is a good starting point for integrating Ecto with NoSQL because it is a document-based engine and the closest one to relational databases.

However, if you have a key-value store, you can use it as a document store by storing the whole document as a value. Riak even has documentation for such: http://docs.basho.com/riak/latest/dev/search/document-store/

The downside is that we would be exposing just part of the Riak API and the query functionality is quite limited. In order to enlighten us, can you tell us a bit more of how you plan to model your data on top of Riak? Which functionalities do you plan to use in the short term?

Thank you for the feedback Ringo!

José Valim

www.plataformatec.com.br

Skype: jv.ptec

Founder and Lead Developer