Tips on adding a Couchbase backend

[Mark Nunberg]

Hi folks!

I apologize in advance if this is the wrong venue for this discussion

I'm trying to determine the best means by which I might be able to add Couchbase (http://www.couchbase.com) support to Django. I've come across something called "django-nonrel" - it seems to have scarce documentation and doesn't seem to have much activity as of late. I've also seen that another non-relational database (Cassandra) has a django backend that bypasses the nonrel fork, though its README (https://github.com/r4fek/django-cassandra-engine) seems to suggest one needs to use cqlengine for this?

Admittedly, I'm fairly new to Django itself - though I maintain the Couchbase Python client, and there's been some demand for this feature.

In relation to traditional RDBMs, Couchbase has:

* SQL-like query language (N1QL) for searching (currently JOINS are only supported where one of the sides is a PK). Note this only works for searches, not updates/inserts

* CRUD operations

* There's also views, but I'm guessing anything expressible as a view can also be expressed in terms of N1QL

Can someone point me in the right direction of what to read first?

Regards,

Mark

[Tim Graham]

There's nothing in the way of official documentation about writing third-party database backends. I guess the best approach would be to look through the source code of database engines in Django as well as the projects you mentioned and learn from that. If we can add any hooks to Django to help out your effort, let us know.

[Shai Berger]

On Sunday 14 June 2015 00:13:33 Mark Nunberg wrote:
> Hi folks!
>
> I apologize in advance if this is the wrong venue for this discussion
>
> I'm trying to determine the best means by which I might be able to add
> Couchbase (http://www.couchbase.com) support to Django. I've come across
> something called "django-nonrel" - it seems to have scarce documentation
> and doesn't seem to have much activity as of late. I've also seen that
> another non-relational database (Cassandra) has a django backend that
> bypasses the nonrel fork, though its README
> (https://github.com/r4fek/django-cassandra-engine) seems to suggest one
> needs to use cqlengine for this?
>

In addition to Tim's notes, please consider that django-cassandra-engine is
not a database-engine in the regular sense -- it does not seem to support
Django models, but only cqlengine models. You can probably use it to build a
Cassandra-based web application, utilizing Django's request-handling
facilities and other non-database-related utilities, but you will not be able
to use most of the Django eco-system. From your description, it is not clear
to me if you can do better for CouchBase, but perhaps you can.

You may also want to look at a project called django-mailer[1], which is an
example of "faking" Django models -- that is, taking something which isn't a
Django model (in this case, mail messages) and wrapping it in enough API to
allow it to be used in the Django admin. This practice is officially supported
since Django 1.8, which made the model metadata APIs public.

HTH,
Shai.

[1] https://github.com/PirosB3/django-mailer

[mnunb...@gmail.com]

Thanks for the advice. I imagine the emailer example is a way to show how to make something that’s not really a Django model into a Django model (similar to what Cassandra does).

Interesting point about not supporting the ecosystem - but the question is how much of the ecosystem (rough estimate) would actually continue to function with some features not present? It would only be worthwhile trying to make a proper database-engine if the rest of django would support it: Specifically, any kind of data modification using SQL is currently unsupported, and JOIN queries requires one of the sides to actually be a PK. Updates and insertions must currently be done using a strict KV API (i.e. no SQL)

I did some brief experimentation doing the nonrel fork. Things seemed to work initially, but it’s a constant struggle, and it involves understanding both Nonrel’s internals and Django’s internals (specifically, the internals of an older version). Is it a reasonable endeavor to try and base this engine off 1.8? This doesn’t seem to be very recent, either https://code.djangoproject.com/wiki/NoSqlSupport

[Aymeric Augustin]

On 14 juin 2015, at 20:23, mnunb...@gmail.com wrote:

It would only be worthwhile trying to make a proper database-engine if the rest of django would support it: Specifically, any kind of data modification using SQL is currently unsupported

This is an well defined limitation. If you can make the ORM work for read queries, some might find it useful, even if write queries don’t work.

and JOIN queries requires one of the sides to actually be a PK.

I believe Django only generates such joins, so you’re safe.

That said, in my view, the Django ORM is a series of layers that bridge the gap between a document-oriented API and SQL. I find it weird to shoehorn NoSQL databases into this framework when they already have a document-oriented API of their own…

An important part is to manage connections properly (I’m not sure what’s adequate for Couchbase) and to provide a solution to run tests (create a test database for tests, drop it when tests are finished). If you implement a database backend, you get this for free. Otherwise, you have to manage it yourself. Here’s a project that does this for ElasticSearch: https://github.com/Exirel/djangoes

I hope this helps.

-- 

Aymeric.

[Götz Bürkle]

Hi Mark,

Maybe the work done to support Google App Engine and its Datastore could also be useful for you. You can find more information about that project on https://github.com/potatolondon/djangae and http://djangae.readthedocs.org/en/latest/db_backend/

Cheers,

Goetz