Django ORM support for NoSql databases

[parisrocks]

Hi Django Users,

I tried Django recently and really liked the simplistic approach for building sites.

But there's no official support for NoSQL databases like Cassandra or MongoDB, there's a great community of NoSQL users waiting for an official ORM support from Django like me. I would say Django with NoSQL ORM support could actually make it more popular compared to its rivals Ruby on Rails or even Java for that reason.

Waiting for official Django ORM support for NoSql databases.

Thanks

[Tom Evans]

ORM stands for Object-Relational Mapper. It is a tool for expressing
relational databases - databases in which objects are related to other
objects - as a way of presenting an object oriented view of that data.

MongoDB (and the vast majority of NoSQL databases) are not relational
databases, they are document oriented databases, they store documents
of an unspecified type and allow you to retrieve documents by id, or
perhaps other attributes of the document.

At no point with MongoDB can you have relationships between two
different documents that can be followed or queried.

Therefore, I'm quite confused what you want from an "ORM" interface to
a NoSQL database. Each document has no relationships, there is nothing
for an "ORM" to do...

NoSQL databases can already be used within django as document stores,
simply install the appropriate python adapter and go nuts.

Cheers

Tom

[Chris Wilson]

Hi all,

On Tue, 17 Dec 2013, Tom Evans wrote:

> On Tue, Dec 17, 2013 at 3:35 PM, parisrocks
> <saravanan....@gmail.com> wrote:
>> Waiting for official Django ORM support for NoSql databases.
>

> MongoDB (and the vast majority of NoSQL databases) are not relational
> databases, they are document oriented databases, they store documents of
> an unspecified type and allow you to retrieve documents by id, or
> perhaps other attributes of the document.
>
> At no point with MongoDB can you have relationships between two
> different documents that can be followed or queried. Therefore, I'm
> quite confused what you want from an "ORM" interface to a NoSQL
> database. Each document has no relationships, there is nothing for an
> "ORM" to do...

I think it would be more helpful to say that it IS possible to store the
ID of a document in a field of another, of course, and that means that
documents CAN have relationships:

{"id": "1", "name": "Pluto", "owner": "2"}
{"id": "2", "name": "Mickey"}

And some NoSQL databases do not provide any facilities for following that
link automatically, as part of a single query, so you would need to make a
separate query to retrieve the owner of an object. Other databases (such
as CouchDB) do support it.

Even SQL does not always make it easy or convenient to follow these links,
especially if one wants to retrieve associated records from multiple
tables without creating a combinatorial explosion. NoSQL databases can
actually do better than SQL here.

There is value in providing storage backends for NoSQL databases in
Django. Several already exist, for example:

* https://code.djangoproject.com/wiki/NoSqlSupport
* http://www.allbuttonspressed.com/projects/django-nonrel

The main issues and solutions are listed on the first of those pages.

There is no particular reason NOT to do it, unless one wants Django's ORM
layer to remain fundamentally tied to SQL. I can't see a good reason to do
that, but maybe others do. In the mean time, this question will continue
to be asked until a position is officially documented, even if it's "we
don't care about NoSQL, just leave us alone."

Cheers, Chris.
--
Aptivate | http://www.aptivate.org | Phone: +44 1223 967 838
Citylife House, Sturton Street, Cambridge, CB1 2QF, UK

Aptivate is a not-for-profit company registered in England and Wales
with company number 04980791.

[Aymeric Augustin]

Hi Chris,

On 17 déc. 2013, at 17:50, Chris Wilson <ch...@aptivate.org> wrote:

> There is no particular reason NOT to do it, unless one wants Django's ORM layer to remain fundamentally tied to SQL.

I don’t understand your sentence. To me it’s as if you were saying “there is no particular reason NOT to relay email with Django, unless one wants Django to remain fundamentally tied to HTTP”.

Django’s ORM is entirely designed to translate between an object oriented API and SQL. That’s what its name says. It achieves this through roughly five layers that bride the abstraction gap.

Django’s ORM is fundamentally tied to SQL. It isn’t a choice, a policy or a wish. It’s a fact.

Its APIs were designed according to the capabilities of SQL databases. I don’t believe they’re well suited to NoSQL storage engines. There’s also more diversity in the NoSQL field, making it less valuable to write a common API. The least common denominator may turn out to be very small.

Of course, you could implement a totally new piece of software that interfaces with both SQL and NoSQL storage engines. You’ll most likely end up with a very different design from Django’s ORM and hopefully a different API too.

> I can't see a good reason to do that, but maybe others do. In the mean time, this question will continue to be asked until a position is officially documented,

The core team doesn’t issue press releases with official positions :) However, as far as I know, most core devs would subscribe to Tom’s answer.

> even if it's "we don't care about NoSQL, just leave us alone.”

You’re framing the debate unfairly. There’s a difference between “we don’t think this is the right solution” and “we don’t care about your problem”. If I didn’t care, I would simply have ignored your email.

--
Aymeric.

[Tom Evans]

On Tue, Dec 17, 2013 at 4:50 PM, Chris Wilson <ch...@aptivate.org> wrote:
> Hi all,
>
>
> On Tue, 17 Dec 2013, Tom Evans wrote:
>
>> On Tue, Dec 17, 2013 at 3:35 PM, parisrocks
>> <saravanan....@gmail.com> wrote:
>>>
>>> Waiting for official Django ORM support for NoSql databases.
>>
>>
>> MongoDB (and the vast majority of NoSQL databases) are not relational
>> databases, they are document oriented databases, they store documents of an
>> unspecified type and allow you to retrieve documents by id, or perhaps other
>> attributes of the document.
>>
>> At no point with MongoDB can you have relationships between two different
>> documents that can be followed or queried. Therefore, I'm quite confused
>> what you want from an "ORM" interface to a NoSQL database. Each document has
>> no relationships, there is nothing for an "ORM" to do...
>
>
> I think it would be more helpful to say that it IS possible to store the ID
> of a document in a field of another, of course, and that means that
> documents CAN have relationships:

I really don't. Yes, some NoSQL stores can do semi-relational things
(like RethinkDB), and you can, as you demonstrate, store raw ids in
the document and use application logic to "make joins", but I think it
clouds the issue and makes people think that NoSQL datastores can
somehow store their relational data in it and perform relational
queries on it, if only those darn developers would write it.

It is nonsensical. The 'R' in the ORM is important; if you cant
perform equivalent relational queries in the same manner as all the
relational databases currently supported by the ORM, then what is the
purpose of attempting to shoe-horn it into the same API?

Relational and document databases serve different purposes, if you are
attempting to do relational logic in a document database, then IMO
you're starting out from an invalid position. Nothing prevents you
from using both concurrently, of course.

I guess the one thing that could be done would be to write a django
API interacting with all the different NoSQL engines, but not
attempting to mimic the ORM. This would give a "django nosql"
solution, without trying to force a square peg into a round hole.
Three downsides to this:
1) SQL backends all generate SQL (duh) which means there is a lot of
shared code in the ORM.
2) Different stores python adapters can have very different
interfaces and features, making it hard to define a base feature set.
3) It's a document database, there is very little involved in
looking up and retrieving a document, it's easier just to use the
specific adapter for the store you want directly.

Cheers

Tom

[Javier Guerra Giraldez]

On Tue, Dec 17, 2013 at 12:16 PM, Aymeric Augustin
<aymeric....@polytechnique.org> wrote:
> Django’s ORM is entirely designed to translate between an object oriented API and SQL. That’s what its name says. It achieves this through roughly five layers that bride the abstraction gap.
>
> Django’s ORM is fundamentally tied to SQL. It isn’t a choice, a policy or a wish. It’s a fact.

hum.... i think you know more than anybody about the ORM, but
recently (March 26) i said something similar on the Django-users list,
just to be immediately corrected by Russel:



On Tue, Mar 26, 2013 at 6:49 PM, Russell Keith-Magee
<rus...@keith-magee.com> wrote:
>
> On Wed, Mar 27, 2013 at 12:46 AM, Javier Guerra Giraldez <jav...@guerrag.com> wrote:
>>
>> On Tue, Mar 26, 2013 at 11:10 AM, Donnie Darko <gitte...@gmail.com> wrote:
>> > I was wondering if there there was a plan to change Django's ORM to
>> > support NoSQL databases?
>>
>>
>> since there's no such thing as "common NoSQL features", each datastore
>> would need its own modifications, so i think the common attitude is
>> that most of these features don't belong in the ORM.
>
>
> I'm not sure where you've got that impression. Django's ORM was
> *specifically* designed in a way to allow for NoSQL backends. Look at the
> architecture of the django.db module -- there's a query module, and a sql
> submodule. There's very little SQL content in the base query module. That's
> so that we can add a "noSQL" module at some later date. We've also been
> aggressive about adding SQL-specific features (e.g., HAVING clauses, GROUP
> BY clauses) to the ORM API.


so, now i don't really know if Django ORM is appropriate for NoSQL or not.

still, IMHO, the important point is about the wide variety of NoSQL
databases. as i said before, that's a term just as precise as
"non-elephant animals".

what most people really want when they ask for NoSQL support is to
handle document-based stores, which are mostly just key-value stores
with some introspection to maintain indexes and a little server-side
operations.

honestly, I think it would be better done as a separate library,
different from the ORM, with an API to do queries and return iterable
objects with fields as element properties. that's enough to be usable
from the templates exactly like querysets. maybe there could be also
another 'model-like' layer to add more functionality on top of the
persistence layer, but unlike current models, they wouldn't define a
database schema....

--
Javier

[Alex Burgel]

On Tuesday, December 17, 2013 12:16:04 PM UTC-5, Aymeric Augustin wrote:

Django’s ORM is entirely designed to translate between an object oriented API and SQL. That’s what its name says. It achieves this through roughly five layers that bride the abstraction gap. 

Django’s ORM is fundamentally tied to SQL. It isn’t a choice, a policy or a wish. It’s a fact. 

I am one of the committers on the django-nonrel project. We have supported backends for mongodb and google appengine. There are others out there, but they're not part of the nonrel project. So knowing what it takes to use the django ORM to work with a NoSQL DB, I hope I can provide some useful information.

There are a few issues that we run into, so you can see that they are not fundamental problems.

1. Things you can do on a SQL system that you can't on NoSQL.

 - The big ones are joins and aggregates (for some kinds of aggregates on some NoSQL systems). My perspective is that this is the trade off you make when choosing your datastore, so you should know this going in.

2. Things you can do on NoSQL that you can't in SQL.

 - This is very database specific, so its hard to give examples, but its the same thing that necessitates raw queries. Some things you either can't do in the ORM or you can do better by hand.

3. Assumptions django makes about the underlying datastore.

 - We have submitted a number of patches to fix these issues, many of which have been merged in. Our fork is becoming less of a fork every day, and I think with django 1.7 (and some changes on our side), we may be able to do away with the fork for at least the app engine backend.

Some specific issues:

 - AutoField is assumed to work only with integers. MongoDB is an outlier here, it uses something called an ObjectId which is an opaque string. This is actually a big one because field validation occurs before you know which database you save to.

 - Inserts vs. updates. Most NoSQL databases don't make the distinction, there is only a PUT. We have a patch that causes the ORM to basically force_insert on every save, but I think we can do away with this and convert the update to a PUT in our compiler.

 - django.contrib app. We've had issues where queries in the contrib apps use joins and other unsupported things. We have patches for these, and would like to get them merged but this is tricky because they are liable to break again if core developers aren't aware of the issues.

So those are the technical issues, not so complicated IMHO. I can provide more details if anyone is interested.

> I can't see a good reason to do that, but maybe others do. In the mean time, this question will continue to be asked until a position is officially documented,

The core team doesn’t issue press releases with official positions :) However, as far as I know, most core devs would subscribe to Tom’s answer.

Fair enough on not issuing policy proclimations :-) But I think it would be helpful to get some feedback on the question: should django try to 'open up' the ORM to make it easier to write backends for NoSQL DBs? Add more hooks, sharper divisions between layers, etc.

--Alex

[Aymeric Augustin]

On 17 déc. 2013, at 19:53, Javier Guerra Giraldez <jav...@guerrag.com> wrote:

> On Tue, Dec 17, 2013 at 12:16 PM, Aymeric Augustin
> <aymeric....@polytechnique.org> wrote:
>> Django’s ORM is entirely designed to translate between an object oriented API and SQL. That’s what its name says. It achieves this through roughly five layers that bride the abstraction gap.
>>
>> Django’s ORM is fundamentally tied to SQL. It isn’t a choice, a policy or a wish. It’s a fact.
>
> hum.... i think you know more than anybody about the ORM, but
> recently (March 26) i said something similar on the Django-users list,
> just to be immediately corrected by Russel:

I’m likely to stand corrected as soon as the sun rises in Perth, then…

--
Aymeric.

[Karen Tracey]

No need to wait that long:

https://groups.google.com/d/msg/django-developers/0IuJssTt8tc/TxdXQ2D0thcJ

Karen

 

 

[Alioune Dia]

Hello
I think there is a difference between an  ORM and  an Document-Object Mapper as
MongoEngine, I don't know if django ORM is  designed to play a role adapted to a Document-Object Mapper, if not you can always use apps to that could be integrated to the django world and also a document mapper
https://github.com/MongoEngine/django-mongoengine/tree/master/django_mongoengine.
Please correct me if I am wrong :)
--Ad

2013/12/17 parisrocks <saravanan....@gmail.com>

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f8779cdb-5ff4-4f44-a623-ae84e8a120ba%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Aymeric Augustin]

On 17 déc. 2013, at 20:38, Karen Tracey <kmtr...@gmail.com> wrote:

https://groups.google.com/d/msg/django-developers/0IuJssTt8tc/TxdXQ2D0thcJ

Interesting. Now I remember reading that message last year. I have three reactions.

First, Russell and I agree that the interest has faded. Currently, the best option is (probably) django-nonrel. We're open to suggestions that help django-nonrel but we aren't working towards a merge.

Second, Russell talks about the ORM API, not about the ORM code. He implies is that little code, if any, would be shared between the relational and the non-relational backends. We agree again.

Third, his examples conveniently ignore relations, which is where both relational databases and Django’s ORM shine. That’s where I disagree. Here’s an interesting example (extracted from a real project):

(Subscription.objects

    .valid_for_reporting()

    .filter(inception_date__gte=self.start_dt, inception_date__lt=self.end_dt)

    .annotate(rentals_count=Count('subscriber__rentals'))

    .annotate(rentals_duration=Sum('subscriber__rentals__dn_duration'))

    .annotate(rentals_distance=Sum('subscriber__rentals__dn_distance'))

    .annotate(rentals_net_amount=Sum('subscriber__rentals__pricing__net_amount'))

    .annotate(rentals_vouchers_net_amount=Sum('subscriber__rentals__pricing__voucher_details__net_amount'))

    .order_by('inception_date')

    .values(...)

)

I have some doubts about the practicality of running such a query on an NoSQL datastore or translating it automatically to a map-reduce job.

So, while it’s technically possible to implement parts of the ORM’s APIs on top of a non-relational database — as proved by the django-nonrel project — there isn’t much enthusiasm because the APIs are optimized for joins, which SQL databases are good at while NoSQL storage engines aren’t.

-- 

Aymeric.

[Russell Keith-Magee]

On Wed, Dec 18, 2013 at 4:41 AM, Aymeric Augustin <aymeric....@polytechnique.org> wrote:

On 17 déc. 2013, at 20:38, Karen Tracey <kmtr...@gmail.com> wrote:

https://groups.google.com/d/msg/django-developers/0IuJssTt8tc/TxdXQ2D0thcJ

Interesting. Now I remember reading that message last year. I have three reactions.

First, Russell and I agree that the interest has faded. Currently, the best option is (probably) django-nonrel. We're open to suggestions that help django-nonrel but we aren't working towards a merge.

Second, Russell talks about the ORM API, not about the ORM code. He implies is that little code, if any, would be shared between the relational and the non-relational backends. We agree again.

Hey - we're two for three already :-)

Third, his examples conveniently ignore relations, which is where both relational databases and Django’s ORM shine. That’s where I disagree. Here’s an interesting example (extracted from a real project):

(Subscription.objects

    .valid_for_reporting()

    .filter(inception_date__gte=self.start_dt, inception_date__lt=self.end_dt)

    .annotate(rentals_count=Count('subscriber__rentals'))

    .annotate(rentals_duration=Sum('subscriber__rentals__dn_duration'))

    .annotate(rentals_distance=Sum('subscriber__rentals__dn_distance'))

    .annotate(rentals_net_amount=Sum('subscriber__rentals__pricing__net_amount'))

    .annotate(rentals_vouchers_net_amount=Sum('subscriber__rentals__pricing__voucher_details__net_amount'))

    .order_by('inception_date')

    .values(...)

)

I have some doubts about the practicality of running such a query on an NoSQL datastore or translating it automatically to a map-reduce job.

Nope - we agree here as well. 

Yes - my examples ignore relations - that's the point. Deep joined queries are *never* going to be API abstracted in a NoSQL store (at least, not in an efficient way, and I don't see the point in building a complex implementation of something inefficient). If you're doing complex queries, you're always going to need to be aware of the underlying store, and lean on those capabilities.

My claim is that complete abstraction of the data store shouldn't be the goal. What we should be aiming for is sufficient API compatibility to allow for two things:

 * ModelForms wrapping a model from a NoSQL data store

 * An admin representation of a NoSQL data model.

These two functions don't require complex relations. They require relatively simple CRUD operations on a single object; they require an implementation of filter, but not an implementation that allows joins of any kind inside a filter clause. Yes, this will cut off some functionality -- some custom filters, for example, won't be possible. It might be necessary to remove total object counts in admin ListViews -- but then, that would be a good performance boost for PostgreSQL as well. 

But broadly speaking, I see no reason why it shouldn't be possible to put store a data model in NoSQL, and visualise the contents of that model in admin. And you should be able to throw up a Form wrapping that data model. 

I see two ways to achieve this goal. The first is to put an alternate backend into Django's existing ORM. This is the approach attempted by Alex in his GSoC project. It's possible, and actually doesn't require that much work; but there are a couple of unresolved issues, mostly around handling of non-integer automatic primary keys. These are problems that we might be worth addressing anyway, because UUID-based auto primary keys would be a worthwhile extension to SQL data stores.

The second approach is to produce a duck type compatible model API. There aren't (or shouldn't be) any instance checks tied to django.db.models.Model; as long as your NoSQL layer implements the same API, you should be able to drop it into the admin or a ModelForm. Taking this approach would have the side effect that we'd have to clean up the formal definition of _meta, which in itself would be a nice goal.  Added benefit -- this approach doesn't actually require any changes to core itself -- the ducks can be completely external to Django. The only changes to core would be whatever cleanups and documentation the project identifies.

The other, unrelated problem isn't technical at all -- its that that enthusiasm for this problem has pretty much dried up in the core team AFAICT. I certainly don't have any call for using a NoSQL store in my life at the moment. However, if a ready-to-use implementation of either of these approaches were to land on our doorstep, I wouldn't resist them being added to core.

Yours,

Russ Magee %-)

[chris.f...@vokalinteractive.com]

On Tuesday, December 17, 2013 8:12:43 PM UTC-6, Russell Keith-Magee wrote:

My claim is that complete abstraction of the data store shouldn't be the goal. What we should be aiming for is sufficient API compatibility to allow for two things:

 * ModelForms wrapping a model from a NoSQL data store

 * An admin representation of a NoSQL data model.

These two functions don't require complex relations. They require relatively simple CRUD operations on a single object; they require an implementation of filter, but not an implementation that allows joins of any kind inside a filter clause. Yes, this will cut off some functionality -- some custom filters, for example, won't be possible. It might be necessary to remove total object counts in admin ListViews -- but then, that would be a good performance boost for PostgreSQL as well. 

But broadly speaking, I see no reason why it shouldn't be possible to put store a data model in NoSQL, and visualise the contents of that model in admin. And you should be able to throw up a Form wrapping that data model. 

I see two ways to achieve this goal. The first is to put an alternate backend into Django's existing ORM. This is the approach attempted by Alex in his GSoC project. It's possible, and actually doesn't require that much work; but there are a couple of unresolved issues, mostly around handling of non-integer automatic primary keys. These are problems that we might be worth addressing anyway, because UUID-based auto primary keys would be a worthwhile extension to SQL data stores.

The second approach is to produce a duck type compatible model API. There aren't (or shouldn't be) any instance checks tied to django.db.models.Model; as long as your NoSQL layer implements the same API, you should be able to drop it into the admin or a ModelForm. Taking this approach would have the side effect that we'd have to clean up the formal definition of _meta, which in itself would be a nice goal.  Added benefit -- this approach doesn't actually require any changes to core itself -- the ducks can be completely external to Django. The only changes to core would be whatever cleanups and documentation the project identifies.

The other, unrelated problem isn't technical at all -- its that that enthusiasm for this problem has pretty much dried up in the core team AFAICT. I certainly don't have any call for using a NoSQL store in my life at the moment. However, if a ready-to-use implementation of either of these approaches were to land on our doorstep, I wouldn't resist them being added to core.

Wouldn't an easy (i.e. straightforward) solution be to add an Django "ODM" that mirrors the ORM wherever it makes sense?  This sounds pretty close to your second solution, except choosing SQL vs NoSQL means users make a more explicit choice whether to use the ODM API vs the ORM API.

[Javier Guerra Giraldez]

On Wed, Dec 18, 2013 at 12:18 PM, <chris.f...@vokalinteractive.com> wrote:
>
> Wouldn't an easy (i.e. straightforward) solution be to add an Django "ODM"
> that mirrors the ORM wherever it makes sense? This sounds pretty close to
> your second solution, except choosing SQL vs NoSQL means users make a more
> explicit choice whether to use the ODM API vs the ORM API.

I think it would be very straightforward to do that at the public API
level, but the ModelForm and the admin use lots of undocumented _meta
fields which would have to be mimicked too.

Still, i'm convinced that this would be the best answer, and a formal
definition of _meta is a big plus, maybe even proportional to the
effort needed.


--
Javier

[Russell Keith-Magee]

Essentially, what you've described here *is* the second solution, with the caveat that in order to maintain basic compatibility with the internals of Django, you intentionally maintain API compatibility between the ORM and ODM with some key aspects of the ORM API - so, for example, MyMongoDBModel.objects.filter() would work, within certain constraints (for example, as long as you only reference local fields).

A richer ODM/document API that better exposes the properties of document-based stores would expand on these basics, but the basics are needed so that it's drop-in compatible with the basics of ModelForm and admin.

However, this doesn't need to be added to Django's Core, and I'd resist adding it to core unless it can actually demonstrate that it's a genuine abstraction (in the same way that the ORM abstracts the differences between SQLite, MySQL and PostgreSQL). I'm not currently confident that NoSQL stores are at a sufficient level of maturity that this sort of abstraction is possible (or plausible) for anything beyond trivial examples. Remember, SQL has been a work in progress for 20 years as a *published* standard (and another 20 prior to that as an informal one) - the common ground in SQL is well understood at this point. The same isn't true of NoSQL.

Yours,

Russ Magee %-)

[Russell Keith-Magee]

Agreed. Independent of whether NoSQL gets added to core, _meta would definitely benefit from an implementation cleanup and formalisation as a backwards-compatible API.

(Actually… this might make a good GSoC project…)

Yours,

Russ Magee %-)