[ GSOC 2016 ] providing standard interface for NoSQL databases

[girish ramnani]

Hello,

I would like to propose providing a standard interface for NoSQL databases . Some the points to consider are:

1. The Fields which are used to define the schema of the sql databases can also be used to provide in No sql databases.
2. In NoSQL dbs ,developers usually store data in denormalized form. so in two related collections a meta tag denormal = True can be provided. Which would make that collection a child of the other model ( related ). In normal scenario the collections will be connected using _id used as a foreign key.
3. providing contrib.mongo with Field such as GridFSField for providing support for GridFS
4. considering the database transaction transaction blocking ( PyMongo provides blocking api for mongo )
   
5. creating Mixins such as FlexibleMixin which would allow Models to have dynamic fields. i.e directly insert fields which are not listed in the Model schema.

there are lot more things (Migrations, settings etc) to consider but before that needed to know that I am going in the right direction or not?. Please suggest.

[Curtis Maloney]

It sounds like by "NoSQL DBS" you mean specifically "Document store
DBMS". Is this correct?

(I'm sure most people know my rant about how "NoSQL" is a misnomer, and
it's really "Non-relational" that people mean - it's not SQL per se
they're avoiding, it's the relational model)

The NoSQL/Non-rel name is really too vague, as it covers "anything
that's not SQL / Relational"... which can cover key/value, document,
graph, log, and even filesystem or email, when it comes down to it.

I think it will help if you clarify what model you _do_ target, not
which you do _not_.

All that said/vented, I agree it would be nice to have a common
abstraction layer atop various document stores. The trick now will be
finding sufficient commonalities.

I feel that the ORM's relation-span/transforms/lookups syntax would be
equally viable on nested documents.

One of the benefits of an abstraction layer is allowing other tools to
generically interact with these new objects. What plans do you have for
how to support non-flat documents in, for instance, Forms?


--
Curtis

On 16/03/16 05:29, girish ramnani wrote:
>
> Hello,
>
> I would like to propose providing a standard interface for NoSQL
> databases . Some the points to consider are:
>

> 1. The /Fields /which are used to define the schema of the sql

> databases can also be used to provide in No sql databases.

> 2. In NoSQL dbs ,developers usually store data in denormalized form. so
> in two related collections a meta tag /denormal = True /can be

> provided. Which would make that collection a child of the other
> model ( related ). In normal scenario the collections will be
> connected using _id used as a foreign key.

> 3. providing contrib.mongo with Field such as GridFSField for providing
> support for GridFS
> 4. considering the database transaction transaction blocking ( PyMongo

> provides blocking api for mongo )

> 5. creating Mixins such as /FlexibleMixin /which would allow Models to

> have dynamic fields. i.e directly insert fields which are not listed
> in the Model schema.
>
>
> there are lot more things (Migrations, settings etc) to consider but
> before that needed to know that I am going in the right direction or
> not?. Please suggest.
>

> --
> You received this message because you are subscribed to the Google
> Groups "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-develop...@googlegroups.com
> <mailto:django-develop...@googlegroups.com>.
> To post to this group, send email to django-d...@googlegroups.com
> <mailto:django-d...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com
> <https://groups.google.com/d/msgid/django-developers/b2477318-9252-4241-a9fe-0972acb1b1cf%40googlegroups.com?utm_medium=email&utm_source=footer>.
> For more options, visit https://groups.google.com/d/optout.

[Russell Keith-Magee]

It would also be worthwhile looking into the prior art on this topic. We had a GSoC project for non-relational data stores a few years back, and came to the conclusion that it probably wasn’t viable. You can dig into the archives to find out why.

If you’re interested in adding non-relational support to Django, my inclination is that you’ll get more traction proposing a specific backend - I outlined what would be involved in my DjangoCon US talk last year:

https://www.youtube.com/watch?v=VgM0qmpHDiE

Yours,

Russ Magee %-)

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/56E88390.40005%40tinbrain.net.

[Anssi Kääriäinen]

There are two goals when it comes to document DB support. First goal
is to allow usage of models written specifically for document oriented
databases. This is something that should be possible to achieve even
if there are some hard problems to solve. The other goal is to allow
usage of any Django model with document oriented databases just like
you would use it with relational databases. This goal is extremely
hard to achieve.

The reason the latter one is hard is that for example ForeignKey and
ManyToManyField do not make much sense with document oriented
databases. In addition most Django models use auto-incrementing
primary keys, and pretty much no distributed database supports those
properly. So, if you take a random Django model (say
django.contrib.auth.models.Permission) it will not be easy to fit that
model into document oriented databases. If I recall correctly, one of
the big problems for the GSoC project Russell mentioned was support
for any Django model with document oriented databases.

I believe there would be lots of uses for models which are written
specifically for document oriented databases. So +1 to the general
idea.

- Anssi

> https://groups.google.com/d/msgid/django-developers/CAJxq84_UfN8OOXOPCTXy0i43s63b51yRP1%3DYdSNHX_yc9qciFg%40mail.gmail.com.

[girish ramnani]

In the beginning I thought about making an api which supports column databases such as cassendra but would mean that i would need to first find commonalities between all 3 ( column , document and relational ) but such commonalities would be lesser and so finally i would require to create a separate api instead of trying to provide it with db.Model.

So i will be target only document based databases like mongo , rethink etc.

There is an opinion I seek from the community.

1 ) Providing a separate Model written specifically for document database would provide easier addition of new features related to real time capabilities which could be used after ASGI gets merged. Also a developer usually knows before writing any code which type of database (relational or non-relational ) so in my opinion there wouldn't be developers who would use sqlite in dev mode and then change it to mongo in production. So would it be better to provide a separate Model for document databases ?

2 ) On the other hand , using the db.Model will be easier to use as the api is already familiar with the developers but using the same api suggests a certain level of similarity between databases which cant be seen in case of relational and document databases.
 
So would the community be open to adding a seperate Model class eg. DocumentModel or Collection .

There are alot of problems which are to be address ( some mentioned by Anssi Kääriäinen ).

The multiple database support,Forms,migrations,Meta,django admin , django auth etc. I would try to address the problems in my proposal as in-depth as possible.

> <mailto:django-developers+unsub...@googlegroups.com>.