Having a MongoDB connector for Django

[Nes Dis]

Hello

I am wondering what is the state of the art on Django having a backend connector for MongoDB database backend. There are a few solutions out there but they don't work as expected. 

A possible solution for this is to have a connector which translates SQL queries created in Django, into MongoDB queries.

I would like to hear the expert opinion from the esteemed members of this group on this concept.

A working solution for this can be found here: djongo. (Django + Mongo = Djongo) The project is hosted on github.

Regards

Nes Dis

[Tom Evans]

Short answer: always use the appropriate tool

Relational databases and document stores have different uses and
purposes. Using a document store like a relational database (eg, with
an ORM (emphasis on the R)) is a bad idea, and using a relational
database as a document store is similarly foolish.

Work out what questions you want to ask of your data, then structure
the data in a way that allows you to query it efficiently.

If the format desired is a document store, I wouldn't attempt to
shoehorn that in to an ORM wrapper, I'd use something like mongothon.

Cheers

Tom

> --
> You received this message because you are subscribed to the Google Groups
> "Django developers (Contributions to Django itself)" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.
> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b0ce04d1-62cb-4765-b850-06c4a5b0607f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Adam Johnson]

I agree, I think forcing Django's ORM to work on MongoDB is not a great idea. Django relies heavily on transactions and other relational goodness.

Have you tried storing JSON in your Postgres/MySQL database? Django can work with that with contrib.postgres/django-mysql 😉

> email to django-developers+unsubscribe@googlegroups.com.
> To post to this group, send email to django-developers@googlegroups.com.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b0ce04d1-62cb-4765-b850-06c4a5b0607f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFHbX1%2BNpZuWrAgcMwkcJQ4Xg4TrM700JRsYV_AnW05_9L3joA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[Tom Forbes]

JSON support in postgres is a great middle ground for storing unstructured documents. 

I don't know the specifics of your application, but I would be surprised if a document orientated database is a perfect fit. The majority of such apps are semi-structured, where you have some relations but also some unstructured data. Postgres is an absolutely perfect fit for this.

On 8 Sep 2017 17:57, "Adam Johnson" <m...@adamj.eu> wrote:

I agree, I think forcing Django's ORM to work on MongoDB is not a great idea. Django relies heavily on transactions and other relational goodness.

Have you tried storing JSON in your Postgres/MySQL database? Django can work with that with contrib.postgres/django-mysql 😉

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1YMFUT_oSGYuj-2Uqu_BSteqFbb_84Uz-zMauoE-m6Kg%40mail.gmail.com.

[Michael Manfre]

Another voice piling on to the "this is not a good idea" train. I inherited a Mongo backed project at my day job that is filled with data that should have been put in a relational database. Mongo is the wrong tool for the job (and we're migrating off it). I tried putting an ORM in front of it to make things a bit easier, but I gave up.  My many years of supporting a 3rd party database backend for Django gave me the insight to realize that it would take a lot of effort and ultimately would probably never work as well as migrating the data to a more appropriate database. It was definitely a lot of fun to tinker on it, but not something I'd ever want to support in production.

Regards,

Michael Manfre

> email to django-develop...@googlegroups.com.
> To post to this group, send email to django-d...@googlegroups.com.

> Visit this group at https://groups.google.com/group/django-developers.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-developers/b0ce04d1-62cb-4765-b850-06c4a5b0607f%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers  (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFHbX1%2BNpZuWrAgcMwkcJQ4Xg4TrM700JRsYV_AnW05_9L3joA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAMyDDM1YMFUT_oSGYuj-2Uqu_BSteqFbb_84Uz-zMauoE-m6Kg%40mail.gmail.com.

[Nes Dis]

If I can summarize the different thoughts for having a Django connector for nosql db like Mongodb:

1.  Don't mix up  relational database with a document store database
   
• Document store DBs also have syntax for performing relation lookups within them. Possibly complex relational operations are not supported but lookups from one table/collection to another is possible
2. Use a completely new wrapper to deal with non-rel DBs
• That's a bit ridiculous, because now I have to reinvent Django ORM, contrib packages and everything under the sun.
3. Use 2 DBs one for contrib packages other for website specific data.
• Managing 2 DBs is a troublesome marriage.
4. Translating SQL to Mongodb syntax is not easy/ Impossible.
• It's not easy, but has been done: djongo. SQL DBs have a lot of features, SQLite has very few, Django can work with DBs that support or don't support transactions.

Does it matter to the end user, what the underlying DB really is? By adding support for other DBs into Django, it's accessible to larger user base. 

[Adam Johnson]

That's a bit ridiculous, because now I have to reinvent Django ORM, contrib packages and everything under the sun.

 

Django can work with DBs that support or don't support transactions. 

I don't know where you get the impression that Django can work with transactionless DB's from. Many of Django's features assume transactional support, e.g. the ORM features update(), delete(), get_or_create(), many-to-many fields, etc. By extension I think the contrib packages all rely on them in some way.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/4826023b-c697-4a0e-b43d-476ad9927673%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Adam

[Aymeric Augustin]

Hello,

This discussions seems to have started with a misunderstanding of what Django is.

Here's a very incomplete list (in advance, apologies to everyone for everything that I'm inevitably going to omit)

1. The Django community

1.a. Django Girls

1.b. Conferences, events

1.c. Processes for keeping the community healthy

1.d. Mailing lists, blogs, SO

etc.

2. The Django ecosystem

2.a. Third party apps

2.b. Books

2.c. Hosting services

2.d. Consulting services

etc.

3. The Django framework

See https://docs.djangoproject.com/en/1.11/ — I'm not going to repeat it all; the ORM is just one of many parts.

Some choices are necessary to keep all this manageable. "We focus on building apps on top of SQL databases" is one of these choices.

The starting point of the discussion was "I'm using MongoDB". Well, this means Django isn't designed to handle your situation. Perhaps you can use it, but you aren't going to get the full benefits.

One of the tradeoffs that you (or someone else) made by picking MongoDB is that you can't use much of the Django ecosystem. You can still use the view and template layers and about half of forms. You can't use the ORM nor anything that depends on it.

At this time, I don't know many people actively involved in Django who thinks it's reasonable and useful to expand the ORM to support MongoDB.

The consensus about "reasonable" is a steady "no". See discussions in the archives of this mailing list or take a closer look at the ORM to understand why.

I think there's a debate aboute "useful". Some people run (or have run) Django on top of MongoDB with various third-party connectors but I haven't heard success stories. (I'd be interested in hearing such stories.)

I see only one way for this situation to change: if someone builds a robust MongoDB backend that works well enough in practice and if it becomes vastly successful. Perhaps that's djongo. There are technical reasons to believe this won't happen, though.

I hope this helps.

-- 

Aymeric.

[Shai Berger]

I would like to add to this discussion just the clarification that, when we say
Django relies on transactions, we don't mean that it is impossible to write a
database backend which doesn't implement them -- but that Django assumes
transactional behavior to protect data integrity. It can work without
transactions, for a very superficial definition of "work", but that operation
will be prone to horrible race conditions and data losses. Similar disclaimers
are valid for several other relational features.

Shai.

[Nes Dis]

Thank you all for your very insightful comments. I personally am a big user/contributor to the framework myself and would like to see it thrive and progress with respect to other competing frameworks.

I am sure most are aware, of this argument about MongoDB increasing in popularity. Several members are of the opinion that not supporting this backend (which in my opinion is not too difficult) will not dent Django's popularity. 

In conclusion, I do hope this is the right decision for Django will take it in the correct direction!

Regards

Nesdis

On Friday, 8 September 2017 18:33:59 UTC+5:30, Nes Dis wrote:

[Josh Smeaton]

I'm in agreement with most of the others - that Django should not strive to achieve compatibility with a bunch of different database types simply to open itself up to a larger user base. The cost you pay is a lack of functionality that takes advantage of the specific features granted by a particular technology choice.

Joins and their different types (LEFT/FULL) are an integral part of the assumptions made about the ORM, as are aggregations, grouping, relationships, indexes, migrations. As you can see with most attempts to shoehorn non-relationship databases into django ORM backends, you lose a lot of functionality. If Django were to adopt these backends as official, then we're stuck in one of two positions. The first is that we hamstring all databases to achieve the lowest common feature set. The second is that we have a bunch of feature flags and documentation that work around the incompatibilities between technologies. Neither of these options are attractive.

If you want to use a non-relational database, that's fine. Django doesn't specifically cater to those technology choices or users, and that's ok. There are other frameworks one may use.

[Anssi Kääriäinen]

My 2 cents on this...

I don't think it would be an unachievable goal to write a MongoDB backend for Django's ORM. However, it wouldn't support any relational feature, and likely would also need to skip support for some other common features, for example AutoField is actually hard to support on MongoDB. The Django ecosystem is very much written on an assumption that the underlying datastore is relational, so this would mean that even if you have MongoDB backend, you can't use it with most contrib models for example.

Even if you can't use the backend with the full ecosystem, such a backend might be very useful for some use cases. My guess is that the most common case would be usage of MongoDB as an additional data store along your relational database.

I believe that most users requesting a MongoDB backend actually would want to see a backend which is a drop in replacement for the SQL backends. Unfortunately, due to the differences in optimal database schema design between relational and document oriented data stores, this is both a bad idea, and almost impossible to implement.

- Anssi

[Patryk Zawadzki]

W dniu poniedziałek, 11 września 2017 05:28:55 UTC+2 użytkownik Nes Dis napisał:

Thank you all for your very insightful comments. I personally am a big user/contributor to the framework myself and would like to see it thrive and progress with respect to other competing frameworks.

I am sure most are aware, of this argument about MongoDB increasing in popularity. Several members are of the opinion that not supporting this backend (which in my opinion is not too difficult) will not dent Django's popularity. 

This is my personal opinion so take with a grain of salt but I believe MongoDB is popular among two very different groups of people:

1) those who think convenience of project setup (ie. I can bootstrap my project within 15 minutes) and believe that going schema-less is saving them time

2) those who deal with terabytes upon terabytes of unstructured data

During my career I've been part of both groups.

All projects that fell under group 1 had to eventually migrate to SQL. MongoDB is easy to setup and convenient to use for proof-of-concept toys but it was built with a very specific goal in mind: to handle data at SCALE. This means there are no transactions (you can't lock data when transactions can take hours to complete), there is no atomicity (all updates are distributed and row updates happen in parallel, you can easily end up with partial successes where some rows are updated and some raise errors, there's no ghost read protection as all updates are final), there is also no way for the database to reject invalid data (as there is no schema enforcement, the only rule is: junk in; junk out). Whatever resources you save by initially choosing MongoDB you pay for ten-fold each time you encounter a problem that is easily solvable in the ACID RDBMS world and that just does not exist in group 2 projects.

All projects that fell under group 2 had to build their logic with MongoDB's architecture in mind. To take advantage of the scalability you need to carefully craft all of your commands. You only select data that you know you'll absolutely need (which means that you do actually have multiple implicit schemas or "interfaces" certain rows conform to), you maintain a fleet of background jobs that recalculate certain denormalizations to keep them in approximate sync with data (as updates can and do happen in parallel and recalculating aggregate queries on terabytes of data can be prohibitively costly), and you do complex updates by sending a JavaScript function along with the query instead of fetching millions of rows and issuing millions of update commands. There would be no advantage for us to be able to use a ModelForm to manage schema-less documents.

Cheers

[Santiago Fraire Willemoës]

I agree with everyone, no relation with relational should not be mixed because they solve different problems.

On the other hand, having an Object Document Manager (instead of an ORM) if required could be a possibility, right? 

I think until a lot of people have that need, it won't happen. 

I've used https://github.com/MongoEngine/mongoengine and it felt like using django's ORM

Regards,

Santiago

[Nes Dis]

I would like to thank everyone for their valuable comments. Simultaneously I would like to comment on some conceptions regarding using MongoDB. Its not accurate to state that relational joins cannot happen in MongoDB. It can be done at the application level. LEFT JOIN and INNER JOIN. A detailed description of this is available.

Admin modules rely heavily on SQL, So cannot be ported to MongoDB. But admin porting to MongoDB is working successfully in djongo.

Regards

Nes Dis

On Friday, 8 September 2017 18:33:59 UTC+5:30, Nes Dis wrote:

[Tom Forbes]

You're right for pointing out that some elements of a relational database can be emulated in a non relational one, like Mongodb. The in-python joins sound s bit scary though.

However I would argue that if you find yourself needing such things your data is relational, and therefore a relational database is perhaps a better fit than a non relational one.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/e0ac0919-0ae6-4a42-ba7c-c558bc42d141%40googlegroups.com.

[Nes Dis]

Hello

I would like to add that the application level joins are scripted such that they are atomic in nature. Which means if there is another thread/process operating in parallel, It will have no impact on final output. Is this why you feel an application join is scary?

Regards

Nes Dis

On Friday, 8 September 2017 18:33:59 UTC+5:30, Nes Dis wrote:

[Tom Evans]

On Fri, Sep 15, 2017 at 6:44 PM, Nes Dis <nes...@gmail.com> wrote:
> I would like to thank everyone for their valuable comments. Simultaneously I
> would like to comment on some conceptions regarding using MongoDB. Its not
> accurate to state that relational joins cannot happen in MongoDB. It can be
> done at the application level. LEFT JOIN and INNER JOIN. A detailed
> description of this is available.

I don't think anyone has said that this cannot be done, but instead
that it is not desirable to do so.

A django ORM adapter that supports Mongo would be used by people who
would not understand that, whilst you can do relational things using
mongo, it will not perform well.

In an earlier email you said:

> 2) Use a completely new wrapper to deal with non-rel DBs

> That's a bit ridiculous, because now I have to reinvent Django ORM,
> contrib packages and everything under the sun.

This is precisely the problem. If a mongodb connector was provided for
Django's ORM, people would expect to be able to use it with all these
packages that are designed to work with relational databases, and will
be annoyed when the project is slow once real amounts of data are
added.

If you want relational data, use a relational database, and use an
object relational mapping API to access it. If you want document data,
use a document store, and use an object document mapping API to access
it (PyMongo provides an ODM).

If you want both, choose one as your primary data store (I prefer
relational), all changes go through there. On modifying the primary
store, update the secondary store, according to your business
requirements (eg, instantly, batched ETL, daily, etc)

Cheers

Tom

[Nes Dis]

I completely agree with you! Having a connector for the backend DB means things will be slow. This must be indicated to the end-user. The engineer must whether having two dbs is a better fit or a single one with some trade-offs.

Regards

Nes Dis