Some bugs in manage multidb

[Alberto Paro]

I'm developing a big application that does some complex mixing of database: SQL and notSQL one.

I'm using the multidb to manage all the stuff in a django manner, using routing to route items in MongoDB and Postgresql.

Some bugs that I've discovered:

1) django is unable to manage different Autofields from integer ones. A patch is to extend the BaseDatabaseOperations

 to manage different types from default one. (Patch available here: http://github.com/aparo/django/commit/62baf86ff872b3a271619a4327d7b08180c89337)

2) the mixing of foreignkey in different backends are not managed correctly (there is a note on documentation on multidb page). A patch that check convert the type of foreign key with expected type is here: http://github.com/aparo/django/commit/edcdc1d9364224fcbc3b810b9d9fa19a10cd537c only the modification to related.py is important.

3) foreign key search and saving in different backends are buggy due to "hints" caching. Disabled cache choose of database on instance (http://github.com/aparo/django/commit/b1c8478f45997df22b3a477c8d3f382613d69246)

1-2) depends that mongodb use string as autofield type

3) django calls the wrong backend if no route is specified for a model and not the default one.

Do I need to open a ticket on trac to commit these patch or we need first to discuss them?

Regarding support to nosql engines, we are able to mix a lot of models between mongodb and postgresql using are backend engine (http://github.com/aparo/django-mongodb-engine) that works also on the current trunk django 1.2 and using routing massively.

I think that there are datamodels that works very well on nonrel db and other that are only suitable for sql: it depends on designer to choose the good solution for a problem.

Congratulations for the 1.2 release.

Hi,

   Alberto Paro

--
You received this message because you are subscribed to the Google Groups "Django developers" group.
To post to this group, send email to django-d...@googlegroups.com.
To unsubscribe from this group, send email to django-develop...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

[Russell Keith-Magee]

On Tue, May 18, 2010 at 5:08 PM, Alberto Paro <albert...@gmail.com> wrote:
> I'm developing a big application that does some complex mixing of database:
> SQL and notSQL one.
> I'm using the multidb to manage all the stuff in a django manner, using
> routing to route items in MongoDB and Postgresql.
> Some bugs that I've discovered:

As a point of order, when something we don't support doesn't work,
it's not a bug - it's a feature request.

> 1) django is unable to manage different Autofields from integer ones. A
> patch is to extend the BaseDatabaseOperations
>  to manage different types from default one. (Patch available here:
> http://github.com/aparo/django/commit/62baf86ff872b3a271619a4327d7b08180c89337)
> 2) the mixing of foreignkey in different backends are not managed correctly
> (there is a note on documentation on multidb page). A patch that check
> convert the type of foreign key with expected type is
> here: http://github.com/aparo/django/commit/edcdc1d9364224fcbc3b810b9d9fa19a10cd537c
> only the modification to related.py is important.
> 3) foreign key search and saving in different backends are buggy due to
> "hints" caching. Disabled cache choose of database on instance
> (http://github.com/aparo/django/commit/b1c8478f45997df22b3a477c8d3f382613d69246)
> 1-2) depends that mongodb use string as autofield type
> 3) django calls the wrong backend if no route is specified for a model and
> not the default one.
> Do I need to open a ticket on trac to commit these patch or we need first to
> discuss them?

In this case, what you're actually proposing is changes that are part
of the way to implementing a noSQL/MongoDB backend to Django. That is
something that will require some additional discussion.

Some of this discussion has already happened. If you search the
archives, you will find many references to NoSQL support (Google
AppEngine and MongoDB are particular favorites). These discussions
have revealed that adding NoSQL support isn't just a matter of some
minor tweaks to AutoField handling; There are some major plumbing
changes that are required.

It's also worth noting that Alex Gaynor has been accepted as a Google
Summer of Code student, and his project is to start work on making the
modifications to the database backend that are required in order to
support NoSQL. He has specifically indicated that MongoDB will be the
backend he is initially targetting.

Regarding the specific proposed fixes: It's very hard to take changes
like this seriously when they aren't accompanied by tests. For
example, change (3) comments out code that exists for a specific
reason. If you run the test suite with that change applied, the
multiple_database tests have 13 failures and 2 errors. Tests aren't an
optional extra -- they're how you prove that you've actually thought
about the problem you're trying to solve.

Yours,
Russ Magee %-)

[Waldemar Kornewald]

Hi Russell,

On May 18, 1:59 pm, Russell Keith-Magee <russ...@keith-magee.com>
wrote:

> On Tue, May 18, 2010 at 5:08 PM, Alberto Paro <alberto.p...@gmail.com> wrote:
> > I'm developing a big application that does some complex mixing of database:
> > SQL and notSQL one.
> > I'm using the multidb to manage all the stuff in a django manner, using
> > routing to route items in MongoDB and Postgresql.
> > Some bugs that I've discovered:
>
> As a point of order, when something we don't support doesn't work,
> it's not a bug - it's a feature request.

At least the ForeignKey bug could be seen as a real bug because it
indeed doesn't use the correct backend. I'm sure you could reproduce
the bug with some other primary_key field that requires backend
interaction (e.g., DecimalField, though this might not make a nice
primary key ;).

> Some of this discussion has already happened. If you search the
> archives, you will find many references to NoSQL support (Google
> AppEngine and MongoDB are particular favorites). These discussions
> have revealed that adding NoSQL support isn't just a matter of some
> minor tweaks to AutoField handling; There are some major plumbing
> changes that are required.

Actually, I've finished the MongoDB backend and it works really well
on Django-nonrel (the patch still hasn't been committed to Alberto's
repository, though). For example, our website (http://
www.allbuttonspressed.com/) which we originally wrote for the App
Engine backend works unmodified on the MongoDB backend (including the
admin UI, BTW) and of course it also works with the SQL backends. All
this still only needs a few minor modifications to Django. Of course,
a cleanup and refactoring of the ORM would be useful to make more
advanced features like JOIN emulation possible (we haven't really
finished that discussion, BTW).

Bye,
Waldemar

[Joshua Russo]

Your solution may work for your situation but you still need to prove it in the test suite, to ensure that it will work in all known scenarios. It's too easy to step on the toes of existing logic.

[Waldemar Kornewald]

On Thu, May 20, 2010 at 2:11 AM, Joshua Russo <josh.r...@gmail.com> wrote:
>> Actually, I've finished the MongoDB backend and it works really well
>> on Django-nonrel (the patch still hasn't been committed to Alberto's
>> repository, though). For example, our website (http://
>> www.allbuttonspressed.com/) which we originally wrote for the App
>> Engine backend works unmodified on the MongoDB backend (including the
>> admin UI, BTW) and of course it also works with the SQL backends. All
>> this still only needs a few minor modifications to Django. Of course,
>> a cleanup and refactoring of the ORM would be useful to make more
>> advanced features like JOIN emulation possible (we haven't really
>> finished that discussion, BTW).
>
> Your solution may work for your situation but you still need to prove it in
> the test suite, to ensure that it will work in all known scenarios. It's too
> easy to step on the toes of existing logic.

Actually, with Django-nonrel all tests pass except for one type of
test: now that our AutoField accepts strings all unit tests that
expect AutoField to raise an exception when you pass a string instead
of an integer will of course fail. So, the actual problem is that the
unit tests don't reflect reality, anymore. We'll leave it at that
until a decision has been made on the direction of NoSQL support in
Django.

Anyway, what really matters is that with the current Django-nonrel
feature set you could easily also support SimpleDB, CouchDB, Redis,
and other backends. I don't see any missing features that stand in the
way of achieving that goal. I hope our work can at least be used as a
starting point for the GSoC NoSQL project.

Bye,
Waldemar Kornewald