[GSOC] Query Refactor Final Status Update
Alex Gaynor
With this past week GSOC has officially come to it's close, and I'm
here to report on the status of the query-refactor. The original
purpose of this branch was to do refactorings to the internals of the
ORM, and produce a prototype backend for a non-relational database to
demonstrate that this was a viable option. At this time far more work
has gone into the latter half of the project, I have developed a fully
functioning MongoDB backend, that demonstrates that possibility of
using the ORM, almost unmodified on non-relational databases.
However, some of the larger refactors that I was originally hoping to
do have ultimately not happened, on the other hand they are
evidentially not necessary for a functioning backend. At this time
there are a number of outstanding tasks, such as: porting the
ListField to work on Postgres, and completing the work on embedded
documents. However the largest open question is what of this work
should be merged into trunk, and what should live external. My
recommendation would be for any changes in Django itself to be merged,
including the new form fields, but for the MongoDB backend (and,
indeed, any future backends) to live external to Django, until such a
time as it obtains a user base anywhere approaching our current
backends, as well as a more individuals dedicated to maintaining it.
Don't hesitate to flame'ly yours,
Alex
--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me
Josh Ourisman
Hey Alex,
Thanks for the great work on this! I haven't had a chance to check it out in a while, but looking forward to diving back in when my schedule allows. I did start looking into porting list fields to postgres, but haven't made much progress yet.
Personally I think your recommendation re merging makes a lot of sense, as much as I would like to see the backend make it into Django. But I'd definitely be interested in helping to finish up the outstanding tasks such as embedded documents. Have you had a chance to look at existing solutions such as mongoengine for the possibility of reusing some of the rework done there? If nothing else it certainly provides a good, working proof of concept.
Cheers,
Josh
--
Josh Ourisman
jo...@joshourisman.com
301-244-9674
Jacob Kaplan-Moss
> My
> recommendation would be for any changes in Django itself to be merged,
> including the new form fields, but for the MongoDB backend (and,
> indeed, any future backends) to live external to Django,
I agree. I'd like to see the first step towards NoSQL support be
Django supporting, but not shipping with, NoSQL backends.
Jacob
Waldemar Kornewald
On Aug 21, 9:01 am, Alex Gaynor <alex.gay...@gmail.com> wrote:
> However the largest open question is what of this work
> should be merged into trunk, and what should live external. My
> recommendation would be for any changes in Django itself to be merged,
> including the new form fields, but for the MongoDB backend (and,
> indeed, any future backends) to live external to Django, until such a
> time as it obtains a user base anywhere approaching our current
> backends, as well as a more individuals dedicated to maintaining it.
merging into Django: There are still a few open design issues:
http://github.com/alex/django/blob/query-refactor/MERGE_CONSIDERATIONS.txt
IMHO, at least the AutoField issue should be dealt with before merging
into trunk, so the API for end-users is finished.
Ideally, also upsert support would be added (it's very easy to do,
anyway).
At some point we'll also need a solution for delegating the deletion
of related objects to the backend. This is needed at least for App
Engine, probably also for HBase, and maybe for some other DBs with
transaction support.
Bye,
Waldemar Kornewald
FlaPer87
I agree about leaving the backend outside django too (IMHO, all
backends should live outside django). Is there any link that points to
the apps you used for tests and to the mongodb backend?
As you might know, We've been working on a django_mongodb_engine that
uses django-nonrel and djangotoolbox. It currently works well (we use
it in production) and it would be great to integrate part of our work
with yours if it is possible.
As Waldemar said, the AutoField issue should be dealt before merging
and I would suggest to leave the non standard AutoField definition to
the backends and then adding an attr into the Meta class that
specifies the class that should be used for AutoFields in case it
isn't the standard one. For example:
class Mymongo(models.Model):
.....
class Meta:
auto_field_class = MyMongodbAutofieldClass
IMHO, this could be an easy and fast way to deal with the AutoField
issue.
On 22 ago, 11:18, Waldemar Kornewald <wkornew...@gmail.com> wrote:
>
> At some point we'll also need a solution for delegating the deletion
> of related objects to the backend. This is needed at least for App
> Engine, probably also for HBase, and maybe for some other DBs with
> transaction support.
Cheers
Flavio Percoco Premoli
Waldemar Kornewald
On Sun, Aug 22, 2010 at 12:40 PM, FlaPer87 <flap...@gmail.com> wrote:
> Hi Alex,
>
> I agree about leaving the backend outside django too (IMHO, all
> backends should live outside django). Is there any link that points to
> the apps you used for tests and to the mongodb backend?
>
> As you might know, We've been working on a django_mongodb_engine that
> uses django-nonrel and djangotoolbox. It currently works well (we use
> it in production) and it would be great to integrate part of our work
> with yours if it is possible.
Yes, you should definitely do that. I already told Alex that the
MongoDB backend has to merge filters sometimes. For example, I think
Alex' MongoDB backend won't work with filter(x__gt=3, x__lt=5), please
correct me if I'm wrong. I implemented support for merging filters in
django_mongodb_engine, so you could reuse at least that.
Apart from that I just remembered a little issue in the current ORM:
is_null queries on ForeignKey generate code which requires a JOIN.
This inefficiency should probably also be fixed before 1.3 stable gets
out. There is an issue for this, already:
http://code.djangoproject.com/ticket/10790
> As Waldemar said, the AutoField issue should be dealt before merging
> and I would suggest to leave the non standard AutoField definition to
> the backends and then adding an attr into the Meta class that
> specifies the class that should be used for AutoFields in case it
> isn't the standard one. For example:
>
> class Mymongo(models.Model):
> .....
>
> class Meta:
> auto_field_class = MyMongodbAutofieldClass
>
> IMHO, this could be an easy and fast way to deal with the AutoField
> issue.
Instead of specifying the AutoField class in Meta you can just use
class Mymongo(models.Model):
id = MyMongodbAutofieldClass()
...
Apart from that, we'd need a plan for maintaining
backward-compatibility because an AutoField that supports string
values has a different validation behavior which might break existing
code. Russell already suggested to check if the user only has SQL
backends and then handle validation accordingly. During the transition
phase a warning would be shown during AutoField validation errors,
telling the user that the validation behavior will change in a future
release and he has to modify his code. If someone uses a NoSQL backend
the new validation behavior would be enabled automatically.
Bye,
Waldemar Kornewald
--
Django on App Engine, MongoDB, ...? Browser-side Python? It's open-source:
http://www.allbuttonspressed.com/blog/django
Russell Keith-Magee
The AutoField issue is the biggest impediment I see to merging code to
trunk. Most of the other changes Alex has proposed are fairly
innocuous (and, with hindsight, probably should have been done as part
of the multi-db changes). Hopefully we'll get a chance to thrash out
these issues at DjangoCon.
Alex - congratulations on another completed Summer of Code, and thanks
for your hard work over the summer!
Yours,
Russ Magee %-)