Alex Gaynor's GSoC project to add multiple database support to Django
is on the final straight. The only piece of the puzzle that is left is
updating contrib.gis - but this hopefully won't require any major
changes outside of the gis tree itself.
Therefore, I'd like to call for feedback and testing of the branch.
This is a very big change to the query engine, but unless your code
relies on the internals of sql.Query, you *should* be able to switch
to using the branch without making any changes to your code. The
release notes give the details of the changes that you need to be
aware of:
If you want to run Django's test suite, you will need to make some
modifications to your test settings file. If you don't make these
changes, you will get failures in the multiple_databases tests. See
the relevant section in the contribution docs for details on the
changes you need to make:
To the best of our knowledge, the full test suite is currently passing
for all databases (except for those known pre-existing failures under
MySQL InnoDB and Oracle). Any discoveries to the contrary will be
gratefully received.
If you actually want to try using multiple databases, basic usage
instructions are here:
What should you expect to see? Well, a while back, Malcolm described
Django's query framework as having the plumbing, but not the porcelain
for multiple database support. At the risk of stretching the metaphor,
this branch adds the porcelain, but not the polished oak seat to go on
top. The settings and API to define multiple databases and to direct
queries to a specific database have all been added by this branch.
However, there isn't any explicit API level support for the common use
cases for multiple databases like sharding or master/slave setups. I
see adding these wrappers as tasks for the broader community, possibly
with a view to inclusion in trunk in v1.3 or later.
So - feedback welcome. It's a big patch, so the more testing we can
get pre-merge, the better.
Also, I hope you won't define an API for overriding the query class in
the multi-db branch. We'll most likely have to define our own API in
the non-relational branch.
Bye,
Waldemar Kornewald
On Thu, Dec 3, 2009 at 5:10 PM, Russell Keith-Magee
> Alex Gaynor's GSoC project to add multiple database support to Django
> is on the final straight. The only piece of the puzzle that is left is
> updating contrib.gis - but this hopefully won't require any major
> changes outside of the gis tree itself.
> Therefore, I'd like to call for feedback and testing of the branch.
> This is a very big change to the query engine, but unless your code
> relies on the internals of sql.Query, you *should* be able to switch
> to using the branch without making any changes to your code. The
> release notes give the details of the changes that you need to be
> aware of:
> If you want to run Django's test suite, you will need to make some
> modifications to your test settings file. If you don't make these
> changes, you will get failures in the multiple_databases tests. See
> the relevant section in the contribution docs for details on the
> changes you need to make:
> To the best of our knowledge, the full test suite is currently passing
> for all databases (except for those known pre-existing failures under
> MySQL InnoDB and Oracle). Any discoveries to the contrary will be
> gratefully received.
> If you actually want to try using multiple databases, basic usage
> instructions are here:
> What should you expect to see? Well, a while back, Malcolm described
> Django's query framework as having the plumbing, but not the porcelain
> for multiple database support. At the risk of stretching the metaphor,
> this branch adds the porcelain, but not the polished oak seat to go on
> top. The settings and API to define multiple databases and to direct
> queries to a specific database have all been added by this branch.
> However, there isn't any explicit API level support for the common use
> cases for multiple databases like sharding or master/slave setups. I
> see adding these wrappers as tasks for the broader community, possibly
> with a view to inclusion in trunk in v1.3 or later.
> So - feedback welcome. It's a big patch, so the more testing we can
> get pre-merge, the better.
> Yours,
> Russ Magee %-)
> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to django-developers+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
You are correct. Thanks for that - I've made the change on github.
> Also, I hope you won't define an API for overriding the query class in
> the multi-db branch. We'll most likely have to define our own API in
> the non-relational branch.
We haven't modified the API for overriding the query class beyond the
techniques that already existed to support GIS (i.e., passing a Query
class into the QuerySet itself).
On Dec 3, 4:10 pm, Russell Keith-Magee <freakboy3...@gmail.com> wrote:
> Alex Gaynor's GSoC project to add multiple database support to Django
> is on the final straight. The only piece of the puzzle that is left is
> updating contrib.gis - but this hopefully won't require any major
> changes outside of the gis tree itself.
> Therefore, I'd like to call for feedback and testing of the branch.
I tried this out yesterday - it's excellent. The combination of the
new DATABASES setting and QuerySet.using() turns out to be exactly
enough to get all sorts of interesting work done. It neatly fulfils
the three requirements set out in the original ticket (replication /
sharding / hosting different applications on different databases).
A couple of observations. Firstly, there's a gotcha involving saving
existing objects to a new database:
e1 = Entry.objects.using('db1').create(title = "A blog post",
body="...")
e2 = Entry.objects.using('db1').create(title = "Another blog post",
body="...")
e3 = Entry.objects.using('db2').create(title = "Another blog post",
body="...")
e4 = Entry.objects.using('db2').create(title = "Another blog post",
body="...")
We now have two blog posts in each database. But, if I do the
following:
e1.save(using = 'db2')
It has the effect of replacing e3 with the values from e2, since the
primary keys collide with each other.
This behaviour makes sense if you think about it for more than a
moment, but still probably counts as a gotcha. Worth mentioning in the
documentation at least.
I could see this pattern biting people since that's the obvious way to
copy an object from one database to another.
Secondly, a question: Is there an idea for how this might be made to
work with the admin interface? I was asked this question at DJUGL last
night and wasn't sure of the answer.
Fantastic work on this, very excited to see it merged to trunk.
On Dec 4, 8:38 am, Simon Willison <si...@simonwillison.net> wrote:
> We now have two blog posts in each database. But, if I do the
> following:
> e1.save(using = 'db2')
> It has the effect of replacing e3 with the values from e2, since the
> primary keys collide with each other.
I meant to say - the fix is clearly to call save() with the
force_insert=True argument, but it's easy to miss that. Worth
explicitly highlighting in the multi-db docs I think.
On Fri, Dec 4, 2009 at 4:38 PM, Simon Willison <si...@simonwillison.net> wrote:
> On Dec 3, 4:10 pm, Russell Keith-Magee <freakboy3...@gmail.com> wrote:
>> Alex Gaynor's GSoC project to add multiple database support to Django
>> is on the final straight. The only piece of the puzzle that is left is
>> updating contrib.gis - but this hopefully won't require any major
>> changes outside of the gis tree itself.
>> Therefore, I'd like to call for feedback and testing of the branch.
> I tried this out yesterday - it's excellent. The combination of the
> new DATABASES setting and QuerySet.using() turns out to be exactly
> enough to get all sorts of interesting work done. It neatly fulfils
> the three requirements set out in the original ticket (replication /
> sharding / hosting different applications on different databases).
> A couple of observations. Firstly, there's a gotcha involving saving
> existing objects to a new database:
> e1 = Entry.objects.using('db1').create(title = "A blog post",
> body="...")
> e2 = Entry.objects.using('db1').create(title = "Another blog post",
> body="...")
> e3 = Entry.objects.using('db2').create(title = "Another blog post",
> body="...")
> e4 = Entry.objects.using('db2').create(title = "Another blog post",
> body="...")
> We now have two blog posts in each database. But, if I do the
> following:
> e1.save(using = 'db2')
> It has the effect of replacing e3 with the values from e2, since the
> primary keys collide with each other.
> This behaviour makes sense if you think about it for more than a
> moment, but still probably counts as a gotcha. Worth mentioning in the
> documentation at least.
> I could see this pattern biting people since that's the obvious way to
> copy an object from one database to another.
Agreed that this could bite people fairly easily.
I'm not sure we can really do much to avoid the problem, though. We
have a few options, but none of them are really appropriate for all
circumstances.
A time of object save, we know the database that the object is
currently on (using _state), and we know if the user has requested a
save on a different database (by comparing with the using argument).
Based on this detail, we could:
1) Do nothing special. This is the currently implement behaviour.
2) Set the primary key of the object to None so that the save on the
new database is guaranteed to be an insertion, rather than a possible
overwrite of an object with the same pk value on the second database.
3) Transform the save into a 'force insert' if the database changes.
This will raise errors if the pk is already in use on
4) Use the PK on the old database and issue a deletion before saving
to the new database.
5) Do 2 and 4.
6) Do 3 and 4
The problem is the conflict between the most appropriate behavior for
different use cases. In the case you describe (copying an object from
one database to another), (5) probably makes the most sense; however,
from the point of view of a master/slave setup, (1) is the best
behaviour.
In fact (4) by makes me a little nervous - If we start modifying
save() to make it copy objects, there will be surprise when objects
related to the object being saved aren't transferred as well. Deleting
data from the old database sounds like a recipe for some really nasty
accidental data loss.
IMHO, it's best to leave entirely in the hands of the end user. By
that reasoning, The current behaviour (1) is actually the right
solution, along with some documentation explaining the problem and
some possible solutions - essentially, the manual interpretations of
(2) and (3).
> Secondly, a question: Is there an idea for how this might be made to
> work with the admin interface? I was asked this question at DJUGL last
> night and wasn't sure of the answer.
...and this is why we put code out for review. You win the internets
for pointing out the obvious omission. For all the spelunking in
django.db internals, the potential effect on admin hadn't occurred to
me.
I can think of a couple of possible approaches to this.
1) Ignore the problem. Admin works on the default database, but
nowhere else. This is certainly less than ideal, but it would be
sufficient for master/slave setups.
2) Use a separate admin deployment for each database. We add a 'using'
argument to admin.Site(), and append .using() the queries that the
admin site issues.
3) Modify the admin list views so that they take a ?using=db GET
argument; also add a pulldown to make it easy to switch to a different
database.
(2) should be a fairly minor change; (3) would be a little more
effort, but shouldn't be impossible. There is also the need for some
mechanics to check whether a table is actually present on a database -
just because auth is in INSTALLED_APPS doesn't mean it's been
synchronized to a particular database.
I'm open to any other suggestions - and for any offers to help out :-)
On Fri, Dec 4, 2009 at 7:31 PM, Russell Keith-Magee
<freakboy3...@gmail.com> wrote:
> IMHO, it's best to leave entirely in the hands of the end user. By
> that reasoning, The current behaviour (1) is actually the right
> solution, along with some documentation explaining the problem and
> some possible solutions - essentially, the manual interpretations of
> (2) and (3).
I've just added some docs to the github branch to cover this situation:
> 1) Do nothing special. This is the currently implement behaviour.
As the default, this seems to be the intended behavior only in rare
cases (overwriting a different object from the original) and will
indeed bite.
There is another option: leave in the hands of the user (as you say)
but make them explicitly say which behavior they want of the
following:
> 2) Set the primary key of the object to None so that the save on the
> new database is guaranteed to be an insertion, rather than a possible
> overwrite of an object with the same pk value on the second database.
> 3) Transform the save into a 'force insert' if the database changes.
> This will raise errors if the pk is already in use on
> 4) Use the PK on the old database and issue a deletion before saving
> to the new database.
If they didn't explicitly indicate which of these to do, then raise an error.
> 1) Ignore the problem. Admin works on the default database, but
> nowhere else. This is certainly less than ideal, but it would be
> sufficient for master/slave setups.
> 2) Use a separate admin deployment for each database. We add a 'using'
> argument to admin.Site(), and append .using() the queries that the
> admin site issues.
> 3) Modify the admin list views so that they take a ?using=db GET
> argument; also add a pulldown to make it easy to switch to a different
> database.
> (2) should be a fairly minor change; (3) would be a little more
> effort, but shouldn't be impossible. There is also the need for some
> mechanics to check whether a table is actually present on a database -
> just because auth is in INSTALLED_APPS doesn't mean it's been
> synchronized to a particular database.
> I'm open to any other suggestions - and for any offers to help out :-)
Just thinking of one more option for this -- what about specifying the
DB on the ModelAdmin level rather than the admin.Site level?
On Fri, Dec 4, 2009 at 10:18 PM, Nan <ringe...@gmail.com> wrote:
>> 1) Ignore the problem. Admin works on the default database, but
>> nowhere else. This is certainly less than ideal, but it would be
>> sufficient for master/slave setups.
>> 2) Use a separate admin deployment for each database. We add a 'using'
>> argument to admin.Site(), and append .using() the queries that the
>> admin site issues.
>> 3) Modify the admin list views so that they take a ?using=db GET
>> argument; also add a pulldown to make it easy to switch to a different
>> database.
>> (2) should be a fairly minor change; (3) would be a little more
>> effort, but shouldn't be impossible. There is also the need for some
>> mechanics to check whether a table is actually present on a database -
>> just because auth is in INSTALLED_APPS doesn't mean it's been
>> synchronized to a particular database.
>> I'm open to any other suggestions - and for any offers to help out :-)
> Just thinking of one more option for this -- what about specifying the
> DB on the ModelAdmin level rather than the admin.Site level?
This is also a possibility. Describing databases at the level of the
ModelAdmin is really only useful for the partitioning case (i.e., auth
models on database X, myapp models on database Y), but that is a use
case we are trying to hit. It shouldn't be to hard to have ModelAdmin
derive using from the site if it isn't defined locally.
Is there a page where one can find a quick summary of the proposed API?
I have some concerns about implementing partitioning through the admin, but
I expect there's something I'm missing.
For those who haven't been, following the conversation closely it'd be nice
to have a quick way to come up to speed.
Sent from a mobile phone, please excuse any typos.
On Dec 4, 2009 9:30 AM, "Russell Keith-Magee" <freakboy3...@gmail.com>
wrote:
On Fri, Dec 4, 2009 at 10:18 PM, Nan <ringe...@gmail.com> wrote: > >> 1)
Ignore the problem. Admin w...
This is also a possibility. Describing databases at the level of the
ModelAdmin is really only useful for the partitioning case (i.e., auth
models on database X, myapp models on database Y), but that is a use
case we are trying to hit. It shouldn't be to hard to have ModelAdmin
derive using from the site if it isn't defined locally.
Yours,
Russ Magee %-)
-- You received this message because you are subscribed to the Google Groups
"Django developers" g...
On Fri, Dec 4, 2009 at 11:40 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> Is there a page where one can find a quick summary of the proposed API?
> I have some concerns about implementing partitioning through the admin, but
> I expect there's something I'm missing.
> For those who haven't been, following the conversation closely it'd be nice
> to have a quick way to come up to speed.
The proposed API is intentionally simple - QuerySets gain a .using()
modifier that lets you direct a query at a specific database; save()
gains a using argument to let you save to a specific database. There
are some configuration changes and to the get_db_prep_* method on
Field to support multiple database setups, but that's about it. Most
of the changes have been in the internals to make the two external
APIs possible.
As noted a couple of messages back, support in the admin was an
oversight. The existing admin will work for the default database, but
will require modifications to support other databases.
For more details, the links I gave in the original call for testing
are probably your best bet:
The very first "Requirement"/ideal feature on the wiki page reads:
"Different models/applications living on different databases, for
example a 'blog' application on db1 and a forum application on db2.
This should include the ability to assign a different database to an
existing application without modifying it, e.g. telling Django where
to keep the django.contrib.auth.User table. "
That sounds perfect - having a way to modify in what database a table
exists on a per-model or per-app basis, without making any changes to
the app itself (so I can choose to put reusable app A in database X
and reusable app B in database Y).
I don't see anything in the docs about if and how this type of
partitioning is supported. From what I understand, it has been
proposed that this be implemented for the admin specifically through
ModelAdmin and/or at the admin.Site level, because in the current
iteration of the code there is no way to use the admin for anything
other than the default database.
To me, specifying the database in the ModelAdmin or admin.Site seems
arbitrary and potentially limiting: For any reusable app on the
market, depending on the value of a particular setting in your urls.py
or admin.py file is a Bad Idea. What if the admin was instead fixed
by providing facilities for the more general case outlined above?
What would this look like? I'm picturing another setting (bleh) that
maps apps and/or models to specific databases. Name choices aside,
this might look something like:
> On Fri, Dec 4, 2009 at 11:40 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> > Is there a page where one can find a quick summary of the proposed API?
> > I have some concerns about implementing partitioning through the admin, but
> > I expect there's something I'm missing.
> > For those who haven't been, following the conversation closely it'd be nice
> > to have a quick way to come up to speed.
> The proposed API is intentionally simple - QuerySets gain a .using()
> modifier that lets you direct a query at a specific database; save()
> gains a using argument to let you save to a specific database. There
> are some configuration changes and to the get_db_prep_* method on
> Field to support multiple database setups, but that's about it. Most
> of the changes have been in the internals to make the two external
> APIs possible.
> As noted a couple of messages back, support in the admin was an
> oversight. The existing admin will work for the default database, but
> will require modifications to support other databases.
> For more details, the links I gave in the original call for testing
> are probably your best bet:
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to django-developers+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
What will be the default action of manage.py syncdb (without the
--database option)? Will it create all tables on all DBs, or just use
"default"? The former would be useful for the simple replication use
case, but wasteful for partitioning, and the latter could get tiresome
unless there is a "--database all" option of which I'm unaware.
I also don't see anywhere in the proposed docs where one could specify
"app -> db" or "app.model -> db" mappings. Without some mechanism for
such mapping, I'm afraid the partitioning case will become error prone
and verbose, both with manage.py commands, and in app code. I'd hate
to have to refactor all of my app's views if I moved some tables to a
new server. Not having such mappings will also require a refactor of
many of the views in existing distributed apps to become "multi-db
enabled" if they don't already accept a QuerySet as an argument (isn't
necessarily a bad thing, but could delay multi-db usefulness). Moving
this problem to settings should (operative word) keep multi-db app
related modification to a minimum, and thus not split all available
apps into supporting and non-supporting categories. It could also
greatly ease deployment if one didn't partition in dev, but did in
staging and production. And this would go a long way toward fixing the
Admin issue for the partitioning case, and for replication most will
only need the admin on the default db anyway.
The mappings could be as simple as a new key on the DATABASES dict:
I thought I saw some discussion of this type of thing a while back.
Was a mechanism for this delayed, removed, undocumented, or am I just
missing something?
On Fri, Dec 4, 2009 at 2:41 PM, Paul McLanahan <pmclana...@gmail.com> wrote:
> What will be the default action of manage.py syncdb (without the
> --database option)? Will it create all tables on all DBs, or just use
> "default"? The former would be useful for the simple replication use
> case, but wasteful for partitioning, and the latter could get tiresome
> unless there is a "--database all" option of which I'm unaware.
> I also don't see anywhere in the proposed docs where one could specify
> "app -> db" or "app.model -> db" mappings. Without some mechanism for
> such mapping, I'm afraid the partitioning case will become error prone
> and verbose, both with manage.py commands, and in app code. I'd hate
> to have to refactor all of my app's views if I moved some tables to a
> new server. Not having such mappings will also require a refactor of
> many of the views in existing distributed apps to become "multi-db
> enabled" if they don't already accept a QuerySet as an argument (isn't
> necessarily a bad thing, but could delay multi-db usefulness). Moving
> this problem to settings should (operative word) keep multi-db app
> related modification to a minimum, and thus not split all available
> apps into supporting and non-supporting categories. It could also
> greatly ease deployment if one didn't partition in dev, but did in
> staging and production. And this would go a long way toward fixing the
> Admin issue for the partitioning case, and for replication most will
> only need the admin on the default db anyway.
> The mappings could be as simple as a new key on the DATABASES dict:
> I thought I saw some discussion of this type of thing a while back.
> Was a mechanism for this delayed, removed, undocumented, or am I just
> missing something?
> Thanks,
> Paul
> --
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to django-developers+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
syncdb only syncs the default database by default.
We will not be adding a setting to pin an application/model to a
specific database. We have already removed the Meta.using option. If
you want to pin an application to a specific database it's fairly
trivial to add a manager and override the save() method to use a
specific DB. Our goal is to design a good long term API for having a
flexible way to define exactly what DB any query goes to, and I
personally have no interest in seeing short term stopgap APIs
implemented.
Alex
-- "I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me
On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor <alex.gay...@gmail.com> wrote:
> We will not be adding a setting to pin an application/model to a
> specific database. We have already removed the Meta.using option. If
> you want to pin an application to a specific database it's fairly
> trivial to add a manager and override the save() method to use a
> specific DB. Our goal is to design a good long term API for having a
> flexible way to define exactly what DB any query goes to, and I
> personally have no interest in seeing short term stopgap APIs
> implemented.
That's fine for apps under my control, but this leaves us without a
way of dealing with 3rd party apps outside of an unfortunate amount of
monkey patching. I certainly don't want the API to be "sort term" or
"stopgap", but I do want this ability. I don't see any other way to
accomplish a heterogeneous admin or partitioning of 3rd party apps. I
know and agree with the reasons that the Meta.using option was
removed, but it wouldn't have solved these issues either. Adding a key
to the DB configs may not be the right way, but I'm sure there is a
good way to accomplish this.
On Fri, Dec 4, 2009 at 3:03 PM, Paul McLanahan <pmclana...@gmail.com> wrote:
> On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor <alex.gay...@gmail.com> wrote:
>> We will not be adding a setting to pin an application/model to a
>> specific database. We have already removed the Meta.using option. If
>> you want to pin an application to a specific database it's fairly
>> trivial to add a manager and override the save() method to use a
>> specific DB. Our goal is to design a good long term API for having a
>> flexible way to define exactly what DB any query goes to, and I
>> personally have no interest in seeing short term stopgap APIs
>> implemented.
> That's fine for apps under my control, but this leaves us without a
> way of dealing with 3rd party apps outside of an unfortunate amount of
> monkey patching. I certainly don't want the API to be "sort term" or
> "stopgap", but I do want this ability. I don't see any other way to
> accomplish a heterogeneous admin or partitioning of 3rd party apps. I
> know and agree with the reasons that the Meta.using option was
> removed, but it wouldn't have solved these issues either. Adding a key
> to the DB configs may not be the right way, but I'm sure there is a
> good way to accomplish this.
Agreed. I was just writing basically the same reply. And in our
defense, support for this functionality is the first thing one sees in
the "Requirements" section on the Multi DB support wiki page:
I see nothing "short term" or "stopgap" about this; rather, it solves
a specific and potentially common use case for multidb support in an
elegant, extensible way that also fixes other issues with the current
implementation (namely, the admin).
That said, figuring out the right way to implement it is still up in
the air. I proposed a separate setting; Paul proposed one integrated
with the DATABASES setting. There are likely better ways to do it and
I'm not attached to any particular one, but the functionality is key.
Database -> Model.Manager -> Queryset.
Save -> Model -> Database
How about making a default Manager that's now "model.Manager" to be a
setting to override?
Like, calling current Manager a DefaultManager, and making "Manager =
load(settings.MODEL_MANAGER) or DefaultManager". Or
Manager.get_queryset calling customizable method. Any.
And provide with django few most usable enhanced managers replacements.
This will solve both master/master, master/slave sharding and "this
table is at DB1, that is at DB2" use case (how to name it in a single
word?).
There should also be such general solution for complex model save
override. Maybe, Or Model.save calling such customizable method.
Alex, you just don't get it seriously yet. Imagine the following:
In your project, you use the plugin application A written by other
people (and 25 other plugins B, C, D, ..., Z!).
It defines model A, and 3 views, each one use model A, with some
forms, templates, etc.
How are you going to change A.save and managers while keeping the views working?
Monkeypatch it? Make patched version for each app? Well, then please
provide a method which will do this override for all models if called
from a single place in project, probably, settings.
In large projects, single point of override on per-project basis is
required, not per-model.
On Sat, Dec 5, 2009 at 2:12 AM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> On Fri, Dec 4, 2009 at 3:03 PM, Paul McLanahan <pmclana...@gmail.com> wrote:
>> On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor <alex.gay...@gmail.com> wrote:
>>> We will not be adding a setting to pin an application/model to a
>>> specific database. We have already removed the Meta.using option. If
>>> you want to pin an application to a specific database it's fairly
>>> trivial to add a manager and override the save() method to use a
>>> specific DB. Our goal is to design a good long term API for having a
>>> flexible way to define exactly what DB any query goes to, and I
>>> personally have no interest in seeing short term stopgap APIs
>>> implemented.
>> That's fine for apps under my control, but this leaves us without a
>> way of dealing with 3rd party apps outside of an unfortunate amount of
>> monkey patching. I certainly don't want the API to be "sort term" or
>> "stopgap", but I do want this ability. I don't see any other way to
>> accomplish a heterogeneous admin or partitioning of 3rd party apps. I
>> know and agree with the reasons that the Meta.using option was
>> removed, but it wouldn't have solved these issues either. Adding a key
>> to the DB configs may not be the right way, but I'm sure there is a
>> good way to accomplish this.
> Agreed. I was just writing basically the same reply. And in our
> defense, support for this functionality is the first thing one sees in
> the "Requirements" section on the Multi DB support wiki page:
> I see nothing "short term" or "stopgap" about this; rather, it solves
> a specific and potentially common use case for multidb support in an
> elegant, extensible way that also fixes other issues with the current
> implementation (namely, the admin).
> That said, figuring out the right way to implement it is still up in
> the air. I proposed a separate setting; Paul proposed one integrated
> with the DATABASES setting. There are likely better ways to do it and
> I'm not attached to any particular one, but the functionality is key.
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to django-developers+unsubscribe@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
-- Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com
On Sat, Dec 5, 2009 at 2:42 AM, Yuri Baburov <burc...@gmail.com> wrote:
> Hi all,
> Database -> Model.Manager -> Queryset.
> Save -> Model -> Database
> How about making a default Manager that's now "model.Manager" to be a
> setting to override?
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.
> And provide with django few most usable enhanced managers replacements.
> This will solve both master/master, master/slave sharding and "this
> table is at DB1, that is at DB2" use case (how to name it in a single
> word?).
> There should also be such general solution for complex model save
> override. Maybe, Or Model.save calling such customizable method.
> Alex, you just don't get it seriously yet. Imagine the following:
> In your project, you use the plugin application A written by other
> people (and 25 other plugins B, C, D, ..., Z!).
> It defines model A, and 3 views, each one use model A, with some
> forms, templates, etc.
> How are you going to change A.save and managers while keeping the views working?
> Monkeypatch it? Make patched version for each app? Well, then please
> provide a method which will do this override for all models if called
> from a single place in project, probably, settings.
> In large projects, single point of override on per-project basis is
> required, not per-model.
> On Sat, Dec 5, 2009 at 2:12 AM, Tobias McNulty <tob...@caktusgroup.com> wrote:
>> On Fri, Dec 4, 2009 at 3:03 PM, Paul McLanahan <pmclana...@gmail.com> wrote:
>>> On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor <alex.gay...@gmail.com> wrote:
>>>> We will not be adding a setting to pin an application/model to a
>>>> specific database. We have already removed the Meta.using option. If
>>>> you want to pin an application to a specific database it's fairly
>>>> trivial to add a manager and override the save() method to use a
>>>> specific DB. Our goal is to design a good long term API for having a
>>>> flexible way to define exactly what DB any query goes to, and I
>>>> personally have no interest in seeing short term stopgap APIs
>>>> implemented.
>>> That's fine for apps under my control, but this leaves us without a
>>> way of dealing with 3rd party apps outside of an unfortunate amount of
>>> monkey patching. I certainly don't want the API to be "sort term" or
>>> "stopgap", but I do want this ability. I don't see any other way to
>>> accomplish a heterogeneous admin or partitioning of 3rd party apps. I
>>> know and agree with the reasons that the Meta.using option was
>>> removed, but it wouldn't have solved these issues either. Adding a key
>>> to the DB configs may not be the right way, but I'm sure there is a
>>> good way to accomplish this.
>> Agreed. I was just writing basically the same reply. And in our
>> defense, support for this functionality is the first thing one sees in
>> the "Requirements" section on the Multi DB support wiki page:
>> I see nothing "short term" or "stopgap" about this; rather, it solves
>> a specific and potentially common use case for multidb support in an
>> elegant, extensible way that also fixes other issues with the current
>> implementation (namely, the admin).
>> That said, figuring out the right way to implement it is still up in
>> the air. I proposed a separate setting; Paul proposed one integrated
>> with the DATABASES setting. There are likely better ways to do it and
>> I'm not attached to any particular one, but the functionality is key.
>> You received this message because you are subscribed to the Google Groups "Django developers" group.
>> To post to this group, send email to django-developers@googlegroups.com.
>> To unsubscribe from this group, send email to django-developers+unsubscribe@googlegroups.com.
>> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
> --
> Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
> MSN: bu...@live.com
-- Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com
On Fri, Dec 4, 2009 at 2:42 PM, Yuri Baburov <burc...@gmail.com> wrote:
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.
If a suggestion like this is going to be implemented, I'd prefer it to
be part of the database backend rather than a setting, because:
1. We already have a ton of DB-related settings, possibly more than we need.
2. Backends already support things like a custom Query class, and so
this would be the logical place for it to go.
3. Encapsulating this as part of a DB backend most likely gives you a
lot more freedom to build in all sorts of complex enterprisey stuff if
you really want it.
-- "Bureaucrat Conrad, you are technically correct -- the best kind of correct."
On Fri, Dec 4, 2009 at 2:34 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> What would this look like? I'm picturing another setting (bleh) that
> maps apps and/or models to specific databases. Name choices aside,
> this might look something like:
I like this. I think it could be made even more simple by using the
"app_label.model_name" pattern in use by other settings (e.g.
ABSOLUTE_URL_OVERRIDES) for the dict keys. This makes the
implementation easier using django.db.models.get_model and
django.db.models.get_models.
I just suggested a way it *could* work, not how it should work.
Are you objecting to my whole plan or only on Manager part of it?
On Sat, Dec 5, 2009 at 2:49 AM, James Bennett <ubernost...@gmail.com> wrote:
> On Fri, Dec 4, 2009 at 2:42 PM, Yuri Baburov <burc...@gmail.com> wrote:
>> Like, calling current Manager a DefaultManager, and making "Manager =
>> load(settings.MODEL_MANAGER) or DefaultManager". Or
>> Manager.get_queryset calling customizable method. Any.
> If a suggestion like this is going to be implemented, I'd prefer it to
> be part of the database backend rather than a setting, because:
> 1. We already have a ton of DB-related settings, possibly more than we need.
Sorry, i don't count this an argument, if a question is "where there
is a single place to put project-wide configuration".
If we decided that settings.py is that storage -- let us be consistent.
> 2. Backends already support things like a custom Query class, and so
> this would be the logical place for it to go.
I will agree with you if you give me an example.
> 3. Encapsulating this as part of a DB backend most likely gives you a
> lot more freedom to build in all sorts of complex enterprisey stuff if
> you really want it.
Ehm.... Don't get it.
I.e, some of my tables are local sqlite database, some are remote
postgresql service, some are mysql on next server, some are in mssql
database accessed through django-pyodbc, and some tables are going to
be splitted across 16 bsddb files with custom backend, sharding based
on key hashes?
Where do I point which table should use what backend and which
connection do I use?
-- Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com
Well, this is goooood.
Few more use cases that will appear on big django installations:
Sometimes, with master/slave replication (or legacy read-only
databases), users write to one connection, but read from (few) others.
Sometimes, with master/master replication, or different sharding
schemes, users write and read from random or key based database
connections.
So I attempted to modify this proposal to make this possible.
On Sat, Dec 5, 2009 at 3:31 AM, Paul McLanahan <pmclana...@gmail.com> wrote:
> On Fri, Dec 4, 2009 at 2:34 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
>> What would this look like? I'm picturing another setting (bleh) that
>> maps apps and/or models to specific databases. Name choices aside,
>> this might look something like:
> I like this. I think it could be made even more simple by using the
> "app_label.model_name" pattern in use by other settings (e.g.
> ABSOLUTE_URL_OVERRIDES) for the dict keys. This makes the
> implementation easier using django.db.models.get_model and
> django.db.models.get_models.
> DATABASE_USING = {
> 'auth' : 'default',
#Isn't it django.contrib.auth? I'm not sure if collision with
'my.superstuff.auth' might happen or not.
'myapp': ('database1', 'database2'), # random access
'myapp3': {'read':('database1', 'database2'),
'write':my_get_app3_connection, 'syncdb':('default', 'database1',
'database2', 'database3')}, # custom function to get RW connection,
custom syncdb since .
'*':my_get_other_connection, # for other 25 apps, use this
connection, not 'default'.
> }
Where my_get_*_connection function accept model name string, access
mode, and will return:
for 'read' and 'write' mode, a database name or connection,
for 'syncdb' mode, a list of database names.
> Any model not specifically mentioned could assume that it should use "default".
Hope it isn't yet toooo complex to understand....
This will also provide super-giper-mega-option:
DATABASE_USING = {'*': my_get_other_connection}
with these 3 modes.
-- Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com
AFAICT adding a setting for the default manager just doesn't work because
then any models with custom managers either (a) would lose their custom
manager, or (b) wouldn't get the manager with using().
On Fri, Dec 4, 2009 at 3:42 PM, Yuri Baburov <burc...@gmail.com> wrote:
> Hi all,
> Database -> Model.Manager -> Queryset.
> Save -> Model -> Database
> How about making a default Manager that's now "model.Manager" to be a
> setting to override?
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.
> And provide with django few most usable enhanced managers replacements.
> This will solve both master/master, master/slave sharding and "this
> table is at DB1, that is at DB2" use case (how to name it in a single
> word?).
> There should also be such general solution for complex model save
> override. Maybe, Or Model.save calling such customizable method.
> Alex, you just don't get it seriously yet. Imagine the following:
> In your project, you use the plugin application A written by other
> people (and 25 other plugins B, C, D, ..., Z!).
> It defines model A, and 3 views, each one use model A, with some
> forms, templates, etc.
> How are you going to change A.save and managers while keeping the views
> working?
> Monkeypatch it? Make patched version for each app? Well, then please
> provide a method which will do this override for all models if called
> from a single place in project, probably, settings.
> In large projects, single point of override on per-project basis is
> required, not per-model.
> On Sat, Dec 5, 2009 at 2:12 AM, Tobias McNulty <tob...@caktusgroup.com>
> wrote:
> > On Fri, Dec 4, 2009 at 3:03 PM, Paul McLanahan <pmclana...@gmail.com>
> wrote:
> >> On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor <alex.gay...@gmail.com>
> wrote:
> >>> We will not be adding a setting to pin an application/model to a
> >>> specific database. We have already removed the Meta.using option. If
> >>> you want to pin an application to a specific database it's fairly
> >>> trivial to add a manager and override the save() method to use a
> >>> specific DB. Our goal is to design a good long term API for having a
> >>> flexible way to define exactly what DB any query goes to, and I
> >>> personally have no interest in seeing short term stopgap APIs
> >>> implemented.
> >> That's fine for apps under my control, but this leaves us without a
> >> way of dealing with 3rd party apps outside of an unfortunate amount of
> >> monkey patching. I certainly don't want the API to be "sort term" or
> >> "stopgap", but I do want this ability. I don't see any other way to
> >> accomplish a heterogeneous admin or partitioning of 3rd party apps. I
> >> know and agree with the reasons that the Meta.using option was
> >> removed, but it wouldn't have solved these issues either. Adding a key
> >> to the DB configs may not be the right way, but I'm sure there is a
> >> good way to accomplish this.
> > Agreed. I was just writing basically the same reply. And in our
> > defense, support for this functionality is the first thing one sees in
> > the "Requirements" section on the Multi DB support wiki page:
> > I see nothing "short term" or "stopgap" about this; rather, it solves
> > a specific and potentially common use case for multidb support in an
> > elegant, extensible way that also fixes other issues with the current
> > implementation (namely, the admin).
> > That said, figuring out the right way to implement it is still up in
> > the air. I proposed a separate setting; Paul proposed one integrated
> > with the DATABASES setting. There are likely better ways to do it and
> > I'm not attached to any particular one, but the functionality is key.
> > You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> > To post to this group, send email to django-developers@googlegroups.com.
> > To unsubscribe from this group, send email to
> django-developers+unsubscribe@googlegroups.com<django-developers%2Bunsubscr ibe@googlegroups.com>
> .
> > For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
> --
> Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
> MSN: bu...@live.com
> --
> You received this message because you are subscribed to the Google Groups
> "Django developers" group.
> To post to this group, send email to django-developers@googlegroups.com.
> To unsubscribe from this group, send email to
> django-developers+unsubscribe@googlegroups.com<django-developers%2Bunsubscr ibe@googlegroups.com>
> .
> For more options, visit this group at
> http://groups.google.com/group/django-developers?hl=en.
On Fri, Dec 4, 2009 at 5:51 PM, Yuri Baburov <burc...@gmail.com> wrote:
> #Isn't it django.contrib.auth? I'm not sure if collision with
> 'my.superstuff.auth' might happen or not.
It is, but the app registry mechanism in the
'django.db.models.loading' module only uses the "app label", which is
the part of the app module name after the last '.'. So, when using
django.db.models.get_model, you would pass it 'auth'. So if you named
your app 'my.superstuff.auth', the app used when calling
get_model('auth', 'user') might not be the one you'd expect.
That's my understanding anyway. I agree that it seems potentially fragile.
