Multiple database support: Request for feedback and testing

48 views
Skip to first unread message

Russell Keith-Magee

unread,
Dec 3, 2009, 11:10:54 AM12/3/09
to Django Developers
Hi all,

Alex Gaynor's GSoC project to add multiple database support to Django
is on the final straight. The only piece of the puzzle that is left is
updating contrib.gis - but this hopefully won't require any major
changes outside of the gis tree itself.

Therefore, I'd like to call for feedback and testing of the branch.

Code is available on github:

http://github.com/alex/django/tree/multiple-db

It is also available on Django's own subversion repository in the
soc2009/multidb branch:

svn co http://code.djangoproject.com/svn/django/branches/soc2009/multidb

This is a very big change to the query engine, but unless your code
relies on the internals of sql.Query, you *should* be able to switch
to using the branch without making any changes to your code. The
release notes give the details of the changes that you need to be
aware of:

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/releases/1.2.txt

If you want to run Django's test suite, you will need to make some
modifications to your test settings file. If you don't make these
changes, you will get failures in the multiple_databases tests. See
the relevant section in the contribution docs for details on the
changes you need to make:

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/internals/contributing.txt

To the best of our knowledge, the full test suite is currently passing
for all databases (except for those known pre-existing failures under
MySQL InnoDB and Oracle). Any discoveries to the contrary will be
gratefully received.

If you actually want to try using multiple databases, basic usage
instructions are here:

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/topics/db/multi-db.txt

What should you expect to see? Well, a while back, Malcolm described
Django's query framework as having the plumbing, but not the porcelain
for multiple database support. At the risk of stretching the metaphor,
this branch adds the porcelain, but not the polished oak seat to go on
top. The settings and API to define multiple databases and to direct
queries to a specific database have all been added by this branch.
However, there isn't any explicit API level support for the common use
cases for multiple databases like sharding or master/slave setups. I
see adding these wrappers as tasks for the broader community, possibly
with a view to inclusion in trunk in v1.3 or later.

So - feedback welcome. It's a big patch, so the more testing we can
get pre-merge, the better.

Yours,
Russ Magee %-)

Waldemar Kornewald

unread,
Dec 3, 2009, 11:20:46 AM12/3/09
to django-developers
Hi Russell,
I just noticed a simple documentation bug. See the comment at the
bottom of this page:
http://github.com/alex/django/commit/aabfee1571d378dd3b7550573e900850d13e1b9b

Also, I hope you won't define an API for overriding the query class in
the multi-db branch. We'll most likely have to define our own API in
the non-relational branch.

Bye,
Waldemar Kornewald
> --
>
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.

Russell Keith-Magee

unread,
Dec 3, 2009, 6:43:05 PM12/3/09
to django-d...@googlegroups.com
On Fri, Dec 4, 2009 at 12:20 AM, Waldemar Kornewald
<wkorn...@gmail.com> wrote:
> Hi Russell,
> I just noticed a simple documentation bug. See the comment at the
> bottom of this page:
> http://github.com/alex/django/commit/aabfee1571d378dd3b7550573e900850d13e1b9b

You are correct. Thanks for that - I've made the change on github.

> Also, I hope you won't define an API for overriding the query class in
> the multi-db branch. We'll most likely have to define our own API in
> the non-relational branch.

We haven't modified the API for overriding the query class beyond the
techniques that already existed to support GIS (i.e., passing a Query
class into the QuerySet itself).

Yours,
Russ Magee %-)

Simon Willison

unread,
Dec 4, 2009, 3:38:39 AM12/4/09
to Django developers
On Dec 3, 4:10 pm, Russell Keith-Magee <freakboy3...@gmail.com> wrote:
> Alex Gaynor's GSoC project to add multiple database support to Django
> is on the final straight. The only piece of the puzzle that is left is
> updating contrib.gis - but this hopefully won't require any major
> changes outside of the gis tree itself.
>
> Therefore, I'd like to call for feedback and testing of the branch.

I tried this out yesterday - it's excellent. The combination of the
new DATABASES setting and QuerySet.using() turns out to be exactly
enough to get all sorts of interesting work done. It neatly fulfils
the three requirements set out in the original ticket (replication /
sharding / hosting different applications on different databases).

A couple of observations. Firstly, there's a gotcha involving saving
existing objects to a new database:

e1 = Entry.objects.using('db1').create(title = "A blog post",
body="...")
e2 = Entry.objects.using('db1').create(title = "Another blog post",
body="...")
e3 = Entry.objects.using('db2').create(title = "Another blog post",
body="...")
e4 = Entry.objects.using('db2').create(title = "Another blog post",
body="...")

We now have two blog posts in each database. But, if I do the
following:

e1.save(using = 'db2')

It has the effect of replacing e3 with the values from e2, since the
primary keys collide with each other.

This behaviour makes sense if you think about it for more than a
moment, but still probably counts as a gotcha. Worth mentioning in the
documentation at least.

I could see this pattern biting people since that's the obvious way to
copy an object from one database to another.

Secondly, a question: Is there an idea for how this might be made to
work with the admin interface? I was asked this question at DJUGL last
night and wasn't sure of the answer.

Fantastic work on this, very excited to see it merged to trunk.

Cheers,

Simon

Simon Willison

unread,
Dec 4, 2009, 3:41:54 AM12/4/09
to Django developers
On Dec 4, 8:38 am, Simon Willison <si...@simonwillison.net> wrote:
> We now have two blog posts in each database. But, if I do the
> following:
>
> e1.save(using = 'db2')
>
> It has the effect of replacing e3 with the values from e2, since the
> primary keys collide with each other.

I meant to say - the fix is clearly to call save() with the
force_insert=True argument, but it's easy to miss that. Worth
explicitly highlighting in the multi-db docs I think.

Russell Keith-Magee

unread,
Dec 4, 2009, 6:31:22 AM12/4/09
to django-d...@googlegroups.com
Agreed that this could bite people fairly easily.

I'm not sure we can really do much to avoid the problem, though. We
have a few options, but none of them are really appropriate for all
circumstances.

A time of object save, we know the database that the object is
currently on (using _state), and we know if the user has requested a
save on a different database (by comparing with the using argument).
Based on this detail, we could:

1) Do nothing special. This is the currently implement behaviour.

2) Set the primary key of the object to None so that the save on the
new database is guaranteed to be an insertion, rather than a possible
overwrite of an object with the same pk value on the second database.

3) Transform the save into a 'force insert' if the database changes.
This will raise errors if the pk is already in use on

4) Use the PK on the old database and issue a deletion before saving
to the new database.

5) Do 2 and 4.

6) Do 3 and 4

The problem is the conflict between the most appropriate behavior for
different use cases. In the case you describe (copying an object from
one database to another), (5) probably makes the most sense; however,
from the point of view of a master/slave setup, (1) is the best
behaviour.

In fact (4) by makes me a little nervous - If we start modifying
save() to make it copy objects, there will be surprise when objects
related to the object being saved aren't transferred as well. Deleting
data from the old database sounds like a recipe for some really nasty
accidental data loss.

IMHO, it's best to leave entirely in the hands of the end user. By
that reasoning, The current behaviour (1) is actually the right
solution, along with some documentation explaining the problem and
some possible solutions - essentially, the manual interpretations of
(2) and (3).

> Secondly, a question: Is there an idea for how this might be made to
> work with the admin interface? I was asked this question at DJUGL last
> night and wasn't sure of the answer.

...and this is why we put code out for review. You win the internets
for pointing out the obvious omission. For all the spelunking in
django.db internals, the potential effect on admin hadn't occurred to
me.

I can think of a couple of possible approaches to this.

1) Ignore the problem. Admin works on the default database, but
nowhere else. This is certainly less than ideal, but it would be
sufficient for master/slave setups.

2) Use a separate admin deployment for each database. We add a 'using'
argument to admin.Site(), and append .using() the queries that the
admin site issues.

3) Modify the admin list views so that they take a ?using=db GET
argument; also add a pulldown to make it easy to switch to a different
database.

(2) should be a fairly minor change; (3) would be a little more
effort, but shouldn't be impossible. There is also the need for some
mechanics to check whether a table is actually present on a database -
just because auth is in INSTALLED_APPS doesn't mean it's been
synchronized to a particular database.

I'm open to any other suggestions - and for any offers to help out :-)

Russ %-)

Russell Keith-Magee

unread,
Dec 4, 2009, 6:57:03 AM12/4/09
to django-d...@googlegroups.com
On Fri, Dec 4, 2009 at 7:31 PM, Russell Keith-Magee
<freakb...@gmail.com> wrote:
> IMHO, it's best to leave entirely in the hands of the end user. By
> that reasoning, The current behaviour (1) is actually the right
> solution, along with some documentation explaining the problem and
> some possible solutions - essentially, the manual interpretations of
> (2) and (3).

I've just added some docs to the github branch to cover this situation:

http://github.com/alex/django/commit/9c162e50c81883e214b60f749de81b79ee10ace6

Russ %-)

Doug Blank

unread,
Dec 4, 2009, 7:58:41 AM12/4/09
to django-d...@googlegroups.com
On Fri, Dec 4, 2009 at 6:31 AM, Russell Keith-Magee
<freakb...@gmail.com> wrote:

[snip]

>  1) Do nothing special. This is the currently implement behaviour.

As the default, this seems to be the intended behavior only in rare
cases (overwriting a different object from the original) and will
indeed bite.

There is another option: leave in the hands of the user (as you say)
but make them explicitly say which behavior they want of the
following:

>  2) Set the primary key of the object to None so that the save on the
> new database is guaranteed to be an insertion, rather than a possible
> overwrite of an object with the same pk value on the second database.
>
>  3) Transform the save into a 'force insert' if the database changes.
> This will raise errors if the pk is already in use on
>
>  4) Use the PK on the old database and issue a deletion before saving
> to the new database.

If they didn't explicitly indicate which of these to do, then raise an error.

Just a thought.

-Doug

[snip]

Nan

unread,
Dec 4, 2009, 9:18:59 AM12/4/09
to Django developers

> 1) Ignore the problem. Admin works on the default database, but
> nowhere else. This is certainly less than ideal, but it would be
> sufficient for master/slave setups.
>
> 2) Use a separate admin deployment for each database. We add a 'using'
> argument to admin.Site(), and append .using() the queries that the
> admin site issues.
>
> 3) Modify the admin list views so that they take a ?using=db GET
> argument; also add a pulldown to make it easy to switch to a different
> database.
>
> (2) should be a fairly minor change; (3) would be a little more
> effort, but shouldn't be impossible. There is also the need for some
> mechanics to check whether a table is actually present on a database -
> just because auth is in INSTALLED_APPS doesn't mean it's been
> synchronized to a particular database.
>
> I'm open to any other suggestions - and for any offers to help out :-)

Just thinking of one more option for this -- what about specifying the
DB on the ModelAdmin level rather than the admin.Site level?

Russell Keith-Magee

unread,
Dec 4, 2009, 9:30:07 AM12/4/09
to django-d...@googlegroups.com
This is also a possibility. Describing databases at the level of the
ModelAdmin is really only useful for the partitioning case (i.e., auth
models on database X, myapp models on database Y), but that is a use
case we are trying to hit. It shouldn't be to hard to have ModelAdmin
derive using from the site if it isn't defined locally.

Yours,
Russ Magee %-)

Tobias McNulty

unread,
Dec 4, 2009, 10:40:44 AM12/4/09
to django-d...@googlegroups.com

Is there a page where one can find a quick summary of the proposed API?

I have some concerns about implementing partitioning through the admin, but I expect there's something I'm missing. 

For those who haven't been, following the conversation closely it'd be nice to have a quick way to come up to speed.

Sent from a mobile phone, please excuse any typos.

On Dec 4, 2009 9:30 AM, "Russell Keith-Magee" <freakb...@gmail.com> wrote:

On Fri, Dec 4, 2009 at 10:18 PM, Nan <ring...@gmail.com> wrote: > >> 1) Ignore the problem. Admin w...

This is also a possibility. Describing databases at the level of the
ModelAdmin is really only useful for the partitioning case (i.e., auth
models on database X, myapp models on database Y), but that is a use
case we are trying to hit. It shouldn't be to hard to have ModelAdmin
derive using from the site if it isn't defined locally.

Yours,
Russ Magee %-)

-- You received this message because you are subscribed to the Google Groups "Django developers" g...

Russell Keith-Magee

unread,
Dec 4, 2009, 10:47:56 AM12/4/09
to django-d...@googlegroups.com
On Fri, Dec 4, 2009 at 11:40 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> Is there a page where one can find a quick summary of the proposed API?
>
> I have some concerns about implementing partitioning through the admin, but
> I expect there's something I'm missing.
>
> For those who haven't been, following the conversation closely it'd be nice
> to have a quick way to come up to speed.

The proposed API is intentionally simple - QuerySets gain a .using()
modifier that lets you direct a query at a specific database; save()
gains a using argument to let you save to a specific database. There
are some configuration changes and to the get_db_prep_* method on
Field to support multiple database setups, but that's about it. Most
of the changes have been in the internals to make the two external
APIs possible.

As noted a couple of messages back, support in the admin was an
oversight. The existing admin will work for the default database, but
will require modifications to support other databases.

For more details, the links I gave in the original call for testing
are probably your best bet:

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/topics/db/multi-db.txt

http://code.djangoproject.com/browser/django/branches/soc2009/multidb/docs/releases/1.2.txt

Yours,
Russ Magee %-)

Tobias McNulty

unread,
Dec 4, 2009, 2:34:46 PM12/4/09
to django-developers
The very first "Requirement"/ideal feature on the wiki page reads:

"Different models/applications living on different databases, for
example a 'blog' application on db1 and a forum application on db2.
This should include the ability to assign a different database to an
existing application without modifying it, e.g. telling Django where
to keep the django.contrib.auth.User table. "

That sounds perfect - having a way to modify in what database a table
exists on a per-model or per-app basis, without making any changes to
the app itself (so I can choose to put reusable app A in database X
and reusable app B in database Y).

I don't see anything in the docs about if and how this type of
partitioning is supported.  From what I understand, it has been
proposed that this be implemented for the admin specifically through
ModelAdmin and/or at the admin.Site level, because in the current
iteration of the code there is no way to use the admin for anything
other than the default database.

To me, specifying the database in the ModelAdmin or admin.Site seems
arbitrary and potentially limiting: For any reusable app on the
market, depending on the value of a particular setting in your urls.py
or admin.py file is a Bad Idea.  What if the admin was instead fixed
by providing facilities for the more general case outlined above?

What would this look like?  I'm picturing another setting (bleh) that
maps apps and/or models to specific databases.  Name choices aside,
this might look something like:

DATABASE_USING = {
    'django.contrib.auth': 'default',
    'myapp': 'database1',
    'myapp2': {
        'Model1': 'database2',
        'Model2': 'database3',
    }
}

The admin could automatically take advantage of this setting to
determine what database to use for a given model.

Cheers,
Tobias
> --
>
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>



--
Tobias McNulty
Caktus Consulting Group, LLC
P.O. Box 1454
Carrboro, NC 27510
(919) 951-0052
http://www.caktusgroup.com

Paul McLanahan

unread,
Dec 4, 2009, 2:41:41 PM12/4/09
to django-d...@googlegroups.com
What will be the default action of manage.py syncdb (without the
--database option)? Will it create all tables on all DBs, or just use
"default"? The former would be useful for the simple replication use
case, but wasteful for partitioning, and the latter could get tiresome
unless there is a "--database all" option of which I'm unaware.

I also don't see anywhere in the proposed docs where one could specify
"app -> db" or "app.model -> db" mappings. Without some mechanism for
such mapping, I'm afraid the partitioning case will become error prone
and verbose, both with manage.py commands, and in app code. I'd hate
to have to refactor all of my app's views if I moved some tables to a
new server. Not having such mappings will also require a refactor of
many of the views in existing distributed apps to become "multi-db
enabled" if they don't already accept a QuerySet as an argument (isn't
necessarily a bad thing, but could delay multi-db usefulness). Moving
this problem to settings should (operative word) keep multi-db app
related modification to a minimum, and thus not split all available
apps into supporting and non-supporting categories. It could also
greatly ease deployment if one didn't partition in dev, but did in
staging and production. And this would go a long way toward fixing the
Admin issue for the partitioning case, and for replication most will
only need the admin on the default db anyway.

The mappings could be as simple as a new key on the DATABASES dict:

DATABASES = {
'default': {
'BACKEND': 'django.db.backends.sqlite3',
'NAME': 'mydatabase',
},
'stuff': {
'BACKEND': 'django.db.backends.sqlite3',
'NAME': 'myotherdatabase',
'PINNED': ['myapp', 'someapp.model'],
},
}

I thought I saw some discussion of this type of thing a while back.
Was a mechanism for this delayed, removed, undocumented, or am I just
missing something?

Thanks,

Paul

Alex Gaynor

unread,
Dec 4, 2009, 2:48:32 PM12/4/09
to django-d...@googlegroups.com
> --
>
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
>

syncdb only syncs the default database by default.

We will not be adding a setting to pin an application/model to a
specific database. We have already removed the Meta.using option. If
you want to pin an application to a specific database it's fairly
trivial to add a manager and override the save() method to use a
specific DB. Our goal is to design a good long term API for having a
flexible way to define exactly what DB any query goes to, and I
personally have no interest in seeing short term stopgap APIs
implemented.

Alex

--
"I disapprove of what you say, but I will defend to the death your
right to say it." -- Voltaire
"The people's good is the highest law." -- Cicero
"Code can always be simpler than you think, but never as simple as you
want" -- Me

Paul McLanahan

unread,
Dec 4, 2009, 3:03:50 PM12/4/09
to django-d...@googlegroups.com
On Fri, Dec 4, 2009 at 2:48 PM, Alex Gaynor <alex....@gmail.com> wrote:
> We will not be adding a setting to pin an application/model to a
> specific database.  We have already removed the Meta.using option.  If
> you want to pin an application to a specific database it's fairly
> trivial to add a manager and override the save() method to use a
> specific DB.  Our goal is to design a good long term API for having a
> flexible way to define exactly what DB any query goes to, and I
> personally have no interest in seeing short term stopgap APIs
> implemented.

That's fine for apps under my control, but this leaves us without a
way of dealing with 3rd party apps outside of an unfortunate amount of
monkey patching. I certainly don't want the API to be "sort term" or
"stopgap", but I do want this ability. I don't see any other way to
accomplish a heterogeneous admin or partitioning of 3rd party apps. I
know and agree with the reasons that the Meta.using option was
removed, but it wouldn't have solved these issues either. Adding a key
to the DB configs may not be the right way, but I'm sure there is a
good way to accomplish this.

Thanks,

Paul

Tobias McNulty

unread,
Dec 4, 2009, 3:12:17 PM12/4/09
to django-developers
Agreed. I was just writing basically the same reply. And in our
defense, support for this functionality is the first thing one sees in
the "Requirements" section on the Multi DB support wiki page:

http://code.djangoproject.com/wiki/MultipleDatabaseSupport#Requirements

I see nothing "short term" or "stopgap" about this; rather, it solves
a specific and potentially common use case for multidb support in an
elegant, extensible way that also fixes other issues with the current
implementation (namely, the admin).

That said, figuring out the right way to implement it is still up in
the air. I proposed a separate setting; Paul proposed one integrated
with the DATABASES setting. There are likely better ways to do it and
I'm not attached to any particular one, but the functionality is key.

Cheers,
Tobias

Yuri Baburov

unread,
Dec 4, 2009, 3:42:57 PM12/4/09
to django-d...@googlegroups.com
Hi all,

Database -> Model.Manager -> Queryset.
Save -> Model -> Database

How about making a default Manager that's now "model.Manager" to be a
setting to override?
Like, calling current Manager a DefaultManager, and making "Manager =
load(settings.MODEL_MANAGER) or DefaultManager". Or
Manager.get_queryset calling customizable method. Any.

And provide with django few most usable enhanced managers replacements.
This will solve both master/master, master/slave sharding and "this
table is at DB1, that is at DB2" use case (how to name it in a single
word?).
There should also be such general solution for complex model save
override. Maybe, Or Model.save calling such customizable method.

Alex, you just don't get it seriously yet. Imagine the following:
In your project, you use the plugin application A written by other
people (and 25 other plugins B, C, D, ..., Z!).
It defines model A, and 3 views, each one use model A, with some
forms, templates, etc.
How are you going to change A.save and managers while keeping the views working?
Monkeypatch it? Make patched version for each app? Well, then please
provide a method which will do this override for all models if called
from a single place in project, probably, settings.

In large projects, single point of override on per-project basis is
required, not per-model.
> --
>
> You received this message because you are subscribed to the Google Groups "Django developers" group.
> To post to this group, send email to django-d...@googlegroups.com.
> To unsubscribe from this group, send email to django-develop...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/django-developers?hl=en.
>
>
>



--
Best regards, Yuri V. Baburov, ICQ# 99934676, Skype: yuri.baburov,
MSN: bu...@live.com

Yuri Baburov

unread,
Dec 4, 2009, 3:47:02 PM12/4/09
to django-d...@googlegroups.com
Yes, and the similar function for syncdb can be written too!

James Bennett

unread,
Dec 4, 2009, 3:49:44 PM12/4/09
to django-d...@googlegroups.com
On Fri, Dec 4, 2009 at 2:42 PM, Yuri Baburov <bur...@gmail.com> wrote:
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.

If a suggestion like this is going to be implemented, I'd prefer it to
be part of the database backend rather than a setting, because:

1. We already have a ton of DB-related settings, possibly more than we need.

2. Backends already support things like a custom Query class, and so
this would be the logical place for it to go.

3. Encapsulating this as part of a DB backend most likely gives you a
lot more freedom to build in all sorts of complex enterprisey stuff if
you really want it.


--
"Bureaucrat Conrad, you are technically correct -- the best kind of correct."

Paul McLanahan

unread,
Dec 4, 2009, 4:31:06 PM12/4/09
to django-d...@googlegroups.com
On Fri, Dec 4, 2009 at 2:34 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> What would this look like?  I'm picturing another setting (bleh) that
> maps apps and/or models to specific databases.  Name choices aside,
> this might look something like:
>
> DATABASE_USING = {
>     'django.contrib.auth': 'default',
>     'myapp': 'database1',
>     'myapp2': {
>         'Model1': 'database2',
>         'Model2': 'database3',
>     }
> }

I like this. I think it could be made even more simple by using the
"app_label.model_name" pattern in use by other settings (e.g.
ABSOLUTE_URL_OVERRIDES) for the dict keys. This makes the
implementation easier using django.db.models.get_model and
django.db.models.get_models.

DATABASE_USING = {
'auth' : 'default',
'myapp' : 'database1',
'myapp2.model1' : 'database2',
'myapp2.model2' : 'database3',
}

Any model not specifically mentioned could assume that it should use "default".

Paul

Yuri Baburov

unread,
Dec 4, 2009, 5:15:12 PM12/4/09
to django-d...@googlegroups.com
Hi James,

I just suggested a way it *could* work, not how it should work.

Are you objecting to my whole plan or only on Manager part of it?

On Sat, Dec 5, 2009 at 2:49 AM, James Bennett <ubern...@gmail.com> wrote:
> On Fri, Dec 4, 2009 at 2:42 PM, Yuri Baburov <bur...@gmail.com> wrote:
>> Like, calling current Manager a DefaultManager, and making "Manager =
>> load(settings.MODEL_MANAGER) or DefaultManager". Or
>> Manager.get_queryset calling customizable method. Any.
>
> If a suggestion like this is going to be implemented, I'd prefer it to
> be part of the database backend rather than a setting, because:
>
> 1. We already have a ton of DB-related settings, possibly more than we need.
Sorry, i don't count this an argument, if a question is "where there
is a single place to put project-wide configuration".
If we decided that settings.py is that storage -- let us be consistent.

> 2. Backends already support things like a custom Query class, and so
> this would be the logical place for it to go.
I will agree with you if you give me an example.

> 3. Encapsulating this as part of a DB backend most likely gives you a
> lot more freedom to build in all sorts of complex enterprisey stuff if
> you really want it.
Ehm.... Don't get it.
I.e, some of my tables are local sqlite database, some are remote
postgresql service, some are mysql on next server, some are in mssql
database accessed through django-pyodbc, and some tables are going to
be splitted across 16 bsddb files with custom backend, sharding based
on key hashes?
Where do I point which table should use what backend and which
connection do I use?

Yuri Baburov

unread,
Dec 4, 2009, 5:51:26 PM12/4/09
to django-d...@googlegroups.com
Hi Paul, Tobias,

Well, this is goooood.
Few more use cases that will appear on big django installations:
Sometimes, with master/slave replication (or legacy read-only
databases), users write to one connection, but read from (few) others.
Sometimes, with master/master replication, or different sharding
schemes, users write and read from random or key based database
connections.
So I attempted to modify this proposal to make this possible.

On Sat, Dec 5, 2009 at 3:31 AM, Paul McLanahan <pmcla...@gmail.com> wrote:
> On Fri, Dec 4, 2009 at 2:34 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
>> What would this look like?  I'm picturing another setting (bleh) that
>> maps apps and/or models to specific databases.  Name choices aside,
>> this might look something like:
>>
>> DATABASE_USING = {
>>     'django.contrib.auth': 'default',
>>     'myapp': 'database1',
>>     'myapp2': {
>>         'Model1': 'database2',
>>         'Model2': 'database3',
>>     }
>> }
>
> I like this. I think it could be made even more simple by using the
> "app_label.model_name" pattern in use by other settings (e.g.
> ABSOLUTE_URL_OVERRIDES) for the dict keys. This makes the
> implementation easier using django.db.models.get_model and
> django.db.models.get_models.
>
> DATABASE_USING = {
>    'auth' : 'default',
#Isn't it django.contrib.auth? I'm not sure if collision with
'my.superstuff.auth' might happen or not.
>    'myapp' : 'database1',
>    'myapp2.model1' : 'database2',
>    'myapp2.model2' : 'database3',
'myapp': ('database1', 'database2'), # random access
'myapp3': {'read':('database1', 'database2'),
'write':my_get_app3_connection, 'syncdb':('default', 'database1',
'database2', 'database3')}, # custom function to get RW connection,
custom syncdb since .
'*':my_get_other_connection, # for other 25 apps, use this
connection, not 'default'.
> }
Where my_get_*_connection function accept model name string, access
mode, and will return:
for 'read' and 'write' mode, a database name or connection,
for 'syncdb' mode, a list of database names.
>
> Any model not specifically mentioned could assume that it should use "default".
Hope it isn't yet toooo complex to understand....

This will also provide super-giper-mega-option:
DATABASE_USING = {'*': my_get_other_connection}
with these 3 modes.

Tobias McNulty

unread,
Dec 4, 2009, 6:04:01 PM12/4/09
to django-d...@googlegroups.com
AFAICT adding a setting for the default manager just doesn't work because then any models with custom managers either (a) would lose their custom manager, or (b) wouldn't get the manager with using().

Cheers,
Tobias

Paul McLanahan

unread,
Dec 4, 2009, 10:56:57 PM12/4/09
to django-d...@googlegroups.com
On Fri, Dec 4, 2009 at 5:51 PM, Yuri Baburov <bur...@gmail.com> wrote:
> #Isn't it django.contrib.auth? I'm not sure if collision with
> 'my.superstuff.auth' might happen or not.

It is, but the app registry mechanism in the
'django.db.models.loading' module only uses the "app label", which is
the part of the app module name after the last '.'. So, when using
django.db.models.get_model, you would pass it 'auth'. So if you named
your app 'my.superstuff.auth', the app used when calling
get_model('auth', 'user') might not be the one you'd expect.

That's my understanding anyway. I agree that it seems potentially fragile.

Paul

Russell Keith-Magee

unread,
Dec 5, 2009, 12:36:39 AM12/5/09
to django-d...@googlegroups.com
On Sat, Dec 5, 2009 at 3:34 AM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> The very first "Requirement"/ideal feature on the wiki page reads:
>
> "Different models/applications living on different databases, for
> example a 'blog' application on db1 and a forum application on db2.
> This should include the ability to assign a different database to an
> existing application without modifying it, e.g. telling Django where
> to keep the django.contrib.auth.User table. "
>
> That sounds perfect - having a way to modify in what database a table
> exists on a per-model or per-app basis, without making any changes to
> the app itself (so I can choose to put reusable app A in database X
> and reusable app B in database Y).
>
> I don't see anything in the docs about if and how this type of
> partitioning is supported.

The underlying idea of multidb is that you can direct any query to any
database you choose. If you want to force a particular model to a
particular database by default, then you install a default manager
that directs queries for that model onto that database of choice:

class MyManager(models.Manager):
def get_query_set(self):
return super(MyManager, self).get_query_set().using('other')

You then use syncdb to synchronize applications onto the databases
where you want them.

Now - I fully acknowledge that this is problematic for models like
contrib.auth, as you don't have the ability to install a default
manager. As I noted in my explanatory notes for the call for feedback,
this iteration is the 'porcelain but not the polished seat' version of
the code. You can direct a contrib.auth query to any database you want
with this iteration; in 1.3, I expect that Django will gain some form
of registration/callback API to allow you control database assignment
on a per-query basis.

> From what I understand, it has been
> proposed that this be implemented for the admin specifically through
> ModelAdmin and/or at the admin.Site level, because in the current
> iteration of the code there is no way to use the admin for anything
> other than the default database.

The idea about having Site bound to a database is consistent with the
view of contrib.admin as an Adminstration interface for your database.
If you have two databases, having two adminstration interfaces isn't a
huge logical jump.

Of course, it would be nice to be able to switch between databases
inside a single admin interface, and I don't see any reason that this
couldn't be done - it will just take more effort to implement.

> To me, specifying the database in the ModelAdmin or admin.Site seems
> arbitrary and potentially limiting: For any reusable app on the
> market, depending on the value of a particular setting in your urls.py
> or admin.py file is a Bad Idea.

I don't follow that logic. Admin sites are deployed into urls.py by
the end user - who knows exactly what databases are available, and
what models are available on those databases.

The only time this would be a problem for Site is if a reusable app
included an admin deployment as part of it's urls.py - but I can't
think of any reusable app that does this.

It is arguably true of ModelAdmin. However, remember if you don't like
the ModelAdmin for contrib.auth, you can unregister that ModelAdmin
and install your own.

> What if the admin was instead fixed
> by providing facilities for the more general case outlined above?
>
> What would this look like?  I'm picturing another setting (bleh) that
> maps apps and/or models to specific databases.  Name choices aside,
> this might look something like:
>
> DATABASE_USING = {
>     'django.contrib.auth': 'default',
>     'myapp': 'database1',
>     'myapp2': {
>         'Model1': 'database2',
>         'Model2': 'database3',
>     }
> }
>
> The admin could automatically take advantage of this setting to
> determine what database to use for a given model.

Alex, Joseph Kocherhans and I discussed this exact idea at Djangocon.
You are correct that it has a lot of potential uses - not only the
admin, but also loaddata/dumpdata, syncdb, and anywhere else that an
iteration over models is required.

However, it's a little bit more complicated than you make out.

This sort of setting is very easy to give as a 10 line example, but in
practice, this isn't what you will require - you will effectively need
to duplicate the contents of INSTALLED_APPS. I have projects in the
wild with dozens of entries in INSTALLED_APS - all running on a single
database. Writing a DATABASE_USING setting like the one you describe
would be extremely cumbersome and annoying.

So, we could hijack INSTALLED_APPS to represent the idea of database
deployment, using tuples/dictionaries rather than strings to define
how apps are deployed. However, this comes at the cost of backwards
compatibility for anyone iterating over INSTALLED_APPS.

Even if we were able to find an acceptable way to represent this
information, consider some of the other use cases for multi-db like
master/slave or sharding. In these cases, every model needs to be
assigned to every database. Using the syntax you propose would involve
a whole lot of repeated typing of lines like:

'django.contrib.auth': ['master','slave1','slave2','slave3','slave4']

You could use DATABASES.keys() here, but even then - there's a lot of
extra configuration required to get the simple case of "just put
everything everywhere".

In the end, the three of us (Alex, Joseph and I) came to the
conclusion that the best approach was to treat the users as
"consenting adults". Rather than engage in a complex configuration
exercise, we can assume everything is everywhere, and provide the
tools during syncdb to prevent particular tables being created in the
case where that isn't the case. We then allow any query on any
database, and assume that the user knows what they are doing.

Looking at this in practical terms, the only code that actually needs
a database-table-lookup facility is code that is engaged in automated
meta-programming, like that used by admin and syncdb. While this is a
very nifty thing to do, it isn't really the common use case.

To be clear - I'm not opposed to having some way to programmatically
determine the models that are available on any given database. Having
this sort of facility would make the admin/syncdb/dumpdata tasks much
easier. It would provide an additional source of error checking in
queries themselves (preventing queries on databases where we know they
can't succeed). However, we need to find a clean way to express this,
and a clean proposal hasn't emerged from the discussions I've had to
date. I'm open to suggestions.

Yours,
Russ Magee %-)

Russell Keith-Magee

unread,
Dec 5, 2009, 12:56:08 AM12/5/09
to django-d...@googlegroups.com
On Sat, Dec 5, 2009 at 4:42 AM, Yuri Baburov <bur...@gmail.com> wrote:
> Hi all,
>
> Database -> Model.Manager -> Queryset.
> Save -> Model -> Database
>
> How about making a default Manager that's now "model.Manager" to be a
> setting to override?
> Like, calling current Manager a DefaultManager, and making "Manager =
> load(settings.MODEL_MANAGER) or DefaultManager". Or
> Manager.get_queryset calling customizable method. Any.

As I noted in my explanatory notes for the call for feedback, this
iteration of multi-db is the 'porcelain but not the polished seat'
version of multi-db. You can direct a contrib.auth query to any
database you want with this iteration. This is a necessary
prerequisite for any more sophisticated API for multi-db.

I fully expect that the 1.3 timeframe will see many proposals for how
to present a more useful API to end users of multidb. Controlling and
configuring Managers will be a big part of that API.

Alex's position is all about avoiding a rush into the second phase. We
all want a pretty multi-db interface. We just don't need to rush the
development of that interface. However, we do need to establish that
the plumbing work that the pretty interface will use is solid and
compatible with existing code.

> Alex, you just don't get it seriously yet.

Yuri - for your benefit: this particular turn of phrase and tone is
quite condescending, and could easily be interpreted as offensive. I
certainly hope that this is just a combination of crossed wires,
enthusiasm, and a language barrier.

Yours
Russ Magee %-)

Johannes Dollinger

unread,
Dec 5, 2009, 3:07:44 AM12/5/09
to django-d...@googlegroups.com

Am 05.12.2009 um 06:36 schrieb Russell Keith-Magee:
> [...]
> [...]

Let me propose a different colored pattern. It's backwards compatible,
dry, feels like the urlconf approach and may evolve into a fix for
#3591:

INSTALLED_APPS = (
app('django.contrib.auth', using=...),
app('myapp'),
app('myapp2', models=(
model('Model1', using=...),
model('Model2', using=...),
),
)

Where `app()` would look like this:

APP_USING = {}

def app(path, **kwargs):
if 'using' in kwargs:
APP_USING[path] = kwargs['using']
...
return path

The downside: It would require an import in settings.py. And the
global dict is not exactly beautiful - but the price for bc.

This is not really an argument for a settings based solution (+0). But
if there will be one, don't make it nested tuples/dicts - the
existance of tuple literals in python does not make s-expressions a
natural representation of configuration.
__
Johannes

Russell Keith-Magee

unread,
Dec 5, 2009, 6:10:14 AM12/5/09
to django-d...@googlegroups.com
The idea of using a function that returns a single string but does
other processing is a novel approach, and one that I hadn't
considered. However, I'm not sure I'm especially fond of the idea of
requiring imports in a settings file, and the syntax you propose is
quite verbose. I'll have to think about this a bit. Thanks for the
suggestion, though.

Yours,
Russ Magee %-)

Waldemar Kornewald

unread,
Dec 5, 2009, 8:05:39 AM12/5/09
to django-developers
On Sat, Dec 5, 2009 at 12:10 PM, Russell Keith-Magee
<freakb...@gmail.com> wrote:
> The idea of using a function that returns a single string but does
> other processing is a novel approach, and one that I hadn't
> considered. However, I'm not sure I'm especially fond of the idea of
> requiring imports in a settings file, and the syntax you propose is
> quite verbose. I'll have to think about this a bit. Thanks for the
> suggestion, though.

Shouldn't all this be more abstract? For our non-relational branch we
thought about something more flexible like a list of query proxies
that sit between QuerySet/Model and sql.Query (QueryData in nonrel)
and that can intercept query execution and instead run their own query
code.

For example, a proxy could emulate unsupported DB features like
select_related for non-relational DBs. You could also use this for
selecting the connection for each query (e.g., sharding by pk) and for
spreading a single query across multiple DBs at the same time (e.g.,
if you want to find all users born in 1980 you'll have to run this
query on all DBs of a sharded users table).

The point is, QuerySet and Model shouldn't care about those details.
They should only provide a high-level abstraction to the DB that is as
expressive and simple as possible. The details can be implemented via
add-ons, so everyone can map the DB abstraction to his real DB setup.

Bye,
Waldemar Kornewald

Russell Keith-Magee

unread,
Dec 5, 2009, 8:58:28 AM12/5/09
to django-d...@googlegroups.com
On Sat, Dec 5, 2009 at 9:05 PM, Waldemar Kornewald <wkorn...@gmail.com> wrote:
> On Sat, Dec 5, 2009 at 12:10 PM, Russell Keith-Magee
> <freakb...@gmail.com> wrote:
>> The idea of using a function that returns a single string but does
>> other processing is a novel approach, and one that I hadn't
>> considered. However, I'm not sure I'm especially fond of the idea of
>> requiring imports in a settings file, and the syntax you propose is
>> quite verbose. I'll have to think about this a bit. Thanks for the
>> suggestion, though.
>
> Shouldn't all this be more abstract? For our non-relational branch we
> thought about something more flexible like a list of query proxies
> that sit between QuerySet/Model and sql.Query (QueryData in nonrel)
> and that can intercept query execution and instead run their own query
> code.

I'm afraid I don't see the connection between what you're describing,
and the problem we actually have.

The problem is determining whether model X is available on database Y.
This is only required for metaprogramming purposes - so, for example,
syncdb knows which models to synchronize onto a particular database,
or an admin interface knows which models can be shown to the end user.

As best as I can make out, you're addressing the problem that I've
said we aren't addressing - that of presenting a useful end-user API
for tasks like master/slave. If I'm mistaken, feel free to correct me
- preferably with some sample code to demonstrate what you're talking
about.

Yours,
Russ Magee %-)

Tobias McNulty

unread,
Dec 5, 2009, 10:15:10 AM12/5/09
to django-developers
On Sat, Dec 5, 2009 at 8:58 AM, Russell Keith-Magee
<freakb...@gmail.com> wrote:
> As best as I can make out, you're addressing the problem that I've
> said we aren't addressing - that of presenting a useful end-user API
> for tasks like master/slave. If I'm mistaken, feel free to correct me
> - preferably with some sample code to demonstrate what you're talking
> about.

Yes - it sounds to me like Waldemar might be talking about the
"registration/callback API to allow you control database assignment on
a per-query basis" that you mention above. I hadn't thought about
this before, but I like the idea and I think it'll be crucial in
implementing some of the more complex partitioning cases.

It seems like it'll be necessary in addition to some sort of
settings-based map of what apps/models go in what databases.

Quick question: will the admin.Site method be useful in addition to
whatever more global method we come up with for designating what
tables go in what databases, or will it be obsolete (and potentially
deprecated) at that point? I can't see a reason that one would want
to modify the database(s) the admin uses irrespective of the global
settings.

Cheers,
Tobias

Waldemar Kornewald

unread,
Dec 5, 2009, 10:33:42 AM12/5/09
to django-developers
Hi Russell,

On Sat, Dec 5, 2009 at 2:58 PM, Russell Keith-Magee
Yes and no. Aren't both tasks (model location and
sharding&master/slave API) highly connected?

We don't know what the API for sharding, master/slave, etc. will look
like, but I have the impression that we're already defining its
settings format and planning changes that won't be necessary with
Django 1.3, anymore. For example, as my suggestion shows, the admin
interface does not have to (and must not!) know about the multi-db
setup because that can be fully abstracted behind an sql.Query proxy
API (actually, it must be abstracted that way because there can be
lots of different ways to do sharding and you don't want high-level
code to deal with those details).

I am just concerned that you'll waste time on implementing something
that will get removed in 1.3 shortly thereafter just because multi-db
isn't ready for real-world use-cases, yet. I'd rather have a temporary
very simple hack that binds a model to a single DB and that doesn't
require any changes to the admin interface itself and code that uses
the admin UI. This could indeed be specified in the settings as was
already suggested. But it should be clear that this is just a
temporary hack and 1.3 will provide a clean solution.

Bye,
Waldemar Kornewald

Russell Keith-Magee

unread,
Dec 5, 2009, 10:51:24 AM12/5/09
to django-d...@googlegroups.com
On Sat, Dec 5, 2009 at 11:15 PM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> On Sat, Dec 5, 2009 at 8:58 AM, Russell Keith-Magee
> <freakb...@gmail.com> wrote:
>> As best as I can make out, you're addressing the problem that I've
>> said we aren't addressing - that of presenting a useful end-user API
>> for tasks like master/slave. If I'm mistaken, feel free to correct me
>> - preferably with some sample code to demonstrate what you're talking
>> about.
>
> Yes - it sounds to me like Waldemar might be talking about the
> "registration/callback API to allow you control database assignment on
> a per-query basis" that you mention above.  I hadn't thought about
> this before, but I like the idea and I think it'll be crucial in
> implementing some of the more complex partitioning cases.
>
> It seems like it'll be necessary in addition to some sort of
> settings-based map of what apps/models go in what databases.

I don't grant that proposition at all. The admin interface serves as a
working example demonstrating that you don't need to use settings to
define the way models are used.

I'm not saying that this is necessarily the best way to solve the
problem - just that a setting-based map isn't an absolute given. There
are many viable alternatives, all of which we can explore when we
actually have working multi-db plumbing in the field.

> Quick question: will the admin.Site method be useful in addition to
> whatever more global method we come up with for designating what
> tables go in what databases, or will it be obsolete (and potentially
> deprecated) at that point?  I can't see a reason that one would want
> to modify the database(s) the admin uses irrespective of the global
> settings.

At this point, I'm prognosticating because I haven't actually written
any code for this - but I don't think the using argument to Site()
would necessarily have to be deprecated. In a single-database admin,
the using argument tells you exactly which database is to be used; if
we update admin to have a fully multi-db interface in the future, the
value of the using argument could easily be interpreted as the
"default" database that is displayed.

Yours,
Russ Magee %-)

Russell Keith-Magee

unread,
Dec 5, 2009, 11:20:52 AM12/5/09
to django-d...@googlegroups.com
On Sat, Dec 5, 2009 at 11:33 PM, Waldemar Kornewald
Well, yes, they are connected - there is a one way dependency. We can
have a solution for model locations that doesn't require a solution
for public API, but not the other way around.

> We don't know what the API for sharding, master/slave, etc. will look
> like, but I have the impression that we're already defining its
> settings format and planning changes that won't be necessary with
> Django 1.3, anymore.

Others in this thread might be convinced of the need for a setting,
but I really *don't* want to introduce a setting at this point in
proceedings.

I asked for feedback, and concern about the absence of a setting
providing database-model assignments is useful feedback. I've
explained why there isn't such a setting in the code at the moment. If
someone is able to come up with an elegant solution to the problem,
I'm interested in hearing it. If I'm discussing settings formats, it's
only in the context of exploring options that we may not have
previously considered.

As I've mentioned a few times now, I'm acutely aware of the fact that
this patch doesn't address public API issues for common use cases, and
I don't want to introduce features in this release that we need to
deprecate in later releases.

> For example, as my suggestion shows, the admin
> interface does not have to (and must not!) know about the multi-db
> setup because that can be fully abstracted behind an sql.Query proxy
> API (actually, it must be abstracted that way because there can be
> lots of different ways to do sharding and you don't want high-level
> code to deal with those details).

Trust me - I don't want to do mindless busy work. However, we need to
have some sort of answer for the admin interface - Django's admin is a
big selling point for Django for some people, so we can't really
introduce a huge new feature, and then say "but you can't use it in
admin". I'm interested in making the least intrusive change that is
possible without hamstringing future multi-db interfaces.

> I am just concerned that you'll waste time on implementing something
> that will get removed in 1.3 shortly thereafter just because multi-db
> isn't ready for real-world use-cases, yet. I'd rather have a temporary
> very simple hack that binds a model to a single DB and that doesn't
> require any changes to the admin interface itself and code that uses
> the admin UI. This could indeed be specified in the settings as was
> already suggested. But it should be clear that this is just a
> temporary hack and 1.3 will provide a clean solution.

In a previous message in this thread, I described a temporary hack to
bind a model to a specific database - override get_query_set() on the
model manager. It's not a perfect solution for all situations, but it
is a workable hack, and it doesn't require any settings, temporary or
otherwise.

Yours,
Russ Magee %-)

Simon Willison

unread,
Dec 5, 2009, 11:35:03 AM12/5/09
to Django developers
On Dec 5, 4:20 pm, Russell Keith-Magee <freakboy3...@gmail.com> wrote:
> Trust me - I don't want to do mindless busy work. However, we need to
> have some sort of answer for the admin interface - Django's admin is a
> big selling point for Django for some people, so we can't really
> introduce a huge new feature, and then say "but you can't use it in
> admin". I'm interested in making the least intrusive change that is
> possible without hamstringing future multi-db interfaces.

It strikes me that the admin issue should be solvable entirely in
django.contrib.admin without any changes to the multidb code itself.
Right now you can get most of the job done using a ModelAdmin
subclass:

from django.contrib import admin
from blog.models import Entry

class EntryAdmin(admin.ModelAdmin):
def save_model(self, request, obj, form, change):
obj.save(using='otherdb')

def queryset(self, request):
return Entry.objects.using('otherdb').all()

admin.site.register(Entry, EntryAdmin)

I haven't tested the above so it's probably missing a few cases
(save_fieldsets for example perhaps) but if we document it I think
it's a good enough solution for the moment. Even if we need to
refactor ModelAdmin a bit to ensure the right hooks are available it
still shouldn't be a massive change.

Cheers,

Simon

Tobias McNulty

unread,
Dec 5, 2009, 12:09:38 PM12/5/09
to django-d...@googlegroups.com
On Sat, Dec 5, 2009 at 10:51 AM, Russell Keith-Magee <freakb...@gmail.com> wrote:
I don't grant that proposition at all. The admin interface serves as a
working example demonstrating that you don't need to use settings to
define the way models are used.

Okay.  Do you grant the proposition that "we will (not necessarily in Django 1.2) need a project-level way to specify the default database(s) to use for queries to a given model," whether it is settings-based or not?

At this point, I'm prognosticating because I haven't actually written
any code for this - but I don't think the using argument to Site()
would necessarily have to be deprecated. In a single-database admin,
the using argument tells you exactly which database is to be used; if
we update admin to have a fully multi-db interface in the future, the
value of the using argument could easily be interpreted as the
"default" database that is displayed.

Maybe I'm just being obtuse, but there are a couple issues with the admin approach to which I can't see a resolution yet:

* Assuming the above proposition is true, the behavior of the API when the global and admin-level configurations conflict is not at all intuitive

* It seems like confining every admin site to a single database would lead to needless partitioning of the admin interface for what might otherwise be a single, cohesive site.

Simon's idea seems like a reasonable (and already supported?) workaround for those who need to modify the admin to use a different database in specific cases.

Cheers,
Tobias

sub...@gmail.com

unread,
Dec 5, 2009, 3:12:56 PM12/5/09
to Django developers
Isn't 'database' going to be an option in a model's Meta? In this
situation, is admin going to attempt to do something different?

-S

On Dec 4, 9:18 am, Nan <ringe...@gmail.com> wrote:
> > 1) Ignore the problem. Admin works on the default database, but
> > nowhere else. This is certainly less than ideal, but it would be
> > sufficient for master/slave setups.
>
> > 2) Use a separate admin deployment for each database. We add a 'using'
> > argument to admin.Site(), and append .using() the queries that the
> > admin site issues.
>
> > 3) Modify the admin list views so that they take a ?using=db GET
> > argument; also add a pulldown to make it easy to switch to a different
> > database.
>
> > (2) should be a fairly minor change; (3) would be a little more
> > effort, but shouldn't be impossible. There is also the need for some
> > mechanics to check whether a table is actually present on a database -
> > just because auth is in INSTALLED_APPS doesn't mean it's been
> > synchronized to a particular database.
>
> > I'm open to any other suggestions - and for any offers to help out :-)
>
> Just thinking of one more option for this -- what about specifying the
> DB on the ModelAdmin level rather than the admin.Site level?

sub...@gmail.com

unread,
Dec 5, 2009, 3:16:44 PM12/5/09
to Django developers
Oh, I see from a later message by Alex that Meta.using was removed.

-1!

-S

Russell Keith-Magee

unread,
Dec 5, 2009, 7:36:54 PM12/5/09
to django-d...@googlegroups.com
On Sun, Dec 6, 2009 at 4:16 AM, sub...@gmail.com <sub...@gmail.com> wrote:
> Oh, I see from a later message by Alex that Meta.using was removed.
>
> -1!

There's a very good reason why this was removed. It isn't a model
level property. Consider - what if contrib.auth.User had a Meta
using='foo' property? If this were the case, you wouldn't be able to
reuse the contrib.auth.User model without defining a 'foo' database.
This would singlehandedly hobble the ability reuse Django
applications.

On top of that, the capability provided by Meta.using can easily be
reproduced with a custom manager. I've given an example of how this
would work in this thread.

Yours,
Russ Magee %-)

Russell Keith-Magee

unread,
Dec 5, 2009, 7:38:41 PM12/5/09
to django-d...@googlegroups.com
On Sun, Dec 6, 2009 at 1:09 AM, Tobias McNulty <tob...@caktusgroup.com> wrote:
> On Sat, Dec 5, 2009 at 10:51 AM, Russell Keith-Magee
> <freakb...@gmail.com> wrote:
>>
>> I don't grant that proposition at all. The admin interface serves as a
>> working example demonstrating that you don't need to use settings to
>> define the way models are used.
>
> Okay.  Do you grant the proposition that "we will (not necessarily in Django
> 1.2) need a project-level way to specify the default database(s) to use for
> queries to a given model," whether it is settings-based or not?

Possibly. :-)

>> At this point, I'm prognosticating because I haven't actually written
>> any code for this - but I don't think the using argument to Site()
>> would necessarily have to be deprecated. In a single-database admin,
>> the using argument tells you exactly which database is to be used; if
>> we update admin to have a fully multi-db interface in the future, the
>> value of the using argument could easily be interpreted as the
>> "default" database that is displayed.
>
> Maybe I'm just being obtuse, but there are a couple issues with the admin
> approach to which I can't see a resolution yet:
> * Assuming the above proposition is true, the behavior of the API when the
> global and admin-level configurations conflict is not at all intuitive

I'm not sure how you can make that assertion given that neither global
nor admin-level configurations for multi-db exist yet.

Might I humbly suggest that we stop speculating about the possible
limitations of a theoretical admin implementation until such time as
an implementation actually exists?

> Simon's idea seems like a reasonable (and already supported?) workaround for
> those who need to modify the admin to use a different database in specific
> cases.

Agreed. All I'm talking about is making it easier to do what Simon has
suggested, but without the need for boilerplate ModelAdmin
definitions. I'm hoping to spend the next couple of days fiddling with
this problem. Once we've got something concrete to argue about, I'll
report back.

Yours,
Russ Magee %-)

Russell Keith-Magee

unread,
Dec 5, 2009, 7:42:09 PM12/5/09
to django-d...@googlegroups.com
I want to poke around this code over the next couple of days. I
suspect that you're probably right - the set of changes won't be too
significant. However, if it is possible to provide a simple
configuration option so that end-users don't have to write boilerplate
save_model() and queryset() definitions, I think that's worth looking
into.

Yours,
Russ Magee %-)
Reply all
Reply to author
Forward
0 new messages