I started from trunk and added things from the multidb branch little
by little, since so much has changed in that area since then.
There is still a lot more that needs to be checked and a number of
things to be redone. References to the default connection object are
all over the codebase, there is still a lot of work left to get all of
them straightened out.

What is working (more or less):
- using existing databases (though there might still be quirks when
using different engines)
- running tests
- some of the management commands (eg sqlall, sqlflush, but loaddata,
inspectdb or syncdb are not quite there yet)

Important to mention is that relations across databases are not
supported.

Anyway, if anyone is interested in helping, please let me know!

Yet another SoC introduction: Getting multi-db done
http://groups.google.com/group/django-developers/browse_thread/thread...

It would be nice to coordinate each one's efforts, to avoid wasting
time. Ben, Daryl, any news?

Koen

On 20 mei, 16:56, "Nicola Larosa (tekNico)" <nicola.lar...@gmail.com>
wrote:

I would still like to get my patch working so others (and myself) can
start testing it.  I won't have time this week, but so far it looks
like I may be able to make some time next week.  If I don't, I see if
I can at least make enough time to write up the API I came up with at
PyCon.

--
Daryl

--
Daryl

Sorry for coming late to the party, but I'm just now catching up on django-dev.

I'm really glad to see you get the ball rolling on multiple db
support, and once I'm dug out from my backlog I'll be happy to start
reviewing code and helping out if I'm needed.

However, before we get to that point, I've got some pretty serious API
concerns with the current approach, so I think I should outline those
before y'all go much further. I don't want you to expend much effort
just to get a -1 smackdown.

The current mechanism of defining "other" databases in the settings
module is just fine, and the underlying mechanism of having
queries/managers "know" their connection is similarly dandy. But the
wheels come off when it comes to the "public" API where users will
choose which connection they use.

As far as I can tell, you've currently provided two hooks to use a
secondary connection: set the model's default connection in the
settings module (which is OK, I suppose, though I might want to
nitpick the syntax a bit), and assigning to ``Model.objects.db``.

This second one is a disaster waiting to happen -- you've had to muddy
things up with threadlocals to work around some problems already. Also
consider the "bookkeeping" you'd need to do to deal with objects
across multiple database simultaneously (think sharding). You'd have
to keep juggling ``Model.objects.db`` and saving old ones... ugh.

Here's how I think it should work:

* I'd like the default connection for each and every object to be the
default database forever and always. I find putting models for default
connections in settings distasteful and I'd rather just a single API
for changing the connection (see below). However, I imagine I'll be in
the minority here so I'm prepared to cede this point if necessary.

* There needs to be an official API to get a model (or perhaps a
manager) which references a different "context" --
``Model.objects.db`` should be read-only. So you'd call some API
method, and get back a sort of proxy object that uses the other
connection. Here's a strawman API::

    >>> from django import db
    >>> from someapp.models import Article

    >>> Article.objects.all()
    [... all Articles from the default database ...]

    >>> ArticlesOnOtherDatabase =
db.get_model_for_other_connection(Article, "private")
    >>> ArticlesOnOtherDatabase.objects.all()
    [... all Articles from the database defined with the "private" key ...]

This should make the threadlocal stuff unnecessary, and (to my eye) is
a lot more sane than assigning the ``Manager.db``. Oh, and please
choose a better better name than
``db.get_model_for_other_connection()``; given that you're building
the bikeshed you might as well paint it, too.

 * I'd like the default connection for each and every object to be the

It's great to see this revived again :-)

Cheers
Ben

On May 21, 2:16 pm, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:

This still leaves questions about how syncdb would be achieved, at
least. But if it could be done, the API seems simple to understand.

Koen

On 21 mei, 23:16, "Jacob Kaplan-Moss" <jacob.kaplanm...@gmail.com>
wrote:

The proposal at the moment deals with having different models live in
different databases - for example, the Forum application lives on DB1
while the Blog application lives on DB2.

I can see how this could be useful, but the two database problems that
keep me up at night are the following:

1. Replication - being able to send all of my writes to one master
machine but spread all of my reads over several slave machines.
Thankfully Ivan Sagalaev's confusingly named mysql_cluster covers this
problem neatly without modification to Django core - it's just an
alternative DB backend which demonstrates that doing this isn't
particularly hard: http://softwaremaniacs.org/soft/mysql_cluster/en/

2. Sharding - being able to put User entries 1-1000 on DB1, whereas
User entries 1001-2000 live on DB2 and so on.

I'd love Django to have built-in abilities to solve #1 - it's a really
important first-step on scaling up to multiple databases, and it's
also massively easier than any other part of the multi-db problem.

I wouldn't expect a magic solution to #2 because it's so highly
dependent on the application that is being built, but at the same time
it would be nice to see a multi-db solution at least take this in to
account (maybe just by providing an easy tool to direct an ORM request
to a specific server based on some arbitrary logic).

I may have misunderstood the proposal, but I think it's vital that the
above two use cases are considered. Even if they can't be solved
outright, providing tools that custom solutions to these cases can be
built with should be a priority for multi-db support.

Cheers,

Usage of several RDBMS in one django application simulatneously

I do not believe this is a trivial change.

My proposal is to keep things simple in a first phase: let's make it
possible to use different databases for different models with the
restriction that relations should not cross databases. Once we get all
that working, we may look at making the query generation deal with
those.

Koen

On 23 mei, 13:00, "Mike Scott" <mic...@gmail.com> wrote:

What has been realised (more or less):
- connection and model registration
- validation that related objects use the same connection
- database_engine specific SQL generation (needs more checking)
- some management commands accept connection parameter, others can
generate output for multiple connections
- syncdb can sync different connections
- transaction management over connections
- some tests (needs a lot more work)

This means point 3 of the discussion at [http://trac.woe-beti.de/wiki/
Discuss] is somewhat working, but definitely needs a lot more testing.

I do need some help with creating tests for all this though.
I have not figured out yet how to create tests that check that the
right SQL is being generated for the backend used. (Nor how to test
the right database was touched by an action, this is quite obvious
manually, but I do not know how to automate this.)

I put some ideas on using multiple databases for copying (backup or
migration) of objects (point 4 of the discussion note) in [http://
trac.woe-beti.de/wiki/APIModelDuplication].

Please comment, add, shoot etc. Any help will be much appreciated.

Anyway, I don't have time to read this thread through with the care it
deserves, but I thought I shouldn't let that stop me from finally
writing a description of the API I proposed at the PyCon sprint.

Each app would have a databases.py file that contains classes used to
define databases connections (in the same manner as classes are used
to define models).  Here's an example:

----

from django.db import connections

class LegacyDatabase(connections.DatabaseConnection):
   engine = 'sqlite3'
   name = '/foo/bar/legacy_db.sqlite3'

----

(And the other DATABASE_* settings (from settings.py) could certainly
be defined as attributes of a DatabaseConnection class.)

JUST FOR TESTING, I propose we allow a database connection to be
specified in a model with a Meta attribute, like this:

----

from django.db import models
from legacy.databases import LegacyDatabase

class LegacyStuff(models.Model):
    ...

    class Meta:
        db_connection = LegacyDatabase

----

Jacob expressed his extreme distaste for this at PyCon--for good
reason.  (We don't want to encourage coupling models to databases.)
But just so we can get a working patch and start testing, I propose we
go with this for now.

Adrian suggested we allow the specification of database connections
per-app using the new app() function being proposed for settings.py.
I haven't seen a description of this since PyCon, but I think it would
look something like:

app(name='legacy', db_connection='LegacyDatabase')

(I'm sure I'm leaving several important arguments out of this example.)

Perhaps one could implement sharding by defining multiple
DatabaseConnection classes in a databases.py file (we could support
these files at the project level in addition to the app level) and
putting them in a list.  Then one could write a function to return the
appropriate database to use and specify that callable in the argument
to the app function (or perhaps as an argument to the url function in
urls.py).

I haven't given any thought to replication.  Perhaps someone who needs
this could think about whether this proposal could somehow make
supporting replication easier (or if it might get in the way), or if
it's simply orthogonal to this.

--
Daryl

On May 22, 9:59 am, Simon Willison <si...@simonwillison.net> wrote:

For case 2, it's very similar, except you need to run the query on all
sides (unless
you can tell it should only go to one) building a chained query-set
union type to hold
the result, and for saves pick the right model  to save to  based on
the condition.

Our approach:

- read_cursor and write_cursor exist. write_cursor is what cursor
would point ot.
- get queries all use the read cursor
- saves all use the write cursor
- we had a list of database connections, which stored the same
settings, just in a tuple format
- reading I believe used something like itertools.cycle but I can't
honestly say without looking at the code

Beyond this, the database itself should handle writing the objects to
the slaves. Django shouldn't even bother.

In regards to multiple databases in general. it is my feeling that
even if it is not good practice, Django _NEEDS_ to support a model
being attached to a database other than the default. So if you have
mydjango_blogs, and mydjango_forums databases, my Forum model always
goes to the write db when it queries, and same for blogs. I myself
like a Meta solution to this as it makes sense.

In MySQL as well, you can optimize things, so that if they use the
same connection, you can just query on that database. It's select X
from mydatabase.mytable. I'm not sure if something similar exists in
other database engines.

On Jun 16, 9:05 pm, mengel <Marc.Men...@gmail.com> wrote: