Multiple Database

[Austin Gabel]

Hello,
I am currently attempting to roll my own multiple database support for django for a new project.  I have attempted to search for examples of this but have not found very much.  Does anyone have some examples on how I can accomplish this?

[Rock]

http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/

[Tim Chase]

There are multiple facets to "multiple database" usage
scenarios...which are you looking for?

1) federation: "I've got data in multiple database sources and I
want my Django app to pull from all of them to give a unified view"

2) load balancing A: "I've got one database, but have replicated
it master-master, and I want my Django instances to spread the
load across these databases"

3) load balancing B: "I've got one database, but have replicated
it master-slave, and I want all my reads to go to the slave DBs,
and all my writes to go to the master DB"

4) sharding: "I've partitioned my data in $HUGE_TABLE across
multiple servers (a-la big-table/hadoop) and need to be able to
handle them transparently"

There may be other multi-db issues that I've missed, but knowing
what you're trying to get the "multi-db" magic-wand to solve, the
list can better help.

-tim

[kron4eg]

But be careful with this piece of code, It's only prototype.
For example, it opens new connection to database each time you
accessing manager :)

[Austin Gabel]

Tim. I'm looking for a federation type of usage.  I have several databases that I need to pull data from into one view.

[Malcolm Tredinnick]

On Wed, 2008-11-26 at 11:46 -0600, Tim Chase wrote:
> > I am currently attempting to roll my own multiple database
> > support for django for a new project. I have attempted to
> > search for examples of this but have not found very much.
> > Does anyone have some examples on how I can accomplish this?
>
>
> There are multiple facets to "multiple database" usage
> scenarios...which are you looking for?
>
> 1) federation: "I've got data in multiple database sources and I
> want my Django app to pull from all of them to give a unified view"
>
> 2) load balancing A: "I've got one database, but have replicated
> it master-master, and I want my Django instances to spread the
> load across these databases"
>
> 3) load balancing B: "I've got one database, but have replicated
> it master-slave, and I want all my reads to go to the slave DBs,
> and all my writes to go to the master DB"

By the way, Tim, unless you're using something like Oracle's
gigabit-speed replication or synchronous replication, this last one is
almost always "all reads to go to the slave DB, unless they're related
to a recent write, in which case they should go to the master".

Asynchronous replication (e.g. MySQL's master-slace situation) has a lag
between between writes and when they appear in the slave. The lag can be
seconds. For example, when Pownce were setting up this type of
replication, they had to be very careful to ensure that after you posted
a note, the reads to display the page came from the master so that the
note actually showed up (people tend to get worried when they post
something and it doesn't appear). Ditto for comments, etc. It took a day
or so of fairly intensive work, tracing through possibilities to get
that mostly working for all cases of "reads associated with a recent
write".

This case, (3), is one or the more common and perceived as an easy way
to balance load, but it has those sorts of hidden traps that need to be
avoided.

Your summary is still spot-on, though, and people talking about
"multiple databases" definitely need to be clear as to which situation
they're talking about (and, when it involves replication, whether it's
synchronous or asynchronous -- although it's almost always the latter).

Regards,
Malcolm