Will future Django releases support multiple databases?

[Thierry]

Right now, Django doesn't seem to support the multiple databases on
the same server or different servers. Will that feature be available
out of the box for future releases?

[Tim Chase]

> Right now, Django doesn't seem to support the multiple
> databases on the same server or different servers. Will that
> feature be available out of the box for future releases?

You may want to eavesdrop on the Django Developers group (as
opposed to this Django Users group) where Alex Gaynor has been
posting updates on his GSoC project to add multi-DB support to
Django[1]. There are several aspects to multi-DB support[2] but
I believe he's addressing some of the more common use-cases. You
can even follow along if you like to live dangerously as I
believe he's got a public repository for his code on GitHub.

-tim

[1]
http://groups.google.com/group/django-developers/search?group=django-developers&q=alex+gaynor+%28multi-db+OR+multiple-database%29

[2]
http://groups.google.com/group/django-users/browse_thread/thread/663046559fd0f9c1/
for my posted concerns

[Alex Gaynor]

I have both a public code repository on github at github.com/alex/django/ as well as a branch in Django's SVN at branches/soc2009/multidb.  The best place to find out about the work is the django-developers list, where I have sent any number of emails discussing both the overall design, as well as the status.

Alex

--
"I disapprove of what you say, but I will defend to the death your right to say it." --Voltaire
"The people's good is the highest law."--Cicero

[Emily Rodgers]

On Jun 25, 6:45 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:
> On Thu, Jun 25, 2009 at 10:24 AM, Tim Chase

> <django.us...@tim.thechases.com>wrote:

> > > Right now, Django doesn't seem to support the multiple
> > > databases on the same server or different servers.  Will that
> > > feature be available out of the box for future releases?
>
> > You may want to eavesdrop on the Django Developers group (as
> > opposed to this Django Users group) where Alex Gaynor has been
> > posting updates on his GSoC project to add multi-DB support to
> > Django[1].  There are several aspects to multi-DB support[2] but
> > I believe he's addressing some of the more common use-cases.  You
> > can even follow along if you like to live dangerously as I
> > believe he's got a public repository for his code on GitHub.
>
> > -tim
>
> > [1]
>

> >http://groups.google.com/group/django-developers/search?group=django-...
>
> > [2]
>
> >http://groups.google.com/group/django-users/browse_thread/thread/6630...

> > for my posted concerns
>
> I have both a public code repository on github at github.com/alex/django/ as
> well as a branch in Django's SVN at branches/soc2009/multidb.  The best
> place to find out about the work is the django-developers list, where I have
> sent any number of emails discussing both the overall design, as well as the
> status.
>
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." --Voltaire
> "The people's good is the highest law."--Cicero

Hi,

Alex - do you know roughly when this is likely to be incorporated into
django? I am working on a project that is going to need to connect to
multiple databases (I am just planning / designing at the moment), and
I want to know whether I should start investigating how to do this
using django 1.0.2 or if that would be a waste of my time.

If it is going to be a while, where would you suggest I start looking
if I need to understand how to do it with django 1.0.2?

Cheers,
Em

[Alex Gaynor]

It will, at a minimum not occur until the end of the summer.  Beyond that I can't make any really useful guesses, other than to say that the aggregation GSOC project from last summer
ultimately made it's way into django trunk in January, about 3-4 months after the official end of GSOC.  That may or may not ultimately be representative of how long it will take multi-db to make it's way back into Django.  A lot of it probably depends on how the 1.2/1.3 releases are scheduled.  For example is 1.2 ends up being a smaller release that's put out in Novemberish I wouldn't expect multi-db to be considered for trunk until 1.3.  Like I said all of that's rather speculative :)

[Emily Rodgers]

Thanks Alex. I don't think I can wait that long so it looks like I am
going to have to get my hands dirty!

Is this [1] a sensible starting point?

[1] http://www.mechanicalgirl.com/view/multiple-database-connection-a-simple-use-case/

[Alex Gaynor]

That blog post is a good source of information, as is: http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/ (which unfortunately appears to be down right now).

[Emily Rodgers]

On Jul 1, 3:44 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:
> On Wed, Jul 1, 2009 at 9:39 AM, Emily Rodgers <

> >http://www.mechanicalgirl.com/view/multiple-database-connection-a-sim...
>
> That blog post is a good source of information, as is:http://www.eflorenzano.com/blog/post/easy-multi-database-support-djan...

> unfortunately appears to be down right now).
>
> Alex
>
> --
> "I disapprove of what you say, but I will defend to the death your right to
> say it." --Voltaire
> "The people's good is the highest law."--Cicero

Thanks - it is great that you are working on this because it is one of
the few things that I find a little frustrating about django
(otherwise I am a big fan and often bore my co-workers who prefer to
code in perl - strange people).

[Emily Rodgers]

On Jul 1, 3:44 pm, Alex Gaynor <alex.gay...@gmail.com> wrote:
> On Wed, Jul 1, 2009 at 9:39 AM, Emily Rodgers <

> >http://www.mechanicalgirl.com/view/multiple-database-connection-a-sim...
>
> That blog post is a good source of information, as is:http://www.eflorenzano.com/blog/post/easy-multi-database-support-djan...

> unfortunately appears to be down right now).

Hi,

I have tried implementing this, but I am running into the problems
that others have about too many idle MySQL connections. Is there an
easy way to get around this?

Alex - you mention something here [1] about needing to track the
connections (that are not the default connection). Should I do
something analogous to the connection_counter attribute of the
django.db.models.Manager class?

Em


[1] http://groups.google.com/group/django-users/browse_thread/thread/6ed797f6b00dca90

[Emily Rodgers]

On Jul 3, 3:29 pm, Emily Rodgers <emily.kate.rodg...@googlemail.com>

OK now I am running into even more problems :(

I am probably being dumb, but I assumed there were only problems with
foreign keys accross the different databases, but I am finding that I
can't follow foreign keys from the secondary database to another table
in the secondary database (even though both tables have the
MultiDbManager defined for them). It just assumes all foreign keys are
for tables in my primary database :(

I may have to set up another script on another django project with a
cli to get the data I want since I am only reading from the second db
(it just seems to wrong!!). Or maybe just use mysqldb?

[Alex Gaynor]

Does your multidb manager have use_for_related_fields = True on it?

[Emily Rodgers]

2009/7/3 Alex Gaynor <alex....@gmail.com>:

Ah - no. Thanks for spotting that!

<trying it out>

Yay - it worked. Thanks Alex :-)

Still stuck on the too many connections thing. Will have to look into
it on Monday though. Any ideas welcome.

Em

[Emily Rodgers]

On Jul 3, 6:36 pm, Emily Rodgers <emily.kate.rodg...@googlemail.com>
wrote:
> 2009/7/3 Alex Gaynor <alex.gay...@gmail.com>:

> >> > that others have about too many idle MySQLconnections. Is there an

> >> > easy way to get around this?
>
> >> OK now I am running into even more problems :(
>
> >> I am probably being dumb, but I assumed there were only problems with
> >> foreign keys accross the different databases, but I am finding that I
> >> can't follow foreign keys from the secondary database to another table
> >> in the secondary database (even though both tables have the
> >> MultiDbManager defined for them). It just assumes all foreign keys are
> >> for tables in my primary database :(
>
> >> I may have to set up another script on another django project with a
> >> cli to get the data I want since I am only reading from the second db
> >> (it just seems to wrong!!). Or maybe just use mysqldb?
>

> > Does your multidbmanagerhave use_for_related_fields = True on it?

>
> Ah - no. Thanks for spotting that!
>
> <trying it out>
>
> Yay - it worked. Thanks Alex :-)
>

> Still stuck on the too manyconnectionsthing. Will have to look into

> it on Monday though. Any ideas welcome.
>
> Em

I am still hitting a brick wall with this 'too-many-connections'
thing :(

I am assuming that I need to be doing something with the Manager
creation_counter, but I can't see where this is being used to close
default connections, so trying to implement something similar is
tricky :(

Can anyone help?

Em

[Emily Rodgers]

On Jul 13, 4:51 pm, Emily Rodgers <emily.kate.rodg...@googlemail.com>

I finally think I have got this working! I have only tested this using
a script that pokes data into a database using django, not using a web
app (yet), so it may need tweaking in that situation (the signal
stuff), but, hopefully that won't be too much of a problem.

Basically I found an example that kimus.linuxus provided [1] that uses
a later version of django (I am using 1.0.2), and used the ideas from
it to adapt Eric Florenzano's example, and here is what I did:

In my models.py:

from project.app.managers import *

class Db1Model(models.Model):
_default_manager = MultiDBManager('db1')
objects = MultiDBManager('db1')
db = 'db1'

class Meta:
abstract = True

# then for all models that use db1:
class MyModelFromDb1(Db1Model):
...


class Db2Model(models.Model):
_default_manager = MultiDBManager('db2')
objects = MultiDBManager('db2')
db = 'db2'

class Meta:
abstract = True

# ditto for models in db2
class MyModelFromDb2(Db2Model):
...

Then in managers.py:

from django.db import models
from django.conf import settings
from django.db.models import sql
from django.db.transaction import savepoint_state
from django.core import signals


try:
import thread
except ImportError:
import dummy_thread as thread

_connections = {}

def close_connection(**kwargs):
global _connections
for connection in _connections.itervalues():
connection.close()
_connections = {}
signals.request_finished.connect(close_connection)

class MultiDBManager(models.Manager):
def __init__(self, database, *args, **kwargs):
self.database = database
self.use_for_related_fields = True
super(MultiDBManager, self).__init__(*args, **kwargs)

def get_query_set(self):
qs = super(MultiDBManager, self).get_query_set()
qs.query.connection = self.get_db_wrapper()
return qs

def get_db_wrapper(self):
if not self.database:
from django import connection
return connection
if not _connections.has_key(self.database):
settings_dict = settings.DATABASES[self.database]
backend = __import__('django.db.backends.%s.base' %
settings_dict['DATABASE_ENGINE'], {}, {}, ['base'])
backup = {}
for key, value in settings_dict.iteritems():
backup[key] = getattr(settings, key)
setattr(settings, key, value)
wrapper = backend.DatabaseWrapper()
wrapper._cursor(settings)
_connections[self.database] = wrapper
for key, value in backup.iteritems():
setattr(settings, key, value)
return _connections[self.database]

def _insert(self, values, return_id=False, raw_values=False):
query = sql.InsertQuery(self.model, self.get_db_wrapper())
query.insert_values(values, raw_values)
ret = query.execute_sql(return_id)
query.connection._commit()
thread_ident = thread.get_ident()
if thread_ident in savepoint_state:
del savepoint_state[thread_ident]
return ret

Then all you need to do is put the database connection details in
settings.py:

DATABASE_ENGINE = 'mysql'
DATABASE_NAME = 'db1'
DATABASE_USER = 'auser'
DATABASE_PASSWORD = 'somepassword'
DATABASE_HOST = ''
DATABASE_PORT = ''


DATABASES = dict(
db1 = dict(
DATABASE_ENGINE=DATABASE_ENGINE,
DATABASE_NAME=DATABASE_NAME,
DATABASE_USER=DATABASE_USER,
DATABASE_PASSWORD=DATABASE_PASSWORD,
DATABASE_HOST=DATABASE_HOST,
DATABASE_PORT=DATABASE_PORT,
),
db2 = dict(
DATABASE_ENGINE="mysql",
DATABASE_NAME='db2',
DATABASE_USER='anotheruser',
DATABASE_PASSWORD='password',
DATABASE_HOST='',
DATABASE_PORT='',
),
)

It seems to have worked for me so far. Obviously this is not a long
term solution, just something to get by with until Alex's code is
ready =)

Em

[1] http://code.djangoproject.com/ticket/1142#comment:65
[2] http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/