Multi-db support

[Jeff Bell]

Hi all,

Am I wasting my time trying to figure out multi-db functionality. I
know that Alex Gaynor is working on a GSOC project that is slatted for
Django 1.2 release that deals with this, but I can't imagine that
being released anytime soon.

I've been able to configure a MultiDBManager for my test site, where I
connect to a local mysql databased for admin, auth, session, etc and
connect to a remote mysql to query 2 models.

My problem occurs during larger queries - I get 'Too many connections'
error. If I run these queries through django's default manager they
are handled fine. Can anyone point me in the right direction, maybe
the source code where django handles this. I'm pretty sure I need to
close the connection in my MultiDBManager but not sure how to
accomplish this.

Thanks in advance

[Russell Keith-Magee]

On Thu, Oct 15, 2009 at 3:08 AM, Jeff Bell <jeffreyme...@gmail.com> wrote:
>
> Hi all,
>
> Am I wasting my time trying to figure out multi-db functionality.  I
> know that Alex Gaynor is working on a GSOC project that is slatted for
> Django 1.2 release that deals with this, but I can't imagine that
> being released anytime soon.

Don't be so sure.

The gsoc2009 multi-db branch that Alex was working on over the summer
is usable, as long as you don't mind a couple of sharp edges. There's
only a small handful of issues left, and most of those are really
about the internals. What is in the branch is reasonably well
documented - the two changes to look for are the new DATABASES
setting, and the using() method on querysets.

There is still some work to be done to make the public interface a
little easier to use for some common use cases; the more feedback we
get on these aspects, the better.

So - if you're feeling adventurous (and if you're willing to try out
multidb on your own, I'm guessing you are), it's worth taking a look
at Alex's branch.

Yours,
Russ Magee %-)

[David De La Harpe Golden]

Jeff Bell wrote:

> My problem occurs during larger queries - I get 'Too many connections'
> error. If I run these queries through django's default manager they
> are handled fine. Can anyone point me in the right direction, maybe
> the source code where django handles this. I'm pretty sure I need to
> close the connection in my MultiDBManager but not sure how to
> accomplish this.
>

If you mean something based on MultiDBManager here:

http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/

I think it makes an awful lot of DatabaseWrappers as-is. I _think_ (but
could be wrong), that seriously lowers connection reuse, so you end up
with a lot of dangling connections.

Here's something further pared down I'm currently testing with django
1.1.1 to tide us over until true multi-db for our own simple
multi-postgresql-database case. It "seems to work", though doesn't
address a bunch of stuff true multi-db would cover - I just needed
(okay, wanted) read-only access to an extra database through django orm,
but you could probably add Eric's _insert. I also only use processes not
threads, with mod_wsgi daemon mode. Doing it this way, you should
probably consider an altconnection1.close() sometimes (it won't happen
implicitly sometimes unlike the main connection), but it's only using
one extra DatabaseWrapper instance per extra database.

from django.conf import settings
from django.db import backend
altconnection1 = backend.DatabaseWrapper(
'DATABASE_HOST': settings.DATABASE_HOST,
'DATABASE_NAME': "altdb1",
'DATABASE_OPTIONS': settings.DATABASE_OPTIONS,
'DATABASE_PASSWORD': '',
'DATABASE_PORT': '',
'DATABASE_USER': settings.DATABASE_USER,
'TIME_ZONE': settings.TIME_ZONE,
})

class ExtDBManager(models.Manager):
def __init__(self, connection, *args, **kwargs):
self.connection = connection
super(ExtDBManager, self).__init__(*args, **kwargs)

def get_query_set(self):
qs = super(ExtDBManager, self).get_query_set()
qs.query.connection = self.connection
return qs

class MyModel1(models.Model):
class Meta:
managed = False
db_table = 'table1'
objects = ExtDBManager(altconnection1)
...

class MyModel2(models.Model):
class Meta:
managed = False
db_table = 'table2'
objects = ExtDBManager(altconnection1)
...

[David De La Harpe Golden]

David De La Harpe Golden wrote:

> Here's something further pared down I'm currently testing with django
> 1.1.1 to tide us over until true multi-db for our own simple
> multi-postgresql-database case. It "seems to work", though doesn't
> address a bunch of stuff true multi-db would cover

> class ExtDBManager(models.Manager):
...

Hmph. In case some poor soul stumbles on this thread in the archives-
Turns out that didn't quite work as it was for many to many fields, as
they try to construct a further subclass of the manager. So, a factory
function that churns out instances of subclasses with the extra
connection as a class field so that the further subclassing inherits the
connection - i.e.

def make_extdbmanager(conn, *args, **kwargs):
if isinstance(conn, basestring):
c = get_connection(conn) # just a convenience function
else:
c = conn
class _ExtDBManager(models.Manager):
connection = c
def get_query_set(self):
qs = super(_ExtDBManager, self).get_query_set()

qs.query.connection = self.connection
return qs

return _ExtDBManager(*args, **kwargs)

Yeah, this is probably all not the best idea ever, but all just to tide
us over.