Creating and using a project-specific database backend?
George Vilches
Now that the database backend refactoring has landed, and DB
functionality is really easy to extend, how does everyone feel about the
possibility of allowing people to specify their own database backends
within their projects (i.e., without modifying the Django source tree in
any way?) I see this as an excellent way for people to increase their
personal database support without needing their specific little extras
stored within the Django source tree (much like adding auth backends
allows to use things like OpenID without internal changes). And, I'm
pretty sure that this can be done without breaking any backwards
compatibility, as in the following scenarios.
Three scenarios I can think of that would make this work (my preferences
below):
Setup: You've created a new backend in <myproject>.backends.mynewbackend
1) Allow DATABASE_ENGINE in settings.py to accept module paths, and if a
name can't be looked up in the Django backends, try it as an __import__
as a module path itself. You could require it to have a "." if you want
to use it as a module path, as well.
DATABASE_ENGINE = 'backends.mynew'
2) settings.py already supports an AUTHENTICATION_BACKENDS, so there's
already precedent for specifying user-defined backends for Django. What
if we add a DATABASE_BACKENDS? Preferably, it would be a dict, so that
if a DATABASE_ENGINE name doesn't match something in django.db.backends
we could just check the DATABASE_BACKENDS dict to see if the key exists,
and throw the same error as would normally be thrown.
DATABASE_BACKENDS = {'mynew': 'backends.mynew', 'mynew2': 'backends.mynew2'}
DATABASE_ENGINE = 'mynew'
3) Require that a folder called "backends" be in the project and just
check the DATABASE_ENGINE name on it just like it's checked for in
django.db.backends._DATABASE_ENGINE_.DatabaseWrapper.
None of these three options would break backwards compatibility at all,
since they could all import from Django first as expected, and they
could still throw the same error that's currently in there.
My preferences: I think #2 is the clearest and the least likely to have
side effects, followed somewhat closely by #3 (convention's okay), and
#1 would be alright but could be a little hard to avoid side effects.
I will happily write the patches for any of these three things, all of
them are pretty small, I just wanted to see which one would be the
preferred solution. Thoughts?
Thanks,
George
Russell Keith-Magee
>
> Folks,
>
> Now that the database backend refactoring has landed, and DB
> functionality is really easy to extend, how does everyone feel about the
> possibility of allowing people to specify their own database backends
> within their projects (i.e., without modifying the Django source tree in
> any way?) I see this as an excellent way for people to increase their
The broad idea seems reasonable to me. There's no point having an
easily pluggable database engine if you can't plug in your own
database :-)
Regarding the approach - I'm inclined to prefer #1. It's simple and
easy to explain, and I don't see that there is that much potential for
side effects. The only clash I can forsee is if you had your own
toplevel module that mirrored the backend names of Django - this
doesn't really strike me as something that will be a common problem,
and if it is, the solution is easy (rename the clashing external
module).
I don't see the point of #2. Any given project settings file will only
ever need 1 database backend, so why keep an index of backends in the
settings file?
DB Backends shouldn't be a project specific thing, so #3 isn't really
a good approach. We should be encouraging people to write standalone
modules that integrate well with Django as a whole, rather than
pushing some sort of 'copy this module into your project' approach.
Yours,
Russ Magee %-)
Brian Harring
>
> On 8/30/07, George Vilches <g...@thataddress.com> wrote:
> >
> > Folks,
> >
> > Now that the database backend refactoring has landed, and DB
> > functionality is really easy to extend, how does everyone feel about the
> > possibility of allowing people to specify their own database backends
> > within their projects (i.e., without modifying the Django source tree in
> > any way?) I see this as an excellent way for people to increase their
>
> The broad idea seems reasonable to me. There's no point having an
> easily pluggable database engine if you can't plug in your own
> database :-)
>
> Regarding the approach - I'm inclined to prefer #1. It's simple and
> easy to explain, and I don't see that there is that much potential for
> side effects.
+1 from me; original patch v2 did this already anyways (we
specifically need it).
So... we can please get the remaining bits of the backend refactoring
pulled in? :)
~harring
George Vilches
> On 8/30/07, George Vilches <g...@thataddress.com> wrote:
>> Folks,
>>
>> Now that the database backend refactoring has landed, and DB
>> functionality is really easy to extend, how does everyone feel about the
>> possibility of allowing people to specify their own database backends
>> within their projects (i.e., without modifying the Django source tree in
>> any way?) I see this as an excellent way for people to increase their
>
> The broad idea seems reasonable to me. There's no point having an
> easily pluggable database engine if you can't plug in your own
> database :-)
>
> Regarding the approach - I'm inclined to prefer #1. It's simple and
> easy to explain, and I don't see that there is that much potential for
> side effects. The only clash I can forsee is if you had your own
> toplevel module that mirrored the backend names of Django - this
> doesn't really strike me as something that will be a common problem,
> and if it is, the solution is easy (rename the clashing external
> module).
Then in this spirit, my patch (against r6049) is at the end of the
message. Notice that the only real change is:
try:
backend = __import__('%s.base' % settings.DATABASE_ENGINE, {},
{}, [''])
except ImportError, e_user:
Everything else is just an indentation correction (it moves the current
error checking under the second import error). The flow of this allows
the internal packages to still be checked first, and then it tries to
import the package directly without the django.db.backends prefix. If
that doesn't work, the error message and handling is exactly as it used
to be.
I've tried it in several existing situations, seems to hold up fine with
existing Django DB backends and finds and imports new backends properly
(and properly errors if it can't do either). Anyone have additional
thoughts on this approach to having replaceable backends, or is this
good as is?
Thanks,
George
Index: django/db/__init__.py
===================================================================
--- django/db/__init__.py (revision 6049)
+++ django/db/__init__.py (working copy)
@@ -10,18 +10,21 @@
try:
backend = __import__('django.db.backends.%s.base' %
settings.DATABASE_ENGINE, {}, {}, [''])
except ImportError, e:
- # The database backend wasn't found. Display a helpful error message
- # listing all possible database backends.
- from django.core.exceptions import ImproperlyConfigured
- import os
- backend_dir = os.path.join(__path__[0], 'backends')
- available_backends = [f for f in os.listdir(backend_dir) if not
f.startswith('_') and not f.startswith('.') and not f.endswith('.py')
and not f.endswith('.pyc')]
- available_backends.sort()
- if settings.DATABASE_ENGINE not in available_backends:
- raise ImproperlyConfigured, "%r isn't an available database
backend. Available options are: %s" % \
- (settings.DATABASE_ENGINE, ", ".join(map(repr,
available_backends)))
- else:
- raise # If there's some other error, this must be an error in
Django itself.
+ try:
+ backend = __import__('%s.base' % settings.DATABASE_ENGINE, {},
{}, [''])
+ except ImportError, e_user:
+ # The database backend wasn't found. Display a helpful error
message
+ # listing all possible database backends.
+ from django.core.exceptions import ImproperlyConfigured
+ import os
+ backend_dir = os.path.join(__path__[0], 'backends')
+ available_backends = [f for f in os.listdir(backend_dir) if not
f.startswith('_') and not f.startswith('.') and not f.endswith('.py')
and not f.endswith('.pyc')]
+ available_backends.sort()
+ if settings.DATABASE_ENGINE not in available_backends:
+ raise ImproperlyConfigured, "%r isn't an available database
backend. Available options are: %s" % \
+ (settings.DATABASE_ENGINE, ", ".join(map(repr,
available_backends)))
+ else:
+ raise # If there's some other error, this must be an error
in Django itself.
get_introspection_module = lambda:
__import__('django.db.backends.%s.introspection' %
settings.DATABASE_ENGINE, {}, {}, [''])
get_creation_module = lambda:
__import__('django.db.backends.%s.creation' % settings.DATABASE_ENGINE,
{}, {}, [''])
Gary Wilson
> On 8/30/07, George Vilches <g...@thataddress.com> wrote:
>> Now that the database backend refactoring has landed, and DB
>> functionality is really easy to extend, how does everyone feel about the
>> possibility of allowing people to specify their own database backends
>> within their projects (i.e., without modifying the Django source tree in
>> any way?) I see this as an excellent way for people to increase their
>
> The broad idea seems reasonable to me. There's no point having an
> easily pluggable database engine if you can't plug in your own
> database :-)
>
> Regarding the approach - I'm inclined to prefer #1. It's simple and
> easy to explain, and I don't see that there is that much potential for
> side effects. The only clash I can forsee is if you had your own
> toplevel module that mirrored the backend names of Django - this
> doesn't really strike me as something that will be a common problem,
> and if it is, the solution is easy (rename the clashing external
> module).
>
> I don't see the point of #2. Any given project settings file will only
> ever need 1 database backend, so why keep an index of backends in the
> settings file?
I suspect that this backend refactoring will also make multi-db functionality
easier to implement. Shouldn't we be looking down the road for this too? In
which case maybe a mixture of #1 and #2 would be good, similar to how the
multi-db branch [1] defines multiple backends.
I guess if method #1 is used now, that code could still be used later on to
load multiple database backends from which ever settings format that is used.
Just thought I would mention it though.
Gary
[1] http://code.djangoproject.com/wiki/MultipleDatabaseSupport
George Vilches
> Russell Keith-Magee wrote:
>> On 8/30/07, George Vilches <g...@thataddress.com> wrote:
>>> Folks,
>>>
>>> Now that the database backend refactoring has landed, and DB
>>> functionality is really easy to extend, how does everyone feel about the
>>> possibility of allowing people to specify their own database backends
>>> within their projects (i.e., without modifying the Django source tree in
>>> any way?) I see this as an excellent way for people to increase their
>> The broad idea seems reasonable to me. There's no point having an
>> easily pluggable database engine if you can't plug in your own
>> database :-)
>>
>> Regarding the approach - I'm inclined to prefer #1. It's simple and
>> easy to explain, and I don't see that there is that much potential for
>> side effects. The only clash I can forsee is if you had your own
>> toplevel module that mirrored the backend names of Django - this
>> doesn't really strike me as something that will be a common problem,
>> and if it is, the solution is easy (rename the clashing external
>> module).
>
> Then in this spirit, my patch (against r6049) is at the end of the
> message. Notice that the only real change is:
And I'm amending this slightly due to something I missed in how the
other modules are imported, we have to adjust the import path for the
introspection/creation/etc. modules as well. Patch at bottom, as before.
The only two changes is that there is now an "import_path" string which
contains the working prefix to the modules (either "django.db.backends"
or empty), and each import after the base uses the one that's appropriate.
Thanks,
George
Index: django/db/__init__.py
===================================================================
--- django/db/__init__.py (revision 6049)
+++ django/db/__init__.py (working copy)
@@ -8,24 +8,29 @@
settings.DATABASE_ENGINE = 'dummy'
try:
- backend = __import__('django.db.backends.%s.base' %
settings.DATABASE_ENGINE, {}, {}, [''])
+ import_path = 'django.db.backends.'
+ backend = __import__('%s%s.base' % (import_path,
settings.DATABASE_ENGINE), {}, {}, [''])
except ImportError, e:
- # The database backend wasn't found. Display a helpful error message
- # listing all possible database backends.
- from django.core.exceptions import ImproperlyConfigured
- import os
- backend_dir = os.path.join(__path__[0], 'backends')
- available_backends = [f for f in os.listdir(backend_dir) if not
f.startswith('_') and not f.startswith('.') and not f.endswith('.py')
and not f.endswith('.pyc')]
- available_backends.sort()
- if settings.DATABASE_ENGINE not in available_backends:
- raise ImproperlyConfigured, "%r isn't an available database
backend. Available options are: %s" % \
- (settings.DATABASE_ENGINE, ", ".join(map(repr,
available_backends)))
- else:
- raise # If there's some other error, this must be an error in
Django itself.
+ try:
+ import_path = ''
+ backend = __import__('%s%s.base' % (import_path,
settings.DATABASE_ENGINE), {}, {}, [''])
+ except ImportError, e_user:
+ # The database backend wasn't found. Display a helpful error
message
+ # listing all possible database backends.
+ from django.core.exceptions import ImproperlyConfigured
+ import os
+ backend_dir = os.path.join(__path__[0], 'backends')
+ available_backends = [f for f in os.listdir(backend_dir) if not
f.startswith('_') and not f.startswith('.') and not f.endswith('.py')
and not f.endswith('.pyc')]
+ available_backends.sort()
+ if settings.DATABASE_ENGINE not in available_backends:
+ raise ImproperlyConfigured, "%r isn't an available database
backend. Available options are: %s" % \
+ (settings.DATABASE_ENGINE, ", ".join(map(repr,
available_backends)))
+ else:
+ raise # If there's some other error, this must be an error
in Django itself.
-get_introspection_module = lambda:
__import__('django.db.backends.%s.introspection' %
settings.DATABASE_ENGINE, {}, {}, [''])
-get_creation_module = lambda:
__import__('django.db.backends.%s.creation' % settings.DATABASE_ENGINE,
{}, {}, [''])
-runshell = lambda: __import__('django.db.backends.%s.client' %
settings.DATABASE_ENGINE, {}, {}, ['']).runshell()
+get_introspection_module = lambda: __import__('%s%s.introspection' %
(import_path, settings.DATABASE_ENGINE), {}, {}, [''])
+get_creation_module = lambda: __import__('%s%s.creation' %
(import_path, settings.DATABASE_ENGINE), {}, {}, [''])
+runshell = lambda: __import__('%s%s.client' % (import_path,
settings.DATABASE_ENGINE), {}, {}, ['']).runshell()
connection = backend.DatabaseWrapper(**settings.DATABASE_OPTIONS)
DatabaseError = backend.DatabaseError