Database configuration errors saying 'ENGINE' isn't set when it is

[Shaun Kruger]

I have a django 1.4.3 application (upgrading to 1.4.14 made no difference) that is behaving oddly.  I know 1.4 isn't officially supported, but I'm hoping to come up with some understanding about what is going wrong so I can get by for the next few weeks until we can start our upgrade to 1.7.  I am also concerned that this problem won't simply be fixed by going to django 1.7 because I have not been able to find any reference to these conditions in my searching.

The application is deployed as a pip installable package into a virtualenv in a directory called vproton.  The local_settings.py file is specified by environment variable and lives in a directory that is specified by the PYTHONPATH environment variable sys.path has everything we expect it to.

I first started investigating based on an odd error coming from a celery worker.

=================

  File "/home/proton/vproton-15e49fb5/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 306, in get_default_columns

    r = '%s.%s' % (qn(alias), qn2(field.column))

  File "/home/proton/vproton-15e49fb5/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 49, in quote_name_unless_alias

    r = self.connection.ops.quote_name(name)

  File "/home/proton/vproton-15e49fb5/local/lib/python2.7/site-packages/django/db/backends/dummy/base.py", line 15, in complain

    raise ImproperlyConfigured("settings.DATABASES is improperly configured. "

ImproperlyConfigured: settings.DATABASES is improperly configured. Please supply the ENGINE value. Check settings documentation for more details.

=================

It took a while, but I was able to figure out that this error was coming up because the 'dummy' backend was being loaded instead of the one that is configured in my settings file.

What has been baffling is that we all have verified that our settings file appears to be configured correctly and that django is loading it correctly.

=================

$ PROTON_LOG_LOCATION=/home/proton/logs DJANGO_SETTINGS_MODULE=local_settings PYTHONPATH=/home/proton/config vproton/bin/django-admin.py shell

Python 2.7.6 (default, Mar 22 2014, 22:59:56) 

[GCC 4.8.2] on linux2

Type "help", "copyright", "credits" or "license" for more information.

(InteractiveConsole)

>>> from django.conf import settings

>>> settings.DATABASES

{'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', 'HOST': '*******', 'PASSWORD': '*******', 'NAME': 'proton_staging', 'USER': 'proton'}}

=================

As I dug deeper I wanted to understand how django's database connections were handled so I began inspecting django.db (https://github.com/django/django/blob/1.4.3/django/db/__init__.py).  I determined that the settings object imported from django.conf seems normal, but things take a problematic turn once I get to ConnectionHandler which is imported from django.db.utils (https://github.com/django/django/blob/1.4.3/django/db/utils.py).  

django.db.connections is initialized by calling ConnectionHandler(settings.DATABASES) and ConnectionHandler sets self.databases equal to the value passed as settings.DATABASES so I expect django.db.settings.DATABASES to match django.db.connections.databases, but they do not match.

=================

>>> import django.db

>>> django.db.settings.DATABASES

{'default': {'ENGINE': 'django.db.backends.postgresql_psycopg2', 'HOST': '*******', 'PASSWORD': '*******', 'NAME': 'proton_staging', 'USER': 'proton'}}

>>> django.db.connections.databases

{'default': {'ENGINE': 'django.db.backends.dummy', 'TEST_MIRROR': None, 'NAME': '', 'TEST_CHARSET': None, 'TIME_ZONE': 'America/Chicago', 'TEST_COLLATION': None, 'PORT': '', 'HOST': '', 'USER': '', 'TEST_NAME': None, 'PASSWORD': '', 'OPTIONS': {}}}

=================

I am now left with a question.  Is there any way that django.db can load a bad django.conf.settings when the settings module is given by environment variable?  As far as I can tell this problem shouldn't even be possible since importing django.conf initializes settings as an instance of LazySettings that loads .  Has anyone else run into this before?  Is this a known issue that would be resolved by just waiting for the 1.7 upgrade we have planned?  Is there anything that I should check for in my settings file that would cause a problem, but not result in any errors coming back to me?

[Shaun Kruger]

After a few more hours of going over this I determined the root cause and now I simply do not understand how this was not a problem in dev instances or any other environments.

This was a circular import issue.

local_settings.py started with an import that fetched defaults from the app.

from <app>.settings import *

<app>/__init__.py imported some error handlers from common.utils (common is one of our 'apps').  common.utils also imported one model which invoked dependencies that tried to import django.db.  

In my testing I was having trouble understanding why when I started a python shell with all of the configuration environment variables I couldn't import django.db without getting an import error on local_settings.  This circular import explains that.  Moving my error handler functions to a different module resolved the circular import and now everything is working.

I was finally able to debug this by inserting a debugging statement at the beginning of django/db/__init__.py that called traceback.print_stack().  The stack trace showed the "from <app>.settings import *" followed by the import in <app>/__init__.py that then imported common.utils that imported a database model and then finally the import of django.db.