Test database, --keepdb parameter, and migrations

[Dan Davis]

There are a cluster of issues that I find difficult to resolve as a user, and as a Django developer, I think there is potential to make them easier through features in Django itself. I'm interested in hearing better work-arounds than I use now, but I'm more interested in helping clarify how the product, e.g. Django, can get better at supporting these scenarios "out-of-the-box".  What I am hoping for is a conversation, not a simple bug report.

"out-of-the-box" Django supports a --keepdb argument.  In many of my environments, I'd like to make this the default in my environment, but I see no *Good* way to do that unless I wrap the test management command, and change the default of the keepdb parameter.    Many developers use django-nose or pytest-django to gain additional capabilities, but REUSE_DB=1 for django-nose and --reuse-db for pytest work differently than --keepdb.   I just filed https://github.com/pytest-dev/pytest-django/issues/704 on pytest-django about that.

It would be nice if "out-of-the-box" Django supported some way to make "keepdb" the default. and some way to override it if it were the default.   Another way to do that would be to make the "CREATEDB" argument of Oracle more general and to support it on other databases.

Another work-around for me would be to overwrite the "migrate" management command to add a "--testdb" option so that it could run migrations on the test database explicitly, or to write my own "testmigrate" management command.

Suppose my database settings include this:

        'external': {

            'ENGINE': 'django.db.backends.oracle',

            'NAME': 'EXTERNAL',

            'USER': external_username,

            'PASSWORD': external_password,

            'TEST': {

                'CREATE_DB': False,

                'CREATE_USER': False,

                'NAME': 'EXTERNAL',

                'USER': external_username,

                'PASSWORD': external_password,

             }

         }

This is almost like the below:

        'external': {

            'ENGINE': 'django.db.backends.oracle',

            'NAME': 'EXTERNAL',

            'USER': external_username,

            'PASSWORD': external_password,

            'TEST': {

                'MIRROR': 'external',

             }

         }

The user of these sort of capability needs to understand database routers, or meta options that are similar.   But that may not be enough - entering "./manage.py migrate --database external" would try to migrate the database, but running tests will always attempt to migrate the database.  Perhaps we need better documentation for "MIRROR",  because it is clearly intended for situations where there are read/write database aliases and read replica databases, not for read-only external databases that are used during tests.

I've fixed this by monkey patching the database creation in a test runner, very similarly to how pytest-django and django-nose do, but that has a code smell that suggests that Django "out of the box" should have some capabilities, maybe in the database router, that prevent migrations for even the DJANGO_MIGRATIONS table.

Another thing that could be done is to provide a cleardb management command  that supports a "--testdb" option, and removes all the definitions, including DDL, for users unable to drop and recreate their database.  This also supports user who have been running with --syncdb, and now they wish to make migrations, or they've made a bunch of migrations and they wish to start fresh and clean things up.  How to clean up on a per-backend basis is a bit complicated, and a place where Django doesn't hide the RDBMS sufficiently and users end-up working through tools such as Toad or SQL Developer to clean things up.

Thoughts?   Has this come up before?