Django ignoring Create_DB setting on test

[Carl Marshall]

Hi all,

I think I've found a bug here - or certainly I have some behaviour that I don't expect and don't want.

My django (2.1.7) project is providing a query/read interface to some existing legacy databases. These are Oracle based (and that brings its own challenges, as they're not current Oracle either) and are only available via a read_only user account. To handle this I've had to duplicate and modify the default django.db.backends.oracle package [1] and create my own that reverts some of the changes made in django 2.0 back to 1.11 equivalents which are then more compatible with the Oracle version I'm working against. This in the main works fine. However, problems start when we get to trying to run the Django test command.

Based on https://docs.djangoproject.com/en/2.1/ref/settings/#test, specifically:

"""

CREATE_DB¶

Default: True

This is an Oracle-specific setting.

If it is set to False, the test tablespaces won’t be automatically created at the beginning of the tests or dropped at the end.

"""

... I have set this to False (and confirmed this is being picked up correctly), and naively assumed this would do what it says. Unfortunately this doesn't seem to be the case as the test runner is still trying to create a test database for this secondary database (the default database for the project is an sqlite db). More unfortunately, because of what looks like Oracle incompatibilities combined with the attempt to create the test database in the face of this setting, is failing because of bad syntax [2]

I'm still trying to dig into where this is all falling over at, but what I'd expect here (and ideally want) is that Test runs, but it only creates a test_database for the default database defined in settings, and ignores the read-only secondary databases defined. Suggestions to create my own DiscoverRunner for the test are looking to be moot, as the issue isn't that I don't want a database created at all, it's that I don't want specific databases to be created during test cycles and that doesn't appear to be something in DiscoverRunner, but in other sub modules elsewhere in the code (and more and more looking like it's in the db.backends.vendor type modules).

Anyone got any insights they can offer here please?

E.g.: There are no migrations in any of the projects apps, so why are we seeing db/migrations/ in the stack trace for this fault, and for this secondary database?

Thanks

Carl

Footnotes

1. Most of this new/duplicated db backend is identical to the Django 2.1.7 package, with the following changes:

base.py

 74:     vendor = 'cerner_oracle'

 75:     display_name = 'Cerner Oracle 11g'

197: return '{0}/{1}@{2}'.format(self.settings_dict['USER'], self.settings_dict['PASSWORD'], self._dsn())

These handle niceties like not being the default Oracle name, and fixes an issue with the connection string that otherwise stops me from getting any database connection.

introspection.py

 - Modifications to the queries to point at tables that can be accessed such as "ALL_TABS" rather than "USER_TABS" as the user account is not the db creator, and thus doesn't have any tables against it in USER_* tables. The upshot of these changes is that I've been able to use the python manage.py inspectdb command to pull out django model classes from the table structures as we'd like.

2.  user$ python manage.py test  OUTPUTS -->

Settings loading from local.py. env file being read=True

Creating test database for alias 'default'...

Creating test database for alias 'cerner_dev'...

Traceback (most recent call last):

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/backends/utils.py", line 83, in _execute

    return self.cursor.execute(sql)

  File "/Users/carl/Projects/diakonia-internal/diakonia-internal-repo/diakonia/atlas/db/backends/cerner/base.py", line 507, in execute

    return self.cursor.execute(query, self._param_generator(params))

cx_Oracle.DatabaseError: ORA-02000: missing ALWAYS keyword

The above exception was the direct cause of the following exception:

Traceback (most recent call last):

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 55, in ensure_schema

    editor.create_model(self.Migration)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 312, in create_model

    self.execute(sql, params or None)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/backends/base/schema.py", line 133, in execute

    cursor.execute(sql, params)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/backends/utils.py", line 68, in execute

    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers

    return executor(sql, params, many, context)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/backends/utils.py", line 85, in _execute

    return self.cursor.execute(sql, params)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/utils.py", line 89, in __exit__

    raise dj_exc_value.with_traceback(traceback) from exc_value

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/backends/utils.py", line 83, in _execute

    return self.cursor.execute(sql)

  File "/Users/carl/Projects/diakonia-internal/diakonia-internal-repo/diakonia/atlas/db/backends/cerner/base.py", line 507, in execute

    return self.cursor.execute(query, self._param_generator(params))

django.db.utils.DatabaseError: ORA-02000: missing ALWAYS keyword

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

  File "manage.py", line 24, in <module>

    execute_from_command_line(sys.argv)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/__init__.py", line 381, in execute_from_command_line

    utility.execute()

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/__init__.py", line 375, in execute

    self.fetch_command(subcommand).run_from_argv(self.argv)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/commands/test.py", line 26, in run_from_argv

    super().run_from_argv(argv)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/base.py", line 316, in run_from_argv

    self.execute(*args, **cmd_options)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/base.py", line 353, in execute

    output = self.handle(*args, **options)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/commands/test.py", line 56, in handle

    failures = test_runner.run_tests(test_labels)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/test/runner.py", line 604, in run_tests

    old_config = self.setup_databases()

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/test/runner.py", line 551, in setup_databases

    self.parallel, **kwargs

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/test/utils.py", line 174, in setup_databases

    serialize=connection.settings_dict.get('TEST', {}).get('SERIALIZE', True),

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/backends/base/creation.py", line 68, in create_test_db

    run_syncdb=True,

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/__init__.py", line 148, in call_command

    return command.execute(*args, **defaults)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/base.py", line 353, in execute

    output = self.handle(*args, **options)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/base.py", line 83, in wrapped

    res = handle_func(*args, **kwargs)

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/core/management/commands/migrate.py", line 203, in handle

    fake_initial=fake_initial,

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/migrations/executor.py", line 91, in migrate

    self.recorder.ensure_schema()

  File "/Users/carl/.virtualenvs/diakonia-internal/lib/python3.7/site-packages/django/db/migrations/recorder.py", line 57, in ensure_schema

    raise MigrationSchemaMissing("Unable to create the django_migrations table (%s)" % exc)

django.db.migrations.exceptions.MigrationSchemaMissing: Unable to create the django_migrations table (ORA-02000: missing ALWAYS keyword)

[Carl Marshall]

I'll note that "CREATE_DB" is slightly misleading, and would be more accurately called "CREATE_SEPARATE_TEST_DB_IN_ORACLE".

The error I'm getting is somewhat misleading also as there's nothing wrong with the ALWAYS keyword, the problem is that the test script is trying to create a table in a database I thought it had been told not to create anything in. Instead it is calling `'CREATE TABLE "DJANGO_MIGRATIONS" ("ID" NUMBER(11) GENERATED BY DEFAULT ON NULL AS IDENTITY NOT NULL PRIMARY KEY, "APP" NVARCHAR2(255) NULL, "NAME" NVARCHAR2(255) NULL, "APPLIED" TIMESTAMP NOT NULL)'` and then rightly falling over because this is a READ-ONLY DB.

The ability to tell Django that a database is Read-Only would be a greatly appreciated feature!

I'm continuing to try and work this issue through though if anyone has solve this (or similar), please tell me how? :)