Database caching and multi db
tiemonster
creates all tables on both of my connections (which it should not do,
if I read the documentation correctly), but also tries to create the
cache table twice, causing the error below. Please let me know if I'm
missing something here. I have two MySQL connections, called default
and datastore, and have my CACHE_BACKEND set to db://cache_table.
Traceback (most recent call last):
File "./manage.py", line 11, in <module>
execute_manager(settings)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/__init__.py", line 438, in execute_manager
utility.execute()
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/__init__.py", line 379, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/base.py", line 191, in run_from_argv
self.execute(*args, **options.__dict__)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/base.py", line 218, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/commands/test.py", line 37, in handle
failures = test_runner.run_tests(test_labels)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/test/simple.py", line 313, in run_tests
old_config = self.setup_databases()
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/test/simple.py", line 270, in setup_databases
connection.creation.create_test_db(self.verbosity, autoclobber=not
self.interactive)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/db/backends/creation.py", line 358, in create_test_db
call_command('createcachetable', cache_name)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/__init__.py", line 166, in call_command
return klass.execute(*args, **defaults)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/base.py", line 218, in execute
output = self.handle(*args, **options)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/base.py", line 318, in handle
label_output = self.handle_label(label, **options)
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/core/management/commands/createcachetable.py", line 50, in
handle_label
curs.execute("\n".join(full_statement))
File "/usr/local/lib/python2.6/dist-packages/Django-1.2.1-py2.6.egg/
django/db/backends/mysql/base.py", line 86, in execute
return self.cursor.execute(query, args)
File "/usr/lib/pymodules/python2.6/MySQLdb/cursors.py", line 166, in
execute
self.errorhandler(self, exc, value)
File "/usr/lib/pymodules/python2.6/MySQLdb/connections.py", line 35,
in defaulterrorhandler
raise errorclass, errorvalue
_mysql_exceptions.OperationalError: (1050, "Table 'cache_table'
already exists")
tiemonster
(http://docs.djangoproject.com/en/1.2/topics/testing/#testing-master-
slave-configurations). While this relationship isn't exactly master-
slave in our situation, the datastore connection is indeed read-only.
Using this setting allowed me to run the unit tests successfully.
Russell Keith-Magee
> It seems that when running unit tests, the test runner not only
> creates all tables on both of my connections (which it should not do,
> if I read the documentation correctly),
Tables will be created on all connections, according to the syncdb
rules of your router (by default, all tables on all databases).
However, the non-default databases will not be flushed at the end of
each test unless you specify multi_db=True in your test case.
This is done for performance reasons; unless you're specifically
testing cross-database behavior, the overhead of setting up and
tearing down multiple databases is quite onerous.
> but also tries to create the
> cache table twice, causing the error below. Please let me know if I'm
> missing something here. I have two MySQL connections, called default
> and datastore, and have my CACHE_BACKEND set to db://cache_table.
It sounds like you may have found a problem (or, a series of related
problems). At present, the cache database backend doesn't have any
specific multi-db awareness. The CACHE_BACKEND setting doesn't allow
you to specify which database the cache backend should operate upon;
the backend itself just gets a cursor on the 'default' backend.
On top of that, the createcachetable management command accepts a
--database argument when it is run from the command line, which allows
you to control where the cache table is created... but the
create_test_db() utility doesn't pass that option when it
programatically invokes createcachetable as part of database setup.
So, when you try to create a test database, and you've specified a
database cache backend, it will repeatedly try to create the database
cache table on the default database.
This explains why TEST_MIRROR fixed your problem. When you set
TEST_MIRROR, you're effectively telling the test infrastructure that
you don't need to create a specific database, so you avoid
accidentally trying to create the cache table twice.
So - yes, this should be logged as a ticket.
As for a fix, there are three parts that are required.
Firstly, create_test_db() should pass the database flag to the
createcachetable command when it is invoked.
Secondly, the cache backend itself needs to be made multi-db aware, so
it doesn't just use the default database connection.
Thirdly, the create_test_db() call needs an additional check -- as
well as checking if the database cache backend is in use, it needs to
check if the test database that is being created actually requires the
cache table.
Conceptually, problems 2 and 3 need to be fixed by the router.
However, the API for the router requires that you provide a model as
the basis for decision making; since the database cache backend
doesn't use a Django model this won't really work well in practice.
A better approach is probably to just add extra arguments to the
CACHE_BACKEND setting when db:// is specified -- so, something like:
CACHE_BACKEND="db://mycachetable?max_entries=100&write_db=foo&read_db=slave1&read_db=slave2"
which would mean that:
* the table would only be created on the database alias "foo"
* one of [slave1, slave2] would be selected at random for read purposes.
When unspecified, read_db would default to the same value as write_db;
if write_db is unspecified, it would default to 'default'.
Yours,
Russ Magee %-)