[Django] #29015: Make Django aware of PostgreSQL database name length limits
Django
-----------------------------------------+------------------------
Reporter: Tadej Janež | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version: 2.0
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
PostgreSQL has a maximum identifier lenght limit that is set to 63 bytes
by default (see https://www.postgresql.org/docs/current/static/runtime-
config-preset.html for more details).
If a user chooses a database name longer than this, PostgreSQL will
truncate the identifier to 63 bytes and operate normally.
Django apparently doesn't notice this and behaves as if the database name
is not truncated.
However, in some cases, Django needs to be aware of this. For example,
when running tests in parallel. Django will fail due to a pre-existing
database with the same name and not finding the template database to
clone.
Here is an example using https://github.com/mdamien/django-tutorial.
First setup up a virtual environment and clone the project:
{{{
mkvirtualenv --python=python3 django-db-name-too-long-bug
pip install Django psycopg2
git clone https://github.com/mdamien/django-tutorial
cd django-tutorial
}}}
Edit `mysite/settings.py` and change database configuration to:
{{{
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'foo',
}
}
}}}
Confirm that the tests run without an error:
{{{
$ ./manage.py test -v 2
Creating test database for alias 'default' ('test_foo')...
Operations to perform:
Synchronize unmigrated apps: messages, staticfiles
Apply all migrations: admin, auth, contenttypes, polls, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
System check identified no issues (0 silenced).
test_detail_view_with_a_future_question
(polls.tests.QuestionIndexDetailTests) ... ok
test_detail_view_with_a_past_question
(polls.tests.QuestionIndexDetailTests) ... ok
test_was_published_recently_with_future_question
(polls.tests.QuestionMethodTests) ... ok
test_was_published_recently_with_old_question
(polls.tests.QuestionMethodTests) ... ok
test_was_published_recently_with_recent_question
(polls.tests.QuestionMethodTests) ... ok
test_index_view_with_a_future_question (polls.tests.QuestionViewTests) ...
ok
test_index_view_with_a_past_question (polls.tests.QuestionViewTests) ...
ok
test_index_view_with_future_question_and_past_question
(polls.tests.QuestionViewTests) ... ok
test_index_view_with_no_questions (polls.tests.QuestionViewTests) ... ok
test_index_view_with_two_past_questions (polls.tests.QuestionViewTests)
... ok
----------------------------------------------------------------------
Ran 10 tests in 0.054s
OK
Destroying test database for alias 'default' ('test_foo')...
}}}
Confirm that the tests when run in parallel run without an error:
{{{
$ ./manage.py test -v 2 --parallel
Creating test database for alias 'default' ('test_foo')...
Operations to perform:
Synchronize unmigrated apps: messages, staticfiles
Apply all migrations: admin, auth, contenttypes, polls, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
Cloning test database for alias 'default' ('test_foo')...
Cloning test database for alias 'default' ('test_foo')...
Cloning test database for alias 'default' ('test_foo')...
System check identified no issues (0 silenced).
test_was_published_recently_with_future_question
(polls.tests.QuestionMethodTests) ... ok
test_was_published_recently_with_old_question
(polls.tests.QuestionMethodTests) ... ok
test_was_published_recently_with_recent_question
(polls.tests.QuestionMethodTests) ... ok
test_detail_view_with_a_future_question
(polls.tests.QuestionIndexDetailTests) ... ok
test_detail_view_with_a_past_question
(polls.tests.QuestionIndexDetailTests) ... ok
test_index_view_with_a_future_question (polls.tests.QuestionViewTests) ...
ok
test_index_view_with_a_past_question (polls.tests.QuestionViewTests) ...
ok
test_index_view_with_future_question_and_past_question
(polls.tests.QuestionViewTests) ... ok
test_index_view_with_no_questions (polls.tests.QuestionViewTests) ... ok
test_index_view_with_two_past_questions (polls.tests.QuestionViewTests)
... ok
----------------------------------------------------------------------
Ran 10 tests in 0.119s
OK
Destroying test database for alias 'default' ('test_foo_1')...
Destroying test database for alias 'default' ('test_foo_2')...
Destroying test database for alias 'default' ('test_foo_3')...
Destroying test database for alias 'default' ('test_foo')...
}}}
Now edit `mysite/settings.py` again and change database name to something
very long. For example, set the following:
{{{
DATABASES = {
'default': {
'ENGINE': 'django.db.backends.postgresql',
'NAME': 'something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy'
'-long-that-will-get-truncated',
}
}
}}}
Running the tests without `--parallel` works fine, even though Django
mistakenly thinks it created a test database with such a long name:
{{{
$ ./manage.py test -v 2
Creating test database for alias 'default' ('test_something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
truncated')...
Operations to perform:
Synchronize unmigrated apps: messages, staticfiles
Apply all migrations: admin, auth, contenttypes, polls, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
System check identified no issues (0 silenced).
test_detail_view_with_a_future_question
(polls.tests.QuestionIndexDetailTests) ... ok
test_detail_view_with_a_past_question
(polls.tests.QuestionIndexDetailTests) ... ok
test_was_published_recently_with_future_question
(polls.tests.QuestionMethodTests) ... ok
test_was_published_recently_with_old_question
(polls.tests.QuestionMethodTests) ... ok
test_was_published_recently_with_recent_question
(polls.tests.QuestionMethodTests) ... ok
test_index_view_with_a_future_question (polls.tests.QuestionViewTests) ...
ok
test_index_view_with_a_past_question (polls.tests.QuestionViewTests) ...
ok
test_index_view_with_future_question_and_past_question
(polls.tests.QuestionViewTests) ... ok
test_index_view_with_no_questions (polls.tests.QuestionViewTests) ... ok
test_index_view_with_two_past_questions (polls.tests.QuestionViewTests)
... ok
----------------------------------------------------------------------
Ran 10 tests in 0.051s
OK
Destroying test database for alias 'default' ('test_something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
truncated')...
}}}
However, running the tests in parallel exposes Django's lack of
understanding of identifier truncation in PostgreSQL:
{{{
$ ./manage.py test -v 2 --parallel
Creating test database for alias 'default' ('test_something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
truncated')...
Operations to perform:
Synchronize unmigrated apps: messages, staticfiles
Apply all migrations: admin, auth, contenttypes, polls, sessions
Synchronizing apps without migrations:
Creating tables...
Running deferred SQL...
Running migrations:
Applying contenttypes.0001_initial... OK
Applying auth.0001_initial... OK
Applying admin.0001_initial... OK
Applying admin.0002_logentry_remove_auto_add... OK
Applying contenttypes.0002_remove_content_type_name... OK
Applying auth.0002_alter_permission_name_max_length... OK
Applying auth.0003_alter_user_email_max_length... OK
Applying auth.0004_alter_user_username_opts... OK
Applying auth.0005_alter_user_last_login_null... OK
Applying auth.0006_require_contenttypes_0002... OK
Applying auth.0007_alter_validators_add_error_messages... OK
Applying auth.0008_alter_user_username_max_length... OK
Applying auth.0009_alter_user_last_name_max_length... OK
Applying polls.0001_initial... OK
Applying sessions.0001_initial... OK
Cloning test database for alias 'default' ('test_something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
truncated')...
Destroying old test database for alias 'default' ('test_something-very-
very-veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-
truncated_1')...
Got an error creating the test database: template database
"test_something-very-very-veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyy" does not
exist
}}}
When Django created the main test database, `"test_something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-truncated"`,
in reality, PostgreSQL created a database named `"test_something-very-
very-veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyy"`.
And when Django tried to create a clone of this test database, it
executed:
{{{
CREATE DATABASE "test_something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-truncated_1"
WITH TEMPLATE "test_something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyyyyy-long-that-will-get-truncated"
}}}
Of course, this doesn't work since PostgreSQL truncated the identifiers
and a database named `"test_something-very-very-
veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyy"` already exists.
PostgreSQL also returns the second error since the template database
`"test_something-very-very-veeeeeeeeeeeeeeeerrrrrrrrrrrrrryyyyyyy"` does
not exist.
I want to note that at first sight it might appear that users won't
encounter this in real life at all, but this is not true.
If, for example, a user sets up the CI system to use a unique database
name that consists of:
{{{
${PROJECT_OWNER}-${PROJECT_NAME}-${BRANCH_NAME}-${BUILD_NUMBER}
}}}
this can quickly get larger than 63 bytes (i.e. characters) and the tests
will fail with the above uninformative error.
--
Ticket URL: <https://code.djangoproject.com/ticket/29015>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Tadej Janež | Owner: nobody
Cleanup/optimization |
Component: Database layer | Version: 2.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* type: Bug => Cleanup/optimization
* stage: Unreviewed => Accepted
* component: Uncategorized => Database layer (models, ORM)
--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:1>
Django
-------------------------------------+-------------------------------------
Type: | Saxena
Cleanup/optimization | Status: assigned
Component: Database layer | Version: 2.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Priyansh Saxena):
* owner: nobody => Priyansh Saxena
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: Tadej Janež | Owner: Priyansh
Type: | Saxena
Cleanup/optimization | Status: assigned
Component: Database layer | Version: 2.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Adam (Chainz) Johnson):
* cc: Adam (Chainz) Johnson (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: Tadej Janež | Owner: Priyansh
Type: | Saxena
Cleanup/optimization | Status: assigned
Component: Database layer | Version: 2.0
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Priyansh Saxena):
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/9600]
--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: Tadej Janež | Owner: Priyansh
Type: | Saxena
Component: Database layer | Version: 2.0
(models, ORM) |
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"6b3d2920438746d260f7d158a91db53450055ae2" 6b3d2920]:
{{{
#!CommitTicketReference repository=""
revision="6b3d2920438746d260f7d158a91db53450055ae2"
Fixed #29015 -- Added an exception if the PostgreSQL database name is too
long.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: Tadej Janež | Owner: Priyansh
Type: | Saxena
Cleanup/optimization | Status: closed
Component: Database layer | Version: 2.0
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"bf17f5e88466e3f571065345f5b2fea0d8af89fe" bf17f5e8]:
{{{
#!CommitTicketReference repository=""
revision="bf17f5e88466e3f571065345f5b2fea0d8af89fe"
Refs #29015 -- Added database name to PostgreSQL database name too long
exception.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:6>
Django
-------------------------------------+-------------------------------------
Reporter: Tadej Janež | Owner: Priyansh
Type: | Saxena
Cleanup/optimization | Status: closed
Component: Database layer | Version: 2.0
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham <timograham@…>):
In [changeset:"cae84904c6a5facc0edd75e030c775de437320e0" cae84904]:
{{{
#!CommitTicketReference repository=""
revision="cae84904c6a5facc0edd75e030c775de437320e0"
[2.1.x] Refs #29015 -- Added database name to PostgreSQL database name too
long exception.
Backport of bf17f5e88466e3f571065345f5b2fea0d8af89fe from master
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/29015#comment:7>