[Django] #28949: Multibyte table name or column name causes miscalculation of the length of index name.
Django
index name.
-------------------------------------+-------------------------------------
Reporter: Pak | Owner: nobody
Youngrok |
Type: Bug | Status: new
Component: | Version: 2.0
Migrations | Keywords: migration multibyte
Severity: Normal | index
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Django migration automatically creates index with name consists of table
name, column names, hash, and suffix. When the length of generated index
name is greater than `self.connection.ops.max_name_length()`, it shortens
the name. However, it calculate length as python string type, so it's
length doesn't match with the length of databases. The length should be
calculated after encoded with the database encoding. Because of this
issue, migration fails with these conditions below:
* long multibyte model names
* two multibyte model related with foreign key
* the foreign key field is CharField(or it's child class)
With these conditions, django migration tries to create two index(one for
normal index, one for `like` index), and the name of those are same except
suffix(the latter has suffix `_like`), and the lengths of both index names
as string are less than max name length but the length of both index names
as bytes are greater than max name length, so name conflict is raised.
long multibyte table name and foreign key name.
Here is the code:
https://github.com/django/django/blob/4420761ea9457d386b2000cf9df5b2f6f88f8f91/django/db/backends/base/schema.py#L873
{{{#!python
index_name = '%s_%s_%s' % (table_name, '_'.join(column_names),
hash_suffix_part)
if len(index_name) <= max_length:
return index_name
}}}
[https://docs.djangoproject.com/en/2.0/ref/databases/#encoding Django
assumes that all databases use UTF-8 encoding], so the code should be
fixed like this:
{{{#!python
index_name = '%s_%s_%s' % (table_name, '_'.join(column_names),
hash_suffix_part)
if len(index_name.encode('utf8')) <= max_length:
return index_name
}}}
The code that shorten the name should be also fixed. Getting a third of
each part and re-joining is not good strategy in multibyte world, it can
also cause miscalculation. I think getting very small amount of table and
column names like 2 or 3 characters and joining them with original hash
can be a safe solution.
--
Ticket URL: <https://code.djangoproject.com/ticket/28949>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
index name.
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Keywords: migration multibyte | Triage Stage: Accepted
index |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:1>
Django
index name.
-------------------------------------+-------------------------------------
| Gautam
Type: Bug | Status: assigned
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Keywords: migration multibyte | Triage Stage: Accepted
index |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Abhishek Gautam
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:2>
Django
index name.
-------------------------------------+-------------------------------------
| Gautam
Type: Bug | Status: assigned
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Keywords: migration multibyte | Triage Stage: Accepted
index |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Abhishek Gautam):
As we just need a unique name for an index can so, can we create
index_name as :
{{{#!python
index_name = '%s%s' % (self._digest(*([table_name] + column_names)),
suffix)
}}}
_digest function will be:
{{{#!python
@classmethod
def _digest(cls, *args):
"""
Generate a 32-bit digest of a set of arguments that can be used to
shorten identifying names.
"""
h = hashlib.md5()
for arg in args:
h.update(force_bytes(arg))
return h.hexdigest()
}}}
Using _digest method we will get 32 byte string and in that we will add
suffix which will give us a length of index_name = 32 + length of suffix.
As suffix length will be very small length of index_name will not be able
to exceed 40 also.
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:3>
Django
index name.
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Keywords: migration multibyte | Triage Stage: Accepted
index |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: Abhishek Gautam => (none)
* status: assigned => new
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:4>
Django
index name.
-------------------------------------+-------------------------------------
| Walls
Type: Bug | Status: assigned
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Keywords: migration multibyte | Triage Stage: Accepted
index |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: (none) => Jacob Walls
* status: new => assigned
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/15273 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:5>
Django
index name.
-------------------------------------+-------------------------------------
| Walls
Type: Bug | Status: assigned
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Keywords: migration multibyte | Triage Stage: Accepted
index |
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* needs_tests: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:6>
Django
index name.
-------------------------------------+-------------------------------------
| Walls
Type: Bug | Status: assigned
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
Keywords: migration multibyte | Triage Stage: Accepted
index |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
* needs_tests: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:7>
Django
index name.
-------------------------------------+-------------------------------------
| Walls
Type: Bug | Status: assigned
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
index | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:8>
Django
index name.
-------------------------------------+-------------------------------------
| Walls
Type: Bug | Status: assigned
Component: Migrations | Version: 2.0
Severity: Normal | Resolution:
index |
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
* stage: Ready for checkin => Accepted
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:9>
Django
index name.
-------------------------------------+-------------------------------------
| Walls
Component: Migrations | Version: 2.0
Severity: Normal | Resolution: wontfix
Keywords: migration multibyte | Triage Stage:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => wontfix
* stage: Accepted => Unreviewed
Comment:
Closing per [https://github.com/django/django/pull/15273 discussion]. We
cannot use `encode()` because identifier limits are express in chars not
bytes, chars that can have 2, 3, 4 bytes. It may also depend on encoding
of the operating system or database, so it's not feasible to prepare a
fully backward compatible solution. I'd say that if you decided to use
non-ASCII chars in identifiers, you actually did this to yourself. Any
solution would be error-prone.
--
Ticket URL: <https://code.djangoproject.com/ticket/28949#comment:10>