[Django] #32595: mysql DatabaseSchemaEditor can't `quote_value` on byte strings
Django
-------------------------------------+-------------------------------------
Reporter: Ryan | Owner: nobody
Siemens |
Type: Bug | Status: new
Component: Database | Version: 3.1
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
Using the the `django.db.backends.mysql` backend with `mysqlclient` fails
to render the schema migration with `sqlmigrate` when using a
`BinaryField` with a binary string default value.
You can recreate this with a simple app using the following model or see
the attached sample project (MD5 = 2be52a23a4001e90728d3b0f5276e087):
{{{
class TestModel(models.Model):
ok = models.BinaryField(default=b'some-blob')
# add this after the initial migration
# not_ok = models.BinaryField(default=b'some-other-blob')
}}}
Then perform the following steps:
1. run initial `manage.py makemigrations`
2. uncomment the `not_ok = models.BinaryField(default=b'some-other-blob')`
3. run `manage.py makemigrations` again to generate the alter table
migration
4. run `manage.py sqlmigrate <app> 0002`
5. assert the following `TypeError` is raised
{{{
Traceback (most recent call last):
File "manage.py", line 22, in <module>
main()
File "manage.py", line 18, in main
execute_from_command_line(sys.argv)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/core/management/__init__.py", line 401, in
execute_from_command_line
utility.execute()
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/core/management/__init__.py", line 395, in execute
self.fetch_command(subcommand).run_from_argv(self.argv)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/core/management/base.py", line 330, in run_from_argv
self.execute(*args, **cmd_options)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/core/management/commands/sqlmigrate.py", line 29, in
execute
return super().execute(*args, **options)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/core/management/base.py", line 371, in execute
output = self.handle(*args, **options)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/core/management/commands/sqlmigrate.py", line 65, in
handle
sql_statements = loader.collect_sql(plan)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/db/migrations/loader.py", line 345, in collect_sql
state = migration.apply(state, schema_editor, collect_sql=True)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/db/migrations/migration.py", line 124, in apply
operation.database_forwards(self.app_label, schema_editor, old_state,
project_state)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/db/migrations/operations/fields.py", line 104, in
database_forwards
schema_editor.add_field(
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/db/backends/mysql/schema.py", line 89, in add_field
super().add_field(model, field)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/db/backends/base/schema.py", line 487, in add_field
self.execute(sql, params)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/db/backends/base/schema.py", line 137, in execute
self.collected_sql.append((sql % tuple(map(self.quote_value, params)))
+ ending)
File "/Users/rsiemens/sandbox/quote_value_bug/.venv/lib/python3.8/site-
packages/django/db/backends/mysql/schema.py", line 55, in quote_value
quoted = self.connection.connection.escape(value,
self.connection.connection.encoders)
TypeError: bytes() argument 2 must be str, not dict
}}}
The error seems to be raised from the mysqlclient `connection.escape`, but
interestingly this works if you alter the
`self.connection.connection.encoders` right before escaping
(https://github.com/django/django/blob/main/django/db/backends/mysql/schema.py#L57)
by doing `self.connection.connection.encoders.pop(bytes)`. Looking at the
default converters mysqlclient sets up
(https://github.com/PyMySQL/mysqlclient/blob/v2.0.1/MySQLdb/converters.py#L106-L139)
I don't see a `bytes` one being added, so I'm guessing django adds it
somewhere along the way.
Using:
Python 3.8.5
MySQL 8.0.23
Django==3.1.7
mysqlclient==2.0.3
--
Ticket URL: <https://code.djangoproject.com/ticket/32595>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* Attachment "example.tar.gz" added.
example app to recreate bug
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Ryan Siemens (added)
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:1>
Django
-------------------------------------+-------------------------------------
Component: Database layer | Version: 3.1
(models, ORM) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => invalid
Comment:
Thanks for this ticket, however that's an issue in
[https://github.com/PyMySQL/mysqlclient/blob/24aaa72c1503fe35cb1d0584ad553b41d10ddb48/MySQLdb/connections.py#L189-L194
mysqlclient] (see also [https://github.com/PyMySQL/mysqlclient/issues/306
mysqlclient#306]). I'm not sure but maybe they should use
`_bytes_literal()`. Please create a new issue in `mysqlclient` and feel-
free to ping me on it.
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:2>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: closed
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Ryan Siemens):
Thanks, Mariusz. I went ahead and opened
https://github.com/PyMySQL/mysqlclient/issues/489
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:3>
Django
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* cc: Claude Paroz (added)
* status: closed => new
* resolution: invalid =>
* stage: Unreviewed => Accepted
Comment:
There seems to be no easy way to fix this in `mysqlclient` at least for
now, see [discussion](https://github.com/PyMySQL/mysqlclient/issues/489).
I'm going to add a workaround in the MySQL backend.
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:4>
Django
-------------------------------------+-------------------------------------
| Felisiak
Type: Bug | Status: assigned
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* owner: nobody => Mariusz Felisiak
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:5>
Django
-------------------------------------+-------------------------------------
| Felisiak
Type: Bug | Status: assigned
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
Comment:
[https://github.com/django/django/pull/14196 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:6>
Django
-------------------------------------+-------------------------------------
| Felisiak
Type: Bug | Status: assigned
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:7>
Django
-------------------------------------+-------------------------------------
| Felisiak
Type: Bug | Status: assigned
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"3c75f1f3cac7985e8a134fc1c33eb6e01639a04b" 3c75f1f]:
{{{
#!CommitTicketReference repository=""
revision="3c75f1f3cac7985e8a134fc1c33eb6e01639a04b"
Refs #32595 -- Added MySQL's SchemaEditor.quote_value() tests for values
with Unicode chars.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:8>
Django
-------------------------------------+-------------------------------------
| Felisiak
Component: Database layer | Version: 3.1
(models, ORM) |
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"f6018c1e63a04e0c12e2ca759e76e05ccf5e09de" f6018c1e]:
{{{
#!CommitTicketReference repository=""
revision="f6018c1e63a04e0c12e2ca759e76e05ccf5e09de"
Fixed #32595 -- Fixed SchemaEditor.quote_value() crash with bytes.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:9>
Django
-------------------------------------+-------------------------------------
| Felisiak
Type: Bug | Status: closed
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"682eba534f15e962d29026bfcac8dcaf4555aac0" 682eba53]:
{{{
#!CommitTicketReference repository=""
revision="682eba534f15e962d29026bfcac8dcaf4555aac0"
[3.2.x] Refs #32595 -- Added MySQL's SchemaEditor.quote_value() tests for
values with Unicode chars.
Backport of 3c75f1f3cac7985e8a134fc1c33eb6e01639a04b from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:10>
Django
-------------------------------------+-------------------------------------
| Felisiak
Type: Bug | Status: closed
Component: Database layer | Version: 3.1
(models, ORM) |
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"d67d48e923eafd8ab7f8cc21985584f9606d9808" d67d48e]:
{{{
#!CommitTicketReference repository=""
revision="d67d48e923eafd8ab7f8cc21985584f9606d9808"
[3.2.x] Fixed #32595 -- Fixed SchemaEditor.quote_value() crash with bytes.
Backport of f6018c1e63a04e0c12e2ca759e76e05ccf5e09de from main
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32595#comment:11>