[Django] #30887: UnicodeEncodeError problem while saving binary to database
Django
-------------------------------------+-------------------------------------
Reporter: Louis | Owner: nobody
Type: Bug | Status: new
Component: Database | Version: 2.0
layer (models, ORM) |
Severity: Normal | Keywords:
Triage Stage: | Has patch: 0
Unreviewed |
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
{{{
from django.db import models
class JobContext(models.Model):
data = models.BinaryField()
class Meta:
verbose_name = 'JobContext'
JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
}}}
----
raise exception below:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/logging/__init__.py", line 1028, in emit
stream.write(msg + self.terminator)
UnicodeEncodeError: 'utf-8' codec can't encode character '\udc80' in
position 173: surrogates not allowed
due to File "/usr/local/lib/python3.7/site-
packages/django/db/backends/utils.py", line 111, in execute
{{{
logger.debug(
'(%.3f) %s; args=%s', duration, sql, params,
extra={'duration': duration, 'sql': sql, 'params': params}
)
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/30887>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
-------------------------------------+-------------------------------------
Reporter: Louis | Owner: nobody
Component: Database layer | Version: 2.0
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => worksforme
Comment:
Hmmm. This seems to be working for me on `master`.
Creating a model as described works as expected.
{{{
In [1]: from app.models import JobContext
In [2]:
JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT INTO
"app_jobcontext" ("data") VALUES (X'800358040000007878787871002E');
args=[<memory at 0x103da5888>] [utils.py:130]
2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT INTO
"app_jobcontext" ("data") VALUES (X'800358040000007878787871002E');
args=[<memory at 0x103da5888>] [utils.py:130]
Out[2]: <JobContext: JobContext object (1)>
}}}
In particular the error isn't thrown at the `logger.debug()` call.
What are your Python/Django versions? (I see you tagged 2.0)
Are you able to provide a sample project that reproduces?
Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:1>
Django
-------------------------------------+-------------------------------------
Reporter: Louis | Owner: nobody
Component: Database layer | Version: 2.0
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Simon Charette):
It would also be useful to know which database backend you were using
because `params` might be massaged to different types based on your
database backend.
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:2>
Django
-------------------------------------+-------------------------------------
Reporter: Louis | Owner: nobody
Component: Database layer | Version: 2.0
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Louis):
Replying to [comment:1 Carlton Gibson]:
> Hmmm. This seems to be working for me on `master`.
>
> Creating a model as described works as expected.
>
> {{{
> In [1]: from app.models import JobContext
>
> In [2]:
JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
> 2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT
INTO "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E');
args=[<memory at 0x103da5888>] [utils.py:130]
> 2019-10-16 17:06:11,151 - DEBUG - django.db.backends - (0.002) INSERT
INTO "app_jobcontext" ("data") VALUES (X'800358040000007878787871002E');
args=[<memory at 0x103da5888>] [utils.py:130]
> Out[2]: <JobContext: JobContext object (1)>
> }}}
>
> In particular the error isn't thrown at the `logger.debug()` call.
>
>> What are your Python/Django versions? (I see you tagged 2.0)
> Python 3.7.4 with Django 2.1.3
>> Are you able to provide a sample project that reproduces?
> To be updated here...
> Thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:3>
Django
-------------------------------------+-------------------------------------
Reporter: Louis | Owner: nobody
Component: Database layer | Version: 2.1
(models, ORM) | Resolution:
Severity: Normal | worksforme
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* version: 2.0 => 2.1
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:4>
Django
-------------------------------------+-------------------------------------
Reporter: Louis | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: closed => new
* resolution: worksforme =>
Comment:
OK, thanks for the follow-up. Let me reopen to investigate.
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:5>
Django
-------------------------------------+-------------------------------------
Reporter: Louis | Owner: nobody
Component: Database layer | Version: 2.1
(models, ORM) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* status: new => closed
* resolution: => invalid
Comment:
Thanks for the docker image. Very helpful!
I think this is a bug with PyMySQL. I suggest moving to `mysqlclient`,
[https://docs.djangoproject.com/en/2.2/ref/databases/#mysql-db-api-drivers
which is the recommended driver now].
I'm able to reproduce the bug with your image but, changing the driver
(and removing the couple of lines from `djangotest/__init__.py`) resolves
the issue:
{{{
root@11f9f5ba1dfb:/usr/src/app# pip freeze
Django==2.1.3
PyMySQL==0.9.3
pytz==2019.3
root@11f9f5ba1dfb:/usr/src/app# pip uninstall PyMySQL
Uninstalling PyMySQL-0.9.3:
Would remove:
/usr/local/lib/python3.7/site-packages/PyMySQL-0.9.3.dist-info/*
/usr/local/lib/python3.7/site-packages/pymysql/*
Proceed (y/n)? y
Successfully uninstalled PyMySQL-0.9.3
root@11f9f5ba1dfb:/usr/src/app# pip install mysqlclient
Collecting mysqlclient
Downloading
https://files.pythonhosted.org/packages/4d/38/c5f8bac9c50f3042c8f05615f84206f77f03db79781db841898fde1bb284/mysqlclient-1.4.4.tar.gz
(86kB)
|████████████████████████████████| 92kB 824kB/s
Building wheels for collected packages: mysqlclient
Building wheel for mysqlclient (setup.py) ... done
Stored in directory:
/root/.cache/pip/wheels/a0/04/57/031b9b01df38999df7dc7f4ee998a98ecdbd2d781f73e3ffbf
Successfully built mysqlclient
Installing collected packages: mysqlclient
Successfully installed mysqlclient-1.4.4
WARNING: You are using pip version 19.1.1, however version 19.3 is
available.
You should consider upgrading via the 'pip install --upgrade pip' command.
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py makemigrations
Migrations for 'demo':
demo/migrations/0001_initial.py
- Create model JobContext
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py migrate
Operations to perform:
Apply all migrations: admin, auth, contenttypes, demo, sessions
Running migrations:
No migrations to apply.
root@11f9f5ba1dfb:/usr/src/app# python3 manage.py shell
Python 3.7.3 (default, Jun 11 2019, 01:05:09)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from demo.models import JobContext
>>> JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
<JobContext: JobContext object (2)>
>>>
now exiting InteractiveConsole...
root@11f9f5ba1dfb:/usr/src/app# pip freeze
Django==2.1.3
mysqlclient==1.4.4
pytz==2019.3
}}}
I'm going to close on that basis.
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:6>
Django
Reporter: Louis | Owner: nobody
Component: Database layer | Version: 2.1
(models, ORM) |
Severity: Normal | Resolution: invalid
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Louis):
Replying to [comment:6 Carlton Gibson]:
> >>> from demo.models import JobContext
> >>>
JobContext.objects.create(data=b'\x80\x03X\x04\x00\x00\x00xxxxq\x00.')
> <JobContext: JobContext object (2)>
> >>>
> now exiting InteractiveConsole...
> root@11f9f5ba1dfb:/usr/src/app# pip freeze
> Django==2.1.3
> mysqlclient==1.4.4
> pytz==2019.3
> }}}
>
> I'm going to close on that basis.
OK, it turned out to be pymysql compatible problem, thanks for the help,
will follow your suggestion then.
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:7>
Django
Reporter: Louis | Owner: nobody
Component: Database layer | Version: 2.1
(models, ORM) |
Keywords: | Triage Stage:
| Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: invalid => duplicate
Comment:
For the record this is tracked in #30380 and was fixed in
a41b09266dcdd01036d59d76fe926fe0386aaade which is part of the upcoming
Django 3.0 release.
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:8>
Django
Reporter: Louis | Owner: nobody
Component: Database layer | Version: 2.1
(models, ORM) |
Severity: Normal | Resolution: duplicate
Keywords: | Triage Stage:
| Unreviewed
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Louis):
Replying to [comment:8 Simon Charette]:
>> For the record this is tracked in #30380 and was fixed in
a41b09266dcdd01036d59d76fe926fe0386aaade which is part of the upcoming
Django 3.0 release.
> good to know, thanks.
--
Ticket URL: <https://code.djangoproject.com/ticket/30887#comment:9>