[Django] #28201: 500 when using a null character, Postgres + Python 3.5.3
Django
-----------------------------------------+------------------------
Reporter: CM Lubinski | Owner: nobody
Type: Bug | Status: new
Component: contrib.admin | Version: 1.11
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
Input with null characters cause Django to crash when saving data.
Reproduction recipe:
0. Setup Django 1.11.1, Postgres (I don't think the version matters) using
Python 3.5 (though this may apply to any 3.x).
1. Create an admin account & login.
2. Navigate to `/admin/auth/group/add/`
3. Using the JavaScript console, execute
{{{
$('#id_name').val("\x00something")
}}}
4. Submit the form. See the error
Result will be similar to:
{{{
dev-api_1 | Internal Server Error: /admin/auth/group/add/
dev-api_1 | Traceback (most recent call last):
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/core/handlers/exception.py", line 41, in inner
dev-api_1 | response = get_response(request)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/core/handlers/base.py", line 187, in _get_response
dev-api_1 | response =
self.process_exception_by_middleware(e, request)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/core/handlers/base.py", line 185, in _get_response
dev-api_1 | response = wrapped_callback(request,
*callback_args, **callback_kwargs)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/contrib/admin/options.py", line 551, in wrapper
dev-api_1 | return self.admin_site.admin_view(view)(*args,
**kwargs)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/utils/decorators.py", line 149, in _wrapped_view
dev-api_1 | response = view_func(request, *args, **kwargs)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
dev-api_1 | response = view_func(request, *args, **kwargs)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/contrib/admin/sites.py", line 224, in inner
dev-api_1 | return view(request, *args, **kwargs)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/contrib/admin/options.py", line 1508, in add_view
dev-api_1 | return self.changeform_view(request, None,
form_url, extra_context)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/utils/decorators.py", line 67, in _wrapper
dev-api_1 | return bound_func(*args, **kwargs)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/utils/decorators.py", line 149, in _wrapped_view
dev-api_1 | response = view_func(request, *args, **kwargs)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/utils/decorators.py", line 63, in bound_func
dev-api_1 | return func.__get__(self, type(self))(*args2,
**kwargs2)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/contrib/admin/options.py", line 1408, in changeform_view
dev-api_1 | return self._changeform_view(request, object_id,
form_url, extra_context)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/contrib/admin/options.py", line 1440, in _changeform_view
dev-api_1 | if form.is_valid():
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/forms/forms.py", line 183, in is_valid
dev-api_1 | return self.is_bound and not self.errors
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/forms/forms.py", line 175, in errors
dev-api_1 | self.full_clean()
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/forms/forms.py", line 386, in full_clean
dev-api_1 | self._post_clean()
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/forms/models.py", line 402, in _post_clean
dev-api_1 | self.validate_unique()
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/forms/models.py", line 411, in validate_unique
dev-api_1 | self.instance.validate_unique(exclude=exclude)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/models/base.py", line 1032, in validate_unique
dev-api_1 | errors =
self._perform_unique_checks(unique_checks)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/models/base.py", line 1129, in _perform_unique_checks
dev-api_1 | if qs.exists():
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/models/query.py", line 668, in exists
dev-api_1 | return self.query.has_results(using=self.db)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/models/sql/query.py", line 517, in has_results
dev-api_1 | return compiler.has_results()
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/models/sql/compiler.py", line 845, in has_results
dev-api_1 | return bool(self.execute_sql(SINGLE))
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/models/sql/compiler.py", line 886, in execute_sql
dev-api_1 | raise original_exception
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/models/sql/compiler.py", line 876, in execute_sql
dev-api_1 | cursor.execute(sql, params)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/debug_toolbar/panels/sql/tracking.py", line 165, in execute
dev-api_1 | return self._record(self.cursor.execute, sql,
params)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/debug_toolbar/panels/sql/tracking.py", line 107, in _record
dev-api_1 | return method(sql, params)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/backends/utils.py", line 80, in execute
dev-api_1 | return super(CursorDebugWrapper,
self).execute(sql, params)
dev-api_1 | File "/usr/src/app/.venv-dev/lib/python3.5/site-
packages/django/db/backends/utils.py", line 65, in execute
dev-api_1 | return self.cursor.execute(sql, params)
dev-api_1 | ValueError: A string literal cannot contain NUL
(0x00) characters.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28201>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
literal cannot contain NUL (0x00) characters is unhandled
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: nobody
Type: Bug | Status: new
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* component: contrib.admin => Database layer (models, ORM)
* severity: Normal => Release blocker
* stage: Unreviewed => Accepted
Comment:
This exception was [https://github.com/psycopg/psycopg2/issues/420
introduced in psycopg2 2.7+]. With psycopg2 < 2.7, there is no exception
and null bytes are silently truncated by PostgreSQL. Other databases that
I tested (SQLite, MySQL, Oracle) allow saving null bytes. This creates
possible cross-database compatibility problems when moving data from those
databases to PostgreSQL, e.g.#28117. I've tentatively closed that ticket
as a duplicate but will reopen it if the solution for this ticket doesn't
address it.
I propose having `CharField` and `TextField` strip null bytes from the
value either a) only on PostgreSQL or b) on all databases. I raised
[https://groups.google.com/d/topic/django-
developers/D1gvXYCezEc/discussion a django-developers thread] to ask for
feedback.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:1>
Django
literal cannot contain NUL (0x00) characters is unhandled
-------------------------------------+-------------------------------------
| Sahu
Type: Bug | Status: assigned
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Biswajit Sahu):
* status: new => assigned
* owner: nobody => Biswajit Sahu
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:2>
Django
literal cannot contain NUL (0x00) characters is unhandled
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Database layer | Version: 1.11
(models, ORM) |
Severity: Release blocker | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* owner: Biswajit Sahu => (none)
* status: assigned => new
Comment:
Please don't work on this issue until there's a consensus about the
solution.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:3>
Django
--------------------------------------+------------------------------------
Reporter: CM Lubinski | Owner: (none)
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* type: Bug => Cleanup/optimization
* component: Database layer (models, ORM) => Forms
* severity: Release blocker => Normal
Comment:
The consensus on the mailing list is to make `CharField` and `TextField`
prohibit null characters.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:4>
Django
--------------------------------------+------------------------------------
Reporter: CM Lubinski | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Rajesh Veeranki):
So the way to achieve this is by adding a default validator to `CharField`
and `TextField` and raise a validation error. Am I right?
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:5>
Django
--------------------------------------+------------------------------------
Reporter: CM Lubinski | Owner: (none)
Type: Cleanup/optimization | Status: new
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
That seems reasonable.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:6>
Django
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Alejandro Zamora Fonseca):
* owner: (none) => Alejandro Zamora Fonseca
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:7>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Alejandro Zamora Fonseca):
Added default null characters validation to CharField and TextField in
order to avoid raising errors on database layer. Added test for both
fields. Tested on PostgreSQL, MySQL, and SQLite backends.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:8>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Alejandro Zamora Fonseca):
Linked PR against master branch
[https://github.com/django/django/pull/6870 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:9>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* needs_better_patch: 0 => 1
* has_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:10>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by Alejandro Zamora Fonseca):
Maybe only adding the null char validator to CharField would be enough?
Because TextField is converted to CharField when Model to Field
translation occurs.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:11>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Correct, there's isn't a `TextField` form field.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:12>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Claude Paroz):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:13>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Component: Forms | Version: 1.11
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Alejandro Zamora Fonseca):
* status: assigned => closed
* resolution: => fixed
Comment:
Should I mark the ticket as "Ready for checkin" ?
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:14>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Alejandro Zamora Fonseca):
* status: closed => new
* resolution: fixed =>
Comment:
Also my apologies, I accidentally modified 'resolution' and 'status'
flags, and I'm not sure how revert it the changes.
Maybe should I 'reopen' the ticket to achieve that?
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:15>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham):
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:14>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
Comment (by Alejandro Zamora Fonseca):
I added documentation and unit tests for the validator.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:15>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Alejandro Zamora Fonseca):
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:16>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: assigned
Component: Forms | Version: 1.11
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Alejandro Zamora Fonseca):
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:17>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Component: Forms | Version: 1.11
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Tim Graham <timograham@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"90d7b912b9c451dfdfb38f5f1f598af3b879257f" 90d7b91]:
{{{
#!CommitTicketReference repository=""
revision="90d7b912b9c451dfdfb38f5f1f598af3b879257f"
Fixed #28201 -- Added ProhibitNullCharactersValidator and used it on
CharField form field.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:18>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: closed
Component: Forms | Version: 1.11
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Vlada Macek):
* cc: Vlada Macek (added)
Comment:
We just ran into this on 1.11 LTS just as the original author. May I ask
whether it is usual to backport fixes like this?
Thank you.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:19>
Django
-------------------------------------+-------------------------------------
Reporter: CM Lubinski | Owner: Alejandro
Type: | Zamora Fonseca
Cleanup/optimization | Status: closed
Component: Forms | Version: 1.11
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Tim Graham):
Per our [https://docs.djangoproject.com/en/dev/internals/release-process
/#supported-versions supported versions policy], 1.11 is only receiving
data loss and security fixes.
--
Ticket URL: <https://code.djangoproject.com/ticket/28201#comment:20>