[Django] #35483: ModelMultipleChoiceField with CharField key throws ValueError if data contains NUL (0x00) characters
43 views
Skip to first unread message
Django
May 27, 2024, 11:13:44 AM5/27/24
to django-...@googlegroups.com
#35483: ModelMultipleChoiceField with CharField key throws ValueError if data
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer | Owner: nobody
Richards |
Type: Bug | Status: new
Component: Forms | Version: 4.2
Severity: Normal | Keywords:
Triage Stage: | ModelMultipleChoiceField null nul
Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
In a form using a ModelMultipleChoiceField with a model that has a
CharField primary key, I'm getting server errors when a request submits
input with a nul (0x00) character. This is resulting from a ValueError
raised by the postgresql backend. It happens whether the data is submitted
as a query string or in the request body.
I've tried adding an explicit ProhibitNullCharactersValidator to the
field, but this fails because the field's clean method runs validators
after evaluating whether the input is a member of the queryset.
A fairly minimal example that exhibits this is:
{{{
class MyModel(models.Model):
slug = models.CharField(primary_key=True)
class MyForm(forms.Form):
field = forms.ModelMultipleChoiceField(queryset=MyModel.objects.all())
def my_view(request):
form = MyForm(data=request.GET)
if form.is_valid():
return HttpResponse("yay")
return HttpResponse("boo", status=400)
}}}
With that running, the following triggers the error:
{{{
$ curl 'http://localhost:8000/my-view?field=hi%00'
}}}
The output from the Django dev server is
{{{
ERROR: django.request:241: Internal Server Error: /my-view
Traceback (most recent call last):
File "/home/dev/.local/lib/python3.9/site-
packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
File "/home/dev/.local/lib/python3.9/site-
packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args,
**callback_kwargs)
File "/workspace/ietf/doc/views_search.py", line 98, in my_view
if form.is_valid():
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/forms.py", line 201, in is_valid
return self.is_bound and not self.errors
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/forms.py", line 196, in errors
self.full_clean()
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/forms.py", line 433, in full_clean
self._clean_fields()
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/forms.py", line 445, in _clean_fields
value = field.clean(value)
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/models.py", line 1590, in clean
qs = self._check_values(value)
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/models.py", line 1623, in _check_values
pks = {str(getattr(o, key)) for o in qs}
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/models/query.py", line 398, in __iter__
self._fetch_all()
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/models/query.py", line 1881, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/models/query.py", line 91, in __iter__
results = compiler.execute_sql(
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
cursor.execute(sql, params)
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 102, in execute
return super().execute(sql, params)
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
ValueError: A string literal cannot contain NUL (0x00) characters.
[27/May/2024 08:02:37] "GET /my-view?field=hi%00 HTTP/1.0" 500 180131
}}}
As a workaround, replacing forms.ModelMultipleChoiceField with
{{{
class MyModelMultipleChoiceField(forms.ModelMultipleChoiceField):
validate_no_nulls = validators.ProhibitNullCharactersValidator()
def clean(self, value):
for item in value:
self.validate_no_nulls(item)
return super().clean(value)
}}}
correctly handles the same input, with only
{{{
[27/May/2024 08:04:22] "GET /my-view?field=hi%00 HTTP/1.0" 400 3
}}}
in the server log.
I'm seeing this with Django 4.2.13 using the postgresql backend with
psycopg2 2.99 against PostgreSQL 14.6.
--
Ticket URL: <https://code.djangoproject.com/ticket/35483>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer | Owner: nobody
Richards |
Type: Bug | Status: new
Component: Forms | Version: 4.2
Severity: Normal | Keywords:
Triage Stage: | ModelMultipleChoiceField null nul
Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-------------------------------------+-------------------------------------
In a form using a ModelMultipleChoiceField with a model that has a
CharField primary key, I'm getting server errors when a request submits
input with a nul (0x00) character. This is resulting from a ValueError
raised by the postgresql backend. It happens whether the data is submitted
as a query string or in the request body.
I've tried adding an explicit ProhibitNullCharactersValidator to the
field, but this fails because the field's clean method runs validators
after evaluating whether the input is a member of the queryset.
A fairly minimal example that exhibits this is:
{{{
class MyModel(models.Model):
slug = models.CharField(primary_key=True)
class MyForm(forms.Form):
field = forms.ModelMultipleChoiceField(queryset=MyModel.objects.all())
def my_view(request):
form = MyForm(data=request.GET)
if form.is_valid():
return HttpResponse("yay")
return HttpResponse("boo", status=400)
}}}
With that running, the following triggers the error:
{{{
$ curl 'http://localhost:8000/my-view?field=hi%00'
}}}
The output from the Django dev server is
{{{
ERROR: django.request:241: Internal Server Error: /my-view
Traceback (most recent call last):
File "/home/dev/.local/lib/python3.9/site-
packages/django/core/handlers/exception.py", line 55, in inner
response = get_response(request)
File "/home/dev/.local/lib/python3.9/site-
packages/django/core/handlers/base.py", line 197, in _get_response
response = wrapped_callback(request, *callback_args,
**callback_kwargs)
File "/workspace/ietf/doc/views_search.py", line 98, in my_view
if form.is_valid():
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/forms.py", line 201, in is_valid
return self.is_bound and not self.errors
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/forms.py", line 196, in errors
self.full_clean()
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/forms.py", line 433, in full_clean
self._clean_fields()
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/forms.py", line 445, in _clean_fields
value = field.clean(value)
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/models.py", line 1590, in clean
qs = self._check_values(value)
File "/home/dev/.local/lib/python3.9/site-
packages/django/forms/models.py", line 1623, in _check_values
pks = {str(getattr(o, key)) for o in qs}
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/models/query.py", line 398, in __iter__
self._fetch_all()
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/models/query.py", line 1881, in _fetch_all
self._result_cache = list(self._iterable_class(self))
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/models/query.py", line 91, in __iter__
results = compiler.execute_sql(
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/models/sql/compiler.py", line 1562, in execute_sql
cursor.execute(sql, params)
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 102, in execute
return super().execute(sql, params)
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/dev/.local/lib/python3.9/site-
packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
ValueError: A string literal cannot contain NUL (0x00) characters.
[27/May/2024 08:02:37] "GET /my-view?field=hi%00 HTTP/1.0" 500 180131
}}}
As a workaround, replacing forms.ModelMultipleChoiceField with
{{{
class MyModelMultipleChoiceField(forms.ModelMultipleChoiceField):
validate_no_nulls = validators.ProhibitNullCharactersValidator()
def clean(self, value):
for item in value:
self.validate_no_nulls(item)
return super().clean(value)
}}}
correctly handles the same input, with only
{{{
[27/May/2024 08:04:22] "GET /my-view?field=hi%00 HTTP/1.0" 400 3
}}}
in the server log.
I'm seeing this with Django 4.2.13 using the postgresql backend with
psycopg2 2.99 against PostgreSQL 14.6.
--
Ticket URL: <https://code.djangoproject.com/ticket/35483>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
May 28, 2024, 2:59:09 AM5/28/24
to django-...@googlegroups.com
#35483: ModelMultipleChoiceField with CharField key throws ValueError if data
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: nobody
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Type: Bug | Status: new
Component: Forms | Version: 4.2
Severity: Normal | Resolution:
Component: Forms | Version: 4.2
Keywords: | Triage Stage: Accepted
ModelMultipleChoiceField null nul |
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Unreviewed => Accepted
Comment:
Replicated, thank you for the report!
My failing test if it's useful
{{{#!diff
diff --git a/tests/model_forms/tests.py b/tests/model_forms/tests.py
index 3f927cb053..16bdbf8d21 100644
--- a/tests/model_forms/tests.py
+++ b/tests/model_forms/tests.py
@@ -2227,6 +2227,12 @@ class ModelMultipleChoiceFieldTests(TestCase):
f = forms.ModelMultipleChoiceField(queryset=Writer.objects.all())
self.assertNumQueries(1, f.clean, [p.pk for p in
persons[1:11:2]])
+ def test_model_multiple_choice_null_characters(self):
+ f =
forms.ModelMultipleChoiceField(queryset=ExplicitPK.objects.all())
+ msg = "Null characters are not allowed."
+ with self.assertRaisesMessage(ValidationError, msg):
+ f.clean(["\x00something"])
+
def test_model_multiple_choice_run_validators(self):
"""
ModelMultipleChoiceField run given validators (#14144).
}}}
Relates to #28201.
--
Ticket URL: <https://code.djangoproject.com/ticket/35483#comment:1>
Django
Jun 8, 2024, 6:31:33 AM6/8/24
to django-...@googlegroups.com
#35483: ModelMultipleChoiceField with CharField key throws ValueError if data
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Type: Bug | Status: assigned
Component: Forms | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
ModelMultipleChoiceField null nul |
Has patch: 1 | Needs documentation: 0
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
ModelMultipleChoiceField null nul |
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by lotvall):
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* has_patch: 0 => 1
* owner: nobody => lotvall
* status: new => assigned
--
Ticket URL: <https://code.djangoproject.com/ticket/35483#comment:2>
Django
Jun 13, 2024, 6:13:02 AM6/13/24
to django-...@googlegroups.com
#35483: ModelMultipleChoiceField with CharField key throws ValueError if data
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: assigned
Component: Forms | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
ModelMultipleChoiceField null nul |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: assigned
Component: Forms | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
ModelMultipleChoiceField null nul |
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):
-------------------------------------+-------------------------------------
* needs_better_patch: 0 => 1
--
Ticket URL: <https://code.djangoproject.com/ticket/35483#comment:3>
Django
Jun 14, 2024, 1:15:57 PM6/14/24
to django-...@googlegroups.com
#35483: ModelMultipleChoiceField with CharField key throws ValueError if data
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: assigned
Component: Forms | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
ModelMultipleChoiceField null nul |
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: assigned
Component: Forms | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
ModelMultipleChoiceField null nul |
Has patch: 1 | Needs documentation: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by lotvall):
-------------------------------------+-------------------------------------
* needs_better_patch: 1 => 0
--
Ticket URL: <https://code.djangoproject.com/ticket/35483#comment:4>
Django
Jun 17, 2024, 4:39:17 AM6/17/24
to django-...@googlegroups.com
#35483: ModelMultipleChoiceField with CharField key throws ValueError if data
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: assigned
Component: Forms | Version: 4.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Ready for
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: assigned
Component: Forms | Version: 4.2
Severity: Normal | Resolution:
ModelMultipleChoiceField null nul | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce):
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* stage: Accepted => Ready for checkin
--
Ticket URL: <https://code.djangoproject.com/ticket/35483#comment:5>
Django
Jun 17, 2024, 6:19:36 AM6/17/24
to django-...@googlegroups.com
#35483: ModelMultipleChoiceField with CharField key throws ValueError if data
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: closed
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Component: Forms | Version: 4.2
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
ModelMultipleChoiceField null nul | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Sarah Boyce <42296566+sarahboyce@…>):
ModelMultipleChoiceField null nul | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
* resolution: => fixed
* status: assigned => closed
Comment:
In [changeset:"38ad710aba885ad26944ff5708ce1a02a446d2d3" 38ad710]:
{{{#!CommitTicketReference repository=""
revision="38ad710aba885ad26944ff5708ce1a02a446d2d3"
Fixed #35483 -- Added NUL (0x00) character validation to
ModelChoiceFields.
Applied the ProhibitNullCharactersValidator to ModelChoiceField and
ModelMultipleChoiceField.
Co-authored-by: Viktor Paripás <viktor....@gmail.com>
Co-authored-by: Vasyl Dizhak <va...@dizhak.com>
Co-authored-by: Arthur Vasconcelos <vasconcel...@gmail.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35483#comment:6>
Django
Jun 17, 2024, 6:23:16 AM6/17/24
to django-...@googlegroups.com
#35483: ModelMultipleChoiceField with CharField key throws ValueError if data
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: closed
Component: Forms | Version: 4.2
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
ModelMultipleChoiceField null nul | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Sarah Boyce <42296566+sarahboyce@…>):
contains NUL (0x00) characters
-------------------------------------+-------------------------------------
Reporter: Jennifer Richards | Owner: lotvall
Type: Bug | Status: closed
Component: Forms | Version: 4.2
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
ModelMultipleChoiceField null nul | checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
In [changeset:"b8983dcf57b46a4391891cecdfc856c9c1c8d363" b8983dc]:
{{{#!CommitTicketReference repository=""
revision="b8983dcf57b46a4391891cecdfc856c9c1c8d363"
[5.1.x] Fixed #35483 -- Added NUL (0x00) character validation to
ModelChoiceFields.
Applied the ProhibitNullCharactersValidator to ModelChoiceField and
ModelMultipleChoiceField.
Co-authored-by: Viktor Paripás <viktor....@gmail.com>
Co-authored-by: Vasyl Dizhak <va...@dizhak.com>
Co-authored-by: Arthur Vasconcelos <vasconcel...@gmail.com>
Backport of 38ad710aba885ad26944ff5708ce1a02a446d2d3 from main.
Applied the ProhibitNullCharactersValidator to ModelChoiceField and
ModelMultipleChoiceField.
Co-authored-by: Viktor Paripás <viktor....@gmail.com>
Co-authored-by: Vasyl Dizhak <va...@dizhak.com>
Co-authored-by: Arthur Vasconcelos <vasconcel...@gmail.com>
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35483#comment:7>
Reply all
Reply to author
Forward
0 new messages