[Django] #32713: URLValidator tests failing on Python versions patched for bpo-43882
Django
-----------------------------------------+------------------------
Reporter: Michał Górny | Owner: nobody
Type: Bug | Status: new
Component: Uncategorized | Version:
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
On Python versions with a fix for [https://bugs.python.org/issue43882
bpo-43882] (i.e. 3.10.0b1 and the 3.9 git branch, not released yet) the
following tests fail:
{{{
======================================================================
FAIL: test_validators (validators.tests.TestValidators) [URLValidator]
(value='http://www.djangoproject.com/\n')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib/python3.7/unittest/case.py", line 546, in subTest
yield
File "/tmp/portage/dev-
python/django-3.2.1/work/Django-3.2.1/tests/validators/tests.py", line
328, in test_validators
validator(value)
File "/usr/lib/python3.7/unittest/case.py", line 203, in __exit__
self._raiseFailure("{} not raised".format(exc_name))
File "/usr/lib/python3.7/unittest/case.py", line 135, in _raiseFailure
raise self.test_case.failureException(msg)
AssertionError: ValidationError not raised
======================================================================
FAIL: test_validators (validators.tests.TestValidators) [URLValidator]
(value='http://[::ffff:192.9.5.5]\n')
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python3.7/unittest/case.py", line 59, in testPartExecutor
yield
File "/usr/lib/python3.7/unittest/case.py", line 546, in subTest
yield
File "/tmp/portage/dev-
python/django-3.2.1/work/Django-3.2.1/tests/validators/tests.py", line
328, in test_validators
validator(value)
File "/usr/lib/python3.7/unittest/case.py", line 203, in __exit__
self._raiseFailure("{} not raised".format(exc_name))
File "/usr/lib/python3.7/unittest/case.py", line 135, in _raiseFailure
raise self.test_case.failureException(msg)
AssertionError: ValidationError not raised
}}}
FWICS, the project is that django rejects URLs based on the split URL
components. However, the bpo-43882 fix changes URL splitting behavior to
strip all instances of LF, CR and tab characters before splitting, so they
never reach the validator.
I'm not sure what the best fix is. One option is to reject URLs containing
the forbidden characters early. Another is to go with the new
recommendation and assume that LF, CR and tabs are to stripped silently.
--
Ticket URL: <https://code.djangoproject.com/ticket/32713>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.
Django
Reporter: Michał Górny | Owner: Mariusz Felisiak
Type: Bug | Status: assigned
Component: Core (Other) | Version:
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* owner: nobody => Mariusz Felisiak
* status: new => assigned
* component: Uncategorized => Core (Other)
* stage: Unreviewed => Accepted
Comment:
Thanks for the report. We noticed this issue.
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:1>
Django
Reporter: Michał Górny | Owner: Mariusz Felisiak
Component: Core (Other) | Version:
Severity: Normal | Resolution: duplicate
Keywords: | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: assigned => closed
* resolution: => duplicate
* stage: Accepted => Unreviewed
Comment:
Duplicate of #32074.
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:2>
Django
Reporter: Michał Górny | Owner: Mariusz Felisiak
Type: Bug | Status: closed
Component: Core (Other) | Version:
Severity: Normal | Resolution: duplicate
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Michał Górny):
For the record, I was wrong that this hasn't been released in 3.9 -- it's
part of 3.9.5.
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:3>
Django
Reporter: Michał Górny | Owner: Mariusz Felisiak
Component: Core (Other) | Version: 2.2
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: closed => new
* version: => 2.2
* resolution: duplicate =>
* stage: Unreviewed => Accepted
Comment:
It looks that it will be backported to all supported versions of Python.
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:4>
Django
Reporter: Michał Górny | Owner: Mariusz
| Felisiak
Component: Core (Other) | Version: 2.2
Severity: Normal | Resolution:
| checkin
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by Mariusz Felisiak):
* status: new => assigned
* has_patch: 0 => 1
* stage: Accepted => Ready for checkin
Comment:
[https://github.com/django/django/pull/14349 PR]
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:5>
Django
Reporter: Michał Górny | Owner: Mariusz
| Felisiak
Component: Core (Other) | Version: 2.2
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Changes (by GitHub <noreply@…>):
* status: assigned => closed
* resolution: => fixed
Comment:
In [changeset:"e1e81aa1c4427411e3c68facdd761229ffea6f6f" e1e81aa]:
{{{
#!CommitTicketReference repository=""
revision="e1e81aa1c4427411e3c68facdd761229ffea6f6f"
Fixed #32713, Fixed CVE-2021-32052 -- Prevented newlines and tabs from
being accepted in URLValidator on Python 3.9.5+.
In Python 3.9.5+ urllib.parse() automatically removes ASCII newlines
and tabs from URLs [1, 2]. Unfortunately it created an issue in
the URLValidator. URLValidator uses urllib.urlsplit() and
urllib.urlunsplit() for creating a URL variant with Punycode which no
longer contains newlines and tabs in Python 3.9.5+. As a consequence,
the regular expression matched the URL (without unsafe characters) and
the source value (with unsafe characters) was considered valid.
[1] https://bugs.python.org/issue43882 and
[2]
https://github.com/python/cpython/commit/76cd81d60310d65d01f9d7b48a8985d8ab89c8b4
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:6>
Django
Reporter: Michał Górny | Owner: Mariusz
| Felisiak
Type: Bug | Status: closed
Component: Core (Other) | Version: 2.2
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"2d2c1d0c97832860fbd6597977e2aae17dd7e5b2" 2d2c1d0c]:
{{{
#!CommitTicketReference repository=""
revision="2d2c1d0c97832860fbd6597977e2aae17dd7e5b2"
[3.2.x] Fixed #32713, Fixed CVE-2021-32052 -- Prevented newlines and tabs
from being accepted in URLValidator on Python 3.9.5+.
In Python 3.9.5+ urllib.parse() automatically removes ASCII newlines
and tabs from URLs [1, 2]. Unfortunately it created an issue in
the URLValidator. URLValidator uses urllib.urlsplit() and
urllib.urlunsplit() for creating a URL variant with Punycode which no
longer contains newlines and tabs in Python 3.9.5+. As a consequence,
the regular expression matched the URL (without unsafe characters) and
the source value (with unsafe characters) was considered valid.
[1] https://bugs.python.org/issue43882 and
[2]
https://github.com/python/cpython/commit/76cd81d60310d65d01f9d7b48a8985d8ab89c8b4
Backport of e1e81aa1c4427411e3c68facdd761229ffea6f6f from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:7>
Django
Reporter: Michał Górny | Owner: Mariusz
| Felisiak
Type: Bug | Status: closed
Component: Core (Other) | Version: 2.2
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"afb23f5929944a407e4990edef1c7806a94c9879" afb23f59]:
{{{
#!CommitTicketReference repository=""
revision="afb23f5929944a407e4990edef1c7806a94c9879"
[3.1.x] Fixed #32713, Fixed CVE-2021-32052 -- Prevented newlines and tabs
from being accepted in URLValidator on Python 3.9.5+.
In Python 3.9.5+ urllib.parse() automatically removes ASCII newlines
and tabs from URLs [1, 2]. Unfortunately it created an issue in
the URLValidator. URLValidator uses urllib.urlsplit() and
urllib.urlunsplit() for creating a URL variant with Punycode which no
longer contains newlines and tabs in Python 3.9.5+. As a consequence,
the regular expression matched the URL (without unsafe characters) and
the source value (with unsafe characters) was considered valid.
[1] https://bugs.python.org/issue43882 and
[2]
https://github.com/python/cpython/commit/76cd81d60310d65d01f9d7b48a8985d8ab89c8b4
Backport of e1e81aa1c4427411e3c68facdd761229ffea6f6f from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:8>
Django
Reporter: Michał Górny | Owner: Mariusz
| Felisiak
Type: Bug | Status: closed
Component: Core (Other) | Version: 2.2
Severity: Normal | Resolution: fixed
Keywords: | Triage Stage: Ready for
| checkin
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
Comment (by Mariusz Felisiak <felisiak.mariusz@…>):
In [changeset:"d9594c4ea57b6309d93879805302cec9ae9f23ff" d9594c4]:
{{{
#!CommitTicketReference repository=""
revision="d9594c4ea57b6309d93879805302cec9ae9f23ff"
[2.2.x] Fixed #32713, Fixed CVE-2021-32052 -- Prevented newlines and tabs
from being accepted in URLValidator on Python 3.9.5+.
In Python 3.9.5+ urllib.parse() automatically removes ASCII newlines
and tabs from URLs [1, 2]. Unfortunately it created an issue in
the URLValidator. URLValidator uses urllib.urlsplit() and
urllib.urlunsplit() for creating a URL variant with Punycode which no
longer contains newlines and tabs in Python 3.9.5+. As a consequence,
the regular expression matched the URL (without unsafe characters) and
the source value (with unsafe characters) was considered valid.
[1] https://bugs.python.org/issue43882 and
[2]
https://github.com/python/cpython/commit/76cd81d60310d65d01f9d7b48a8985d8ab89c8b4
Backport of e1e81aa1c4427411e3c68facdd761229ffea6f6f from main.
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/32713#comment:9>