[Django] #36923: URLField mangles mailto: URLs

[Django]

#36923: URLField mangles mailto: URLs
---------------------------------+-----------------------------------------
Reporter: waveywhite | Type: Uncategorized
Status: new | Component: Forms
Version: 5.2 | Severity: Normal
Keywords: mailto urlfield | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+-----------------------------------------
Having derived from django.forms.URLField and django.models.URLField to
set up a form that can handle mailto URLs, I'm finding that the URL gets
mangled. Entering "mailto:te...@example.com" results in
"mailto://te...@example.com".

This was encountered while creating models for a Wagtail site. Wagtail
uses Django's models and form fields within its framework.

My modified URLFields:

{{{
from django import forms
from django.db import models

class ContactUrlFormField(forms.URLField):
default_validators = [
validators.URLValidator(schemes=['http', 'https', 'mailto',
'tel'])
]

class ContactUrlField(models.URLField):
default_validators = [
validators.URLValidator(schemes=['http', 'https', 'mailto',
'tel'])
]

def formfield(self, **kwargs):
return super().formfield(
**{
"form_class": ContactUrlFormField,
**kwargs,
}
)
}}}

NB. It would be great to have a way of defining which URL schemes are
allowed as an option in the model field and then have it picked up by the
form field automatically.
--
Ticket URL: <https://code.djangoproject.com/ticket/36923>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36923: URLField mangles mailto: URLs
---------------------------------+--------------------------------------

Reporter: waveywhite | Owner: Vishy Algo
Type: Uncategorized | Status: assigned
Component: Forms | Version: 5.2
Severity: Normal | Resolution:

Keywords: mailto urlfield | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+--------------------------------------

Changes (by Vishy Algo):

* owner: (none) => Vishy Algo
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/36923#comment:1>

[Django]

#36923: URLField mangles mailto: URLs
---------------------------------+--------------------------------------
Reporter: waveywhite | Owner: Vishy Algo

Type: Bug | Status: assigned

Component: Forms | Version: 5.2
Severity: Normal | Resolution:

Keywords: mailto urlfield | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+--------------------------------------

Changes (by Jacob Walls):

* stage: Unreviewed => Accepted
* type: Uncategorized => Bug

Comment:

Thanks for the report. Looks like the mangling is happening
[https://github.com/django/django/blob/380d77cccefbe185ddb3f9368d8fdeb7b7cf7108/django/forms/fields.py#L801-L808
here]. We shouldn't assume that "if a domain is not provided, that the
path segment contains the domain" for mailto links.

Tentatively accepting, since this is a security-sensitive area, and we
have to make sure however we adjust this heuristic can't be fooled.
--
Ticket URL: <https://code.djangoproject.com/ticket/36923#comment:2>

[Django]

#36923: URLField mangles mailto: URLs
---------------------------------+--------------------------------------
Reporter: waveywhite | Owner: Vishy Algo
Type: Bug | Status: assigned
Component: Forms | Version: 5.2
Severity: Normal | Resolution:
Keywords: mailto urlfield | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+--------------------------------------

Comment (by Vishy Algo):

Replying to [comment:2 Jacob Walls]:

The current normalization logic unconditionally assumes a hierarchical
structure, forcing // on opaque schemes like mailto: and tel:.

We could restrict this domain enforcement to an allowlist of standard web
schemes (http, https, ftp, ftps) to prevent mangling valid URIs that use
data segments. Note that we already default the scheme to http if none is
provided.

What's your take on this approach?
--
Ticket URL: <https://code.djangoproject.com/ticket/36923#comment:3>

[Django]

#36923: URLField mangles mailto: URLs
---------------------------------+--------------------------------------
Reporter: waveywhite | Owner: Vishy Algo
Type: Bug | Status: assigned
Component: Forms | Version: 5.2
Severity: Normal | Resolution:
Keywords: mailto urlfield | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
---------------------------------+--------------------------------------

Comment (by Jacob Walls):

Seems reasonable!
--
Ticket URL: <https://code.djangoproject.com/ticket/36923#comment:4>

[Django]

#36923: URLField mangles mailto: URLs

-------------------------------------+-------------------------------------
Reporter: waveywhite | Owner: Natalia
| Bidart

Type: Bug | Status: assigned
Component: Forms | Version: 5.2
Severity: Normal | Resolution:
Keywords: mailto urlfield | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------------+-------------------------------------
Changes (by Jacob Walls):

* owner: Vishy Algo => Natalia Bidart

Comment:

Hi Vishy. Natalia mentioned she has a local stash that solves this issue.
Do you mind if I assign it to her? I don't want you to duplicate any
effort.
--
Ticket URL: <https://code.djangoproject.com/ticket/36923#comment:5>

[Django]

#36923: URLField mangles mailto: URLs
-------------------------------------+-------------------------------------
Reporter: waveywhite | Owner: Natalia
| Bidart
Type: Bug | Status: assigned
Component: Forms | Version: 5.2
Severity: Normal | Resolution:
Keywords: mailto urlfield | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Vishy Algo):

Replying to [comment:5 Jacob Walls]:

> Hi Vishy. Natalia mentioned she has a local stash that solves this
issue. Do you mind if I assign it to her? I don't want you to duplicate
any effort.

That's fine Jacob. However, I wanna give a heads up on this:

While verifying scheme in {{{ django.core.validators.URLValidator }}}, the
delimiter used in {{{ :// }}}. I think we should just use a semi-colon
(":"), which is standard scheme delimiter according to [https://www.rfc-
editor.org/rfc/rfc3986.html#section-1.2.3 RFC 3986 Standards].

Please have a look at this, and provide your opinion.

{{{
#!diff
--git a/django/core/validators.py b/django/core/validators.py
index 534acdd904..18239e3ad4 100644
--- a/django/core/validators.py
+++ b/django/core/validators.py
@@ -166,7 +166,7 @@ class URLValidator(RegexValidator):
if self.unsafe_chars.intersection(value):
raise ValidationError(self.message, code=self.code,
params={"value": value})
# Check if the scheme is valid.
- scheme = value.split("://")[0].lower()
+ scheme = value.split(":")[0].lower()
if scheme not in self.schemes:
raise ValidationError(self.message, code=self.code,
params={"value": value})
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36923#comment:6>

[Django]

#36923: URLField mangles mailto: URLs
-------------------------------------+-------------------------------------
Reporter: waveywhite | Owner: Natalia
| Bidart
Type: Bug | Status: assigned
Component: Forms | Version: 5.2
Severity: Normal | Resolution:
Keywords: mailto urlfield | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Jacob Walls):

Good question. The `URLValidator` is out of scope. I only accepted the
ticket for removing the mangling inside `to_python()`, but I didn't
clarify that well enough. (We have other tickets, some of which were
wontfixed, for extending `URLValidator` to work with other schemes, e.g.
#36054, #25593, and #25594.)
--
Ticket URL: <https://code.djangoproject.com/ticket/36923#comment:7>