Re: [Django] #36809: Allow EmailValidator to customize error messages depending on the part that failed validation

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: Daniel E
Type: | Onetti
Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Changes (by Mike Edmunds):

* cc: Mike Edmunds (added)

Comment:

Natalia's proposal would also simplify implementation of #27029 EAI
address validation. (Or help developers who need that functionality sooner
subclass EmailValidator themselves.)

Django already uses "recipient" to mean a complete email address, possibly
including a friendly display name (e.g.,
django.core.mail.EmailMessage.all_recipients()). The official RFC 5322
term for the part before the @ is ''local-part'', but other common terms
are ''user'' (e.g., existing EmailValidator code), ''username'' (e.g.,
Python's email.headerregistry.Address.username), or ''mailbox''. I'd maybe
go with `validate_username()` and `validate_domain()` to align with
Python's email package.
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:6>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: Daniel E
Type: | Onetti
Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

Thank you Mike for the validation! I appreciate it. Your naming proposal
sounds great!
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:7>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: Daniel E
Type: | Onetti
Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by jaffar Khan):

* cc: jaffar Khan (added)

Comment:

I want to work on this ticket as the current owner seem to no longer
active.
I think it is more genuine way to separate checks of username and domain
as Natalia proposed. I defined two methods **validate_username()** and
**validate_domain()** as:

{{{
def validate_username(self, user_part):
if not self.user_regex.match(user_part):
raise ValidationError(self.message, code=self.code,
params={"value": user_part})

def validate_domain(self, domain_part):
if domain_part not in self.domain_allowlist and not
self.domain_regex.match(domain_part):
raise ValidationError(self.message, code=self.code,
params={"value": domain_part})
}}}

then inside call, I called these two methods and removed the username and
domain checks as it looks no longer necessary:


{{{
def __call__(self, value):
# The maximum length of an email is 320 characters per RFC 3696
# section 3.
if not value or "@" not in value or len(value) > 320:
raise ValidationError(self.message, code=self.code,
params={"value": value})

user_part, domain_part = value.rsplit("@", 1)

self.validate_username(user_part)
self.validate_domain(domain_part)
}}}

I think it will be a more developer-friendly way. Please let me know
whether to create a PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:8>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: Daniel E
Type: | Onetti
Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by jaffar Khan):

I checked **docs/ref/validators.txt**, and there is no description about
methods of classes, so I don't think documentation changes are needed.
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:9>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: Daniel E
Type: | Onetti
Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by jaffar Khan):

I opened a PR [https://github.com/django/django/pull/20616] based on above
description.
There are some linter failures, If the modified changes are acceptable
then I will polish the PR.
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:10>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------

Reporter: Daniel E Onetti | Owner: jaffar
Type: | Khan

Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by jaffar Khan):

* owner: Daniel E Onetti => jaffar Khan

--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:11>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: jaffar
Type: | Khan
Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mike Edmunds):

Natalia, in attempting to review jaffar Khan's PR, I realized that while I
agree with these statements in theory, I'm having trouble putting them
into practice:

Replying to [comment:3 Natalia Bidart]:
> … Looking at the other validators in this module, there is a clear
pattern of separating parsing, normalization, and validation into distinct
steps or methods, even when the final error surface remains simple.
`EmailValidator` is somewhat inconsistent here: it already decomposes the
value into user and domain components, but does so inline inside
`__call__`, with no clear extension points. Adding a single extra param
addresses the symptom rather than the structure.
>
> I think a more Django aligned approach would be to re-evaluate
`EmailValidator` as a whole and give it clearer, overridable validation
hooks, while keeping full backward compatibility. For example, introducing
explicit methods like `validate_recipient()` and `validate_domain()`,
called from `__call__`, would allow customization via subclassing without
re-parsing the email or wrapping errors. …

I couldn't tell which validators you were referring to that separate
parsing, normalization and validation. EmailValidator follows the pattern
of the RegexValidators: it has some (overridable) regex and list
properties, and it does pretty much ''everything'' in its `__call__()`
method. Unlike the RegexValidators, it also has one extension point
method: `validate_domain_part()` allows replacing the domain validation
logic (but not `domain_allowlist` exceptions or ValidationError
construction). I ''believe'' the intent was to support stricter domain
validation via subclassing (e.g., for users who felt DomainNameValidator
was too lax or wanted to disallow numeric domain literals).

If a refactored EmailValidator looks something like:

{{{#!python
class EmailValidator:
def __call__(self, value):
... # pre-validation omitted
username, domain = value.rsplit("@", 1) # or
self.parse_parts(value) ?
self.validate_username(username, value)
self.validate_domain(domain, value)

def validate_domain(self, domain, value):
if domain not in self.domain_allowlist and not
some_other_logic_on(domain):

raise ValidationError(self.message, code=self.code,
params={"value": value})
}}}

… then the original ticket request to have access to the domain in the
error message ''does'' seem to require wrapping errors (or duplicating
logic from the superclass):

{{{#!python
class EmailValidatorWithBetterErrorMessages(EmailValidator):
def validate_domain(self, domain, value):
try:
super().validate_domain(domain, value)
except ValidationError as error:
# change "code" and add domain to the error params
raise ValidationError(self.message, code="invalid-domain",
params={"value": value, "domain": domain}) from error
}}}

… and substituting the domain validation logic requires duplicating some
of the superclass code:

{{{#!python
class EmailValidatorWithCustomDomainValidation(EmailValidator):
def validate_domain(self, domain, value):
if domain in self.domain_allowlist:
return # duplicate allowlist logic from superclass
if not idna.utils.is_valid_domain(domain):

raise ValidationError(self.message, code=self.code,
params={"value": value})
}}}

… which makes me think I've misunderstood what you were envisioning in
reworking EmailValidator to improve subclassing hooks.
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:12>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: jaffar
Type: | Khan
Cleanup/optimization | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 1
Needs tests: 1 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by jaffar Khan):

Thanks Mike for the review. I think defining subclass is not a good option
which makes duplication of code as you describe. Defining explicit methods
like I did will makes the code more readable and will avoid duplication.
In my current PR I made some changes like the methods
**validate_domain()** and **validate_username()** will return boolean so
we need to wrap the methods inside **__call__**. Now I am confused about
do I have to make these two methods to raise **ValidationError** then no
need for wrapping inside **__call__** or left as it is?
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:13>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------

Reporter: Daniel E Onetti | Owner: Natalia
| Bidart
Type: New feature | Status: assigned

Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0

Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Changes (by Natalia Bidart):

* needs_better_patch: 1 => 0
* needs_docs: 1 => 0
* needs_tests: 1 => 0
* owner: jaffar Khan => Natalia Bidart
* type: Cleanup/optimization => New feature


Old description:

> Currently, the EmailValidator only provides the full 'value' in the
> params dictionary when a ValidationError is raised. However, in many use
> cases, developers need to customize error messages based specifically on
> the domain part of the email (e.g., "The domain %(domain_part)s is not
> allowed").
>
> This change adds 'domain_part' to the params dictionary in
> EmailValidator.__call__, bringing it in line with how other validators
> provide decomposed parts of the validated value. This allows for more
> granular and helpful error messages for end users.
>
> I have already prepared a patch with tests and verified that it passes
> all style (flake8) and functional checks.

New description:

Currently, the EmailValidator only provides the full 'value' in the params
dictionary when a ValidationError is raised. However, in many use cases,
developers need to customize error messages based specifically on the
domain part of the email (e.g., "The domain %(domain_part)s is not
allowed").

This change adds 'domain_part' to the `params` dictionary in
`EmailValidator.__call__`, bringing it in line with how other validators
provide decomposed parts of the validated value. This allows for more
granular and helpful error messages for end users.

I have already prepared a patch with tests and verified that it passes all
style (flake8) and functional checks.

--
Comment:

Replying to [comment:12 Mike Edmunds]:

> I couldn't tell which validators you were referring to that separate
parsing, normalization and validation.

I made this reasoning a while ago, and since you asked, I kept snoozing
the email. Sorry! But I think I meant something like this:

* `URLValidator` starts with a parsing stage (`value.split("://")` for
the scheme, `urlsplit(value)` into components), then normalizes
(`.lower()` on the scheme), then validates each piece (scheme in
`schemes`, hostname length, IPv6 in the netloc), raising as it goes.
* `DecimalValidator` also has a parsing phase (`value.as_tuple()` into
digits/exponent), then normalizes (derives `digits` / `decimals` /
`whole_digits`), then validates each against the limits, raising a
distinct code per failure from its `messages` dict.
* `FileExtensionValidator` is smaller but IMO has the same pattern:
parse+normalize (`Path(value.name).suffix[1:].lower()`), then validate
against `allowed_extensions`.

Granted those keep the concerns inline in `__call__()` and are not split
by methods, but I think the "stages" are defined. The `BaseValidator`
family is the one that turns them into dedicated methods: `clean()` is the
normalize stage, `compare()` is the validate stage (the bool decision),
and `__call__()` owns the raise.

> [...] what you were envisioning in reworking EmailValidator to improve
subclassing hooks.

I don't think I had a specific implementation in mind when I wrote that,
but last night (and a lot of today, this was more time consuming than
anticipated, but it brought :sparkles:) I spent some time putting together
a more concrete plan. I went looking for precedents and used your use
cases as a guide (thanks for those BTW). Some thoughts:

1. **The hooks are a bool decision** A `validate_domain()` that ''only''
raises fuses the ''decision'' and the ''error'' into one method; fine for
replacing the logic, but for your `EmailValidatorWithBetterErrorMessages`
it forces exactly the wrapping you flagged. So the default hook returns a
bool and `__call__()` owns the raise. The key addition: each hook receives
the full `value`, so a subclass that ''does'' want a custom message can
override `validate_domain()` and even potentially raise a complete
`ValidationError` straight from the hook (with `value` in `params`, custom
code, etc), no wrapping. I deprecated `validate_domain_part()` in favor of
the nwe `validate_domain()` and added `validate_username()` (the names
align with Python's `email.headerregistry.Address`, per your suggestion).

2. **The default error `code` stays `"invalid"`.** There's a single
`code` for the whole validator, so today every failure raises `"invalid"`
and there's no way to give ''only'' the domain part its own code. The
`messages` dict idiom I wanted to borrow (`DecimalValidator`, and
`default_error_messages` across forms and model fields) ''does'' give each
failure its own code, but those validators were built with distinct codes
from scratch; bringing it to `EmailValidator` would mean the domain
failure starts emitting a ''new'' code instead of `"invalid"`, a
backwards-incompatible change with no clean deprecation path (I couldn't
find a precedent for changing an already-emitted code value, with or
without deprecation -- ideas welcome here!). What the new hooks ''do''
give users an opt-in: because they receive `value`, a subclass can raise
its own per-part code from `validate_domain()` (e.g.
`code="invalid_domain"`) without changing the default for everyone else.

So what's left puts the failed `username` / `domain` in `params`,
deprecates `validate_domain_part()` in favor of `validate_domain()`, and
adds `validate_username()`; both returning a bool by default but receiving
the full address: https://github.com/django/django/pull/21568
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:14>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: Natalia
| Bidart
Type: New feature | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Natalia Bidart):

Apparently, a solution for ticket #27029 would look like this (llm-
generated) following my PR:
{{{#!diff
@deconstructible
class EmailValidator:
message = _("Enter a valid email address.")
code = "invalid"
...
user_regex = _lazy_re_compile(
# dot-atom
r"(^[-!#$%&'*+/=?^_`{}|~0-9A-Z]+(\.[-!#$%&'*+/=?^_`{}|~0-9A-Z]+)*\Z"
...
)
+ # RFC 6531: a UTF-8 local part, i.e. the dot-atom class plus any non-
ASCII.
+ smtputf8_user_regex = _lazy_re_compile(
+ <something something>
+ r'*"\Z)',
+ re.IGNORECASE,
+ )
domain_allowlist = ["localhost"]

- def __init__(self, message=None, code=None, allowlist=None):
+ def __init__(self, message=None, code=None, allowlist=None,
allow_smtputf8=False):
if message is not None:
self.message = message
if code is not None:
self.code = code
if allowlist is not None:
self.domain_allowlist = allowlist
+ self.allow_smtputf8 = allow_smtputf8

def validate_username(self, username, value):
- return bool(self.user_regex.match(username))
+ regex = self.smtputf8_user_regex if self.allow_smtputf8 else
self.user_regex
+ return bool(regex.match(username))

def validate_domain(self, domain, value):
if self.domain_regex.match(domain):
return True
+ if self.allow_smtputf8:
+ # Accept an internationalized (U-label) domain via its IDNA
form.
+ try:
+ domain.encode("idna")
+ return True
+ except UnicodeError:
+ pass
literal_match = self.literal_regex.match(domain)
...
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:15>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: Natalia
| Bidart
Type: New feature | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------

Comment (by Mike Edmunds):

Replying to [comment:14 Natalia Bidart]:
> [...] So what's left puts the failed `username` / `domain` in `params`,

deprecates `validate_domain_part()` in favor of `validate_domain()`, and
adds `validate_username()`; both returning a bool by default but receiving
the full address: https://github.com/django/django/pull/21568

That looks great to me. I left a couple of questions and comments in the
PR.

Also, one option for enabling distinct codes (without going through
deprecation) might be something like:

{{{#!python
class EmailValidator(...):
code = "invalid"
code_username = code
code_domain = code

def __call__(value):
...
if not self.validate_username(user_part, value):
raise ValidationError(..., code=self.code_username, ...)
}}}

…so a custom subclass could keep all the built-in validation logic and
just override the codes:

{{{#!python
class EmailValidatorWithDistinctCodes(EmailValidator):
code_username = "invalid-username"
code_domain = "invalid-domain"
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:16>

[Django]

#36809: Allow EmailValidator to customize error messages depending on the part that
failed validation
-------------------------------------+-------------------------------------
Reporter: Daniel E Onetti | Owner: Natalia
| Bidart
Type: New feature | Status: assigned
Component: Core (Other) | Version: dev
Severity: Normal | Resolution:
Keywords: EmailValidator | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-------------------------------------+-------------------------------------
Comment (by Mike Edmunds):

Replying to [comment:15 Natalia Bidart]'s LLM:

> Apparently, a solution for ticket #27029 would look like this (llm-
generated) following my PR:
> {{{#!diff
> @deconstructible
> class EmailValidator:

> [...]

> + # RFC 6531: a UTF-8 local part, i.e. the dot-atom class plus any
non-ASCII.
> + smtputf8_user_regex = _lazy_re_compile(
> + <something something>
> + r'*"\Z)',
> + re.IGNORECASE,
> + )

> [...]

> - def __init__(self, message=None, code=None, allowlist=None):
> + def __init__(self, message=None, code=None, allowlist=None,
allow_smtputf8=False):

> [...]
> + self.allow_smtputf8 = allow_smtputf8
>
> [...]:

> def validate_domain(self, domain, value):
> if self.domain_regex.match(domain):
> return True
> + if self.allow_smtputf8:
> + # Accept an internationalized (U-label) domain via its IDNA
form.
> + try:
> + domain.encode("idna")
> + return True
> + except UnicodeError:
> + pass
> literal_match = self.literal_regex.match(domain)
> ...
> }}}

If I may speak directly to your LLM for a moment (or to anyone thinking
about copying and pasting that code into a PR for #27029): (1) Naming-
wise, let's try to avoid confusing the rfc6531 "smtputf8" SMTP protocol
extension with rfc6532 internationalized email headers (which allows utf8
in email addresses, among other things). Although those RFCs are related,
the `EmailValidator` is about addresses not SMTP. (2) That `<something
something>` in your regex is doing some heavy lifting. (3) The
`domain_name_regex` borrowed from `DomainNameValidator` already allows
IDNA U-labels. You've either managed to reintroduce a variation on the
dead code that was removed in 54059125956789ad4c19b77eb7f5cde76eec0643, or
you've broken IDNA 2008 domains by insisting on Python's IDNA 2003
encoding. 😀
--
Ticket URL: <https://code.djangoproject.com/ticket/36809#comment:17>