[Django] #33969: Improve django.core.mail.messages EAI processing

[Django]

#33969: Improve django.core.mail.messages EAI processing
-----------------------------------------+------------------------------
Reporter: j-bernard | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Mail) | Version: 4.0
Severity: Normal | Keywords: EAI IDNA RFC
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------------
This ticket is the third and last of a list of tickets aiming at bringing
Email Address Internationalization (EAI) compliance to Django by
supporting International Domain Name (IDN) with regards to the latest
standard (IDNA 2008) and fixing some processing on internationalized
domains or email addresses.
Previous tickets: #33967, #33968

`sanitize_address` transforms the email address domain in Punycode
regardless of the email server's compliance with EAI.

The main issue here is that the conversion is performed with the
deprecated IDNA 2003 standard instead of IDNA 2008 (see the previous
ticket for more information) but this conversion should also be skipped
for consistency with the user input and only performed if the server does
not support Unicode email addresses.

The logic (with the backend using Python smtplib) would then be:
- try to send the message, regardless of the presence of Unicode in the
address (Python smtplib will add the right options if you use the
send_message method else the SMTPUTF8 option should be provided)
- if smtplib.SMTPNotSupportedError is raised
- if the local-part is ASCII only, convert the domain to A-Label
using and IDNA 2008 compliant library and retry
- else return failure

--
Ticket URL: <https://code.djangoproject.com/ticket/33969>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#33969: Improve django.core.mail.messages EAI processing

-------------------------------+--------------------------------------

Reporter: j-bernard | Owner: nobody
Type: Uncategorized | Status: new
Component: Core (Mail) | Version: 4.0

Severity: Normal | Resolution:

Keywords: EAI IDNA RFC | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-------------------------------+--------------------------------------

Comment (by Florian Apolloner):

> try to send the message, regardless of the presence of Unicode in the
address (Python smtplib will add the right options if you use the
send_message method else the SMTPUTF8 option should be provided)

Can you link the code in question?

> if smtplib.SMTPNotSupportedError is raised

What does this error tell us? Given a standard setup your "mail server" is
often a simple relay running on localhost which probably doesn't give you
errors often. Whether or how the server it relays to does domain
internationalization is unknown to it.

I am not deep into mail standards, so any details you could provide here
would be helpful. I also do think that we have some leeway when changing
stuff here because international domain names are not used often. We just
should make clear to not introduce any security issues (like sending mail
to a different domain all of a sudden -- though I am not sure if we can
prevent that realistically because for instance `ß` encodes differently in
the standards iirc).

--
Ticket URL: <https://code.djangoproject.com/ticket/33969#comment:1>

[Django]

#33969: Improve django.core.mail.messages EAI processing
------------------------------+--------------------------------------
Reporter: j-bernard | Owner: nobody

Type: New feature | Status: closed
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution: needsinfo

Keywords: EAI IDNA RFC | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

------------------------------+--------------------------------------
Changes (by Mariusz Felisiak):

* status: new => closed
* type: Uncategorized => New feature
* version: 4.0 => dev
* resolution: => needsinfo

--
Ticket URL: <https://code.djangoproject.com/ticket/33969#comment:2>

[Django]

#33969: Improve django.core.mail.messages EAI processing
------------------------------+--------------------------------------
Reporter: j-bernard | Owner: nobody
Type: New feature | Status: closed
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution: needsinfo
Keywords: EAI IDNA RFC | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------+--------------------------------------

Comment (by j-bernard):

Replying to [comment:1 Florian Apolloner]:

> > try to send the message, regardless of the presence of Unicode in the
address (Python smtplib will add the right options if you use the
send_message method else the SMTPUTF8 option should be provided)
>
> Can you link the code in question?

[https://github.com/django/django/blob/main/django/core/mail/message.py#L116
This line] transforms the domain to A-Label.

>
> > if smtplib.SMTPNotSupportedError is raised
>
> What does this error tell us? Given a standard setup your "mail server"
is often a simple relay running on localhost which probably doesn't give
you errors often. Whether or how the server it relays to does domain
internationalization is unknown to it.

This error means the mail server does not support the SMTPUTF8 option and
won't be able to process a `to` address with Unicode.

>
> I am not deep into mail standards, so any details you could provide here
would be helpful. I also do think that we have some leeway when changing
stuff here because international domain names are not used often. We just
should make clear to not introduce any security issues (like sending mail
to a different domain all of a sudden -- though I am not sure if we can
prevent that realistically because for instance `ß` encodes differently in
the standards iirc).

IDNA 2008 was created to lower some security issues with the old IDNA 2003
standard. And as you mentioned, keeping both standards active creates even
more issues. Many already moved to the new standard and Django could
clearly make a difference by being also compliant. Without that kind of
move, IDNs will keep being "not used often".

--
Ticket URL: <https://code.djangoproject.com/ticket/33969#comment:3>

[Django]

#33969: Improve django.core.mail.messages EAI processing
------------------------------+--------------------------------------
Reporter: j-bernard | Owner: nobody
Type: New feature | Status: closed
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution: needsinfo
Keywords: EAI IDNA RFC | Triage Stage: Unreviewed
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
------------------------------+--------------------------------------

Comment (by Mike Edmunds):

I looked into this earlier today as part of ticket #35581, and was
surprised to find that IDNA 2003 is probably still the correct choice for
sending email. Documenting my findings here.

The problem is for domains containing one of the
[https://www.unicode.org/reports/tr46/#Deviations "deviation characters"]
where the two IDNA versions differ. For instance:

IDNA 2003: otto@​faß.example → otto@​fass.example
IDNA 2008: otto@​faß.example → otto@​xn--fa-hia.example

If those two domains are owned by different people, and Django uses a
different version of IDNA than Otto expects, Otto's email could go to the
wrong person. Big problem.

So the question is, what version of IDNA does Otto expect? Browsers have
all updated to IDNA 2008: if you enter http://faß.example, you will end up
at http://xn--fa-hia.example, not http://fass.example. (You can try this
with the .de equivalents to those domains, which are currently parked at
different registrars.)

I had assumed email should match the browsers, and be using IDNA 2008 by
now. (And I was thinking that Django's ''not'' using it for email
addresses was a serious security issue.) I was wrong.

In testing earlier today, I found both Gmail and Outlook.com are still
using IDNA 2003 for domains in address headers: both treat
otto@​faß.example as otto@​fass.example. (They might be using IDNA 2008,
but with UTS #​46 "transitional processing" enabled, which retains the
IDNA 2003 encoding for the deviation characters.)

Bottom line: we wouldn't want to switch Django's sanitize_address() to use
IDNA 2008 encoding (at least not without transitional processing), because
''that'' would actually introduce a security issue, by sending Otto's
email to an unexpected domain.

Also, If I'm understanding correctly, part of the request here is to be
able to get Django's EmailMessage.message().as_string() to generate a
message that hasn't had ''any'' encoding applied to the addresses, for use
with SMTPUTF8. (That is, `To: jörg@faß.example` should stay just like
that, not turn into `To: =?utf-8?q?j=C3=B6rg?=@fass.example`.) I'm hoping
to address that as part of #35581, if
[https://docs.python.org/3/library/email.policy.html#email.policy.SMTPUTF8
`email.policy.SMTPUTF8`] is used for EmailMessage.message().

Note that Django's SMTP EmailBackend doesn't currently support SMTPUTF8.
That's probably best handled as a separate new feature request. (Or could
also be implemented by a third-party custom EmailBackend.)
--
Ticket URL: <https://code.djangoproject.com/ticket/33969#comment:4>