Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

I want to propose updating django.core.mail to replace use of Python's legacy email.message.Message (and other legacy email APIs) with email.message.EmailMessage (and other modern APIs).

If there's interest, I can put together a more detailed proposal and/or ticket, but was hoping to get some initial feedback first. (I searched for relevant discussions in django-developers and the issue tracker, and didn't find any. Apologies if this has come up before.)

[Note: I maintain django-anymail, which implements Django integration with several transactional email service providers.]

Background

Since Python ~3.6, Python's email package has included two largely separate implementations:

• a "modern (unicode friendly) API" based on email.message.EmailMessage and email.policy.default
• a "legacy API" providing compatibility with older Python versions, based on email.message.Message and email.policy.compat32

(See https://docs.python.org/3/library/email.html, especially toward the end.)

django.core.mail currently uses the legacy API.

Why switch?

There are no plans to deprecate Python's legacy email API, and it's working, so why change Django?

• Fewer bugs: The modern API fixes a lot of bugs in the legacy API. My understanding is legacy bugs will generally not be fixed. (And in fact, there are some cases where the legacy API deliberately replicates earlier buggy behavior.)

  Django #35497 is an example of a legacy bug (with a problematic proposed workaround) which is just fixed in the modern API.

• Simpler, safer code: A substantial portion of django.core.mail's internals implements workarounds and security fixes for problems in the legacy API. This would be greatly simplified—maybe eliminated completely—using the modern API.

  Examples: the modern API prevents CR/NL injections in message headers. It serializes and folds address headers properly—even ones with unicode names. It enforces single-instance headers where appropriate.

• Easier to work with: The modern API adds some nice conveniences for developers working in django.core.mail, third-party library developers, and (depending on what we choose to expose) users of Django's mail APIs.

  Examples: populating the "Date" header with a datetime or an address header with Address objects—without needing intricate knowledge of email header formats. Using email.policy to generate a 7-bit clean serialization (without having to muck about with the MIME parts).

Concerns & risks

Compatibility and security, of course…

• Backwards compatibility (for API users): django.core.mail largely insulates callers from the underlying Python email package. There are a few places where this leaks (e.g., attachments allows legacy email MIMEBase objects), but in general the switch should be transparent. (And I have some ideas for supporting the other cases.)

• Backwards compatibility (for third-party libraries): Some libraries may use internals I'd propose removing (e.g., SafeMIME and friends); we'd handle this through deprecation.

• Backwards compatibility (bug-level): There's probably some code out there that unintentionally depends on legacy email bugs (or the specific ways Django works around them). I don't have any examples, but I also don't have a good solution for when they surface. Plus, while Python's modern email API is pretty mature at this point, there are still new bugs being reported against it. Email is complicated.

• Security: As noted above, the modern API should be more secure than the legacy one. But we also have a nice collection of email security tests—which mostly don't depend on internal implementation. We'd keep these.

[Mohamed El-Kalioby]

+1, I like to help in this

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f410ad79-a034-4275-88a7-22e7626c06fdn%40googlegroups.com.

[Arthur Pemberton]

Would this be designed to be compatible with "Proposal 14: Background Workers"?

--

[Mike Edmunds]

> Would this be designed to be compatible with "Proposal 14: Background Workers"?

I wouldn't expect this to impact background workers one way or the other. The same goes for the async email proposal(s) floating around.

Virtually all of this work would occur in django.core.mail.message, where Python's `email` APIs are used. A goal is to avoid changes that would break existing email backends (Django or third-party).

The background workers proposal will implement a new background SMTP EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend doesn't directly use Python's `email` APIs, and there should be no reason for background SMTP to be any different. (It might be helpful to know that Python's `email` library isn't involved in sending email; that's handled by Python's `smtplib`, which Django uses only in the SMTP EmailBackend.)

The existing SMTP EmailBackend does use one function I expect will become deprecated: django.core.mail.message.sanitize_address(). I haven't yet investigated whether that use is still necessary, or whether it's there to get around past bugs/limitations in Python's smtplib. If any of it is still needed for SMTP, I'd probably want to move that code into the SMTP EmailBackend(s).

- Mike

[Arthur Pemberton]

>  The background workers proposal will implement a new background SMTP EmailBackend (in django.core.mail.backends).

I had missed that fact. Thanks for the explanation.

I for one think that this is a good proposal -- this would modern transactional email sending.

- Arthur

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8f4716b9-0856-46bd-af16-0ce715f195b2n%40googlegroups.com.

[Adam Johnson]

After my comment in the steering council vote and in-person conversation with Jake, I believe the SMTP backend will not be implemented for DEP 14 : https://forum.djangoproject.com/t/steering-council-vote-on-background-tasks-dep-14/31131/20

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CA%2BX4dQSCwndDFcOid0Lyi4PtUJw264DTZK-r6vSNNoRh%2B4gBTQ%40mail.gmail.com.

[Ronny V.]

Hi all!

Jakob Rief pointed this discussion out to me. I've been going around lately to make some advertisement for my idea of class-based emails.

I've implemented a package called "django-pony-express" which in a nutshell provides to things:

* A class-based way of creating new emails (very similar to class-based views)

* A test suite for easy unit-testing of emails (we are currently working on bringing this to Django core)

Here's a documentation link: https://django-pony-express.readthedocs.io/en/latest/features/introduction.html#create-a-single-email

The idea is to create emails like you create views. All important stuff is encapsulated but you can overwrite everything you need. There are a bunch of examples in the docs.

Apart from not having to deal with low-level email API stuff (which is a pain IMHO), it provides lots of neat improvements I had to (re-)invent in every project I was working on.

What do you people think about this? I got quite good feedback in Vigo and in my opinion, the solution is very Django-esque.

I'd be happy to go into more detail if required but wanted to keep this first commend brief.

Best from Cologne

Ronny

[Mike Edmunds]

Hi Ronny,

django-pony-express looks really interesting, thanks. (I'm going to add a link in django-anymail's "you probably don't need proprietary ESP templates" docs.)

I think django-pony-express is very complementary to this proposal, both as a third-party library and if some version of class-based emails finds its way into Django core. 

Here's my mental model of Django's email sending functionality:

• A django.core.mail.message.EmailMessage (or EmailMultiAlternatives*) is a list of ingredients for building an email message—it's mostly just a "bucket of properties" that specifies what should end up in the message.**
• An EmailBackend is responsible for transmitting that EmailMessage to a server that will actually send an email—an SMTP server, an email service provider's HTTP API, etc. Each backend implements its own recipe to bake the Django EmailMessage ingredients into the form expected by its server—an RFC 2822+ message for SMTP, a JSON API payload for an ESP, etc.
• Because a Django EmailMessage can be "a pain" (you're not wrong!), there are convenience APIs that simplify constructing and sending it. Some come with Django: send_mail(), send_mass_mail(), mail_admins(). Some come from third party libraries: django-pony-express, django-templated-mail, etc. Many get built (and repeatedly re-built) by developers in individual Django projects.

To stretch your class-based view analogy, Django's EmailMessage is roughly akin to Django's HttpResponse. You can write a view function that builds up an HttpResponse from scratch. For a lot of common cases, though, it's much easier to work one of Django's helper functions or class-based View subclasses. But however you go about it, every view eventually has to return an HttpResponse that Django can hand off to its "http backend" (wsgi/asgi).

Similarly, there are several ways to build a Django EmailMessage. At the end of the day, though, all of these need to return a django.core.mail.EmailMessage that Django can hand off to an email backend for sending.

- Mike

_____

* I suspect the EmailMultiAlternatives/EmailMessage distinction is no longer helpful. I'm fairly certain it's confusing to users, and I know it adds complexity for third-party email backends. Given that modern Python email convenience methods handle all the multipart restructuring, I'll propose collapsing EmailMultiAlternatives/EmailMessage into a single class as part of this work.

** Along with the "bucket of properties," Django's EmailMessage also implements serialization to RFC 2822+ format in as_message(). This is used by many—but certainly not all—email backends. And (just to come full circle here in the footnotes), the bulk of work in this proposal is actually about updating as_message() and the private helpers it calls.

[Mike Edmunds]

> After my comment in the steering council vote and in-person conversation with Jake, I believe the SMTP backend will not be implemented for DEP 14

Then I can improve my earlier response: I am confident this proposal will have no impact on background workers. :-)

- Mike

[Mike Edmunds]

Since the early feedback seems positive (though we're still waiting for more votes), here's some additional detail on the changes I think would be involved to update django.core.mail to use Python's modern email API.

(See my earlier message for background on Python's legacy vs. modern email APIs and why updating is useful.)

Note: Django and Python both have classes named EmailMessage. I'm using "Django EmailMessage" to refer to django.core.mail.message.EmailMessage, and "Python EmailMessage" (or sometimes just "modern API") to refer to Python's modern email.message.EmailMessage.

Necessary work

1. Update tests.mail.tests to use modern email APIs

   Where a test relies on legacy implementation details, try to rewrite it to be implementation agnostic if possible; otherwise try to retain the spirit of the test using modern APIs.

   Retain all security related tests, with updates as appropriate. (Even where we know Python's modern API handles the security for us, it doesn't hurt to double check.)

   Probably need to add some cases for existing behavior not currently covered by tests, particularly around attachments and alternatives.

   Remove a test case only if it's truly no longer relevant and can't be usefully updated. (And perhaps leave behind a brief comment explaining why.)

   (Legacy email APIs used in tests.mail.tests: email.charset, email.header.Header, MIMEText, parseaddr. Also message_from_… currently defaults to legacy policy.compat32.)

2. Update django.core.mail.message.EmailMessage to use modern email APIs

   Change Django's EmailMessage.message() to construct a modern email.message.EmailMessage rather than a SafeMIME object (which is based on legacy email.message.Message with policy=compat32). Add a message(policy=default) param forwarded to Python's EmailMessage constructor.

   Hoist alternative part handling from Django's EmailMultiAlternatives into Django's base EmailMessage to simplify the code. (More on this below.)

   In _create_alternatives(), use modern add_alternative() and friends to replace legacy SafeMIME objects.

   In _create_attachments(), use modern add_attachment(). Handle (legacy Python) MIMEBase objects as deprecated, and convert to modern equivalent. This is a relatively complex topic; I'll post a separate message about it later. (I also have some questions about how best to handle content-disposition "inline" and whether we want to somehow support multipart/related.)

   Remove the private Django EmailMessage methods _create_mime_attachment() and _create_attachment() (without deprecation).

   (Legacy APIs used in django.core.mail.message: email.message.Message, email.charset, email.encoders, email.header, email.mime, email.utils)

3. Deprecate unused internal APIs from django.core.mail.message

   Django will no longer need these (Python's modern email API covers their functionality), but they may be in use by third-party libraries:

   • utf8_charset
   • utf8_charset_qp
   • RFC5322_EMAIL_LINE_LENGTH_LIMIT
   • BadHeaderError^ (more details below)
   • ADDRESS_HEADERS
   • forbid_multi_line_headers()^
   • sanitize_address() (more details below)
   • MIMEMixin
   • SafeMIMEMessage
   • SafeMIMEText^
   • SafeMIMEMultipart^

   (Items marked ^ are exposed in django.core.mail via __all__. I haven't looked into the reason for that.)

4. Update django.core.mail.__init__ to avoid EmailMultiAlternatives, and django.core.mail.backend.smtp to replace sanitize_address. (More details below.)

5. Update docs

   • deprecation of legacy MIMEBase in attachments list, what to do instead
   • eliminate EmailMessage/EmailMultiAlternatives distinction
   • deprecation of internal legacy items from #3

Tricky bits and things requiring longer discussion

• Legacy MIMEBase attachments: I'll post a separate message about this sometime later. (Maybe next week. It's a long topic, plus I'm still experimenting.)

• EmailMultiAlternatives: I'd like to get rid of the distinction between Django's EmailMessage and EmailMultiAlternatives, and just move the logic for alternative parts into base EmailMessage. (And then simplify the docs.)

  EmailMultiAlternatives is additional complexity for users, and it isn't really helpful with Python's modern add_alternative() API.

  We could deprecate EmailMultiAlternatives, but that would be a very noisy deprecation. (Everything sending plaintext + html email today has to use EmailMultiAlternatives.) My preference would just be to leave a stub class with a doc string:

  class EmailMultiAlternatives(EmailMessage):
      # TODO: add doc string explaining what this used to be,
      # and suggesting just using EmailMessage instead.
      pass

• BadAddressHeader: is a ValueError subclass raised only by forbid_multi_line_headers()—which would be deprecated. (The modern email API raises a ValueError for CR/NL in headers.)

  Do we want to issue a deprecation warning for this? The warning has to be on import, not on use. (There's a way to do that with a module-level __getattr()__, which I see we've done before in db.models.enums.)

  Otherwise I'm inclined to just write BadAddressHeader = ValueError in the deprecated code. (forbid_multi_line_headers() will warn about deprecation when it's called.)

  Same question about all the deprecated constants (like RFC5322_EMAIL_LINE_LENGTH_LIMIT).

• sanitize_address(): This function is called from two places in Django:

  • forbid_multi_line_headers() (which would be deprecated)
  • Django's SMTP EmailBackend to process the "envelope" from and recipient addresses (which may not be the same as the From, To, etc. message headers—e.g., bcc is only in envelope recipients, not message headers)

  sanitize_address() does a bunch of things:

  • Raises ValueError some forms of invalid addresses
  • Checks again for CR/NL, and raises ValueError (not BadAddressHeader)
  • Encodes/re-encodes non-ascii display names to ascii RFC 2047 (using a mix of legacy and modern email APIs that may be the source of some bugs)
  • Encodes/re-encodes non-ascii addr-specs to ascii RFC 2047 mailbox + punycode domain

  The modern email APIs handle all of this for message headers. (Except the punycode part, which is arguably wrong in message header fields.)

  However, I believe at least some of this is necessary for smtplib. In particular, the envelope from and recipient addresses have to be ascii only. (Unless SMTP.sendmail() is called with the "SMTPUTF8" option—which is not currently possible with Django's SMTP EmailBackend—and the SMTP server supports that option.)

  I'd suggest we deprecate sanitize_address(), and create a simpler private function in the SMTP EmailBackend to perform the SMTP-specific ascii conversion.

• Other stuff: I haven't yet investigated what to do with:

  • Django EmailMessage.encoding and settings.DEFAULT_CHARSET
  • Django EmailMultiAlternatives.alternative_subtype (when overridden)

[Tom Carrick]

I'm in favour of this change, and nice that you're thinking about the future, but if you're going to write a ticket for this I would focus it on purely the strictly necessary parts to update to the new API, and when that is done, make more proposals to simplify the API as you suggest.

I say this just with the goal of getting something that is easy to agree on merged, before dealing with potentially contentious things like EmailMultiAlternatives, MIMEBase, etc.

Ignore me if this is irrelevant and those things are indeed necessary, my knowledge on the topic is a bit lacking.

Tom

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0041c2f3-2ba4-4e76-b7df-b3dc025eb4ean%40googlegroups.com.

[Paolo Melchiorre]

I agree with the approach suggested by Tom.

And thanks for proposing this enhancement.

Ciao,

Paolo

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAHoz%3DMbHArxHYnnEbWQi7AdL%3D9%2ByAheUQZQ1UoPkfS2QOvNXkA%40mail.gmail.com.

[Mike Edmunds]

> focus it on purely the strictly necessary parts to update to the new API … before dealing with potentially contentious things like EmailMultiAlternatives, MIMEBase, etc.

Appreciate the advice. I'll try to keep EmailMultiAlternatives and see whether that makes the new code hard to follow.

Sadly, MIMEBase attachments are a documented part of Django's EmailMessage API (and have been necessary for certain kinds of attachments), so they'll have to be dealt with one way or another.

- Mike

[Florian Apolloner]

Hi Mike,

overall the plan sounds good. I especially like the approach to "fix" the tests first. This can happen in an extra merge request and reviewed independently so we can be sure that we are still testing what we want to test before moving to the tricky parts.

On Thursday, June 27, 2024 at 3:28:06 AM UTC+2 Mike Edmunds wrote:

1. Deprecate unused internal APIs from django.core.mail.message

   Django will no longer need these (Python's modern email API covers their functionality), but they may be in use by third-party libraries:

   • utf8_charset
   • utf8_charset_qp
   • RFC5322_EMAIL_LINE_LENGTH_LIMIT
   • BadHeaderError^ (more details below)
   • ADDRESS_HEADERS
   • forbid_multi_line_headers()^
   • sanitize_address() (more details below)
   • MIMEMixin
   • SafeMIMEMessage
   • SafeMIMEText^
   • SafeMIMEMultipart^

   (Items marked ^ are exposed in django.core.mail via __all__. I haven't looked into the reason for that.)

Are all of those documented? If not we can simply remove them (especially if the deprecation implementation turns out to be a PITA). 

Cheers,

Florian

[Ronny V.]

Adding my support for the suggested approach. Start small and central and step-by-step. 👍

> (I'm going to add a link in django-anymail's "you probably don't need proprietary ESP templates" docs.)

Thanks! Much appreciated!

And thx on your thoughts about the class-based emails. I do agree that we should start with the proposed changes. Then we'll worry about the high-level stuff.

Best regards from Cologne

Ronny

PS: Sorry for misspelling Jacob Rief in my earlier posting!