Proposal to upgrade django.core.mail to Python's modern email API

[Mike Edmunds]

I want to propose updating django.core.mail to replace use of Python's legacy email.message.Message (and other legacy email APIs) with email.message.EmailMessage (and other modern APIs).

If there's interest, I can put together a more detailed proposal and/or ticket, but was hoping to get some initial feedback first. (I searched for relevant discussions in django-developers and the issue tracker, and didn't find any. Apologies if this has come up before.)

[Note: I maintain django-anymail, which implements Django integration with several transactional email service providers.]

Background

Since Python ~3.6, Python's email package has included two largely separate implementations:

• a "modern (unicode friendly) API" based on email.message.EmailMessage and email.policy.default
• a "legacy API" providing compatibility with older Python versions, based on email.message.Message and email.policy.compat32

(See https://docs.python.org/3/library/email.html, especially toward the end.)

django.core.mail currently uses the legacy API.

Why switch?

There are no plans to deprecate Python's legacy email API, and it's working, so why change Django?

• Fewer bugs: The modern API fixes a lot of bugs in the legacy API. My understanding is legacy bugs will generally not be fixed. (And in fact, there are some cases where the legacy API deliberately replicates earlier buggy behavior.)

  Django #35497 is an example of a legacy bug (with a problematic proposed workaround) which is just fixed in the modern API.

• Simpler, safer code: A substantial portion of django.core.mail's internals implements workarounds and security fixes for problems in the legacy API. This would be greatly simplified—maybe eliminated completely—using the modern API.

  Examples: the modern API prevents CR/NL injections in message headers. It serializes and folds address headers properly—even ones with unicode names. It enforces single-instance headers where appropriate.

• Easier to work with: The modern API adds some nice conveniences for developers working in django.core.mail, third-party library developers, and (depending on what we choose to expose) users of Django's mail APIs.

  Examples: populating the "Date" header with a datetime or an address header with Address objects—without needing intricate knowledge of email header formats. Using email.policy to generate a 7-bit clean serialization (without having to muck about with the MIME parts).

Concerns & risks

Compatibility and security, of course…

• Backwards compatibility (for API users): django.core.mail largely insulates callers from the underlying Python email package. There are a few places where this leaks (e.g., attachments allows legacy email MIMEBase objects), but in general the switch should be transparent. (And I have some ideas for supporting the other cases.)

• Backwards compatibility (for third-party libraries): Some libraries may use internals I'd propose removing (e.g., SafeMIME and friends); we'd handle this through deprecation.

• Backwards compatibility (bug-level): There's probably some code out there that unintentionally depends on legacy email bugs (or the specific ways Django works around them). I don't have any examples, but I also don't have a good solution for when they surface. Plus, while Python's modern email API is pretty mature at this point, there are still new bugs being reported against it. Email is complicated.

• Security: As noted above, the modern API should be more secure than the legacy one. But we also have a nice collection of email security tests—which mostly don't depend on internal implementation. We'd keep these.

[Mohamed El-Kalioby]

+1, I like to help in this

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/f410ad79-a034-4275-88a7-22e7626c06fdn%40googlegroups.com.

[Arthur Pemberton]

Would this be designed to be compatible with "Proposal 14: Background Workers"?

--

[Mike Edmunds]

> Would this be designed to be compatible with "Proposal 14: Background Workers"?

I wouldn't expect this to impact background workers one way or the other. The same goes for the async email proposal(s) floating around.

Virtually all of this work would occur in django.core.mail.message, where Python's `email` APIs are used. A goal is to avoid changes that would break existing email backends (Django or third-party).

The background workers proposal will implement a new background SMTP EmailBackend (in django.core.mail.backends). The existing SMTP EmailBackend doesn't directly use Python's `email` APIs, and there should be no reason for background SMTP to be any different. (It might be helpful to know that Python's `email` library isn't involved in sending email; that's handled by Python's `smtplib`, which Django uses only in the SMTP EmailBackend.)

The existing SMTP EmailBackend does use one function I expect will become deprecated: django.core.mail.message.sanitize_address(). I haven't yet investigated whether that use is still necessary, or whether it's there to get around past bugs/limitations in Python's smtplib. If any of it is still needed for SMTP, I'd probably want to move that code into the SMTP EmailBackend(s).

- Mike

[Arthur Pemberton]

>  The background workers proposal will implement a new background SMTP EmailBackend (in django.core.mail.backends).

I had missed that fact. Thanks for the explanation.

I for one think that this is a good proposal -- this would modern transactional email sending.

- Arthur

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8f4716b9-0856-46bd-af16-0ce715f195b2n%40googlegroups.com.

[Adam Johnson]

After my comment in the steering council vote and in-person conversation with Jake, I believe the SMTP backend will not be implemented for DEP 14 : https://forum.djangoproject.com/t/steering-council-vote-on-background-tasks-dep-14/31131/20

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CA%2BX4dQSCwndDFcOid0Lyi4PtUJw264DTZK-r6vSNNoRh%2B4gBTQ%40mail.gmail.com.

[Ronny V.]

Hi all!

Jakob Rief pointed this discussion out to me. I've been going around lately to make some advertisement for my idea of class-based emails.

I've implemented a package called "django-pony-express" which in a nutshell provides to things:

* A class-based way of creating new emails (very similar to class-based views)

* A test suite for easy unit-testing of emails (we are currently working on bringing this to Django core)

Here's a documentation link: https://django-pony-express.readthedocs.io/en/latest/features/introduction.html#create-a-single-email

The idea is to create emails like you create views. All important stuff is encapsulated but you can overwrite everything you need. There are a bunch of examples in the docs.

Apart from not having to deal with low-level email API stuff (which is a pain IMHO), it provides lots of neat improvements I had to (re-)invent in every project I was working on.

What do you people think about this? I got quite good feedback in Vigo and in my opinion, the solution is very Django-esque.

I'd be happy to go into more detail if required but wanted to keep this first commend brief.

Best from Cologne

Ronny

[Mike Edmunds]

Hi Ronny,

django-pony-express looks really interesting, thanks. (I'm going to add a link in django-anymail's "you probably don't need proprietary ESP templates" docs.)

I think django-pony-express is very complementary to this proposal, both as a third-party library and if some version of class-based emails finds its way into Django core. 

Here's my mental model of Django's email sending functionality:

• A django.core.mail.message.EmailMessage (or EmailMultiAlternatives*) is a list of ingredients for building an email message—it's mostly just a "bucket of properties" that specifies what should end up in the message.**
• An EmailBackend is responsible for transmitting that EmailMessage to a server that will actually send an email—an SMTP server, an email service provider's HTTP API, etc. Each backend implements its own recipe to bake the Django EmailMessage ingredients into the form expected by its server—an RFC 2822+ message for SMTP, a JSON API payload for an ESP, etc.
• Because a Django EmailMessage can be "a pain" (you're not wrong!), there are convenience APIs that simplify constructing and sending it. Some come with Django: send_mail(), send_mass_mail(), mail_admins(). Some come from third party libraries: django-pony-express, django-templated-mail, etc. Many get built (and repeatedly re-built) by developers in individual Django projects.

To stretch your class-based view analogy, Django's EmailMessage is roughly akin to Django's HttpResponse. You can write a view function that builds up an HttpResponse from scratch. For a lot of common cases, though, it's much easier to work one of Django's helper functions or class-based View subclasses. But however you go about it, every view eventually has to return an HttpResponse that Django can hand off to its "http backend" (wsgi/asgi).

Similarly, there are several ways to build a Django EmailMessage. At the end of the day, though, all of these need to return a django.core.mail.EmailMessage that Django can hand off to an email backend for sending.

- Mike

_____

* I suspect the EmailMultiAlternatives/EmailMessage distinction is no longer helpful. I'm fairly certain it's confusing to users, and I know it adds complexity for third-party email backends. Given that modern Python email convenience methods handle all the multipart restructuring, I'll propose collapsing EmailMultiAlternatives/EmailMessage into a single class as part of this work.

** Along with the "bucket of properties," Django's EmailMessage also implements serialization to RFC 2822+ format in as_message(). This is used by many—but certainly not all—email backends. And (just to come full circle here in the footnotes), the bulk of work in this proposal is actually about updating as_message() and the private helpers it calls.

[Mike Edmunds]

> After my comment in the steering council vote and in-person conversation with Jake, I believe the SMTP backend will not be implemented for DEP 14

Then I can improve my earlier response: I am confident this proposal will have no impact on background workers. :-)

- Mike

[Mike Edmunds]

Since the early feedback seems positive (though we're still waiting for more votes), here's some additional detail on the changes I think would be involved to update django.core.mail to use Python's modern email API.

(See my earlier message for background on Python's legacy vs. modern email APIs and why updating is useful.)

Note: Django and Python both have classes named EmailMessage. I'm using "Django EmailMessage" to refer to django.core.mail.message.EmailMessage, and "Python EmailMessage" (or sometimes just "modern API") to refer to Python's modern email.message.EmailMessage.

Necessary work

1. Update tests.mail.tests to use modern email APIs

   Where a test relies on legacy implementation details, try to rewrite it to be implementation agnostic if possible; otherwise try to retain the spirit of the test using modern APIs.

   Retain all security related tests, with updates as appropriate. (Even where we know Python's modern API handles the security for us, it doesn't hurt to double check.)

   Probably need to add some cases for existing behavior not currently covered by tests, particularly around attachments and alternatives.

   Remove a test case only if it's truly no longer relevant and can't be usefully updated. (And perhaps leave behind a brief comment explaining why.)

   (Legacy email APIs used in tests.mail.tests: email.charset, email.header.Header, MIMEText, parseaddr. Also message_from_… currently defaults to legacy policy.compat32.)

2. Update django.core.mail.message.EmailMessage to use modern email APIs

   Change Django's EmailMessage.message() to construct a modern email.message.EmailMessage rather than a SafeMIME object (which is based on legacy email.message.Message with policy=compat32). Add a message(policy=default) param forwarded to Python's EmailMessage constructor.

   Hoist alternative part handling from Django's EmailMultiAlternatives into Django's base EmailMessage to simplify the code. (More on this below.)

   In _create_alternatives(), use modern add_alternative() and friends to replace legacy SafeMIME objects.

   In _create_attachments(), use modern add_attachment(). Handle (legacy Python) MIMEBase objects as deprecated, and convert to modern equivalent. This is a relatively complex topic; I'll post a separate message about it later. (I also have some questions about how best to handle content-disposition "inline" and whether we want to somehow support multipart/related.)

   Remove the private Django EmailMessage methods _create_mime_attachment() and _create_attachment() (without deprecation).

   (Legacy APIs used in django.core.mail.message: email.message.Message, email.charset, email.encoders, email.header, email.mime, email.utils)

3. Deprecate unused internal APIs from django.core.mail.message

   Django will no longer need these (Python's modern email API covers their functionality), but they may be in use by third-party libraries:

   • utf8_charset
   • utf8_charset_qp
   • RFC5322_EMAIL_LINE_LENGTH_LIMIT
   • BadHeaderError^ (more details below)
   • ADDRESS_HEADERS
   • forbid_multi_line_headers()^
   • sanitize_address() (more details below)
   • MIMEMixin
   • SafeMIMEMessage
   • SafeMIMEText^
   • SafeMIMEMultipart^

   (Items marked ^ are exposed in django.core.mail via __all__. I haven't looked into the reason for that.)

4. Update django.core.mail.__init__ to avoid EmailMultiAlternatives, and django.core.mail.backend.smtp to replace sanitize_address. (More details below.)

5. Update docs

   • deprecation of legacy MIMEBase in attachments list, what to do instead
   • eliminate EmailMessage/EmailMultiAlternatives distinction
   • deprecation of internal legacy items from #3

Tricky bits and things requiring longer discussion

• Legacy MIMEBase attachments: I'll post a separate message about this sometime later. (Maybe next week. It's a long topic, plus I'm still experimenting.)

• EmailMultiAlternatives: I'd like to get rid of the distinction between Django's EmailMessage and EmailMultiAlternatives, and just move the logic for alternative parts into base EmailMessage. (And then simplify the docs.)

  EmailMultiAlternatives is additional complexity for users, and it isn't really helpful with Python's modern add_alternative() API.

  We could deprecate EmailMultiAlternatives, but that would be a very noisy deprecation. (Everything sending plaintext + html email today has to use EmailMultiAlternatives.) My preference would just be to leave a stub class with a doc string:

  class EmailMultiAlternatives(EmailMessage):
      # TODO: add doc string explaining what this used to be,
      # and suggesting just using EmailMessage instead.
      pass

• BadAddressHeader: is a ValueError subclass raised only by forbid_multi_line_headers()—which would be deprecated. (The modern email API raises a ValueError for CR/NL in headers.)

  Do we want to issue a deprecation warning for this? The warning has to be on import, not on use. (There's a way to do that with a module-level __getattr()__, which I see we've done before in db.models.enums.)

  Otherwise I'm inclined to just write BadAddressHeader = ValueError in the deprecated code. (forbid_multi_line_headers() will warn about deprecation when it's called.)

  Same question about all the deprecated constants (like RFC5322_EMAIL_LINE_LENGTH_LIMIT).

• sanitize_address(): This function is called from two places in Django:

  • forbid_multi_line_headers() (which would be deprecated)
  • Django's SMTP EmailBackend to process the "envelope" from and recipient addresses (which may not be the same as the From, To, etc. message headers—e.g., bcc is only in envelope recipients, not message headers)

  sanitize_address() does a bunch of things:

  • Raises ValueError some forms of invalid addresses
  • Checks again for CR/NL, and raises ValueError (not BadAddressHeader)
  • Encodes/re-encodes non-ascii display names to ascii RFC 2047 (using a mix of legacy and modern email APIs that may be the source of some bugs)
  • Encodes/re-encodes non-ascii addr-specs to ascii RFC 2047 mailbox + punycode domain

  The modern email APIs handle all of this for message headers. (Except the punycode part, which is arguably wrong in message header fields.)

  However, I believe at least some of this is necessary for smtplib. In particular, the envelope from and recipient addresses have to be ascii only. (Unless SMTP.sendmail() is called with the "SMTPUTF8" option—which is not currently possible with Django's SMTP EmailBackend—and the SMTP server supports that option.)

  I'd suggest we deprecate sanitize_address(), and create a simpler private function in the SMTP EmailBackend to perform the SMTP-specific ascii conversion.

• Other stuff: I haven't yet investigated what to do with:

  • Django EmailMessage.encoding and settings.DEFAULT_CHARSET
  • Django EmailMultiAlternatives.alternative_subtype (when overridden)

[Tom Carrick]

I'm in favour of this change, and nice that you're thinking about the future, but if you're going to write a ticket for this I would focus it on purely the strictly necessary parts to update to the new API, and when that is done, make more proposals to simplify the API as you suggest.

I say this just with the goal of getting something that is easy to agree on merged, before dealing with potentially contentious things like EmailMultiAlternatives, MIMEBase, etc.

Ignore me if this is irrelevant and those things are indeed necessary, my knowledge on the topic is a bit lacking.

Tom

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/0041c2f3-2ba4-4e76-b7df-b3dc025eb4ean%40googlegroups.com.

[Paolo Melchiorre]

I agree with the approach suggested by Tom.

And thanks for proposing this enhancement.

Ciao,

Paolo

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAHoz%3DMbHArxHYnnEbWQi7AdL%3D9%2ByAheUQZQ1UoPkfS2QOvNXkA%40mail.gmail.com.

[Mike Edmunds]

> focus it on purely the strictly necessary parts to update to the new API … before dealing with potentially contentious things like EmailMultiAlternatives, MIMEBase, etc.

Appreciate the advice. I'll try to keep EmailMultiAlternatives and see whether that makes the new code hard to follow.

Sadly, MIMEBase attachments are a documented part of Django's EmailMessage API (and have been necessary for certain kinds of attachments), so they'll have to be dealt with one way or another.

- Mike

[Florian Apolloner]

Hi Mike,

overall the plan sounds good. I especially like the approach to "fix" the tests first. This can happen in an extra merge request and reviewed independently so we can be sure that we are still testing what we want to test before moving to the tricky parts.

On Thursday, June 27, 2024 at 3:28:06 AM UTC+2 Mike Edmunds wrote:

1. Deprecate unused internal APIs from django.core.mail.message

   Django will no longer need these (Python's modern email API covers their functionality), but they may be in use by third-party libraries:

   • utf8_charset
   • utf8_charset_qp
   • RFC5322_EMAIL_LINE_LENGTH_LIMIT
   • BadHeaderError^ (more details below)
   • ADDRESS_HEADERS
   • forbid_multi_line_headers()^
   • sanitize_address() (more details below)
   • MIMEMixin
   • SafeMIMEMessage
   • SafeMIMEText^
   • SafeMIMEMultipart^

   (Items marked ^ are exposed in django.core.mail via __all__. I haven't looked into the reason for that.)

Are all of those documented? If not we can simply remove them (especially if the deprecation implementation turns out to be a PITA). 

Cheers,

Florian

[Ronny V.]

Adding my support for the suggested approach. Start small and central and step-by-step. 👍

> (I'm going to add a link in django-anymail's "you probably don't need proprietary ESP templates" docs.)

Thanks! Much appreciated!

And thx on your thoughts about the class-based emails. I do agree that we should start with the proposed changes. Then we'll worry about the high-level stuff.

Best regards from Cologne

Ronny

PS: Sorry for misspelling Jacob Rief in my earlier posting!

[Mike Edmunds]

On Friday, June 28, 2024 at 1:07:35 PM UTC-7 Florian Apolloner wrote:

> Are all of those documented? If not we can simply remove them (especially if the deprecation implementation turns out to be a PITA).

It sort of depends on the definition of "documented." I've dug into real-world usage; results and proposals below.

Incidentally, I thought there was (used to be?) a policy that internal undocumented APIs were fair game for use by third-party libraries, subclassing, etc., so long as they didn't start with an underscore. (But "private" underscore APIs could have breaking changes at any time.) Am I remembering that wrong? Or was internal API stability only guaranteed for patch-level releases?

Should go through deprecation process (in my opinion, sorted roughly by real-world usage)

• BadHeaderError is mentioned in the docs as potentially raised during sending
  • Exposed in django.core.mail.__all__
  • Real-world use: ~2300+ cases in GitHub code search, seems to be primarily error handling
  • Proposal: deprecate on import. (Modern Python email API raises ValueError for newlines in headers. In our deprecation, we'll need `BadHeaderError = ValueError`—rather than subclassing ValueError—to preserve behavior of existing error handling code.)
  • Suggested replacement for docs: change `except BadHeaderError` to `except ValueError`
    
    
• sanitize_address() is not documented, but is widely used
  • Not exposed in django.core.mail (have to import from django.core.mail.message)
  • Real-world use: ~430 cases in GitHub code search, often in custom EmailBackend implementations or for special case handling that needs knowledge of how django.core.mail handles addresses.
  • Proposal: deprecate. (Have to keep implementation around anyway as part of other deprecated code.)
  • Suggested replacement for docs: none (it's an internal, undocumented API). Or we could say it depends on the use case: just skip it if no longer relevant, use something like Python's modern email Address object if you need formatting cleanup, or copy sanitize_address() into your code and adapt for your needs. (Incidentally, it's already pretty common to copy and adapt sanitize_address().)
    
    
• SafeMIMEText and SafeMIMEMultipart are mentioned in the docs as the return type of EmailMessage.message(), but not further documented
  • Exposed in django.core.mail.__all__
  • Real-world use: ~240 cases in GitHub code search, both for type checking and for constructing text or message attachments
  • Proposal: deprecate (on import, to catch type checking uses)
  • Suggested replacement for docs: django.mail.EmailMessage.message() will now return an email.message.EmailMessage; to construct text or message attachments, switch to modern email.message.EmailMessage
  • [Tip for django-stubs: SafeMIMEText, SafeMIMEMultipart, and email.message.EmailMessage are all subclasses of email.message.Message]
    
    
• forbid_multi_line_headers() is not documented, but seems to be an intentional part of the public API for implementing pluggable email backends
  • Exposed in django.core.mail as part of ticket-10355 "add support for email backends," and added to __all__ by Tim Graham via ticket-21302
  • Real-world use: ~100 cases in GitHub code search, mostly false positives; at least one real use in openedx/edx-platform to construct a from address
  • Proposal: deprecate. (Have to keep implementation around anyway as part of deprecated SafeMIME*.)
  • Suggested replacement for docs: modern Python email already detects CR/NL in headers; or use `if '\r' in value or '\n' in value` if your code needs to check it directly

Can remove without deprecation

(We'll have to keep these around as part of the implementation of things that are being deprecated, but can move them to a place that would break imports in existing code.)

• MIMEMixin and SafeMIMEMessage are not documented, not exposed in django.core.mail
  • Used to implement SafeMIMEText and SafeMIMEMultipart
  • Real-world use: virtually nonexistent - GitHub code search
  • Proposal: remove without deprecation
    
• utf8_charset, utf8_charset_qp, RFC5322_EMAIL_LINE_LENGTH_LIMIT are not documented, not exposed in django.core.mail
  • Used to implement MIMEMixin, SafeMIME*; not exposed in django.core.mail
  • Real-world use: virtually nonexistent - GitHub code search is mainly false positives, one use in third-party library
  • Proposal: remove without deprecation
    
• ADDRESS_HEADERS is not documented, not exposed in django.core.mail
  • Used to implement forbid_multi_line_headers()
  • Real-world use: virtually nonexistent - GitHub code search is mainly false positives, one use in third-party code
  • Proposal: remove without deprecation

[Real-world GitHub code search numbers are clearly not exact. I've tried to catch wrapped import statements and ignore Django forks and vendored copies, but it's not perfect. The search expressions find false positives (overcount) in comments and obsolete branches, as well as forks of popular libraries. But they miss (undercount) some creative approaches and any usage outside of a public repo hosted on GitHub.]

- Mike

[Mike Edmunds]

About MIMEBase attachments: I propose we continue supporting them in Django, without deprecation, for now. We can investigate adding support for the equivalent in Python's modern email API later, as a separate proposal.

More details…

On Wednesday, June 26, 2024 at 6:28:06 PM UTC-7 I wrote:

> Tricky bits and things requiring longer discussion…

> Legacy MIMEBase attachments: I'll post a separate message about this sometime later.

Since 2007, Django has supported and documented using MIMEBase objects in Django's EmailMessage.attachments list. 

Using MIMEBase is necessary for any "complex attachment" that can't be simply expressed as filename + content data + mimetype. This includes things like text attachments with a different charset than the main message, inline images (which require Content-Disposition: inline and Content-ID MIME headers), adding params to the Content-Type header, or in general anything where you need additional control over the attachment's MIME headers and content encoding.

MIMEBase is part of Python's legacy email API. The modern email replacement is add_attachment(), which offers everything we need for simple attachments. It also supports complex attachments via "contentmanager" kwargs like `disposition` for the Content-Disposition header, `cid` for Content-ID, and `params` for Content-Type extensions. But add_attachment() doesn't support MIMEBase.

So as part of this proposal, we'll handle simple filename+content+mimetype attachments through modern add_attachment(). I had originally planned to treat MIMEBase attachments as deprecated, and find a way to convert them to modern add_attachment() kwargs. And I was going to propose a way to somehow include add_attachment() kwargs directly in Django's EmailMessage.attachments, as the non-deprecated way to specify complex attachments. I still think this is the right long-term direction.

But in the interests of limiting scope for the current proposal, I think we can treat that as a separate project, and just continue to support MIMEBase attachments for now:

• From what I can tell, it's still possible to attach legacy MIMEBase objects to a modern Python EmailMessage, and get results that work at least as well as they did with entirely legacy APIs.
• You can also opt into the modern API when creating a MIMEBase object, by passing policy=email.policy.default to its constructor. This might avoid some obscure bugs that occur with the legacy APIs. I think Django's docs should suggest this approach "for improved compatibility" or something like that.
• When the time comes to support complex Django EmailMessage.attachments using Python's modern add_attachment() kwargs, I think we should also look into supporting Python EmailMessage's add_related(). Right now, there's no way to get Django to send a properly constructed multipart message with html, inline images, and attachments: the inline images end up in the wrong part. Most email clients aren't that picky, and display the inlines anyway (if they handle inline images at all), but it would be helpful to have a way to generate the correct message structure. Just… later.

- Mike

[Mike Edmunds]

Thanks to everyone for the feedback so far.

It looks like all the responses to date are generally positive, and I haven't seen any objections, so I'm going to open a ticket.

(Of course, additional feedback—positive or negative—and advice is very much appreciated.)

Cheers,

Mike

[Jörg Breitbart]

We recently had quite a struggle with programming a newsletter extension
for django-cms, mainly around the html-mails with CID embedded images.

I am not deep into the details, still want to express my support for the
idea to reshape the email abstraction in django, thus +1.

Cheers,
Jörg

[Florian Apolloner]

On Saturday, July 6, 2024 at 12:30:32 AM UTC+2 Mike Edmunds wrote:

Incidentally, I thought there was (used to be?) a policy that internal undocumented APIs were fair game for use by third-party libraries, subclassing, etc., so long as they didn't start with an underscore. (But "private" underscore APIs could have breaking changes at any time.) Am I remembering that wrong? Or was internal API stability only guaranteed for patch-level releases?

The API stability contract is documented here: https://docs.djangoproject.com/en/5.0/misc/api-stability/ -- Everything documented is considered part of the API, the rest is private.So yeah, it seems like you are remembering that wrong. Your suggested deprecations look okay. I have no strong feelings on `sanitize_address/forbid_multi_line_headers` -- if you want to deprecate instead of simply remove that is okay with me.

Cheers,

Florian

[Mike Edmunds]

Ticket created: https://code.djangoproject.com/ticket/35581

[Mike Edmunds]

On Saturday, July 6, 2024 at 11:17:35 AM UTC-7 Florian Apolloner wrote:

On Saturday, July 6, 2024 at 12:30:32 AM UTC+2 Mike Edmunds wrote:

Incidentally, I thought there was (used to be?) a policy that internal undocumented APIs were fair game for use by third-party libraries, subclassing, etc., so long as they didn't start with an underscore. (But "private" underscore APIs could have breaking changes at any time.) Am I remembering that wrong? Or was internal API stability only guaranteed for patch-level releases?

The API stability contract is documented here: https://docs.djangoproject.com/en/5.0/misc/api-stability/ -- Everything documented is considered part of the API, the rest is private.So yeah, it seems like you are remembering that wrong. 

Heh. The policy was changed in ticket 19728—prior to that, stable "public APIs" included "everything documented …, and all methods that don't begin with an underscore" [emphasis added]. So apparently I haven't looked at Django's api-stability docs in over a decade. (And had a really obscure detail get stuck in my memory. :-) )

- Mike

[Pankaj Kumar]

Hi Sir,
       I hope this message finds you well.

I am writing to inform you that we have reviewed and decided to accept the proposal to upgrade django.core.mail to Python's modern email API. We believe that this upgrade will bring significant improvements to our system's performance and maintainability.

I would also like to extend my sincere apologies for the delay in our response. I received the proposal seven days ago, and I regret not being able to get back to you sooner. Thank you for your patience and understanding in this matter.

We are looking forward to the successful implementation of this upgrade and are excited about the enhancements it will bring to our project. Please let us know the next steps and how we can assist in the process.

Thank you once again for your proposal and for your continued support.

Best regards,

- PANKAJ NEEMA

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/95e993a0-4b61-47d8-a365-d63676187f43n%40googlegroups.com.

[Othniel Davidson]

Thank you for the update 

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAPD84HnRTfZ-z1R5%3D0zNeLwdnbaECgRbCHOFV9TiFfK9bC0dQw%40mail.gmail.com.

[Adam Johnson]

I wish this discussion was on the forum so I could mark Pankaj’s response as spam 😅

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAPRrvgxwxop48-ZFZdKvuLxG0CYUbMD4uFCaSWC2P-q2_fxeiQ%40mail.gmail.com.

[Mike Edmunds]

> I wish this discussion was on the forum so I could mark Pankaj’s response as spam 😅

And here I was thinking it was another +1 (that also gently poked fun at my own overly-verbose tendencies). <g>

Speaking of the forum, I've taken the "combine EmailMessage and EmailMultiAlternatives" idea that was dropped from this proposal, and incorporated it into a separate proposal that's looking for feedback over there: https://forum.djangoproject.com/t/proposal-simplify-sending-testing-html-in-emailmessage/32844. (Apologies for the cross-post; I get the impression some of the people with opinions on django.core.mail are maybe here on django-developers but not the forum. There are also a couple of other interesting email-related threads active over there.)

- Mike

[Mike Edmunds]

PRs are now open:

1. Updated test cases (on existing legacy implementation): https://github.com/django/django/pull/18502
2. Change to modern email API: https://github.com/medmunds/django/pull/2

A few changes of note from the original plan, based on things discovered during implementation:

• I did end up deprecating MIMEBase attachments. Modern email has a MIMEPart class that can be used in a similar fashion.
  
• Modern email insists that text/* bodies and attachments should end with a newline, and will add one for you if missing. I think that's probably OK, but it affects a lot of our tests (which use strings like "Content" that don't include trailing newlines). More details in PR#2 comments, and suggestions are welcome if this seems like a problem.
• These additional, undocumented django.core.mail features will be retired without a deprecation period (but will raise errors on attempted use, and have been identified in the release notes):
  • EmailMessage.mixed_subtype overrides: modern email uses multipart/mixed
  • EmailMultiAlternatives.alternative_subtype overrides: modern email uses multipart/alternative
  • Using a legacy email.charset.Charset object for the undocumented EmailMessage.encoding: modern email doesn't support that. (But EmailMessage.encoding will continue to support string charset names as it has in the past, also still undocumented.)

Finally, although Python's modern email API is generally less buggy than the legacy API, it does have some issues of its own. (Including a security issue in address header generation that was publicly reported 5+ years ago, and that probably needs to be resolved before we'd want to merge the second PR.)

- Mike