[Django] #35514: Dictionary based EMAIL_PROVIDERS settings

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------------------+------------------------
Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new
Component: Uncategorized | Version: dev
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
As discussed in https://groups.google.com/g/django-
developers/c/R8ebGynQjK0/m/Tu-o4mGeAQAJ and during the sprints at Django
Con Europe 2024 (Carton Gibson, Natalia Bidart, Jacob Rief), we now have
consensus that we want to add this feature, even though this proposal has
been rejected in https://code.djangoproject.com/ticket/22734 10 years ago.

Reason for this change of opinion is that nowadays developers want to use
different email backends and that the number of configuration settings for
email providers has been steadily growing over the years.

So we want to replace all the settings starting with `EMAIL_...`and
replace them against a dictionary based approach such as:

{{{
EMAIL_PROVIDERS = {
"default": {
"BACKEND": "…",
"HOST": "…",
...
},
}
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35514>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings

-----------------------------+------------------------------------

Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new

Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+------------------------------------
Changes (by Adam Johnson):

* component: Uncategorized => Core (Mail)
* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:1>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+------------------------------------
Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+------------------------------------

Comment (by Adam Johnson):

Great to see this get a ticket. Good luck Jacob, if you are planning on
taking it up.

To bikeshed on the setting name, I think `EMAIL_PROVIDERS` is a little
unclear. One might configure several backends using the same actual
provider, such as for different domains. I think it would work to use
`EMAILS` (like `DATABASES`, `CACHES`, `STORAGES`). Yes, it’s a little
confusing that the setting configures email backends, not actual emails,
but the same could be said for the other settings.

When this ticket is done, it would also be possible to add a fixer to
django-upgrade to rewrite the settings, like the existing
[https://github.com/adamchainz/django-upgrade?tab=readme-ov-file#storages-
setting STORAGES fixer]. If you feel brave, please give it a try!
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:2>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings

-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned

Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+--------------------------------------
Changes (by Jacob Rief):

* owner: nobody => Jacob Rief
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:3>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

[[https://github.com/anymail/django-anymail django-anymail] maintainer
here]

I'm excited to see this getting traction. There are several common, real-
world use cases for it.

For the setting name, I'd suggest `EMAIL_BACKENDS`, since it configures
email backends, and a lot of existing docs and tutorials talk about
"backends." Also, `EMAIL_PROVIDERS` and `EMAILS` could imply configuration
for both sending ''and receiving'' email. (But I don't feel that
strongly.)

Handful of questions:

1) Are these settings provided as kwargs to the backend's constructor
(with the names lowercased)? E.g., with:

{{{#!python
EMAIL_PROVIDERS = {
"default": {
"BACKEND": "django.core.mail.backends.smtp.EmailBackend",
"HOST": "smtp-relay.gmail.com",
"USER": "a...@example.com",
"PASSWORD": env["GMAIL_APP_PASSWORD"],
},
"transactional": {
"BACKEND": "anymail.backends.mailgun.EmailBackend",
"API_KEY": env["MAILGUN_API_KEY"],
"SENDER_DOMAIN": "app.example.com",
},
}
}}}

am I assuming correctly that a request for the "transactional" provider
would end up creating:

{{{#!python
connection = anymail.backends.mailgun.EmailBackend(
api_key="...",
sender_domain="app.example.com",
# (and no host, user, or password args)
)
}}}

2) How does a caller request a particular provider? Is there a new
argument to `send_mail` and friends? (`name="..."`? `provider="..."`?)

3) It would be helpful if the logging `AdminEmailHandler` and
`mail_managers` could be easily configured to use a different provider
than django.contrib.auth. E.g., maybe admin/manager email uses
`EMAIL_PROVIDERS["admin"]` if present. (It's fairly common to want
internal SMTP for admin notifications, but a transactional email service
provider for password resets. And usually you want the transactional ESP
to be the default.)

In general, for third-party libraries that send email, do you envision
them adding a new setting to select an email provider
(`ALLAUTH_EMAIL_PROVIDER = "transactional"`)? Or maybe django-allauth
would want to try `provider="allauth"` first and fall back to
`provider="default"`? Or…?

4) Is `DEFAULT_FROM_EMAIL` affected by this at all? There's an argument
the default from_email will often need to vary by provider, but that might
require changes to all email backends. The simplest answer is no, it's a
global default across all providers. (Ditto `SERVER_EMAIL` for
admin/manager notifications.)

Again, I'm glad to see this proposal moving forward, and happy to test
with django-anymail when the time comes. (Anymail backends
[https://anymail.dev/en/stable/installation/#anymail-settings-
reference:~:text=override%20most%20settings%20on%20a%20per%2Dinstance%20basis%20by%20providing%20keyword%20args%20where%20the%20instance%20is%20initialized
already support constructor kwargs] for their settings, so if I'm
understanding the first item correctly, it should "just work.")
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:4>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Mike Edmunds):

* cc: Mike Edmunds (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:5>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Hrushikesh Vaidya):

* cc: Hrushikesh Vaidya (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:6>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

[Withdrawing my earlier comment about the settings name: I've warmed to
`EMAIL_PROVIDERS`. It's descriptive and reflects common technical jargon:
"Email Service Provider" is how most providers of email sending APIs
describe themselves. And it naturally leads to a terse, meaningful param
name: `provider="<key>"`—vs. something like `name=`, `backend_id=`,
`connection_name=`, `backend=` (already used for something else), or
`email=` (which would practically guarantee confusion).]
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:7>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------
Comment (by Mike Edmunds):

Related to my question (1) [#comment:4 above], is there going to be a
certain set of distinguished keys that can be used at the root of a
provider definition, like `HOST` and `USER`, while other parameters have
to be put inside `OPTIONS` (like with the current `DATABASES` setting)?
I'm, uh, "asking for a friend" who maintains a bunch of non-SMTP email
backends that tend to require params like `API_KEY` or `SERVER_TOKEN` but
couldn't care less about `USER` or `PASSWORD`.

I'm bringing this up because—just like `DATABASES`—I suspect we'll
eventually want to allow email provider configuration that ''isn't'' a
backend param. For example, a hypothetical `MESSAGE_DEFAULTS` feature
could address both my question (4) above about provider-specific
`DEFAULT_FROM_EMAIL`/`SERVER_EMAIL` as well as a [/ticket/35365#comment:12
suggestion in #36365] to allow setting header options:

{{{#!python
EMAIL_PROVIDERS = {
"admin": {
"BACKEND": "django.core.mail.backends.smtp.EmailBackend",
"OPTIONS": {
"host": "smtp-relay.gmail.com",
"user": "a...@corp.example.com",
"password": env["GMAIL_APP_PASSWORD"],
},
"MESSAGE_DEFAULTS": {
"from_email": "app-ope...@corp.example.com",
"reply_to": "i...@corp.example.com",
"headers": {"Auto-Submitted": "auto-generated"},
},
},
"default": {
"BACKEND": "anymail.backends.mailersend.EmailBackend",
"OPTIONS": {
"api_token": env["MAILERSEND_API_TOKEN"],
},
"MESSAGE_DEFAULTS": {
"from_email": "nor...@app.example.com",
# TODO: upgrade our MailerSend account to allow headers
# "headers": {"Auto-Submitted": "auto-generated"},
},
},
}
}}}

(Here I've just pushed ''all'' backend params into `OPTIONS`, to avoid
relegating non-SMTP backends to second class. Also, to be clear, I'm not
proposing `MESSAGE_DEFAULTS` become part of this work; just asking how we
leave room for configuring a feature like it.)
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:8>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Jacob Rief):

Created a draft pull request for this ticket:
https://github.com/django/django/pull/18421
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:9>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------
Comment (by Jacob Rief):

@Mike Edmunds

addressing your questions from June 10th, 2024

ad 1) yes, the provider name usually will be lowercased, jsut as with
databases, caches, etc.

ad 2) I would prefer `send_name(provider="…"`, …)`.

ad 3) good point, should maybe be implemented in a separate issue.

ad 4) currently I did neither address `DEFAULT_FROM_EMAIL` nor
`EMAIL_SUBJECT_PREFIX`.

addressing your questions from June 25th, 2024

first) On the sprints at DjangoCon Europe we (Natalia, Carlton, Jacob)
brainstormed for meaningful names. The conclusion was that
`EMAIL_PROVIDERS` was the best choice.

second) hmm, didn't think about this yet. Currently it would behave just
like using the setting `EMAIL_…` but moved into a dictionary based
setting.
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:10>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------
Comment (by Jacob Rief):

@Mike Edmunds

in your second post on 2023-06-25 you made a good point about using a sub-
dict `OPTIONS` to maintain non-SMTP email backends. I now changed the code
to support this.

We probably should consider use-cases, where users want to be informed
through messaging apps rather than email. Signal, WhatApp, Telegram, etc.
all offer an API for this purpose and in Django we should allow easy
integration for them. Therefore I fully back your proposal.
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:11>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

@Jacob Rief Appreciate the responses.

You raise a really interesting idea about non-email messaging apps—I
actually hadn't considered that case. (It probably merits its own
discussion at some point.)

I know of three cases where ''existing'' third-party libraries deal with
Django email and are likely to be impacted by EMAIL_PROVIDERS. I'd hope we
can consider all of these in the design:

A. Email backends that call ESPs' HTTP APIs to send email. (This is what I
meant by "non-SMTP email backends.")
* Examples: [https://pypi.org/project/django-ses/ django-ses],
[https://postmarker.readthedocs.io/en/stable/django.html postmarker],
[https://anymail.dev django-anymail] (disclosure: I maintain django-
anymail)
* These backends typically have their own settings (such as
`POSTMARK_API_KEY`) that are different from Django's existing `EMAIL_*`
settings
* I think your latest PR update to support `OPTIONS` effectively handles
their needs. (And in many cases, these libraries won't even need to be
updated. That's great!)

B. "Wrapper" email backends which add functionality (such as an
asynchronous send queue) and then connect to another email backend for the
actual sending
* Examples: [https://pypi.org/project/django-mailer/ django-mailer],
[https://pypi.org/project/django-celery-email/ django-celery-email],
[https://pypi.org/project/django-post-office/ django-post-office]
* These packages typically have their own setting to specify the
"wrapped" backend and then call Django's
`get_connection(backend=setting_value)`. E.g.: with django-mailer you set
`EMAIL_BACKEND = "mailer.backend.DbBackend"` and then
`MAILER_EMAIL_BACKEND = "django.core.mail.backends.stmp.EmailBackend"`.
* These libraries should continue to work for now with deprecated
`EMAIL_*` settings, but will start to break as the deprecated settings are
removed. We should probably deprecate the `backend` param to
get_connection() now, and add a new `get_connection(provider=...)` option.
(Or deprecate get_connection() and add a new get_provider() to replace
it.)
* Wrapper email backends will need updates to properly use
EMAIL_PROVIDERS. For example, django-mailer could replace its
`MAILER_EMAIL_BACKEND` setting with a new `provider` option:
{{{#!python
EMAIL_PROVIDERS = {
"default:" {
"BACKEND": "mailer.backend.DbBackend",
"OPTIONS": {
# django-mailer needs a NEW option for its wrapped
provider:
"provider": "smtp",
},
},
"smtp": {
"BACKEND": "django.core.mail.backends.stmp.EmailBackend",
"OPTIONS": { ... },
}
}
}}}
* Or, could we think of a way django-mailer could generically wrap
''all'' defined EMAIL_PROVIDERS (except itself), without the user needing
to specify a wrapped version of each?

C. Third-party libraries that send email
* Examples: [https://pypi.org/project/django-newsletter/ django-
newsletter]; email confirmation in [https://pypi.org/project/django-
allauth/ django-allauth] and [https://pypi.org/project/django-user-
accounts/ django-user-accounts]; password reset email in
django.contrib.auth
* As I mentioned earlier, a common use case is wanting a transactional
ESP for password resets, internal SMTP for admin messages, and a bulk ESP
for newsletters
* How would we expect these libraries to allow specifying EMAIL_PROVIDER
aliases for the different types of mail they send?
* (django.utils.log.AdminEmailHandler is another case of this, and
already supports an
[https://docs.djangoproject.com/en/5.0/ref/logging/#handlers:~:text=By%20setting%20the%20email_backend%20argument%20of%20AdminEmailHandler
email_backend option]. I'd suggest it should also have an `email_provider`
option, and as you say that could be a separate ticket.)

For B & C, I'm just suggesting that we think through what we might expect
those packages to do, and then make sure the EMAIL_PROVIDERS feature will
be compatible with those expectations. (Does that make sense?)


[btw, I agree the names `EMAIL_PROVIDERS` and `provider="<alias>"` are the
best choice. My earlier comments on naming were in response to [#comment:1
comment:1] from Adam Johnson, and I came to the same conclusion as you and
Natalia and Carlton.]
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:12>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Jacob Rief):

Now, that I touched Django's mail sending code, I noticed that the current
implementation has a serious flaw: `get_connection(backend=…)` returns a
configured email backend. This means that whenever one calls this
function, the connection parameters must be provided together. The
consequence is that typical configuration settings, such as hostname,
user, password, etc. must be provided from inside the calling code, rather
that from a configuration file, aka `settings.py`. I therefore left the
`backend` parameter in `get_connection` but made it mutually exclusive
with the new parameter `provider`. In my opinion we should add a
deprecation warning whenever someone uses `backend`. What do you think?

Anyway, the current implementation from my pull request seems to work now.
I still have to add/change the documentation. Does anybody want to review
this pull request?
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:13>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

> In my opinion we should add a deprecation warning whenever someone uses

backend. What do you think?

Agreed. (It's also going to cause problems for wrapper email backends
after the deprecated `EMAIL_*` settings are removed—see item B in
[#comment:12 comment:12].)
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:14>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Jacob Rief):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:15>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Sarah Boyce):

* needs_better_patch: 0 => 1

Comment:

Marking as "Patch needs improvement" following the review from Mike and
that some merge conflicts need resolving
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:16>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

(I'll try to create an updated patch once some of the other changes to
mail have been merged.)
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:17>