[Django] #35514: Dictionary based EMAIL_PROVIDERS settings

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------------------+------------------------
Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new
Component: Uncategorized | Version: dev
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
As discussed in https://groups.google.com/g/django-
developers/c/R8ebGynQjK0/m/Tu-o4mGeAQAJ and during the sprints at Django
Con Europe 2024 (Carton Gibson, Natalia Bidart, Jacob Rief), we now have
consensus that we want to add this feature, even though this proposal has
been rejected in https://code.djangoproject.com/ticket/22734 10 years ago.

Reason for this change of opinion is that nowadays developers want to use
different email backends and that the number of configuration settings for
email providers has been steadily growing over the years.

So we want to replace all the settings starting with `EMAIL_...`and
replace them against a dictionary based approach such as:

{{{
EMAIL_PROVIDERS = {
"default": {
"BACKEND": "…",
"HOST": "…",
...
},
}
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35514>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings

-----------------------------+------------------------------------

Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new

Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+------------------------------------
Changes (by Adam Johnson):

* component: Uncategorized => Core (Mail)
* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:1>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+------------------------------------
Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+------------------------------------

Comment (by Adam Johnson):

Great to see this get a ticket. Good luck Jacob, if you are planning on
taking it up.

To bikeshed on the setting name, I think `EMAIL_PROVIDERS` is a little
unclear. One might configure several backends using the same actual
provider, such as for different domains. I think it would work to use
`EMAILS` (like `DATABASES`, `CACHES`, `STORAGES`). Yes, it’s a little
confusing that the setting configures email backends, not actual emails,
but the same could be said for the other settings.

When this ticket is done, it would also be possible to add a fixer to
django-upgrade to rewrite the settings, like the existing
[https://github.com/adamchainz/django-upgrade?tab=readme-ov-file#storages-
setting STORAGES fixer]. If you feel brave, please give it a try!
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:2>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings

-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned

Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+--------------------------------------
Changes (by Jacob Rief):

* owner: nobody => Jacob Rief
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:3>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

[[https://github.com/anymail/django-anymail django-anymail] maintainer
here]

I'm excited to see this getting traction. There are several common, real-
world use cases for it.

For the setting name, I'd suggest `EMAIL_BACKENDS`, since it configures
email backends, and a lot of existing docs and tutorials talk about
"backends." Also, `EMAIL_PROVIDERS` and `EMAILS` could imply configuration
for both sending ''and receiving'' email. (But I don't feel that
strongly.)

Handful of questions:

1) Are these settings provided as kwargs to the backend's constructor
(with the names lowercased)? E.g., with:

{{{#!python
EMAIL_PROVIDERS = {
"default": {
"BACKEND": "django.core.mail.backends.smtp.EmailBackend",
"HOST": "smtp-relay.gmail.com",
"USER": "a...@example.com",
"PASSWORD": env["GMAIL_APP_PASSWORD"],
},
"transactional": {
"BACKEND": "anymail.backends.mailgun.EmailBackend",
"API_KEY": env["MAILGUN_API_KEY"],
"SENDER_DOMAIN": "app.example.com",
},
}
}}}

am I assuming correctly that a request for the "transactional" provider
would end up creating:

{{{#!python
connection = anymail.backends.mailgun.EmailBackend(
api_key="...",
sender_domain="app.example.com",
# (and no host, user, or password args)
)
}}}

2) How does a caller request a particular provider? Is there a new
argument to `send_mail` and friends? (`name="..."`? `provider="..."`?)

3) It would be helpful if the logging `AdminEmailHandler` and
`mail_managers` could be easily configured to use a different provider
than django.contrib.auth. E.g., maybe admin/manager email uses
`EMAIL_PROVIDERS["admin"]` if present. (It's fairly common to want
internal SMTP for admin notifications, but a transactional email service
provider for password resets. And usually you want the transactional ESP
to be the default.)

In general, for third-party libraries that send email, do you envision
them adding a new setting to select an email provider
(`ALLAUTH_EMAIL_PROVIDER = "transactional"`)? Or maybe django-allauth
would want to try `provider="allauth"` first and fall back to
`provider="default"`? Or…?

4) Is `DEFAULT_FROM_EMAIL` affected by this at all? There's an argument
the default from_email will often need to vary by provider, but that might
require changes to all email backends. The simplest answer is no, it's a
global default across all providers. (Ditto `SERVER_EMAIL` for
admin/manager notifications.)

Again, I'm glad to see this proposal moving forward, and happy to test
with django-anymail when the time comes. (Anymail backends
[https://anymail.dev/en/stable/installation/#anymail-settings-
reference:~:text=override%20most%20settings%20on%20a%20per%2Dinstance%20basis%20by%20providing%20keyword%20args%20where%20the%20instance%20is%20initialized
already support constructor kwargs] for their settings, so if I'm
understanding the first item correctly, it should "just work.")
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:4>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Mike Edmunds):

* cc: Mike Edmunds (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:5>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Hrushikesh Vaidya):

* cc: Hrushikesh Vaidya (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:6>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

[Withdrawing my earlier comment about the settings name: I've warmed to
`EMAIL_PROVIDERS`. It's descriptive and reflects common technical jargon:
"Email Service Provider" is how most providers of email sending APIs
describe themselves. And it naturally leads to a terse, meaningful param
name: `provider="<key>"`—vs. something like `name=`, `backend_id=`,
`connection_name=`, `backend=` (already used for something else), or
`email=` (which would practically guarantee confusion).]
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:7>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------
Comment (by Mike Edmunds):

Related to my question (1) [#comment:4 above], is there going to be a
certain set of distinguished keys that can be used at the root of a
provider definition, like `HOST` and `USER`, while other parameters have
to be put inside `OPTIONS` (like with the current `DATABASES` setting)?
I'm, uh, "asking for a friend" who maintains a bunch of non-SMTP email
backends that tend to require params like `API_KEY` or `SERVER_TOKEN` but
couldn't care less about `USER` or `PASSWORD`.

I'm bringing this up because—just like `DATABASES`—I suspect we'll
eventually want to allow email provider configuration that ''isn't'' a
backend param. For example, a hypothetical `MESSAGE_DEFAULTS` feature
could address both my question (4) above about provider-specific
`DEFAULT_FROM_EMAIL`/`SERVER_EMAIL` as well as a [/ticket/35365#comment:12
suggestion in #36365] to allow setting header options:

{{{#!python
EMAIL_PROVIDERS = {
"admin": {
"BACKEND": "django.core.mail.backends.smtp.EmailBackend",
"OPTIONS": {
"host": "smtp-relay.gmail.com",
"user": "a...@corp.example.com",
"password": env["GMAIL_APP_PASSWORD"],
},
"MESSAGE_DEFAULTS": {
"from_email": "app-ope...@corp.example.com",
"reply_to": "i...@corp.example.com",
"headers": {"Auto-Submitted": "auto-generated"},
},
},
"default": {
"BACKEND": "anymail.backends.mailersend.EmailBackend",
"OPTIONS": {
"api_token": env["MAILERSEND_API_TOKEN"],
},
"MESSAGE_DEFAULTS": {
"from_email": "nor...@app.example.com",
# TODO: upgrade our MailerSend account to allow headers
# "headers": {"Auto-Submitted": "auto-generated"},
},
},
}
}}}

(Here I've just pushed ''all'' backend params into `OPTIONS`, to avoid
relegating non-SMTP backends to second class. Also, to be clear, I'm not
proposing `MESSAGE_DEFAULTS` become part of this work; just asking how we
leave room for configuring a feature like it.)
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:8>