[Django] #35514: Dictionary based EMAIL_PROVIDERS settings

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------------------+------------------------
Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new
Component: Uncategorized | Version: dev
Severity: Normal | Keywords:
Triage Stage: Unreviewed | Has patch: 0
Needs documentation: 0 | Needs tests: 0
Patch needs improvement: 0 | Easy pickings: 0
UI/UX: 0 |
-----------------------------------------+------------------------
As discussed in https://groups.google.com/g/django-
developers/c/R8ebGynQjK0/m/Tu-o4mGeAQAJ and during the sprints at Django
Con Europe 2024 (Carton Gibson, Natalia Bidart, Jacob Rief), we now have
consensus that we want to add this feature, even though this proposal has
been rejected in https://code.djangoproject.com/ticket/22734 10 years ago.

Reason for this change of opinion is that nowadays developers want to use
different email backends and that the number of configuration settings for
email providers has been steadily growing over the years.

So we want to replace all the settings starting with `EMAIL_...`and
replace them against a dictionary based approach such as:

{{{
EMAIL_PROVIDERS = {
"default": {
"BACKEND": "…",
"HOST": "…",
...
},
}
}}}
--
Ticket URL: <https://code.djangoproject.com/ticket/35514>
Django <https://code.djangoproject.com/>
The Web framework for perfectionists with deadlines.

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings

-----------------------------+------------------------------------

Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new

Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+------------------------------------
Changes (by Adam Johnson):

* component: Uncategorized => Core (Mail)
* stage: Unreviewed => Accepted

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:1>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+------------------------------------
Reporter: Jacob Rief | Owner: nobody
Type: New feature | Status: new
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+------------------------------------

Comment (by Adam Johnson):

Great to see this get a ticket. Good luck Jacob, if you are planning on
taking it up.

To bikeshed on the setting name, I think `EMAIL_PROVIDERS` is a little
unclear. One might configure several backends using the same actual
provider, such as for different domains. I think it would work to use
`EMAILS` (like `DATABASES`, `CACHES`, `STORAGES`). Yes, it’s a little
confusing that the setting configures email backends, not actual emails,
but the same could be said for the other settings.

When this ticket is done, it would also be possible to add a fixer to
django-upgrade to rewrite the settings, like the existing
[https://github.com/adamchainz/django-upgrade?tab=readme-ov-file#storages-
setting STORAGES fixer]. If you feel brave, please give it a try!
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:2>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings

-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned

Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0

-----------------------------+--------------------------------------
Changes (by Jacob Rief):

* owner: nobody => Jacob Rief
* status: new => assigned

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:3>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

[[https://github.com/anymail/django-anymail django-anymail] maintainer
here]

I'm excited to see this getting traction. There are several common, real-
world use cases for it.

For the setting name, I'd suggest `EMAIL_BACKENDS`, since it configures
email backends, and a lot of existing docs and tutorials talk about
"backends." Also, `EMAIL_PROVIDERS` and `EMAILS` could imply configuration
for both sending ''and receiving'' email. (But I don't feel that
strongly.)

Handful of questions:

1) Are these settings provided as kwargs to the backend's constructor
(with the names lowercased)? E.g., with:

{{{#!python
EMAIL_PROVIDERS = {
"default": {
"BACKEND": "django.core.mail.backends.smtp.EmailBackend",
"HOST": "smtp-relay.gmail.com",
"USER": "a...@example.com",
"PASSWORD": env["GMAIL_APP_PASSWORD"],
},
"transactional": {
"BACKEND": "anymail.backends.mailgun.EmailBackend",
"API_KEY": env["MAILGUN_API_KEY"],
"SENDER_DOMAIN": "app.example.com",
},
}
}}}

am I assuming correctly that a request for the "transactional" provider
would end up creating:

{{{#!python
connection = anymail.backends.mailgun.EmailBackend(
api_key="...",
sender_domain="app.example.com",
# (and no host, user, or password args)
)
}}}

2) How does a caller request a particular provider? Is there a new
argument to `send_mail` and friends? (`name="..."`? `provider="..."`?)

3) It would be helpful if the logging `AdminEmailHandler` and
`mail_managers` could be easily configured to use a different provider
than django.contrib.auth. E.g., maybe admin/manager email uses
`EMAIL_PROVIDERS["admin"]` if present. (It's fairly common to want
internal SMTP for admin notifications, but a transactional email service
provider for password resets. And usually you want the transactional ESP
to be the default.)

In general, for third-party libraries that send email, do you envision
them adding a new setting to select an email provider
(`ALLAUTH_EMAIL_PROVIDER = "transactional"`)? Or maybe django-allauth
would want to try `provider="allauth"` first and fall back to
`provider="default"`? Or…?

4) Is `DEFAULT_FROM_EMAIL` affected by this at all? There's an argument
the default from_email will often need to vary by provider, but that might
require changes to all email backends. The simplest answer is no, it's a
global default across all providers. (Ditto `SERVER_EMAIL` for
admin/manager notifications.)

Again, I'm glad to see this proposal moving forward, and happy to test
with django-anymail when the time comes. (Anymail backends
[https://anymail.dev/en/stable/installation/#anymail-settings-
reference:~:text=override%20most%20settings%20on%20a%20per%2Dinstance%20basis%20by%20providing%20keyword%20args%20where%20the%20instance%20is%20initialized
already support constructor kwargs] for their settings, so if I'm
understanding the first item correctly, it should "just work.")
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:4>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Mike Edmunds):

* cc: Mike Edmunds (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:5>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Hrushikesh Vaidya):

* cc: Hrushikesh Vaidya (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:6>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

[Withdrawing my earlier comment about the settings name: I've warmed to
`EMAIL_PROVIDERS`. It's descriptive and reflects common technical jargon:
"Email Service Provider" is how most providers of email sending APIs
describe themselves. And it naturally leads to a terse, meaningful param
name: `provider="<key>"`—vs. something like `name=`, `backend_id=`,
`connection_name=`, `backend=` (already used for something else), or
`email=` (which would practically guarantee confusion).]
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:7>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------
Comment (by Mike Edmunds):

Related to my question (1) [#comment:4 above], is there going to be a
certain set of distinguished keys that can be used at the root of a
provider definition, like `HOST` and `USER`, while other parameters have
to be put inside `OPTIONS` (like with the current `DATABASES` setting)?
I'm, uh, "asking for a friend" who maintains a bunch of non-SMTP email
backends that tend to require params like `API_KEY` or `SERVER_TOKEN` but
couldn't care less about `USER` or `PASSWORD`.

I'm bringing this up because—just like `DATABASES`—I suspect we'll
eventually want to allow email provider configuration that ''isn't'' a
backend param. For example, a hypothetical `MESSAGE_DEFAULTS` feature
could address both my question (4) above about provider-specific
`DEFAULT_FROM_EMAIL`/`SERVER_EMAIL` as well as a [/ticket/35365#comment:12
suggestion in #36365] to allow setting header options:

{{{#!python
EMAIL_PROVIDERS = {
"admin": {
"BACKEND": "django.core.mail.backends.smtp.EmailBackend",
"OPTIONS": {
"host": "smtp-relay.gmail.com",
"user": "a...@corp.example.com",
"password": env["GMAIL_APP_PASSWORD"],
},
"MESSAGE_DEFAULTS": {
"from_email": "app-ope...@corp.example.com",
"reply_to": "i...@corp.example.com",
"headers": {"Auto-Submitted": "auto-generated"},
},
},
"default": {
"BACKEND": "anymail.backends.mailersend.EmailBackend",
"OPTIONS": {
"api_token": env["MAILERSEND_API_TOKEN"],
},
"MESSAGE_DEFAULTS": {
"from_email": "nor...@app.example.com",
# TODO: upgrade our MailerSend account to allow headers
# "headers": {"Auto-Submitted": "auto-generated"},
},
},
}
}}}

(Here I've just pushed ''all'' backend params into `OPTIONS`, to avoid
relegating non-SMTP backends to second class. Also, to be clear, I'm not
proposing `MESSAGE_DEFAULTS` become part of this work; just asking how we
leave room for configuring a feature like it.)
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:8>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Jacob Rief):

Created a draft pull request for this ticket:
https://github.com/django/django/pull/18421
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:9>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------
Comment (by Jacob Rief):

@Mike Edmunds

addressing your questions from June 10th, 2024

ad 1) yes, the provider name usually will be lowercased, jsut as with
databases, caches, etc.

ad 2) I would prefer `send_name(provider="…"`, …)`.

ad 3) good point, should maybe be implemented in a separate issue.

ad 4) currently I did neither address `DEFAULT_FROM_EMAIL` nor
`EMAIL_SUBJECT_PREFIX`.

addressing your questions from June 25th, 2024

first) On the sprints at DjangoCon Europe we (Natalia, Carlton, Jacob)
brainstormed for meaningful names. The conclusion was that
`EMAIL_PROVIDERS` was the best choice.

second) hmm, didn't think about this yet. Currently it would behave just
like using the setting `EMAIL_…` but moved into a dictionary based
setting.
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:10>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------
Comment (by Jacob Rief):

@Mike Edmunds

in your second post on 2023-06-25 you made a good point about using a sub-
dict `OPTIONS` to maintain non-SMTP email backends. I now changed the code
to support this.

We probably should consider use-cases, where users want to be informed
through messaging apps rather than email. Signal, WhatApp, Telegram, etc.
all offer an API for this purpose and in Django we should allow easy
integration for them. Therefore I fully back your proposal.
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:11>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

@Jacob Rief Appreciate the responses.

You raise a really interesting idea about non-email messaging apps—I
actually hadn't considered that case. (It probably merits its own
discussion at some point.)

I know of three cases where ''existing'' third-party libraries deal with
Django email and are likely to be impacted by EMAIL_PROVIDERS. I'd hope we
can consider all of these in the design:

A. Email backends that call ESPs' HTTP APIs to send email. (This is what I
meant by "non-SMTP email backends.")
* Examples: [https://pypi.org/project/django-ses/ django-ses],
[https://postmarker.readthedocs.io/en/stable/django.html postmarker],
[https://anymail.dev django-anymail] (disclosure: I maintain django-
anymail)
* These backends typically have their own settings (such as
`POSTMARK_API_KEY`) that are different from Django's existing `EMAIL_*`
settings
* I think your latest PR update to support `OPTIONS` effectively handles
their needs. (And in many cases, these libraries won't even need to be
updated. That's great!)

B. "Wrapper" email backends which add functionality (such as an
asynchronous send queue) and then connect to another email backend for the
actual sending
* Examples: [https://pypi.org/project/django-mailer/ django-mailer],
[https://pypi.org/project/django-celery-email/ django-celery-email],
[https://pypi.org/project/django-post-office/ django-post-office]
* These packages typically have their own setting to specify the
"wrapped" backend and then call Django's
`get_connection(backend=setting_value)`. E.g.: with django-mailer you set
`EMAIL_BACKEND = "mailer.backend.DbBackend"` and then
`MAILER_EMAIL_BACKEND = "django.core.mail.backends.stmp.EmailBackend"`.
* These libraries should continue to work for now with deprecated
`EMAIL_*` settings, but will start to break as the deprecated settings are
removed. We should probably deprecate the `backend` param to
get_connection() now, and add a new `get_connection(provider=...)` option.
(Or deprecate get_connection() and add a new get_provider() to replace
it.)
* Wrapper email backends will need updates to properly use
EMAIL_PROVIDERS. For example, django-mailer could replace its
`MAILER_EMAIL_BACKEND` setting with a new `provider` option:
{{{#!python
EMAIL_PROVIDERS = {
"default:" {
"BACKEND": "mailer.backend.DbBackend",
"OPTIONS": {
# django-mailer needs a NEW option for its wrapped
provider:
"provider": "smtp",
},
},
"smtp": {
"BACKEND": "django.core.mail.backends.stmp.EmailBackend",
"OPTIONS": { ... },
}
}
}}}
* Or, could we think of a way django-mailer could generically wrap
''all'' defined EMAIL_PROVIDERS (except itself), without the user needing
to specify a wrapped version of each?

C. Third-party libraries that send email
* Examples: [https://pypi.org/project/django-newsletter/ django-
newsletter]; email confirmation in [https://pypi.org/project/django-
allauth/ django-allauth] and [https://pypi.org/project/django-user-
accounts/ django-user-accounts]; password reset email in
django.contrib.auth
* As I mentioned earlier, a common use case is wanting a transactional
ESP for password resets, internal SMTP for admin messages, and a bulk ESP
for newsletters
* How would we expect these libraries to allow specifying EMAIL_PROVIDER
aliases for the different types of mail they send?
* (django.utils.log.AdminEmailHandler is another case of this, and
already supports an
[https://docs.djangoproject.com/en/5.0/ref/logging/#handlers:~:text=By%20setting%20the%20email_backend%20argument%20of%20AdminEmailHandler
email_backend option]. I'd suggest it should also have an `email_provider`
option, and as you say that could be a separate ticket.)

For B & C, I'm just suggesting that we think through what we might expect
those packages to do, and then make sure the EMAIL_PROVIDERS feature will
be compatible with those expectations. (Does that make sense?)


[btw, I agree the names `EMAIL_PROVIDERS` and `provider="<alias>"` are the
best choice. My earlier comments on naming were in response to [#comment:1
comment:1] from Adam Johnson, and I came to the same conclusion as you and
Natalia and Carlton.]
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:12>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Jacob Rief):

Now, that I touched Django's mail sending code, I noticed that the current
implementation has a serious flaw: `get_connection(backend=…)` returns a
configured email backend. This means that whenever one calls this
function, the connection parameters must be provided together. The
consequence is that typical configuration settings, such as hostname,
user, password, etc. must be provided from inside the calling code, rather
that from a configuration file, aka `settings.py`. I therefore left the
`backend` parameter in `get_connection` but made it mutually exclusive
with the new parameter `provider`. In my opinion we should add a
deprecation warning whenever someone uses `backend`. What do you think?

Anyway, the current implementation from my pull request seems to work now.
I still have to add/change the documentation. Does anybody want to review
this pull request?
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:13>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 0 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

> In my opinion we should add a deprecation warning whenever someone uses

backend. What do you think?

Agreed. (It's also going to cause problems for wrapper email backends
after the deprecated `EMAIL_*` settings are removed—see item B in
[#comment:12 comment:12].)
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:14>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted

Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 0
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Jacob Rief):

* has_patch: 0 => 1

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:15>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0

Needs tests: 0 | Patch needs improvement: 1

Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Sarah Boyce):

* needs_better_patch: 0 => 1

Comment:

Marking as "Patch needs improvement" following the review from Mike and
that some merge conflicts need resolving
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:16>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

(I'll try to create an updated patch once some of the other changes to
mail have been merged.)
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:17>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Johanan Oppong Amoateng):

What is the updates on this? I would want to tag myself and work on it but
i want to know the status whether Jacob Rief is still working on it
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:18>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Mike Edmunds):

Replying to [comment:18 Johanan Oppong Amoateng]:

> What is the updates on this? I would want to tag myself and work on it
but i want to know the status whether Jacob Rief is still working on it

I believe Jacob Rief's PR is actually quite close—with the caveat that
(for a feature of this scope) "quite close" will likely still involve ''a
lot'' of work to get over the finish line. There are (I think) a couple of
mechanical issues and at least a couple of design decisions needed to move
this forward.

The mechanical issues:
- The PR needs to be rebased and conflicts resolved with the current code
on main.
- The PR has exceeded the number of commits and comments where GitHub's PR
review is usable. It would help to squash and open a new PR.

The design decisions:
A. To what extent do we want to support mixing new EMAIL_PROVIDERS
settings with deprecated EMAIL_BACKEND and EMAIL_HOST/USER/etc. settings?
[[BR]][[BR]]My suggestion would be it's ''all or nothing:'' once a
settings file contains EMAIL_PROVIDERS, the user has opted into the new
approach and any attempt to set ''or access'' the old EMAIL_BACKEND,
EMAIL_HOST, etc. becomes a hard error (not just a deprecation warning).
One implication is users would need to wait to switch to EMAIL_PROVIDERS
until ''all'' email backends they use have been updated to be providers-
aware, which would impact anyone using a "wrapper" email backend like
django-celery-email or django-mailer.
[[BR]][[BR]](More detail in
[https://github.com/django/django/pull/18421#pullrequestreview-2624473570
this PR comment] and the ones following, plus several related suggested
edits.)

B. Can we require that (Django's built-in) EmailBackend instances ''must''
be obtained by calling mail.get_connection(), or do we need to continue
allowing code to construct them directly?
[[BR]][[BR]]If we can require get_connection(), it simplifies the
backwards compatibility code in each EmailBackend somewhat. But it
requires updating nearly all of Django's existing EmailBackend tests,
which tend to call the constructor directly. (The current PR includes most
of those test updates; we'd likely want to pull them into a separate
"refs" commit first to prove they run correctly before this change.)
[[BR]][[BR]]The relevant docs start at
[https://docs.djangoproject.com/en/6.0/topics/email/#obtaining-an-
instance-of-an-email-backend Obtaining an instance of an email backend]:
"The `get_connection()` function in `django.core.mail` returns an instance
of the email backend that you can use." But they go on to document
available options for backends.smtp.EmailBackend() using the class
constructor format.
[[BR]][[BR]]I don't know whether that means we have documented
smtp.EmailBackend as a class, so the constructor must remain callable for
backwards compatibility. Or if it was just a convenient docutils syntax
for describing SMTP EmailBackend options that can be used as keywords to
get_connection(). (I can't imagine why real-world code would want to call
an EmailBackend constructor directly rather than going through
get_connection().)

C. I think there was also some question about maybe deprecating the
`backend` param to get_connection(). Or how and where to support new
provider alias params alongside existing `connection` params in other
django.core.mail APIs. (But I'm blanking on the details right now.)

(I'm hoping to have some time to look at this again more closely next
year, probably late January at the earliest.)
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:19>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Comment (by Jacob Rief):

Sorry for the huge delay. I just rebased the code base to the main branch,
which is the current base for Django-6.1. All unit tests are working
again.

Next, I have to address some issues @medmunds mentioned in GitHub's pull
request. I hope to find some time during Christmas holidays.

As for the design decisions:

A:
Yes, having an all or nothing approach is the only viable option, imo.

B & C:
I have to reread the code to get some answers.

AFAIK, you can't squash commits after having merged another branch into
your current branch. So creating a fresh PR probably is the only option.
However, before doing so I'd like being sure that the pull requst will be
merged. Otherwise I have to repeat this again and again.
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:20>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Adam Johnson):

* cc: Adam Johnson (added)

--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:21>

[Django]

#35514: Dictionary based EMAIL_PROVIDERS settings
-----------------------------+--------------------------------------
Reporter: Jacob Rief | Owner: Jacob Rief
Type: New feature | Status: assigned
Component: Core (Mail) | Version: dev
Severity: Normal | Resolution:
Keywords: | Triage Stage: Accepted
Has patch: 1 | Needs documentation: 0
Needs tests: 0 | Patch needs improvement: 1
Easy pickings: 0 | UI/UX: 0
-----------------------------+--------------------------------------

Changes (by Natalia Bidart):

* cc: Jacob Rief (added)

Comment:

Hi Jacob R and Mike,

Thank you both for the work and discussion on this feature. As Django
Fellows, Jacob Walls and I want to encourage you to move this forward,
since we believe this work it's valuable and we'll do our best to help get
it merged for the 6.1 feature freeze (set to May 20th).

** About the implementation approach **

I'd like to propose aligning `EMAIL_PROVIDERS` with Django's existing
patterns for `DATABASES`, `CACHES`, and `STORAGES`. These all share a
consistent API:

{{{#!python
# Dict-like handler with named aliases for providers
from django.db import connections
from django.core.cache import caches
from django.core.files.storage import storages
from django.core.mail import providers # proposed

# Access by alias
db_conn = connections['analytics']
cache = caches['session']
storage = storages['s3']
provider = providers['marketing'] # proposed

# Lazy proxy for default provider
from django.db import connection
from django.core.cache import cache
from django.core.files.storage import default_storage
from django.core.mail import provider # proposed
}}}

This pattern would replace `get_connection(backend=...)` with
`providers["alias"]`, by having:
- Named provider configuration in `EMAIL_PROVIDERS`
- Consistent API across Django's connection systems
- Natural (?) solution for wrapper backends: configuration stays
declarative storing provider "alias" names in `OPTIONS`, then resolve them
via `providers["alias"]` at runtime, matching the pattern used with
multiple databases/caches/storages.

I think the wrapper backend issue (django-mailer, etc.) resolves cleanly
with this pattern (as far as I understand, let me know if I got that
wrong!). We can deprecate `get_connection(backend=...)` in favor of
`providers["alias"]` (hand-to-hand with `EMAIL_*` settings deprecation),
while MAYBE maintaining `get_connection()` with no arguments for backwards
compatibility. This may need further thinking.

For the ESP integration, and per Mike's feedback, I read that the
`OPTIONS` approach solved the `API_KEY` issue for `django-anymail` and
similar backends well. This appears resolved?

** PR and commit structure, next steps **

For the PR, we strongly recommend structuring commits as self-contained
units of functionality, each complete with relevant tests and
documentation. This allows:
- Reviewing each piece in isolation
- Merging incrementally when possible
- Easier identification of issues during review

For a feature this size, incremental review makes an enormous difference.
Think of commits as logical building blocks rather than chronological
changes.

So, concretely, next steps would be opening a fresh branch against current
`main` with a clean, well-structured commit history. That would help move
this forward efficiently. Jacob R, if you can prepare this, we'll
definitely try to prioritize review. Mike, your continued feedback on the
integration side would be highly appreciated. Please let us know if you
have questions about this approach!
--
Ticket URL: <https://code.djangoproject.com/ticket/35514#comment:22>