Proposal on how add configuration options to Email Backends

[Jacob Rief]

This proposal attempts to address the following issues:

https://code.djangoproject.com/ticket/6989  (14 years old)

https://github.com/django/django/pull/13728

and

https://code.djangoproject.com/ticket/31885

https://github.com/django/django/pull/13305

Currently all the parameters to configure the email backend are located

in the global namespace, typically starting with EMAIL_...

This in the past led to discussions whenever a new configuration parameter

was required, because it added another entry to the yet too long list of configuration

settings.

Since Django allows us to use more than one email backend concurrently,

why don't we configure them in a similar way as we already do it with

database, caching or template engine backends?

Such a configuration setting then may look like:

EMAIL = [

    {

        'BACKEND': 'django.core.mail.backends.smtp.EmailBackend',

        'HOST': 'smtp.example.org',

        'HOST_USER': 'smtp_user',

        'HOST_PASSWORD': 'smtp_secret',

        'PORT': 25,

        'FQDN': 'example.org',

        # all other related directives 

    }

]

This would allow each email backend to "bring" its own configuration settings

contained in their own dictionary. Adding extra settings to this dict will certainly

lead to less discussions as we currently have.

Personally, I would prefer SMTP = {...} as the main configuration directive, but EMAIL seems to be more popular in Django.

This would mean that the configuration directives starting from here will have

to be replaced, by their counterparts as explained before.

[Tim Graham]

The idea was pursued in https://code.djangoproject.com/ticket/22734 but ultimately abandoned.

[Jacob Rief]

Well, that ticket is 8 years old and in the meantime other email backends have emerged, requiring different configuration options.

I made this proposal after attempting to fix a 14 year old open ticket #6989 but this was ultimately postponed, see comment by

Carlton Gibson on https://github.com/django/django/pull/13728#issuecomment-987762791

To summarize the discussion from 7 years ago

Collin Anderson wrote:

I don't see any benefit to moving email settings to a dictionary. It is helpful for databases and caches because there can be multiple backends.  

It makes the popular "from local_settings import *" convention harder to use. What's wrong with 6 individual settings? If the goal is to allow

multiple email backends, then let's make that the ticket goal.

 

and Carl Meyer replied:

I agree with Collin; unless we are adding new capabilities (i.e. multiple configured email backends, which it seems nobody really wants), it's hard 

to find any actual benefit from this change to justify the churn (and the additional complexities of working with dictionary settings in partial-override scenarios).

 

why shouldn't it makes sense to have different email backends? If you have a staging system you may want to use you local SMTP-relay, while in production

you may for instance use AWSs SES service. That service may require additional configuration settings not available in the local smtp backend.

I can also imagine that in some situations it may make sense to have two email backends concurrently. We maybe should rethink about that.

[Florian Apolloner]

Hi Jacob,

I wouldn't be opposed to move email configuration into a dictionary (somewhere between -0 and +0). Although if we plan to do that we should rethink all the existing session variables and other as well I guess and figure out if we should move more settings to dictionaries.

> why shouldn't it makes sense to have different email backends? If you have a staging system you may want to use you local SMTP-relay, while in production

you may for instance use AWSs SES service.

This specific example at hand is imo not a good motivator to add support for multiple backends because the settings would imo be different. Take databases as an example: You also don't have staging/production in there but switch the actual values in the default database.

> `EMAIL = [...]`

I am not sure a list makes sense here and would go for similarity with CACHES & DATABASES since you'd usually identify the backend via a unique name or so. Also DATABASES & CACHES have an OPTIONS dict which is the passed on to the backend, I think we should follow suit here. 

> Personally, I would prefer SMTP = {...}

I do not think SMTP would be a good fit because many services allow HTTP submission, so what we are sending is actually an email and smtp is just a protocol implementation in the backend of AWS SES or so.

As for other email backends that do require different options: I do not see an issue when they simply take `EMAIL_AWS_SES_KEY` and document it as such; this doesn't require us to add more flexibility to email backends…

So I guess it boils down to the following questions:

 * Do we want to support multiple (at the same time) email backends, if yes we would move to a settings dict anyways…

 * If the answer to the above is no, what value does putting it into a single dict give us?

In the past I think I have argued for a SECURITY_HEADERS (or similar) dict because it allows us to check the dictionary keys easily for typos; emails probably don't suffer from that problem as badly as security related settings.

I hope this can get the discussion going.

Cheers,

Florian

[st...@jigsawtech.co.uk]

I've a big +1 on changing email config to a dictionary to support multiple backends as it's very much a common occurrence for both clients of mine and for my own businesses. Most of the use cases are when they main site sends emails from no-reply@ such as for password resets but then when alternative email are required for sales and/or customer service email address where it's handled via the website. Currently I end up creating a custom settings.py dictionary to store the settings so I can then refer to that using the connection for swapping the backend to send from.

[Florian Apolloner]

I do not understand why you would need multiple email backends to send from different addresses. Can you elaborate on that?

[Matthias Kestenholz]

Maybe we're doing something stupid or there's a misunderstanding but we had a recent use case for this: The same website runs on several domains, one domain per language and each domain has its own email address. We were using an email relay (Mailgun in this case but the same will probably be true for Sendgrid, SES etc.) with different authentication parameters for each sender address.

It wasn't that hard to instantiate one EmailBackend per domain/language but it would definitely be nice if something like this was supported out of the box.

Best regards,

Matthias

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/aec69c4e-a7c2-4002-bb24-21ab11d89343n%40googlegroups.com.

[Alexander Schulze]

+1 from our side as well.

Another common scenario for us is mail server reputation: We use different providers for transactional and newsletter emails as well as different servers depending on the users plan (enterprise customers have a more reliable but more expensive mail server opposed to basic customers).

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CANvPqgDhevLyPGfXi6R0EgDxDb%2B-cCXOVgzVOX2xAyNDcPiUBQ%40mail.gmail.com.

[Steven Mapes]

I have clients that use different mailboxes sometimes with totally different providers (O365, Google, their own mailservers) so they need different SMTP settings. In some cases they also read from the mailboxes, granted this isn't from Django builtin functionality but in those cases it uses the same credentials.

Steve Mapes Steven Mapes T/A Jigsaw Tech
	E: st...@jigsawtech.co.uk P: +44(0)7974220046 W: https://www.jigsawtech.co.uk/

The entire content of this email message is confidential. This also applies to any files attached to it. This email is intended for an individual or entity to whom they are addressed. In case you are not the addressee of this email, and you have received it in error, contact the system manager immediately. The information could be very sensitive, and it is intended for the specific addressee. This email should not be disseminated, distributed or copied. If you have received this email and it was not for you, please notify the sender by email immediately and afterward delete this email from your system. Disclosing, copying, distributing, or taking any action in reliance on the email content is strictly prohibited.

On Jan 30 2022, at 3:38 pm, Florian Apolloner <f.apo...@gmail.com> wrote:

I do not understand why you would need multiple email backends to send from different addresses. Can you elaborate on that?

--

You received this message because you are subscribed to a topic in the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-developers/R8ebGynQjK0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to django-develop...@googlegroups.com.

[Steven Mapes]

I should add I also have other instances my clients use a custom AWS SES backend I wrote used for various types of sends (bulk, newsletters) but generally sends where they need SNS notifications to trigger HTTPS postbacks for certain results from the send such as bounced mails, spam reports etc where those the requests go back to the django server to be handled whereas other mail sends may go via O365 / Google for general, often internal, notifications and emails send.

I have another client who has two mail providers where one is the primary and if that fails due to the SMTP server being down, the platform othen fails over to re-send via the backup SMTP server.

Steve Mapes Steven Mapes T/A Jigsaw Tech
	E: st...@jigsawtech.co.uk P: +44(0)7974220046 W: https://www.jigsawtech.co.uk/

The entire content of this email message is confidential. This also applies to any files attached to it. This email is intended for an individual or entity to whom they are addressed. In case you are not the addressee of this email, and you have received it in error, contact the system manager immediately. The information could be very sensitive, and it is intended for the specific addressee. This email should not be disseminated, distributed or copied. If you have received this email and it was not for you, please notify the sender by email immediately and afterward delete this email from your system. Disclosing, copying, distributing, or taking any action in reliance on the email content is strictly prohibited.

On Jan 30 2022, at 3:38 pm, Florian Apolloner <f.apo...@gmail.com> wrote:

I do not understand why you would need multiple email backends to send from different addresses. Can you elaborate on that?

[Adam Johnson]

Another situation you might want multiple backends is when switching providers. Rather than big-bang swap *all* email sends to a new provider, you might want to move only low-value emails first, or a percentage of your user base, and iterate.

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CF847694-1156-4BA3-BC92-5F42D434B955%40getmailspring.com.

[Florian Apolloner]

Okay then,

some of the things like sender reputation and different bounce hooks came to my mind as well, but it is good to hear confirmation from others. I think the next steps would be to create a new ticket to add support for *multiple* email backends and then work from that (I would only link the old ticket since it's scope was mainly putting the config into a dict as opposed to multiple backends). Given that there are plenty of +1 here already I think we already have our implementors? :)

Some items that I like to see addressed in a PR:

 * Backwards compat

 * Similarity to Caches & Databases (ie so we don't invent yet another syntax)

 * Support for connection aliases (default/…) in send_email % friends (basically everything taking a connection now should probably take aliases as well)

Cheers,

Florian

[Jacob Rief]

On Monday, January 31, 2022 at 7:55:47 AM UTC+1 f.apo...@gmail.com wrote:

Okay then,

some of the things like sender reputation and different bounce hooks came to my mind as well, but it is good to hear confirmation from others. I think the next steps would be to create a new ticket to add support for *multiple* email backends and then work from that (I would only link the old ticket since it's scope was mainly putting the config into a dict as opposed to multiple backends). Given that there are plenty of +1 here already I think we already have our implementors? :)

Some items that I like to see addressed in a PR:

 * Backwards compat

 * Similarity to Caches & Databases (ie so we don't invent yet another syntax)

 * Support for connection aliases (default/…) in send_email % friends (basically everything taking a connection now should probably take aliases as well)

OK. Since I made the proposal, it then is probably up to me to create the ticket.  

[Mariusz Felisiak]

It seems we have a consensus here to support multiple email backends. Jacob, feel-free to create a new ticket.

Best,
Mariusz

[Jacob Rief]

There is a ticket for this feature request now:

https://code.djangoproject.com/ticket/35514#ticket