#10355 Add support for email backends
Andi Albrecht
I'm interested in working on #10355 "Add support for email backends."
IMHO it's an good idea to make the email backend configurable. There
are at least two use cases I can think of. The first is to send email
with other services than SMTP, like App Engine as noted in the
ticket's description. The second is to deliver email asynchronously,
like the django-mailer application does already.
The ticket currently needs a design decision, so my question is what
the actual concerns to change this are.
I would propose the following changes. It's a very simplistic approach
that tries to keep the current API as much as possible:
Add a new setting EMAIL_BACKEND. A string that can be resolved to a
class. Default should be the current SMTP implementation.
Provide a base class for mail backends. A mail backend must provide
the method send_messages(email_messages) and must return the number of
messages sent to provide backward compatibility. The constructor of a
mail backend should take at least the keyword argument fail_silently
(default: False). What I'm a bit unsure about are additional
constructor arguments. Currently the SMTP backend allows in addition
host, port, username, password and use_tls. Those are very
SMTP-specific, but only username and password are used by the
mail.send_mail* APIs. It would be an agreement to allow username and
password in addition to fail_silently to not break the send_mail* API.
The SMTP backend could accept host, port and use_tls as extra keywords
again to provide backward compatibility for code that directly uses
SMTPConnection (within Django SMTPConnection is not used outside
django.core.mail). I would suggest to rename SMTPConnection to
SMTPBackend, but only if this would break too much third-party code as
SMTPConnection is mentioned in the docs.
The test utils could be refactored to use a TestMailBackend instead of
monkey-patching the mail module, but this would be a fairly invisible
change as it would just change the way how the test utils provide the
mailbox attribute in the mail module for unittests.
A nice addition to Django would be to include parts of django-mailer
to provide a backend for asynchronous mail delivery. That could be the
models and command line script to deliver mails. While thinking about
this, django-mailer has a nice option for setting priorities to mails.
IMO Django's API for sending mails should be as simple as possible
(like it is today), but a priority option would be a nice addition
even if the SMTP backend (as the default) wouldn't respect this option
when sending mails.
Adding a App Engine backend is not covered here. I think it's up to
third-party code to add support for sending mails on App Engine.
However, such a backend would be pretty easy to implement.
It would be nice to get some feedback on this suggestions and in a
first step to find a design decision about #10355. Even if the
decision is to not change anything for good reasons :)
Regards,
Andi
Russell Keith-Magee
Albrecht<albrec...@googlemail.com> wrote:
>
> Hi,
>
> I'm interested in working on #10355 "Add support for email backends."
>
> IMHO it's an good idea to make the email backend configurable. There
> are at least two use cases I can think of. The first is to send email
> with other services than SMTP, like App Engine as noted in the
> ticket's description. The second is to deliver email asynchronously,
> like the django-mailer application does already.
>
> The ticket currently needs a design decision, so my question is what
> the actual concerns to change this are.
I think the original concern was the time required to triage the
ticket :-) The original report is fairly light on detail, so I suspect
Jacob punted the proposal to DDN in a fit of rapid triage.
However, your proposal gives us a bit more detail to work with. On the
whole, I have to say I like what I see. This is the sort of thing I'd
like to see more of in Django - less specific feature enhancements
baked into the core, and more opening of interfaces that let people
plug their own capabilities in where they need to.
> I would propose the following changes. It's a very simplistic approach
> that tries to keep the current API as much as possible:
To qualify this - "as much as possible" must include "100% backwards
compatible with 0 code changes with all public advertised interfaces".
> Add a new setting EMAIL_BACKEND. A string that can be resolved to a
> class. Default should be the current SMTP implementation.
No problem with this, and follows the precedent set by other backends
(such as the cache backend)
> Provide a base class for mail backends. A mail backend must provide
> the method send_messages(email_messages) and must return the number of
> messages sent to provide backward compatibility. The constructor of a
> mail backend should take at least the keyword argument fail_silently
> (default: False). What I'm a bit unsure about are additional
> constructor arguments. Currently the SMTP backend allows in addition
> host, port, username, password and use_tls. Those are very
> SMTP-specific, but only username and password are used by the
> mail.send_mail* APIs. It would be an agreement to allow username and
> password in addition to fail_silently to not break the send_mail* API.
I'll need to look into this in more detail. I'm hesitant to lock down
APIs like this too much - the more rigid you make an interface, the
harder it becomes to use it in interesting way. I'm sure that some
insights will emerge as a result of writing the code.
> The SMTP backend could accept host, port and use_tls as extra keywords
> again to provide backward compatibility for code that directly uses
> SMTPConnection (within Django SMTPConnection is not used outside
> django.core.mail). I would suggest to rename SMTPConnection to
> SMTPBackend, but only if this would break too much third-party code as
> SMTPConnection is mentioned in the docs.
Breaking _any_ third party code is a non-starter. We can deprecate the
old name and provide an alias from the old name to the new name, but
historical usage must continue without modification.
> The test utils could be refactored to use a TestMailBackend instead of
> monkey-patching the mail module, but this would be a fairly invisible
> change as it would just change the way how the test utils provide the
> mailbox attribute in the mail module for unittests.
This is an interesting suggestion all by itself. It also dovetails
nicely with #8638. That ticket was on the list for v1.1, and
originally proposed including a 'test email server'. Ultimately, it
was decided not to include an email server, but to document the ways
you could use existing mail servers.
A 'test email backend' (as well as a dummy no-mail-at-all backend, and
a log-to-console/log-to-file backend) is a different way to target the
underlying problem posed by #8638.
> A nice addition to Django would be to include parts of django-mailer
> to provide a backend for asynchronous mail delivery. That could be the
> models and command line script to deliver mails. While thinking about
> this, django-mailer has a nice option for setting priorities to mails.
> IMO Django's API for sending mails should be as simple as possible
> (like it is today), but a priority option would be a nice addition
> even if the SMTP backend (as the default) wouldn't respect this option
> when sending mails.
I'm aware of the existence of django-mailer, but I haven't looked into
it in detail. At least for the first cut, I'd be more comfortable
seeing django-mailer-esque capabilities living as a community project.
Then, maybe in the v1.3 timeframe, we can look at adding it as a
'batteries included' backend.
That said - if a prototype django-mailer backend were to be developed
as part of this development effort, it would go a long way to proving
that the backend API you are proposing is sufficiently flexible. The
code doesn't need to be trunk-ready, but if you can prove that the
needs of a complex email backend can be met with your proposed
interface, it goes a long way to proving that your interface is rich
enough.
> Adding a App Engine backend is not covered here. I think it's up to
> third-party code to add support for sending mails on App Engine.
> However, such a backend would be pretty easy to implement.
This is exactly the right approach. Again, it would be good to develop
a prototype backend as a proof of concept of the interface, but that
implementation doesn't need to be complete or trunk-ready.
> It would be nice to get some feedback on this suggestions and in a
> first step to find a design decision about #10355. Even if the
> decision is to not change anything for good reasons :)
On the whole, I think you're on the right track here. I like the idea,
and your design proposal needs some minor tweaking and finesse, but on
the whole makes sense. It's certainly worth putting this ticket on the
v1.2 feature list for formal consideration.
Yours,
Russ Magee %-)
Thomas Guettler
Andi Albrecht schrieb:
> Hi,
>
> I'm interested in working on #10355 "Add support for email backends."
>
> IMHO it's an good idea to make the email backend configurable. There
> are at least two use cases I can think of. The first is to send email
> with other services than SMTP, like App Engine as noted in the
> ticket's description. The second is to deliver email asynchronously,
> like the django-mailer application does already.
Yes, I think it is a good idea.
It would be nice to have several backends active, too. In my applications
error-mails of uncaught exceptions should take a different backend than emails
for application users.
Thomas
--
Thomas Guettler, http://www.thomas-guettler.de/
E-Mail: guettli (*) thomas-guettler + de
Zachary Voase
On 21 Aug 2009, at 05:34, Andi Albrecht wrote:
> Hi,
>
> I'm interested in working on #10355 "Add support for email backends."
>
> IMHO it's an good idea to make the email backend configurable. There
> are at least two use cases I can think of. The first is to send email
> with other services than SMTP, like App Engine as noted in the
> ticket's description. The second is to deliver email asynchronously,
> like the django-mailer application does already.
I wholeheartedly agree.
> The ticket currently needs a design decision, so my question is what
> the actual concerns to change this are.
>
> I would propose the following changes. It's a very simplistic approach
> that tries to keep the current API as much as possible:
>
> Add a new setting EMAIL_BACKEND. A string that can be resolved to a
> class. Default should be the current SMTP implementation.
>
> Provide a base class for mail backends. A mail backend must provide
> the method send_messages(email_messages) and must return the number of
> messages sent to provide backward compatibility. The constructor of a
> mail backend should take at least the keyword argument fail_silently
> (default: False). What I'm a bit unsure about are additional
> constructor arguments. Currently the SMTP backend allows in addition
> host, port, username, password and use_tls. Those are very
> SMTP-specific, but only username and password are used by the
> mail.send_mail* APIs. It would be an agreement to allow username and
> password in addition to fail_silently to not break the send_mail* API.
> The SMTP backend could accept host, port and use_tls as extra keywords
> again to provide backward compatibility for code that directly uses
> SMTPConnection (within Django SMTPConnection is not used outside
> django.core.mail). I would suggest to rename SMTPConnection to
> SMTPBackend, but only if this would break too much third-party code as
> SMTPConnection is mentioned in the docs.
This I disagree with slightly. My main concern is the single-backend
architecture; many websites will probably want to use more than one
method for sending e-mail. In addition, if mail backends only need to
implement one method, why not just have EMAIL_BACKEND refer to a
callable instead?
In terms of the SMTP-specific settings (host, port, username, password
and use_tls), I personally feel that those parameters should be in the
settings module and not in the code. Although at the moment,
django.core.mail will use certain settings if said parameters are left
out.
I think a slightly better architecture would be this:
* Make full use of the existing Django signals framework to send e-
mail. Have callables which implement 'send_messages()' connect to a
'sendmail' signal through the use of a simple decorator or connector.
These callables will connect with the 'sender' keyword argument, which
will be a string which uniquely identifies that backend; so AppEngine
might be 'app_engine', SMTP 'smtp', and so on.
* To send a message, just send on the 'sendmail' signal, with a
'sender' equal to whatever backend you want to use for that particular
message or batch of messages. Failing silently can be done with the
signals framework, using 'send_robust()' to catch errors and then
deciding whether or not to raise them later. Of course, nice wrappers
would be provided over all of this, so that the code which sends mail
does not need to know the signals API.
* E-mail backends can, by default, configure themselves from the
project settings. Using signals means that it’s also quite easy to
dynamically create a backend, connect it to the dispatcher, send some
signals, and then disconnect it.
What do you think about this? I might work on a simple proof-of-
concept reusable application.
Regards,
Zack
Tim Chase
> and use_tls), I personally feel that those parameters should be in the
> settings module and not in the code. Although at the moment,
> django.core.mail will use certain settings if said parameters are left
> out.
Having email settings defined globally precludes the ability to
do things like create a web email application for which each user
has their own email (usually SMTP/IMAP/POP) settings. While yes,
I could see having some defaults at the settings.py level, it
should be possible to override those at the call level.
-tim
Zachary Voase
On 21 Aug 2009, at 11:39, Tim Chase wrote:
> Having email settings defined globally precludes the ability to
> do things like create a web email application for which each user
> has their own email (usually SMTP/IMAP/POP) settings. While yes,
> I could see having some defaults at the settings.py level, it
> should be possible to override those at the call level.
It would be easy to do so. Since signal receivers can be dynamically
connected and disconnected, it would be relatively trivial to
implement a function which would dynamically create a receiver with a
UUID, use it for sending messages via that single connection, and then
destroy the receiver again afterwards.
But still, the top-level functions for sending e-mails should be
backend-agnostic, which means they shouldn’t accept any SMTP-specific
arguments, and so developers should really be putting their default
configuration into the settings module.
--
Zack
Andi Albrecht
access to the SMTP protocol to handle errors more accurate than an API
that just sends out mails and simply fails (optional silently) when
any error occurs. If you want to include this in Django's mail API, I
suppose you end up with a smtplib clone :)
POP and IMAP are totally different topics.
>
> -tim
>
>
>
>
>
> >
>
Russell Keith-Magee
I'm not sure I agree with your assertion of "many"."Some" might be
accurate. "Your" is probably more accurate :-)
I've got many websites in the field, and not one of them has needed
anything more than trivial email handling. We've managed to get to
v1.1 and AppEngine support is the first time that pluggable email
backends have really been raised as an issue.
This is hardly surprising. After all, email is email. You have an SMTP
server, you connect to it, you send your mail. AppEngine is a weird
case in that they provide an email-sending API rather than using SMTP,
but that's an artefact of the platform. Once you have one email
sending capability, I find it hard to believe that most people will
need a second.
I don't doubt that there are applications that will require more than
one mail server, but I'm comfortable calling them edge cases. If you
have highly specialized mail requirements, then it makes sense that
you should have a highly specialized mail server handling.
That said, there isn't really that much difference between the simple
and complex case - it's just a matter of defaults.
Django needs to have a default Email backend, guaranteed available.
EmailMessage.send() uses the 'default' backend - essentially just
calling backend.send_messages([msg])
backend.send_messages() also exists as a direct call.
SMTPConnection().send_messages() is really just a shortcut for
instantiating and using an SMTP connection with the default settings.
You're not compelled to use the default connection though. You could
instantiate multiple instances of different backends, and use them to
call other_backend.send_messages().
I see this working almost exactly as the Cache backend works right
now. There is a base interface for caching. There are several cache
backends; dummy and locmem are handy for testing, but if you're
serious, the only one that gets used is memcached. There is a default
cache instantiated as a result of the CACHE_BACKEND setting. For most
people, this is all you will ever need. However, if you want to
instantiate (and use) multiple caches (e.g., if you want to get really
fancy about cache overflow and expiry policies), you can.
> In addition, if mail backends only need to
> implement one method, why not just have EMAIL_BACKEND refer to a
> callable instead?
Persistence of settings. s1 = Backend(host, port, username password)
can be configured once, then s1 can be reused whenever needed.
> I think a slightly better architecture would be this:
>
> * Make full use of the existing Django signals framework to send e-
> mail. Have callables which implement 'send_messages()' connect to a
> 'sendmail' signal through the use of a simple decorator or connector.
What exactly is the use case where you need to have the 1-n
cardinality provided by signals? The only use I can think of would be
to have message delivery determined by the message itself - i.e., a
message could be sent by SMTP server 1, SMTP server 2, or the
AppEngine Server, depending on the recipient/content.
If this is the use case, I put it to you that this is an _extreme_
edge case - and one that could be handled using a custom mail backend
that does content-based dispatch - i.e., a mail backend that wraps
other mail backend instances.
Yours,
Russ Magee %-)
Andi Albrecht
that determines the backend to use implies that someone who wants to
use the mail API already needs to know which backends are available
and has to decide which backend he wants to use. I think it's up to
the administrator to choose which backend should be used for sending
mails. The send_mail functions should be agnostic of this setting.
OTOH signals could be a nice addition and could be used for example to
log failed messages or to modify messages before sending. But I
wouldn't use them to actually send the message.
Andi Albrecht
make the mail module a package? Preserving the current namespace, of
course...
Russell Keith-Magee
Albrecht<albrec...@googlemail.com> wrote:
>
> On Fri, Aug 21, 2009 at 7:23 AM, Russell
> Keith-Magee<freakb...@gmail.com> wrote:
>> A 'test email backend' (as well as a dummy no-mail-at-all backend, and
>> a log-to-console/log-to-file backend) is a different way to target the
>> underlying problem posed by #8638.
>
> A dummy backend/log backend sounds good. I'll add them. Is it time to
> make the mail module a package? Preserving the current namespace, of
> course...
I think that's pretty much inevitable. Again, the cache backend is a
good model here.
Yours,
Russ Magee %-)
Andi Albrecht
"connection" instance attribute. It's documented as "An SMTPConnection
instance. Use this parameter if you want to use the same connection
for multiple messages. If omitted, a new connection is created when
send() is called."
So persistence is needed here to allow multiple calls to
EmailMessage.send() to share the same connection. I'm reading
"connection" here as "network connection", otherwise it wouldn't make
much sense as the creation of SMTPConnection, and possible other
backends likewise, is pretty cheap.
Actually the documentation lacks some explanation on this part - and
therefore it took me a while to understand the connection attribute :)
After reading the docs I've had expected that the following example
uses a single network connection to send mails:
conn = SMTPConnection()
msg1 = EmailMessage(connection=conn).send()
msg2 = EmailMessage(connection=conn).send()
But each call to send() opens and closes a new network connection. To
have a single network connection you'll have to do the following:
conn = SMTPConnection()
conn.open()
msg1 = EmailMessage(connection=conn).send()
msg2 = EmailMessage(connection=conn).send()
conn.close()
If we want to have an instance of the mail backend on module level
similar to the backends in the cache or db module those calls to
open() and close() become nasty as we've might get problems with
thread-safety here (assuming that the relation between backend and
network connection is 1:1).
I would propose that backends have those open() and close() methods
that do nothing in the base implementation. Only backends that require
a connection on network level should overwrite those methods to manage
network connections. I think in most cases (that is mail.send_mail*,
backend.send_messages and EmailMessage.send without setting the
connection instance attribute) the backend handles network connections
(if needed) internally. The open() and close() methods are just needed
for the example outlined above and the docs should be updated
accordingly. (A use case for this pattern is if you want to use a
single network connection and if you want to do things after each
message is sent. backend.send_message([mgs1, msg2]) is not an option
then.)
Instead of having a backend instance on module level, the mail module
should provide a get_backend(fail_silently=False) function that
returns an backend instance. This would avoid problems with concurrent
calls to open/close on a global backend instance.
Russell Keith-Magee
It sounds like you've got a good handle on the problem. I can't see
anything fundamentally wrong with what you are proposing. You are
correct that the 'instance' problem doesn't really apply to the cache,
so different handling is appropriate here.
My only issue is one of nomenclature. While you will be defining an
SMTP backend and dummy backend, etc, and those classes should be
referred to as Backends, it doesn't make much sense to me to talk
about obtaining an instance of a backend or get_backend();
get_connection() would seem more appropriate terminology.
So, the public interface in django.core.mail will look a little like:
def get_connection(*args, **kwargs):
module = <resolve backend module from settings>
return module.EmailBackend(*args, **kwargs)
SMTPConnection = smtp.EmailBackend
and the get_connection call in EmailMessage, send_mail and
send_mass_mail would make a call to the get_connection utliity, rather
than instantiating SMTPConnection directly as they do now. At the end
of the day, it's actually quite a low impact change.
Yours,
Russ Magee %-)
Andi Albrecht
the mail module into a package and provides the backend functionality.
Documented functions and classes are of course available in the
top-level module. Undocumented methods like make_msgid() are not
available in django.core.mail anymore, but an additional import could
be added easily.
Two demo backends are attached too. One implements a backend for
sending mails through App Engine's mail API, the other implements a
backend for django-mailer. In case someone is interested, I've made
the full examples of both backends available on bitbucket:
http://bitbucket.org/andialbrecht/django_mailbackend_examples/
Andi
On Sat, Aug 22, 2009 at 9:15 AM, Russell
Russell Keith-Magee
Albrecht<albrec...@googlemail.com> wrote:
>
> I've attached a patch to the issue on the tracker. The patch splits up
> the mail module into a package and provides the backend functionality.
> Documented functions and classes are of course available in the
> top-level module. Undocumented methods like make_msgid() are not
> available in django.core.mail anymore, but an additional import could
> be added easily.
>
> Two demo backends are attached too. One implements a backend for
> sending mails through App Engine's mail API, the other implements a
> backend for django-mailer. In case someone is interested, I've made
> the full examples of both backends available on bitbucket:
> http://bitbucket.org/andialbrecht/django_mailbackend_examples/
Thanks Andi. I've left comments on the ticket.
Yours,
Russ Magee %-)
Mat Clayton
Mat
--
Matthew Clayton | Founder/CEO
Wakari Limited
twitter http://www.twitter.com/matclayton
email m...@wakari.co.uk
mobile +44 7872007851
skype matclayton
Russell Keith-Magee
> I'm sure you have this covered but thought I would throw it into the mix. We
> are currently having lots of problems with blacklisting and spam filters
> blocking our email servers, one way we are investigating to avoid this is to
> outsource the email sending to trusted 3rd party services. We don't send
> unsolicited email (believe it or not), but being a social site, we send a
> fair amount of opt in emails, and for some reason this seems to be having an
> impact on our registration confirmation emails getting blocked which is a
> real problem for us. Something like this mail backend system seems an ideal
> solution to allow us to try and test alternatives. Just thought I would
> raise this as another reason why a pluggable backend would be a good idea,
> it would certainly help in situations like ours.
I'm not sure that this is the same problem.
The goal of having pluggable email backends is to allow for mechanisms
other than SMTP for sending mail. The main driver for this is to
support AppEngine, but there are also benefits when testing.
The situation you are describing is where you have multiple mail
servers, but they're all SMTP. In this case, you're still only using
one backend - you just have different configuration parameters
(hostname, port, password, etc).
I presume the change SMTP server is something you need to be able to
do without restarting the server or updating settings. If this is the
case, this is already supported by instantiating multiple
SMTPConnection instances. The logic required to pick the right
SMTPConnection is something you would need to code yourself, as it
would be entirely site dependent. Having multiple backends won't help
you here - with Andi's code, all you will be doing is using
get_connection('smtp') to instantiate a connection, rather than using
SMTPConnection directly.
The backend framework that has been proposed would allow you to write
a 'server-shifting backend' - that is, an email backend that
instantiates other email backends. This isn't really any different to
what you can do right now, except that it allows you to use the
send_mail() shortcut as well as EmailMessage().
Yours,
Russ Magee %-)
Mat Clayton
Mat