Best practices for sending bulk (+4000) mail in Django / Python
gorans
I've been sending out email to a list of around 8000 people on a
monthly basis using django and was wondering whether there is a way to
optimise the process.
Currently, I open a mail connection [ mail_connection =
SMTPConnection(fail_silently=True) ] and then loop through each
message calling the .send() method.
however, this takes about 2 seconds per message which is ridiculous!
Is there are better way to do mass emailing to our subscribers?
Cheers
Goran
Malcolm Tredinnick
On Wed, 2008-04-23 at 07:03 -0700, gorans wrote:
> Hi,
>
> I've been sending out email to a list of around 8000 people on a
> monthly basis using django and was wondering whether there is a way to
> optimise the process.
>
> Currently, I open a mail connection [ mail_connection =
> SMTPConnection(fail_silently=True) ] and then loop through each
> message calling the .send() method.
>
> however, this takes about 2 seconds per message which is ridiculous!
Not to mention that with that many messages, there are going to be
errors -- some permanent, some transient. Django's email system isn't
designed to be a mass mailer on that scale. It's a whole other class of
problem.
> Is there are better way to do mass emailing to our subscribers?
One immediate possibility that jumps to mind is to set up some mailing
list software. Django sends email to the mailing list address and it
then goes out to everybody. You'll need to configure things so that
other people cannot post to the list, etc, but that's just a matter of
configuration.
If you use mailman (written in Python; well tested in production
settings), you could fairly easily write some code to synchronise the
mailman subscription list with whatever database table you are currently
using to store the subscribers, I suspect.
Another possibility is to use some sort of queueing system or service so
that the emails are inserted into the queue and then sent over an
extended period of time.
Regards,
Malcolm
>
> Cheers
>
> Goran
>
> >
>
--
I've got a mind like a... a... what's that thing called?
http://www.pointy-stick.com/blog/
Doug B
combination of what Malcom suggested. I setup a model to queue the
messages (pickled email.MIMEMultipart) with a priority, and a cron
that runs occasionally to dump N messages per mx domain over time so
the mailing trickles out. to the local mailserver that actually does
the sending. Since the connection to the local mailserver is done
once and messages are batched it should eliminate the overhead of
sending one by one.
Then for bounce detection, import from Mailman and let it check the
POP3 account of the sending account.
I was really surprised at how easy it was. Python has all of the mail
stuff build in, and Mailman adds the bounce detection just by
importing its module. My point is don't be afraid to roll your own
solution, its not hard at all. For example the pop3 check/bounce
detection which might seem complicated is just a few lines of code:
def check_pop3_bounces(server,user,password):
messages=[]
s=poplib.POP3(server)
s.user(user)
s.pass_(password)
resp, items, octets = s.list()
bounced=Set()
# items will be a list of 'MSG ID, SIZE'
for item in items:#[:5]:
id,size=item.split()
resp,text,octets = s.retr(id)
text = string.join(text, "\n")
file = StringIO.StringIO(text)
message=email.message_from_file(file)
addr=BouncerAPI.ScanMessages('somename',message)
s.dele(id) # delete it, we'll forward non-junk
if isinstance(addr,BouncerAPI._Stop):
pass # junk message
elif addr:
for a in addr:
bounced.add(a)
else:
valid_messages.append(message)
s.quit()
return bounced,valid_messages
James Tauber
On Thu, 24 Apr 2008 00:11:02 +1000, "Malcolm Tredinnick"
<mal...@pointy-stick.com> said:
> Another possibility is to use some sort of queueing system or service so
> that the emails are inserted into the queue and then sent over an
> extended period of time.
django-mailer is intended to be used for exactly this. It's not usable
yet, but could always do with more people driving it to completion.
See http://code.google.com/p/django-mailer/
James
--
James Tauber http://jtauber.com/
journeyman of some http://jtauber.com/blog/