Sending mass, yet personalised email newsletters using Django

[Patrick J. Anderson]

I'm in the process of writing a simple newsletter managers with a web
interface and since I've been a Django user for the past 2 years, I
thought of using Django for this task.

Some of the main requirements are:

* Personalised HTML/text messages
* Manual delivery twice a week
* Message logging/archiving
* Amount of newsleter issues: ~50
* Amount of subscribers: 500-5000

I was wondering about sending personalised mass mail and scaling issues,
since I assume that each email message would need to be rendered
separately to insert subscribers' first and last name, and other related
information and sent using its own SMTPConnection.

Can Django handle this task reliably? If so, I'd be interested in best,
most efficient ways of doing so from other Django users who have done it
before.

I'd appreciate your comments, suggestions, tips and any other advice you
might offer.

[Doug B]

I can't claim best way, but the way we do it is to have a few django
models for storing the messages, and some help from the email, and
smptlib libraries.

Models:
Mailserver: model for storing connection information for connecting to
a standard smtp server, with mixin class using smtplib to provide smtp
connection.
BulkMail: Unrendered Template, and M2M relationship to users. Fk to
Mailserver. This model generates Queued messages, in batches until
all recipients have a message queued (we trickle messages out).
QueuedMessage: Rendered message (pickled email.MimeMultipart object),
Fkeys to user, Bulkmail (if message was generated by the BulkMailer)
and Mailserver.

Every 15min a Bulk Mail cron to generate messages into the queue.
Depending on options one bulk mail object may trickle out the messages
until all are sent.
Every minute a cron runs to check for pending QueuedMessage objects,
and sends them to a real smtp server in batches [something like
mailserver_object.process_queue(priority=N,limit=100)]
Another cron using poplib to check the sending POP3 account for
unsubscribe requests, and for bounces using GNU Mailman (it's python,
and pretty simple to throw on your path and use the BouncerAPI for
bounce detection). Anything that isn't an unsub or bounce gets cloned/
forwarded to a human read pop account.
The crons do some checking using memcache okeys to make sure only one
process is doing mail at any one time, and no messages get stuck in a
loop.

All in all, it's been working great. It takes very little time to
generate the messages and lets a real mail server handle the heavy
lifting/retries.

[alex.v...@gmail.com]

Hello Patrick,

On Nov 11, 1:53 am, "Patrick J. Anderson" <pat.j.ander...@gmail.com>
wrote:

>
> Some of the main requirements are:
>
> * Personalised HTML/text messages
> * Manual delivery twice a week
> * Message logging/archiving
> * Amount of newsleter issues: ~50
> * Amount of subscribers: 500-5000

We (as company) has created 6 different versions of Newsletter for our
customers. They are
all commercial, but we plan to release some non-commercial version
too, as part of our e-commerce
solution, djWarehouse.

What about the Newsletter, I can explain the architecture for you, so
you would not fall into problems which had us to rewrite our
Newsletter 6 times until we get satisfied with results.


We have the following models:
Newsletter (name, added, sent, status (NEW, READY, SCHEDULING,
SENDING, SENT)
MessageSet (name, newsletter, category, status (same as in
newsletter))
Message (messageset foreign key, marker_tag, subject, from name, from
email, format, message)
MessageCopy(user, messageset, message)
MessageResponse(message, url, count)

This newsletter is used on some of our sites, with thousands of
subscribers. (100000-200000). The logick is the following:
customers are subscribed to some specific categories on site. We have
choice to send general newsletter to all customers, or customize
messages by categories. MesageSet - a number of messages sent to
specific category[ies]. Every newsletter can contain a few message
sets. Why so complicated? Because we need to know what information is
more interesting to end users. Counting the results in MessageResponse
we find out how many customers clicked the links in one or another
newsletter, and know what customers are more interested in - this data
is used for next monthly newsletter from the site.

Now, the process of sending newsletter. That is one of the most
difficult parts of the process, since the SMTP protocol and message
daemons are not allowing to do many things, so we have to organize
things ourself. For example, since our sites have such a large users
base, Yahoo, Gmail, and specifically Hotmail were blocking our legal
newsletter every time, because from a few hundreds thousands
subscribers, a large part belongs to those online services (a few
thousands) and those services will automatically put you in "block
list" as soon as you try to send messages too quick.

That is why, we don't just send mails out via SMTP. The process of
sending is the following:
1. When admin press "send" newsletter button, this only triggers the
process to save data to MesageCopy.
2. An external program, run in 'daemon' mode, watches the table, and
maintains the list
of exeternal web services (Yahoo, MSN, etc). This daemon controls
that SMTP send out rate should
not exceed some specific amount for specific service.

So far, this architecture works out the best. We are using this 6th
version of improved newsletter for last 1.5 years, without any
problems.

I can not post the whole code here because the program is commercial,
but... some smaller version will be released soon. Anyway, most of
customers need to only send mails to a few thousands customers, not a
few hundreds thousands, so some advanced functions are just not needed
for an average customer.

Hope that information is useful.

Best Regards.
Alex

[ilmarik]

I would like to share with you some different ideology...
I don't like to use any external software to work aside with main
application. Very often, it's just imposible to use cron on client's
hyper-hostings.
The idea is simple, use ajax.

I got very simple mechanism that uses some javascript to send
asynchronous requests to the sending script. I store newsletters
prepared earlier(with tinymce) in one table, subscribers (split in
groups) in another and smtp accounts in another.
To send emails, user have to prepare sending session. He choose smtp
account, subscribers group and newsletter from combobox'es and click
prepare. combined id's populate session table in amount of rows equal
to emails in chosen group.
with little optimalization, that process is very fast even for
hundreds of thousands of emails.
finally, user clicks to send emails and little js script send
asynchronously a sending request. after every email succesfully sent,
it's information in session table is updated to check it as send.
pros.
- you never lost any email because of timeout
- you can show progress bar to user
- you may start and stop whole process whenever needed
cons
- user have to prepare and start sending process manualy. sometimes it
may be needed to do it absolutely automatic.

I hope it would help

[Patrick J. Anderson]

Thanks for your comments and tips.

I was actually thinking already about doing it with javascript by sending
AJAX requests. I was considering some sort of cron jobs, but this would
not be my preference.

With sending emails via AJAX requests one consideration was to time it so
sending messages doesn't overwhelm Apache. My solution to this would be
to segment the list by region, since I will have that information. This
should split the load somewhat. But I'd have to test it first to see what
kind of burden it'll put on the server.

The other main issue is rendering and creating about 1000 or more
messages in the database with a click of a button. This could be a
bottleneck, but perhaps I could segment that as well.