Building a mailer service

[Tauren Mills]

I have a webapp that needs to send out customized mail to its users. I
have thousands of users and need to send messages to them such as
account activation notices, changed password notices, daily status
reports (activity summary, action items they need to do, etc.), and
alerts/reminders at specific times. Each message that is sent out is
customized for a particular user. Once every few weeks there might be
mass-emailing of the same message to everyone that announces new
features or changes to the webapp.

Currently, the webapp is not on GAE, but may be in the future. At the
moment I'm considering using GAE as a platform for the mailer service.
This is so that I can scale up the mailer service as needed without
having to maintain the infrastructure inhouse.

The basic idea is that I would build an application on GAE that has a
RESTful interface. My app could then add messages to the send queue
via that interface. The GAE mailer app would then use GAE's Mail
api's to send out the messages. My webapp would also have a RESTful
interface that the mailer service would use to communicate bak
information about errors, bounces, and so forth.

1. Is this a reasonable sounding solution? Any suggestions for
improvements? What are the drawbacks?

2. I'm assuming there isn't anything that would prevent me from having
DKIM and other technologies to help reduce my mailings as being
flagged as spam. Is this a valid assumption?

3. I need to have bounce detection support and was planning to use
VERP to do so. Since it appears GAE may limit the sending email
addresses, I'm concerned this may not be possible. Is there any way
to use GAE mail apis and support VERP?

4. Are there any existing services that already do this? Any open
source projects that would help get me part way?

Thanks!

[Jeff Schnitzer]

One thing to watch out for is that GAE's mailer won't send
multipart/related emails, so you cannot embed images in your mail.
You can link to external images but that's nearly useless these days.

Considering your VERP requirement, you're probably better off running
the email-sending portion of your app in EC2 or Rackspace Cloud (or
whatever).

Jeff

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

[Tauren Mills]

Jeff,

Thanks for your feedback. I'm thinking you are probably right that GAE
isn't the best solution, especially since you are confirming my
concerns about VERP. It's too bad this isn't possible, because it
seems like GAE would provide a fairly effective way to implement this
solution in an extremely scalable manner.

I may be mistaken, but it seems like an advantage to doing this on GAE
via the Mail API is that mail is sent using a mail server that Google
manages. This means that it is unlikely to get blacklisted by other
ISPs. I've had troubles with my mail servers being blacklisted because
of certain customers abusing it. This shouldn't be a problem for my
webapp, since the mail server will only send mail to registered users.
Bottom line is that mail servers in general are something I really
don't care to hassle with.

Also, I don't think that EC2 would be the best platform. Without
reverse DNS control and truly static IPs, hosting a mail server might
be problematic. As far as I know, they don't provide a Mail API, so
it would be back to simply managing my own mail server and the
headaches that go with it.

Rackspace Cloud might work since they provide a static IP and their
Fanatical Support person told me they can change reverse DNS for me.
But I'm still managing my own mail server. They do not provide an
SMTP server for cloud users.

I just came across a service called JangoSMTP that seems to do much of
what I need. Unfortunately, if I want to send a single email to all my
1000 users each day, I'm looking at spending $180/mo or so. But it
includes an API, supports DKIM, handles bounces, and even supports
feedback loops.
http://www.jangosmtp.com/How-It-Works.asp

It looks like I may just have to manage my own server using a
traditional dedicated or VPS server.

Tauren

[Jeff Schnitzer]

FWIW, I run the "offsite" portions of my application (proxies for
apple's push notification service and proxies for services that get
ratelimited from appengine) in Rackspace Cloud on a single $11/month
virtual server. Works great - static IP, forward and reverse DNS.

Here's one of my opensource projects, it may help you process bounce
notifications:

http://code.google.com/p/subethasmtp/

Good luck,
Jeff

[Tauren Mills]

Jeff,

That's great to hear! I actually am already considering using
subethasmtp to build my mailer with. However, I'm currently leaning
toward using James just because I found a resource explaining how to
easily do VERP with it:
http://cephas.net/blog/2006/06/09/using-apache-james-and-javamail-to-implement-variable-envelope-return-paths/

Is this fairly easy to accomplish with subethasmtp as well? The fact
that you recommend it, are using it, and developing it makes me want
to really reconsider looking into it further.

Thanks again, you've provided some great insights!
Tauren

[Jeff Schnitzer]

That article is crazy talk. Here's the code (all of it) using SubEthaSMTP:

class BounceListener implements SimpleMessageListener {

public boolean accept(String from, String recipient) {
return recipient.startsWith("deals-");
}

public void deliver(String from, String recipient, InputStream data)
throws TooMuchDataException, IOException {
int atIndex = recipient.indexOf("@");
String rec = recipient.substring(0, atIndex).replaceAll("=",
"@").replaceAll("deals-", "");

YourCode.markAsBad(rec);
}

public void main(String[] args) {
SMTPServer smtpServer = new SMTPServer(new
SimpleMessageListenerAdapter(new BounceListener()));
smtpServer.start();
}
}

Just "java -cp . BounceListener" and you're running.

I hate complicated frameworks.

Jeff

[Tauren Mills]

Jeff,

Sweet. I too thought that it seemed awfully bloated to use James for
this. I'll certainly give it a whirl with SubEthaSMTP.

Thanks,
Tauren

On Feb 24, 6:17 pm, Jeff Schnitzer <j...@infohazard.org> wrote:
> That article is crazy talk.  Here's the code (all of it) using SubEthaSMTP:
>
> class BounceListener implements SimpleMessageListener {
>
>         public boolean accept(String from, String recipient) {
>                 return recipient.startsWith("deals-");
>         }
>
>          public void deliver(String from, String recipient, InputStream data)
> throws TooMuchDataException, IOException {
>                 int atIndex = recipient.indexOf("@");
>                 String rec = recipient.substring(0, atIndex).replaceAll("=",
> "@").replaceAll("deals-", "");
>
>                 YourCode.markAsBad(rec);
>         }
>
>         public void main(String[] args) {
>                 SMTPServer smtpServer = new SMTPServer(new
> SimpleMessageListenerAdapter(new BounceListener()));
>                 smtpServer.start();
>         }
>
> }
>
> Just "java -cp . BounceListener" and you're running.
>
> I hate complicated frameworks.
>
> Jeff
>
>
>
> On Wed, Feb 24, 2010 at 2:22 PM, Tauren Mills <yowza...@gmail.com> wrote:
> > Jeff,
>
> > That's great to hear! I actually am already considering using
> > subethasmtp to build my mailer with. However, I'm currently leaning
> > toward using James just because I found a resource explaining how to
> > easily do VERP with it:

> >http://cephas.net/blog/2006/06/09/using-apache-james-and-javamail-to-...