Re: managing bulk emails from app engine app

[Joakim]

Split the work into manageable chunks and create one TaskQueue task per chunk. TaskQueue jobs have a request time limit of ten minutes [0], which may well be enough to send all your emails.

[0]: https://developers.google.com/appengine/docs/java/backends/overview

Joakim

On Saturday, July 7, 2012 7:19:12 PM UTC+2, Vivek Kumar wrote:

Hello

We have around 10000 registered users with us who are blood donors. We want to send them periodic emails like every month to remind them or their account status etc. 

Obviously, a single request cannot process so many emails. 

So, what is the suggested way to handle this use case without hitting request time limits etc on app engine?

our app is in GAE for java

Vik

[Jeff Schnitzer]

I find it most convenient to make the 'manageable chunk' one single email.

It's a map/reduce-type problem. You want to map the "enqueue email
task" across every user. The simplest solution is to iterate across
the users and enqueue a task for each. You can do it in batches of
100 (actually QueueConstants.maxTasksPerAdd()); I find the Guava
Lists.partition() and Lists.transform() methods really handy here.

With a keys-only query you can probably iterate across 100k records in
the ~60s before a query times out. If not, you can enqueue the
mapping process. You could even use the google-provided MapReduce
toolkit.

Ultimately you end up with a zillion "email this user" tasks which you
can run as parallel as you wish.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/google-appengine/-/Oah95qLypI8J.
>
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to
> google-appengi...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/google-appengine?hl=en.

[Per]

Hi Vik,

technical details aside, keep in mind that the spam filter settings on App Engine can be pretty strict. We have been sending mails for 18 months now, increasing the number gradually, and we also used DKIM signing. Still, suddenly some time last week our mails stopped getting sent. From all I can tell some Google mechanism decided that we're a spammer. We have inquired with Google instantly, but of course the GAE customer service is poor as always and never got back to us. So we switched to an external email provider now (mailgun, but there are dozens out there), and I'm quite happy with that, since tracking of emails is a lot more convenient too. It's not free though.

Long story short, if you all of a sudden send 10k mails, be prepared to get filtered by Google because they think you're a spammer. Make sure to spread the load, and track what you have sent already, so you know what you'd need to resend if push comes to shove.

Cheers,
Per

[Richard Watson]

Offload both transactional and bulk email to a 3rd-party email specialist.  It's their core competence, and you're likely to be able to send bulk email with one api call once your list is set up.  If you have issues you'll have better insight into what those are.  You can get delivery and read reports, and automatic remove-me-from-your-list handling.

[Jeff Schnitzer]

I should add that we also use a third-party email system. Ultimately
it's no harder to make a REST call to an external service than to make
a call to MailService, and many services offer a much more robust
solution. GAE's mail system has silly limitations like the inability
to send emails with inline (Content-ID) image attachments.

Just make sure that your 'unit of work' is the same as a single
urlfetch. You wouldn't want to put 100 REST calls in a single task -
imagine it failed at #50, then the whole task will retry and re-spam
#1-#49. If your mail service does "send this message to these 100
people", great; otherwise stick to one-email-per-task. We do
one-email-per-task.

Jeff

> --
> You received this message because you are subscribed to the Google Groups
> "Google App Engine" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/google-appengine/-/m4NlskwCrF8J.

[Vivek Kumar]

Hie

Actually that was my worry as well like sending through app engine may mark emails spam and thus wasting our efforts.

We ocassionaly send newsletter to all our subscribers and for that we are already using VerticalResponse.com which gives us 10000 emails credit every month for free being a non profit.

Right now, we want to handle use cases like:

" Sending email once a month to all  the blood donors to verify/update their profile"

So, if we use any 3rd party system then any recommendations? If non -profit friendly then that would be awesome!

Vik

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[pdknsk]

> suddenly some time last week our mails stopped getting sent. From all I can
> tell some Google mechanism decided that we're a spammer. We have inquired

http://code.google.com/p/googleappengine/issues/detail?id=6181

[Vivek Kumar]

Hie Jeff

Which thrid party system do you guys use? please share the details like how u r using their service to send emails from your gae app

Vik

On Sunday, 8 July 2012 09:13:15 UTC-7, Jeff Schnitzer wrote:

> To post to this group, send email to google-appengine@googlegroups.com.

> To unsubscribe from this group, send email to

> google-appengine+unsubscribe@googlegroups.com.

[Vivek Kumar]

hie can some one help us on suggesting third party emails systems which are good to go to integrate with app engine?

Vik

[Richard Watson]

Sendgrid, or Google "sendgrid vs" and you'll see comparisons.  Note that some specialise in bulk email, some transactional, some do both.

[Stephen Lewis]

I'm not the best person to provide a comparison between services because we've only tried Sendgrid. That said, they've been excellent for our needs - we've sent just shy of 2 million mails via them so far.

Stephen

[Vivek Kumar]

Hey Guys 

Thanks for the reply...

Any experience with Amazon SES ? Pricing looks good to me however, I am not sure it works smooth with google app engine for java? Please advise

Thankx and Regards

Vik
Founder
http://www.sakshum.org
http://blog.sakshum.org

To view this discussion on the web visit https://groups.google.com/d/msg/google-appengine/-/gwKTwZsdkIoJ.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Jeff Schnitzer]

On Thu, Jul 12, 2012 at 10:51 PM, Vivek Kumar <vik...@gmail.com> wrote:
> Hie Jeff
>
> Which thrid party system do you guys use? please share the details like how
> u r using their service to send emails from your gae app

Sorry I missed this earlier. We use messagebus.com. We're very happy
with them, although I don't think their service is generally available
to the public yet.

Jeff

[Vivek Kumar]

Jeff

Does it play nice with google app engine for java?  Also how the pricing looks like? As you said not yet public so emailed them.

Thankx and Regards

Vik
Founder
http://www.sakshum.org
http://blog.sakshum.org

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Jeff Schnitzer]

It's a simple REST API. Works fine with GAE. I'm sure they all do.

Jeff