Sending email in batch

[Branko Vukelic]

I'm currently trying to figure out the best way to send batches of
emails using tasks. There are three ways I can imagine doing it:

1. Enqueue a task passing it a list of addressees, and the task
processes the first one, and enqueues a task passing it the rest

2. Enqueue a task passing it the whole list, and the task processes N
addresses, and enqueues a task passing it the rest

3. Enqueue a task passing it the whole list, task enqueues another
task with all items except first N items, and processes the first N
items after that (which I guess would imagine to parallel processing).

How would you do it?

--
Branko Vukelić
bra...@herdhound.com

Lead Developer
Herd Hound (tm) - Travel that doesn't bite
www.herdhound.com

[nischalshetty]

If you're using the appengine mail api be sure you want that or something along the lines of Amazons mail service because I remember reading a thread where one of the appengine devs said the email service is not meant for large amount of emails (clients would block your email and many others might take it to be spam)

How about querying the datastore and making use of a cursor. Pass the cursor value to the new tasks. Will that work for you? 

[nischalshetty]

Here is the discussion about using a different dedicated email service in case you are into sending a lot of emails https://groups.google.com/d/msg/google-appengine/azhX_BKiNiE/WYrM9blv1CIJ

[Branko Vukelic]

On Sun, Jun 26, 2011 at 9:41 PM, nischalshetty
<nischal...@gmail.com> wrote:
> If you're using the appengine mail api be sure you want that or something
> along the lines of Amazons mail service because I remember reading a thread
> where one of the appengine devs said the email service is not meant for
> large amount of emails (clients would block your email and many others might
> take it to be spam)

Yes I know about this. We will use SendGrid after Beta release, but
for Alpha, we plan to use the GAE mail API simply to keep things
rolling.

> How about querying the datastore and making use of a cursor. Pass the cursor
> value to the new tasks. Will that work for you?

The mailing lists are generated dynamically. So once the list is
compiled from the data in the datastore, I would not like to store it,
and then repeatedly read it from the datastore each time a task is
executed. What I do is:

mailing_list = generate_list(from=some_data)
task.add(url='/url/to/task',
params={'mailing_list': mailing_list.join(',')})

and in the task handler (when I want to do a batch of 50 emails in one task):

mailing_list = params.POST.get('mailing_list').split(',')
to_process = mailing_list[50:]
the_rest = mailing_list[:50]

or in case I want to go one by one:

to_process = mailing_list.pop(0)
the_rest = mailing_list

enqueue the_rest, and process to_process. I haven't tested this yet,
but I'm wondering if it's better to go one-by-one or many-by-many.

[Nischal Shetty]

I would suggest you do it in parallel. Create n tasks with x emails in each of them. It would be faster that way! 

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.

To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

--
-Nischal

twitter: NischalShetty

facebook: Nischal

    

[Branko Vukelic]

Ok, thanks for all the answers. I'll do it that way.

[Nischal Shetty]

Also remember that in a batch you can add a maximum of 100 tasks.

    

[Branko Vukelic]

I've decided to process 50 emails per task. I've looked over the logs and I'm guessing that's far more than enough limit to for the task to finish in 10 minutes.The task itself checks if there's anything left (at the very top) and queues another task. It seems to be working nicely. Not sure if that's the proper way of doing parallel processing, but I'm happy with the solution.