Email & Idempotence

[Joseph Lebel]

"When implementing the code for Tasks (as worker URLs within your
app), it is important that you consider whether the task is
idempotent. App Engine's Task Queue API is designed to only invoke a
given task once, however it is possible in exceptional circumstances
that a Task may execute multiple times (e.g. in the unlikely case of
major system failure). Thus, your code must ensure that there are no
harmful side-effects of repeated execution." -http://code.google.com/
appengine/docs/python/taskqueue/overview.html

According to this part of the documentation, it would seem that
setting up an email-sending task-queue is impossible. After all,
sending someone the same email thousands of times would be disastrous.
Yet the documentation also gives email sending as an example use of
the Task Queue.

This seems to be contradictory. How can I create a Task Queue for
sending email without running into problems of idempotence?

Thanks for your help.

[Tim Hoffman]

Well could yo record if you sent any particular email, and
not resend it ?

[Jeff Schnitzer]

Or you can just ignore the problem, because it's really quite slight.

Code defensively - do all the work (that might fail) up front in your
task and send the email as a last step. The only problem would be
something going wrong in the email send process itself (or the task
service wrapup). Could this theoretically result in an extra email to
the user? Sure. Will it result in 1,000 emails to the user? It
seems pretty unlikely.

Jeff

> --
> You received this message because you are subscribed to the Google Groups "Google App Engine" group.
> To post to this group, send email to google-a...@googlegroups.com.
> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.
>
>

[Eli Jones]

Like Tim says... if you're really worried about sending out a duplicate e-mail from time to time... then you'll need to design your mailer task to do something like this:

1.  Mailer task starts.. gets a batch of work from some datastore Model that serves as your queue.

2.  Process a row of work, and after you successfully send out an e-mail for that row delete the row from your queue.

Then just do step 2 over and over until you finish all of your work.. or if some error forces the task to retry.. then it should get a new batch of work from your queue.. which should now contain only entities that have not been processed.

The only case where you'd get a duplicate email sent for a row of work would be if the email successfully was sent out.. but an error occurred when trying to remove the row from your queue.

And, even in that case, you could add try: except: blocks to do everything possible to avoid that (just retry the db.delete() from your queue, etc). (Then, only the 30 second limit would be a concern.. but you can handle that as well).

Then again, you really have to be adamant about avoiding the potential duplicate e-mail issue to do this sort of thing.

[Joseph Lebel]

I guess all of you are technically correct. However, the solutions
given were either: a) introduce a tremendous amount of latency/CPU
usage in terms of putting/checking/deleting things from the datastore
b) live with it. I think it's kind of weird for them to use this
specific case as an example, when it clearly isn't the best example.

I guess I'll just have to hope that everything goes okay, and write
lots of try/except clauses in my code to make sure the "little things"
don't mess stuff up.

Thanks anyways,

> > google-appengi...@googlegroups.com<google-appengine%2Bunsubscrib e...@googlegroups.com>

[Jeff Schnitzer]

This problem isn't particular to appengine. There is no such thing as
"transactional" email. If you're doing any bit of transactional work
that might be retried (as with message queueing system, or just a user
who gets an error message and hits the Do button again), you have to
be somewhat careful with when you send emails. And even then, you're
(very) occasionally going to send out duplicate emails.

Try to send your emails as a final step. If you want to be super
extra careful, create another task that does nothing but call the
email send service. If you can use transactions, enlist this task
creation into the transaction.

Jeff

> To unsubscribe from this group, send email to google-appengi...@googlegroups.com.

[Tim Hoffman]

I would pass of the email sending to a task queue, and wrap the
success of the send in a transaction
and only update to yes if the email send call returns ok.

At least you off load your work from the primary transaction.

Remember email isn't really a reliable service, and even if you do
mark it as sent it may never reach the recipient.

Pushing the send email workload and recording if you sent it into a
task should not really cost you to much
in end user response times/latency.

As someone else put it email is not transactional in any sense. Its
very much a
"throw it over the fence and hope it gets there in the next few days"

T

[GAEfan]

This just happened to me... I sent a single email to the taskqueue.
I received this twice:

"DeadlineExceededError: The API call mail.Send() took too long to
respond and was cancelled."

error. However, the same email was sent 3 times, successfully.
That's bad enough, but I would hate to have it send a few thousand
times!

Could this have anything to do with the new queue limit of 50/second?
Should a mail queue be set to a smaller number?

queue:
- name: mail-queue
rate: 50/s
bucket_size: 10

[GAEfan]

Further... I sent the email to a customer, with a bcc to myself (Gmail
via Google Apps account). The customer got 3 emails; I got 2. So, it
timed out sending the bcc to my Gmail account. I could remove the
bcc, and just send a separate email to myself, but that gets me to the
quota twice as fast, and seems unnecessary.

[GAEfan]

OK, just received the 3rd bcc, 34 minutes later.

So the question is, how can I put an email in the taskqueue, and have
the task not re-executed if the mail.send() takes too long?

[Joseph Lebel]

The reason for this is because appengine retries the task if it
returns an error...

One way to work around this is to put the email sending in a try/
except clause so that if it does get an error it would not still
return okay. And, at the same time, you can do whatever you want to
process the error.

[Julian Namaro]

What about:

try:
mail.send_mail( ...)
except Exception, e:
logging.error( str(e) )
return

[Brian Miller]

Any updates on this subject based on updates to GAE or current Recommended / Best Practices

2017 vs. 2010 ?

We have some Java code running in GAE that sends e-mails using javax.mail

but which does not yet use a Task Queue to do this.

We occasionally experience missing e-mails with no server log to explain it.

When debugging and adding try/catch blocks and server logs, I noticed

that we occasionally see the "com.google.apphosting.api.ApiProxy$ApiDeadlineExceededException: The API call mail.Send() took too long to respond and was cancelled."

exception, but this is a red herring because for the specific cases I've looked at where

we got that Exception, the e-mail WAS in fact successfully sent/received, actually.

I'm looking into options for making the e-mail sending more robust, and am considering

introducing the Java Task Queue API into this portion of our code, if it will help and not cause new problems.

Thanks for any input

Brian Miller