Saving incoming email to datastore

[Phil Hodey]

I need to handle incoming email in an app I am building. I have setup the incoming mail handler and this is working well.

Next stage is to persist the mail I to the Datastore. Is there an accepted best practice for handling this? I was planning on pickling the incoming mail to persist it. Is there a better approach?

[Phil Hodey]

I've realised storing in the datastore is a bad idea given the size limitation of a Datastore entity. Instead I'm saving the raw email to gcs with a Datastore entity giving just email headers. Seems to work well.

[Alex Martelli]

Good point: an incoming email (inc. attachments) can be up to 31+MB -- far above the largest datastore entity, 1MB. So, stashing the mail away in GCS (and, the raw form does seem best), with metadata in the datastore, seems the best strategy.

Alex

On Wed, Dec 9, 2015 at 8:18 AM, Phil Hodey <phil....@hcstech.co.uk> wrote:

I've realised storing in the datastore is a bad idea given the size limitation of a Datastore entity. Instead I'm saving the raw email to gcs with a Datastore entity giving just email headers. Seems to work well.

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To unsubscribe from this group and stop receiving emails from it, send an email to google-appengi...@googlegroups.com.
To post to this group, send email to google-a...@googlegroups.com.
Visit this group at http://groups.google.com/group/google-appengine.
To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/e85a44f8-db26-49ba-888c-ae22eb2feb77%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[PK]

Therefore the body over a threshold should also come in the GCS to start with, not as a 31MB blob. Please star this improvement. :-)

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/CAE46Be_Z2GApJMiObOS0X-_4omZ997G99sHkv_iPhoWX22FNxA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

PK

p...@gae123.com

[Alex Martelli]

Yes PK, I've already starred your FR -- because the sending half of it should have no issues, i.e be certainly helpful w/no downside. As I mentioned, however, the *receiving* part is very problematic: you're essentially giving anybody knowing your app's email address unlimited, unchecked write access to your GCS bucket, up to 31.5 MB at a time -- sounds pretty iffy to me. Surely the ability to write in your GCS bucket should be limited somehow to whitelisted mail senders only -- whence the need for an API for your app to initiate, revoke, and check registrations into said whitelist (which must be held by the system to accept or bounce emails to your app) -- smells an order of magnitude more complex than the sending part AND no doubt the registration requirement for senders would also rule out some use cases (impossible to say if those would be use cases you had specifically in mind, since you don't list any example use cases at all in the FR).

If email reception could just be dispatched to an appropriate module (without forcing you to upsize all instances of the default module just for email-receiving tasks, a too-onerous solution), then *your* app's code could perform all the business logic to bounce the request (with whatever error message if you deem that any such msg is appropriate), or silently drop it, or accept it (and then process it however it's appropriate for your app, including but not limited to shunting it off to GCS) -- based on sender, size, app state, and whatever other meta-data and app info you desire -- a HUGE increase in flexibility over blindly stashing it in GCS.

So, if you and I were both in a PM/Eng team developing GAE, we'd agree on the "send from GCS" feature, but argue in detail how best to handle *receiving* mail. Unfortunately, neither of us is part of such a team, so the best we can do is try to bring it to said team's attention -- and for that purpose nothing works better than the FR getting more stars (it might help if it was broken in two FRs, sending and receiving, since they're really separate and with very different impact and complexity issues).

Alex

To view this discussion on the web visit https://groups.google.com/d/msgid/google-appengine/BA51B2CF-09F4-445D-9B8C-F97002B15E52%40gae123.com.