Email Parser

[Cynthia]

I'm brand new to Google App Engine and I'm not really sure I can get
something that will do what I want. Or maybe I need to go elsewhere. I
am a programmer by trade but I am still a beginner in many areas.

I am looking for a way to receive emails and parse out certain
information, then save the info to a DB or a Spreadsheet. A Google
Docs Spreadsheet would be fine, if that's possible.

Would Google App Engine be able to do this? Which language would be
best? Maybe something like this already exists that I could use?

It's for a club membership list. People sign up with an HTML form, and
the form emails the information to an email address. I want to pluck
the data out of the email and have it save automatically.

Thanks,
Cynthia

[Rishi Arora]

My app heavily relies on processing received emails, and even though I'm just as good (or bad) at python as I am at Java, I chose python because if its flexibility in string processing.  The emails my app receives are in hundreds of different formats and these formats change very often.  Python offers ease of initial development, and ease of quick and frequent changes.  Of course, these are personal preferences - experience Java developers can possibly do things much faster than I can.

But this is only part 1 of your problem set.  Part is being able to save generated and organized data in a Google Spreadsheet.  I haven't developed any such thing myself, but in my related research, I found things that might interest you here (GData Services): http://code.google.com/appengine/docs/python/howto/usinggdataservices.html.

Lastly, why would you have an email step in the middle at all.  Your current HTML form that people submit - where is it hosted?  If it isn't on Google App Engine, you can migrate it to GAE, and have the form directly write to Google Docs spreadsheet.  If migrating your HTML form to GAE isn't viable either, consider making your form send URL encoded data in HTTP Post requests to GAE, in a pre-formatted way so its easier to parse this data and store in a spreadsheet.

Cynthia

--
You received this message because you are subscribed to the Google Groups "Google App Engine" group.
To post to this group, send email to google-a...@googlegroups.com.
To unsubscribe from this group, send email to google-appengi...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/google-appengine?hl=en.

[Cynthia]

Thanks for your reply Rishi,
Our darn website is hosted on iPage. I'm about the only technical
person in our very small, and new, club, and they don't want to change
to a new host. I get moans and groans and no one wants to change,
they want an easy Drag and Drop interface, though it drives me nuts
and I'd move hosts in a heartbeat if I could.

So.. I have thought of modifying the form itself so it can write
directly to a Database, but I'm not sure if my changes to the code
would remain if someone else updates the pages with all their
automatic junk. So the fewer changes I can make to the HTML itself,
the better. The least intrusive way of doing this would be, I think,
to catch the emails that the form generates and then do what I want
with them. But alas, I've never done such a thing.

Cynthia

On Dec 1, 1:10 pm, Rishi Arora <rishi.ar...@ship-rack.com> wrote:
> My app heavily relies on processing received emails, and even though I'm
> just as good (or bad) at python as I am at Java, I chose python because if
> its flexibility in string processing.  The emails my app receives are in
> hundreds of different formats and these formats change very often.  Python
> offers ease of initial development, and ease of quick and frequent changes.
>  Of course, these are personal preferences - experience Java developers can
> possibly do things much faster than I can.
>
> But this is only part 1 of your problem set.  Part is being able to save
> generated and organized data in a Google Spreadsheet.  I haven't developed
> any such thing myself, but in my related research, I found things that

> might interest you here (GData Services):http://code.google.com/appengine/docs/python/howto/usinggdataservices....

[Xavier Sanchez]

Hi.

I think you don't need GAE. I think you have 2 options:

1. Create a new form with Google Docs. Embed it in your web and get the results directly at a Spreadsheet.
2. If you receive the mails at a Gmail account you can use Google Apps Script to process mail and get data into a Spreadsheet. I've not tried this myself but I know you an process mail labels, so I suppose you can parse the content too.

Xavi.

[Brandon Wirtz]

Unless there is a reason for the form to send an email, you would be better
to put the forms "target" on appengine and convert the post data to a
database. Java or python can do this relatively easily, if you don't speak
either now however I don't know as this is a super easy first task.

The DataStorage is easy, but making a spreadsheet out of the data (if you
want it to be editable) is not going to be simple.

Personally if I were doing the project you describe I would Check out PHP
Form Maker (Pro?) or one of the other form maker software suites depending
on what you or your Sys admin/web host determines would be easiest in your
web configuration.

Not that I want to dissuade you from GAE, just don't think that the learning
curve for the task is well balanced.

-Brandon

Brandon Wirtz
BlackWaterOps: President / Lead Mercenary

Work: 510-992-6548
Toll Free: 866-400-4536
IM: dra...@gmail.com (Google Talk)
Skype: drakegreene
BlackWater Ops

Thanks,
Cynthia

--

[Cynthia]

Thanks for the replies. Maybe I can do this another way. I'm reading
up on Procmail, maybe that's what I need.
Cynthia

On Dec 1, 2:15 pm, "Brandon Wirtz" <drak...@digerat.com> wrote:
> Unless there is a reason for the form to send an email, you would be better
> to put the forms "target" on appengine and convert the post data to a
> database.  Java or python can do this relatively easily, if you don't speak
> either now however I don't know as this is a super easy first task.
>
> The DataStorage is easy, but making a spreadsheet out of the data (if you
> want it to be editable) is not going to be simple.
>
> Personally if I were doing the project you describe I would Check out PHP
> Form Maker (Pro?)  or one of the  other form maker software suites depending
> on what you or your Sys admin/web host determines would be easiest in your
> web configuration.
>
> Not that I want to dissuade you from GAE, just don't think that the learning
> curve for the task is well balanced.
>
> -Brandon
>
> Brandon Wirtz
> BlackWaterOps: President / Lead Mercenary
>
> Work: 510-992-6548
> Toll Free: 866-400-4536

> IM: drak...@gmail.com (Google Talk)

[Ikai Lan (Google)]

Cynthia,

Here are some procmail docs:

http://sunsite.utk.edu/UNIX-help/email/procmail.html

Having used this before, I will tell you that it is not the easiest thing in the world to debug and write for. Procmail lets you pipe raw email data into a different process, but you still need to use a program that can successfully parse the email, attachments, headers and lots of other weird email spec goodness.

You definitely *don't* need Google App Engine, but if I am reading your requirements right, you can get away with using the incoming email API:

http://code.google.com/appengine/docs/python/mail/receivingmail.html

And making a call to the spreadsheets API:

http://code.google.com/apis/spreadsheets/

I'd start by trying to get incoming email working. The nice thing about using GAE's incoming mail API is that it makes processing email SUPER easy. The downside is that you don't get a ton of choices about what addresses you can use (your-...@appspotmail.com and others).

--

Ikai Lan 
Developer Programs Engineer, Google App Engine

plus.ikailan.com | twitter.com/ikai

[Rishi Arora]

I solved the problem of email address choices by creating a Gmail account (or a Google Apps account) that uses Gmail filters to forward emails to ap...@appspotmail.com.  It has worked great, with one exception.  If Gmail marks an incoming email as spam, it won't get forwarded.  This requires manual intervention.