auto-processing of attachments

[Ken Mankoff]

Hi,

I'm working in an Exchange environment. I can send/receive Exchange emails using DavMail, but wonder if mu can help handle calendar attachments.

Many emails have an attachment, all with the same name, "vcal-3.vcs". If I process these (save to Desktop, run a vcal perl script I found[1]), I see:

>
> event start: 2017-09-14 09:15 Central European Standard Time
> event end: 2017-09-14 10:00 Central European Standard Time
>
> summary:
> SUMMARY;SnapPy etc
>

Is there any way to automate me seeing this information? Ideally I'd like for mu to detect there is an attachment that matches a name or regex, and if so run a command, and then display the output of that command at the top of the message. If not, I will I guess make a key-combo that runs a function that does this for me (or displays results in a new temp buffer rather than injecting into the message). But it would be nice if it were automated.

Thanks,

-k.


[1] http://waynemorrison.com/software/vcal

[Eduardo Mercovich]

Hi Ken.

> [...] but wonder if mu can help handle calendar attachments.

> Many emails have an attachment, all with the same name,

> "vcal-3.vcs". [...]

Does this reference help to process that calendar entry?
https://www.gnu.org/software/emacs/manual/html_node/emacs/Importing-Diary.html#Importing-Diary

I am interested in that too...

> Is there any way to automate me seeing this information? Ideally
> I'd like for mu to detect there is an attachment that matches a
> name or regex, and if so run a command, and then display the
> output of that command at the top of the message. If not, I will
> I guess make a key-combo that runs a function that does this for
> me (or displays results in a new temp buffer rather than
> injecting into the message). But it would be nice if it were
> automated.

I asked a short time ago about automation too (Subject: Sequence
automation and mail filtering) but still don't know about how to
do it.
Maybe we can help each other.
Do we have a way to trigger a function after, for example, we have
new mail? :)


--
Eduardo Mercovich

Donde se cruzan tus talentos
con las necesidades del mundo,
ahí está tu vocación.
(Anónimo)

[Ken Mankoff]

Dear Eduardo,

On 2017-09-15 at 03:02, Eduardo Mercovich <eduardo....@gmail.com> wrote:
>> [...] but wonder if mu can help handle calendar attachments. Many
>> emails have an attachment, all with the same name, "vcal-3.vcs".
>> [...]
>
> Does this reference help to process that calendar entry?
> https://www.gnu.org/software/emacs/manual/html_node/emacs/Importing-Diary.html#Importing-Diary

Not initially. I get the error message: "diary-from-outlook: Don’t know how to snarf in ‘mu4e-view-mode’"

After a bit longer using mu4e + exchange, I realize the solution should be a bit more generic and also handle "winmail.dat" attachments.

It is beyond my lisp abilities, but perhaps the recent work by @gambhiro that removes, modifies, and re-attaches attachments might be a good starting point for a generic attachment processor: https://github.com/gambhiro/mu4e-detach

> I asked a short time ago about automation too (Subject: Sequence
> automation and mail filtering) but still don't know about how to do
> it. Maybe we can help each other. Do we have a way to trigger a
> function after, for example, we have new mail? :)

I'm going to solve this. It'll take a while. I'll begin with manually-triggered workflows, and move to automation from there as necessary. I'll post back with updates and code as it develops.

-k.

[Eduardo Mercovich]

Hello Ken.

>>> [...] but wonder if mu can help handle calendar attachments.
>>> Many emails have an attachment, all with the same name,
>>> "vcal-3.vcs". [...]

>> Does this reference help to process that calendar entry?
>> https://www.gnu.org/software/emacs/manual/html_node/emacs/Importing-Diary.html#Importing-Diary

> Not initially. I get the error message: "diary-from-outlook:
> Don’t know how to snarf in ‘mu4e-view-mode’"

Pity.

> After a bit longer using mu4e + exchange, I realize the solution
> should be a bit more generic and also handle "winmail.dat"
> attachments.
> It is beyond my lisp abilities, but perhaps the recent work by
> @gambhiro that removes, modifies, and re-attaches attachments
> might be a good starting point for a generic attachment
> processor: https://github.com/gambhiro/mu4e-detach

A little seach about processing those the mail
attachments/invitations showed up gcalcli -useful only if you use
google calendars- that can process them:
https://github.com/insanum/gcalcli#importing-vcsvcalics-files-from-exchange-or-other

Maybe that could help?

>> I asked a short time ago about automation too [...] Maybe we

>> can help each other. Do we have a way to trigger a function
>> after, for example, we have new mail? :)

> I'm going to solve this. It'll take a while. I'll begin with
> manually-triggered workflows, and move to automation from there
> as necessary. I'll post back with updates and code as it
> develops.

If I can help, please don't hesitate to ping me.

Best regards...

[Ken Mankoff]

CLI parsers:
+ vcs files can be parsed with vcal[1]
+ winmail.dat with TNEF[2]

Are you on Win/OS X/Linux?

My short-term goals:

+ winmail.dat: have it open correctly

+ vcs: when opening, have it display the event (pop up a notification, new buffer, whatever)

My long-term goals:

+ winmail.dat: Replace attachment with extracted file

+ vcs: display easily, and have special Org capture that creates an Org EVENT (how I manage my calendar) from the file.

[1] http://waynemorrison.com/software/vcal
[2]https://github.com/verdammelt/tnef

-k.

[Dirk-Jan C. Binnema]

I think we can automate at least some of this; it seems `vcal' doesn't
read from stdin, but we can fix that with a little shell-script, which
we'll call `showcall.sh'.:

--8<---------------cut here---------------start------------->8---
#!/bin/sh
cat > /tmp/vcal-3.vcs
vcal -all /tmp/vcal-3.vcs
--8<---------------cut here---------------end--------------->8---

(this assumes that vcal is available in your $PATH). With this you can
use mu4e's attachment actions, ie. `A | 1 RET showcall.sh RET`, and that
can be easily put e.g. in a keyboard macro.

Now, it would indeed be nice if we could fully automated that (and, if
possible, integrate it with e.g. org-mode. Same with meeting
invitations). That'd require a bit more glue / elisp though.

Kind regards,
Dirk.

--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

[Eduardo Mercovich]

Hello Ken.

[...]

> Are you on Win/OS X/Linux?

Debian. :)

> My short-term goals:
> + winmail.dat: have it open correctly
> + vcs: when opening, have it display the event (pop up a
> notification, new buffer, whatever)

I've been looking at solutions for calendar/events managing in
emacs-orgmode and found quite some references and methods.

However, I'll cite the main ones here only, just in case someone
found them useful:

+ Google Calendar Synchronization; worg >
http://orgmode.org/worg/org-tutorials/org-google-sync.html

+ Google Calendar integration in OrgMode; by König von Haunstetten
>
https://koenig-haunstetten.de/2017/01/02/google-calendar-integration-in-orgmode/

+ Google Calendar, Org Agenda; by Mike Zamansky >
http://cestlaz.github.io/posts/using-emacs-26-gcal/#.WG7PabGX_dR

+ How I view my google calendar agenda in Emacs; by Ben Maughan >
http://pragmaticemacs.com/emacs/how-i-view-my-google-calendar-agenda-in-emacs/

Each has a different approack for a different workflow.

> My long-term goals:
> + winmail.dat: Replace attachment with extracted file

Sorry to ask, but why you want to replace the attachment? Is'nt it
enough to add the info to the calendar?

> + vcs: display easily, and have special Org capture that creates
> an Org EVENT (how I manage my calendar) from the file.
> [1] http://waynemorrison.com/software/vcal
> [2]https://github.com/verdammelt/tnef

That's why I referenced gcalcli before: it works quite well and
simple in the command line, so if we translate an org event to
gcalcli add command, it's done.
From https://github.com/insanum/gcalcli#usage

--8<---------------cut here---------------start------------->8---
add
add a detailed event to a calendar
- a single --calendar must specified
- the "--details url" option will show the event link
- example: gcalcli --calendar 'Eric Davis'
--title 'Analysis of Algorithms Final'
--where UCI
--when '12/14/2012 10:00'
--duration 60
--description 'It is going to be hard!'
--reminder 30
--who 'bo...@example.com'
add--8<---------------cut
here---------------end--------------->8---

Of course, gcalcli it's not the only tool.
Variety is good. :)

Best...

[Ken Mankoff]

On 2017-09-16 at 17:55, Eduardo Mercovich <eduardo....@gmail.com> wrote:
> I've been looking at solutions for calendar/events managing in
> emacs-orgmode and found quite some references and methods.

I think we're trying to solve different problems. I already have a calendar management system and workflow in Org. I just want a way to easily include VCS files.

> + winmail.dat: Replace attachment with extracted file
>
> Sorry to ask, but why you want to replace the attachment? Is'nt it
> enough to add the info to the calendar?

Yes it is. But it would be nice to see the email as it is meant to be seen (or how Exchange viewers would see it).

> That's why I referenced gcalcli before: it works quite well and
> simple in the command line, so if we translate an org event to
> gcalcli add command, it's done.

It is done if you use Google Calendar. I don't.

-k.

[Eduardo Mercovich]

Hello Dirk.

>> I'm working in an Exchange environment. I can send/receive
>> Exchange emails using DavMail, but wonder if mu can help handle
>> calendar attachments.
>> Many emails have an attachment, all with the same name,
>> "vcal-3.vcs". If I process these (save to Desktop, run a vcal
>> perl script I found[1]), I see:

>> [...] But it would be nice if it were automated.

> I think we can automate at least some of this; it seems `vcal'
> doesn't read from stdin, but we can fix that with a little
> shell-script, which we'll call `showcall.sh'.:
> --8<---------------cut
> here---------------start------------->8---
> #!/bin/sh
> cat > /tmp/vcal-3.vcs
> vcal -all /tmp/vcal-3.vcs
> --8<---------------cut
> here---------------end--------------->8---
> (this assumes that vcal is available in your $PATH). With this
> you can use mu4e's attachment actions, ie. `A | 1 RET
> showcall.sh RET`, and that can be easily put e.g. in a keyboard
> macro.

Great!

> Now, it would indeed be nice if we could fully automated that
> (and, if possible, integrate it with e.g. org-mode. Same with
> meeting invitations). That'd require a bit more glue / elisp
> though.

A possible workflow would be:
. When the message is opened, mu4e scans the attachments.
. If one is recognized as an event, presents the info and asks the
User if it wants to add it to it's calendar file (ideally, should
be one of the agenda files).
. If the User says "yes", it creates the entry. If it's not
configured, it asks for one.

Another variable is to use an event template from org, but we
depend there to have than configured...

[Charles-H. Schulz]

Hello Eduardo, hello Dirk,

It would really be useful - so far people using CaldDav and calendars that are not explicitely GCal are somewhat left out in the Emacs ecosystem...

Best,

Charles.

[Charles-H. Schulz]

Ken,

I'm exactly in the same situation - see my previous email in the thread. However, parsing this gcalcli, I was wondering what was so specific about Google Calendar settings that could not be configured for any standard CalDav provider. Everything with gcalcli seems to revolve around the use of the Google Calendar API, which makes sense of course and anything besides that is really standard (it could work with any other provider). I don't know the Google Calendar internals so I did not study its API entirely but I did notice this: https://developers.google.com/google-apps/calendar/caldav/v2/guide

It seems the API lets you access GCal pretty much like any other CalDav provider if you want it to use it in that mode. It even looks like not going through the Google API may be easier, but I could well be wrong. I've set out to ask the author .

Apologies for this slightly OT comment,

Charles.

  -k.

--
You received this message because you are subscribed to the Google Groups "mu-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Eduardo Mercovich]

Hi Charles.

[...]

>> A possible workflow would be:
>> . When the message is opened, mu4e scans the attachments.
>> . If one is recognized as an event, presents the info and asks
>> the User if it wants to add it to it's calendar file (ideally,
>> should be one of the agenda files).
>> . If the User says "yes", it creates the entry. If it's not
>> configured, it asks for one.

>> Another variable is to use an event template from org, but we
>> depend there to have than configured...

> It would really be useful - so far people using CaldDav and
> calendars that are not explicitely GCal are somewhat left out in
> the Emacs ecosystem...

Totally true, I agree completely.

I think that mu4e to an org event is the 1st and most important
step.
After that, it's not about mu4e since depends on how each person
has it's things arranged.

WDYT?

[Eduardo Mercovich]

Hi Ken, Charles.

>> That's why I referenced gcalcli before: it works quite well and
>> simple in the command line, so if we translate an org event to
>> gcalcli add command, it's done.

> It is done if you use Google Calendar. I don't.

And I'm going out of it.

Thant's why in the following mail I rethought it and proposed to
go just up to an org event. :)

> I'm exactly in the same situation - see my previous email in the

> thread. However, parsing this gcalcli, [...] Everything with

> gcalcli seems to revolve around the use of the Google Calendar

> API [...] It seems the API lets you access GCal pretty much like

> any other CalDav provider if you want it to use it in that mode.
> It even looks like not going through the Google API may be
> easier, but I could well be wrong. I've set out to ask the
> author .

That would be very good.

> Apologies for this slightly OT comment,

Not needed at all, we are working on this puzzle together and each
one has a different piece. :)

[Ken Mankoff]

Hi Dirk & list,

I think I've made some progress, but have also run into a block.

My current plan is to create a wrapper for mbsync that will do the following:

1) Run mbsync
2) Find the new messages (if any)
3) Process them (replace any winmail.dat with the proper attachment)
4) Then exit, at which point mu-index would continue as normal, but see the only the modified messages.

My first attempt at this mbsync wrapper would be:

a) Touch a file
b) Run mbsync
c) Use find to search in a subset of folders (where new mail might go) for files newer than the touched file.

Then, for each new message, run

mu extract ${MSG} winmail.dat --target-dir=/path/to/somewhere

For each ${MSG} where a winmail.dat was attached (and extracted by 'mu extract'), I can decode the winmail.dat to its proper contents using the 'tnef' utility.

The last bit that I have not been able to solve this evening is how to re-attach the files. I can swap winmail.dat with its tnef-extracted contents using altermime:

altermime --input=${MSG} --replace=winmail.da --with=${FILE}

But when I test this by running 'mu extract' the modified file, it does not work. I do get a file named correctly (e.g. it is whatever was inside winmail.dat), but the file is corrupt. I think I need to do something with mime-enconding the file before I attach it, but I'm not sure what or how. Perhaps someone (Dirk?) with more familiarity with the internals of emails and their attachments can help?

Thanks,

-k.

[Ken Mankoff]

Hi List,

Just in case anyone else is getting VCS files (Outlook emails) as attachments and wants them imported seamlessly into Org, I have an alpha version of this workflow that works for me.

On OS X, I made an Automator App called vcs-to-org-capture.app, put it in ~/bin/, associate this app with .vcs file extensions, and its sole contents is a shell script action: "~/bin/vcs-to-org-capture.sh $@"

The vcs-to-org-capture.sh shell script contents follows. It uses vcal.pl (https://github.com/hangyan/Code/blob/master/perl/vcal.pl) to parse the VCS file, extracts the date, start time, stop time, summary, and location, and then calls org-protocol with all of that. It also puts the entire VCS (text version) in the body of the capture.


#!/bin/sh

cd ~/tmp/
cp $@ ~/tmp/vcal.vcs

~/bin/vcal.pl -all ~/tmp/vcal.vcs > ~/tmp/vcal.txt

DATE=$(grep "event start: " vcal.txt | cut -c10-25 | tr -dc '0--9')
T0=$(grep "event start: " vcal.txt | cut -c 25- | tr -dc '0-:9')
T1=$(grep "event end: " vcal.txt | cut -c 25- | tr -dc '0-:9')

SUMMARY=$(grep "SUMMARY" vcal.txt | cut -d';' -f2)
LOCATION=$(grep "LOCATION" vcal.txt | cut -d';' -f2)

ALL=$(cat ~/tmp/vcal.txt)

TITLE=${SUMMARY}\ [${LOCATION}]
DT=$(echo \<${DATE} ${T0}\-${T1}\>)

~/bin/emacsclient "org-protocol://capture://V/${TITLE}/${DT}/${ALL}" &


The org-protocol capture template I use is the following, where I use "** EVENT" for calendar events.

("V" "VCS via org-protocol"
entry (file+olp "/path/to/calendar.org" "Events")
"** EVENT %:link\n %:description %? \n\n%:initial")


This works for me if I double-click on a VCS file on the desktop, or <RET> on one as an attachment in mu4e. As I write this email, I realize it should do one more thing, which is provide a link back to the original mu4e email that had the VCS attached. I don't think this is possible with the current workflow that uses the OS X Automator app.

I hope this is helpful to someone...

-k.