Beyond mu4e 1.0 : a few ideas

[Charles-H. Schulz]

Hello,

I am quite aware that this is going to be one of those "what if" emails and that I am not a developer however I'd like yo throw a few ideas in this list about possible improvements/new features for mu/mu4e.

1.semi-automatic indexing of messages for specific content:

It's all the rage these days with Google Inbox,  Edison Mail etc. clients that index emails directly according to predefined categories ( trsvel, finances, shopping etc.) and in my experience it can be really helpful. Surprisingly enough these enhanced indexing capabilities do not all (if any) rely on fancy AI or machine learning. To a large extent they work through advanced inbox indexing. So I went to have a look at Xapian because marking messages/threads may not provide the automatic kind of results that are expected in this case. Xapian seems to allow the access to any kind of data of the documents/files it indexes: URLs, metadata and also actual words inside a file. This could enable mu specifically to search for specific predefined expressions that can be of course customised by the user: "travel, plane tickets, city, vacations, bank, mortgage, payments..." A simple predefined bookmark with a specific search pattern could then be automatically added to mu4e default view after the initial indexing of the inbox(es).

2. Attachment of files stored on a remote storage: 

Read: easily attach files from Dropbox, Google Drive or other services by a simple command. This should be possible as long as the user can paste the relevant URL/URI from any of these services and that they allow the sharing of the particular file.

3. Indexing of calendar entries:

This has to be clarified somewhat: most shared calendaring interaction happening inside Emacs is a) done on Google Calendar b) not using open standards such as CalDAV. Calfw only works through Google APIs for instance and its suthor was quite clear to me about this. But for anyone using shared calendar with CalDAV Emacs is a hard place to be. As it turns out it is possible to rely on mail indexing to refile calendar entries. How? Services like Kolab and KolabNow do use CalDAV as a communication and calendaring protocol but do use IMAP as storage for calendar entries. I am not suggesting mu4e should turn into a calendar client but I am suggesting that as calendar entries already appear as messages in the inbox mu4e should be able to properly index them and allow their refiling to tools such as OrgMode.

Feedback is most welcome,

Charles-H. Schulz.

[Ken Mankoff]

On 2017-10-24 at 21:00, Charles-H. Schulz <charles....@gmail.com> wrote:
> 1.semi-automatic indexing of messages for specific content:

Bookmarks

> 2. Attachment of files stored on a remote storage:

Possible with old DropBox API and a custom function/key-combo. I haven't solved this with new DropBox API. May work with Google Drive? Sticking to the unix philosophy, of doing 1 thing and doing it well, this doesn't need to be a part of mu4e to still work well with mu4e. This is an emacs level issue.

> 3. Indexing of calendar entries:
>
> This has to be clarified somewhat: most shared calendaring interaction
> happening inside Emacs is a) done on Google Calendar b) not using open
> standards such as CalDAV. Calfw only works through Google APIs for instance
> and its suthor was quite clear to me about this.

I use Calfw with Org. But yes, I'd like to see this improved.

-k.

[Charles-H. Schulz]

Hello Ken,

On Tue, Oct 24, 2017 at 11:19 PM Ken Mankoff <man...@gmail.com> wrote:

On 2017-10-24 at 21:00, Charles-H. Schulz <charles....@gmail.com> wrote:
> 1.semi-automatic indexing of messages for specific content:

Bookmarks

Bookmarks are only the end result of this idea: These pre-configured bookmarks would be shipped with the standard version of mu4e; also they would rely on the fact that the initial indexing of the MailDirs would be complemented by specific searches for pre-defined keywords : travel, plane ticket, train, trip, etc. That alone is a project in itself, but it is well worth giving some thought and adding it as a feature if possible in mu/mu4e. I'm happy to help in any capacity I can (finding keywords and their weight for instance).

 

> 2. Attachment of files stored on a remote storage:

Possible with old DropBox API and a custom function/key-combo. I haven't solved this with new DropBox API. May work with Google Drive? Sticking to the unix philosophy, of doing 1 thing and doing it well, this doesn't need to be a part of mu4e to still work well with mu4e. This is an emacs level issue.

I did not know about that API limitation with DropBox. That specifically may be an emacs problem, but adding the ability to attach a file from an URL in mu4e's own message mode (not the Emacs message mode) with a documented key combination is the idea. Still: you're right that most of this may be an Emacs issue first.

> 3. Indexing of calendar entries:
>
> This has to be clarified somewhat: most shared calendaring interaction
> happening inside Emacs is a) done on Google Calendar b) not using open
> standards such as CalDAV. Calfw only works through Google APIs for instance
> and its suthor was quite clear to me about this.

I use Calfw with Org. But yes, I'd like to see this improved.

Thanks - this is a feature potentially interesting to anyone using anything else than Google Calendar as its own online calendaring tool.

Best,

Charles.

[Yuri D'Elia]

On Tue, Oct 24 2017, Charles-H. Schulz wrote:
> 1.semi-automatic indexing of messages for specific content:

I use dbacl server-side and some custom scripts for automatic retraining
based on the current IMAP folder. On the client side I only use a
customized refiling function.

It should be possible to do also locally, as long as you have some hooks
during sync (offlineimap does, but isync is limited - which is why I
went for server-side processing).

> any) rely on fancy AI or machine learning. To a large extent they work
> through advanced inbox indexing. So I went to have a look at Xapian because

dbacl actually trains bayes models with whatever content you feed it to.
Commercial alternatives solutions I've seen (including "Clutter" from
office365) are absolute garbage in comparison.

Highly recommended.

> 2. Attachment of files stored on a remote storage:

I'm partial to fully open solutions here.

Use FeX (https://fex.belwue.de/) or DL
(https://www.thregr.org/~wavexx/software/dl/). I'm partial to the
latter, as I'm the author, but it generates self-expiring links. The cli
makes it work as a one-liner returning an URL.

I wanted to intergrate it into mu4e-compose, but never got to it.
Ideally, it would parse attachment tags on the send hook, and convert
the attachment to a link automatically, pretty much how Thunderbird does
with it's "filelink" feature.

> 3. Indexing of calendar entries:

Another highly partial answer here.

I decode and use VCS files using khal's printics/import
(https://github.com/pimutils/khal), and otherwise use ikhal as a
calendaring front-end for CalDAV.

I'm not too happy with CalDAV itself, and calendaring in email as a
whole. It's a sloppy experience full of horrid protocols. For my
personal organization I use TaskWarrior (which includes recurring
tasks), and only use CalDAV because of Android.

Again, I've longing this idea to write a script to convert TW tasks into
a caldav automatically, so that I do not have to use khal at all.

[Charles-H. Schulz]

Hello Yuri,

Le jeu. 26 oct. 2017 à 20:41, Yuri D'Elia <wav...@thregr.org> a écrit :

On Tue, Oct 24 2017, Charles-H. Schulz wrote:
> 1.semi-automatic indexing of messages for specific content:

I use dbacl server-side and some custom scripts for automatic retraining
based on the current IMAP folder. On the client side I only use a
customized refiling function.

It should be possible to do also locally, as long as you have some hooks
during sync (offlineimap does, but isync is limited - which is why I
went for server-side processing).

> any) rely on fancy AI or machine learning. To a large extent they work
> through advanced inbox indexing. So I went to have a look at Xapian because

dbacl actually trains bayes models with whatever content you feed it to.
Commercial alternatives solutions I've seen (including "Clutter" from
office365) are absolute garbage in comparison.

Highly recommended.

Okay... What do you think of using Dirk-Jan's new query framework and turning a few predefined queries into bookmarks? It seems Dirk-Jan's recent work could not have come at a best moment.

Even better question: what does Dirk-Jan think about the original suggestion? :-)

> 2. Attachment of files stored on a remote storage:

I'm partial to fully open solutions here.

Use FeX (https://fex.belwue.de/) or DL
(https://www.thregr.org/~wavexx/software/dl/). I'm partial to the
latter, as I'm the author, but it generates self-expiring links. The cli
makes it work as a one-liner returning an URL.

I wanted to intergrate it into mu4e-compose, but never got to it.
Ideally, it would parse attachment tags on the send hook, and convert
the attachment to a link automatically, pretty much how Thunderbird does
with it's "filelink" feature.

I am not fond of services like DropBox for a variety of reasons. I happen to use Hubic, SendFirefox and even GDrive from a perspective of a personal ratio of laziness/actual concern but my point here is simply that a lot of people use these commercial services and therefore that feature could be quite needed.

> 3. Indexing of calendar entries:

Another highly partial answer here.

I decode and use VCS files using khal's printics/import
(https://github.com/pimutils/khal), and otherwise use ikhal as a
calendaring front-end for CalDAV.

Never heard of Khal. It could really be useful if it were ported on Emacs...

I'm not too happy with CalDAV itself, and calendaring in email as a
whole. It's a sloppy experience full of horrid protocols. For my
personal organization I use TaskWarrior (which includes recurring
tasks), and only use CalDAV because of Android.

Again, I've longing this idea to write a script to convert TW tasks into
a caldav automatically, so that I do not have to use khal at all.

Well my concern here is somewhat reduced. I am not suggesting mu4e to gain a calendar component but to be able to parse calendar entries and invites in a way that can be reused (refiled for instance) by tools such as OrgMode.

Thanks,

Charles.

[Yuri D'Elia]

On Fri, Oct 27 2017, Charles-H. Schulz wrote:
>> dbacl actually trains bayes models with whatever content you feed it to.
>> Commercial alternatives solutions I've seen (including "Clutter" from
>> office365) are absolute garbage in comparison.
>>
>> Highly recommended.
>
> Okay... What do you think of using Dirk-Jan's new query framework and
> turning a few predefined queries into bookmarks? It seems Dirk-Jan's
> recent work could not have come at a best moment.

From your original question, I assumed you wanted something more
advanced to cut yourself with :-)

The new parser doesn't change significantly what you can/could do with
bookmarks already.

The way I'm using dbacl is by automatically filing messages into
different folders using it as a multi-category classifier. I use
"folders" as categories, with their content training the filter.

dbacl will classify messages about fishing automatically if you train it
with fish-related material, in a way that cannot be captured with a
keyword-based filter.

> I am not fond of services like DropBox for a variety of reasons. I
> happen to use Hubic, SendFirefox and even GDrive from a perspective of
> a personal ratio of laziness/actual concern but my point here is
> simply that a lot of people use these commercial services and
> therefore that feature could be quite needed.

Thunderbird did it right with their "filelink" feature. It doesn't tie
to any service by itself, but just provides some hooks to do it. It
should be even easier to implement on top of the current compose mode.

> Well my concern here is somewhat reduced. I am not suggesting mu4e to gain
> a calendar component but to be able to parse calendar entries and invites
> in a way that can be reused (refiled for instance) by tools such as OrgMode.

I would be more in favor of a mailcap-like feature, where I can
associate a filter to a mime type, so I can convert it to text *before*
it is indexed.

This would allow to search metadata associated with attachments, without
introducing specific hacks related to calendaring.

[Dirk-Jan C. Binnema]

On Wednesday Oct 25 2017, Charles-H Schulz wrote:

> 1.semi-automatic indexing of messages for specific content:
>
> It's all the rage these days with Google Inbox, Edison Mail etc. clients
> that index emails directly according to predefined categories ( trsvel,
> finances, shopping etc.) and in my experience it can be really helpful.
> Surprisingly enough these enhanced indexing capabilities do not all (if
> any) rely on fancy AI or machine learning. To a large extent they work
> through advanced inbox indexing. So I went to have a look at Xapian because
> marking messages/threads may not provide the automatic kind of results that
> are expected in this case. Xapian seems to allow the access to any kind of
> data of the documents/files it indexes: URLs, metadata and also actual
> words inside a file. This could enable mu specifically to search for
> specific predefined expressions that can be of course customised by the
> user: "travel, plane tickets, city, vacations, bank, mortgage, payments..."
> A simple predefined bookmark with a specific search pattern could then be
> automatically added to mu4e default view after the initial indexing of the
> inbox(es).

I'd imagine having some 'new-message-hook' in which the user can do any
additional processing they'd like, extract data etc. Xapian is just a
data store, any extra information has to be provided by clients.

> 2. Attachment of files stored on a remote storage:
>
> Read: easily attach files from Dropbox, Google Drive or other services by a
> simple command. This should be possible as long as the user can paste the
> relevant URL/URI from any of these services and that they allow the sharing
> of the particular file.

On Linux at least, with Nautilus (and presumably other file managers)
you can drag files into your messages. Nautilus also supports Google
Drive; however dragging directly from Google Drive into mu4e does not
work yet. Just emacs needs a tiny bit of glue.

> 3. Indexing of calendar entries:
>
> This has to be clarified somewhat: most shared calendaring interaction
> happening inside Emacs is a) done on Google Calendar b) not using open
> standards such as CalDAV. Calfw only works through Google APIs for instance
> and its suthor was quite clear to me about this. But for anyone using
> shared calendar with CalDAV Emacs is a hard place to be. As it turns out it
> is possible to rely on mail indexing to refile calendar entries. How?
> Services like Kolab and KolabNow do use CalDAV as a communication and
> calendaring protocol but do use IMAP as storage for calendar entries. I am
> not suggesting mu4e should turn into a calendar client but I am suggesting
> that as calendar entries already appear as messages in the inbox mu4e
> should be able to properly index them and allow their refiling to tools
> such as OrgMode.

It'd be useful to do that... I think the aforementioned
'new-message-hook' should help here.

Anyway, thanks, it's good to keep such use cases in mind, so some future
mu can make them possible!

Kind regards,
Dirk.

--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

[Charles-H. Schulz]

Hello Dirk-Jan,

Le 28 oct. 2017 14:00, "Dirk-Jan C. Binnema" <dj...@djcbsoftware.nl> a écrit :

On Wednesday Oct 25 2017, Charles-H Schulz wrote:

> 1.semi-automatic indexing of messages for specific content:
>
> It's all the rage these days with Google Inbox,  Edison Mail etc. clients
> that index emails directly according to predefined categories ( trsvel,
> finances, shopping etc.) and in my experience it can be really helpful.
> Surprisingly enough these enhanced indexing capabilities do not all (if
> any) rely on fancy AI or machine learning. To a large extent they work
> through advanced inbox indexing. So I went to have a look at Xapian because
> marking messages/threads may not provide the automatic kind of results that
> are expected in this case. Xapian seems to allow the access to any kind of
> data of the documents/files it indexes: URLs, metadata and also actual
> words inside a file. This could enable mu specifically to search for
> specific predefined expressions that can be of course customised by the
> user: "travel, plane tickets, city, vacations, bank, mortgage, payments..."
> A simple predefined bookmark with a specific search pattern could then be
> automatically added to mu4e default view after the initial indexing of the
> inbox(es).

I'd imagine having some 'new-message-hook' in which the user can do any
additional processing they'd like, extract data etc. Xapian is just a
data store, any extra information has to be provided by clients.

I see, it seems quite useful. Do you think mu4e could even ship with a few pre-set hooks of that kind?

> 2. Attachment of files stored on a remote storage:
>
> Read: easily attach files from Dropbox, Google Drive or other services by a
> simple command. This should be possible as long as the user can paste the
> relevant URL/URI from any of these services and that they allow the sharing
> of the particular file.

On Linux at least, with Nautilus (and presumably other file managers)
you can drag files into your messages. Nautilus also supports Google
Drive; however dragging directly from Google Drive into mu4e does not
work yet. Just emacs needs a tiny bit of glue.

Now I think I may be describing something way overblown. After all one can simply copy-paste the exact URL of a file stored on these services.

> 3. Indexing of calendar entries:
>
> This has to be clarified somewhat: most shared calendaring interaction
> happening inside Emacs is a) done on Google Calendar b) not using open
> standards such as CalDAV. Calfw only works through Google APIs for instance
> and its suthor was quite clear to me about this. But for anyone using
> shared calendar with CalDAV Emacs is a hard place to be. As it turns out it
> is possible to rely on mail indexing to refile calendar entries. How?
> Services like Kolab and KolabNow do use CalDAV as a communication and
> calendaring protocol but do use IMAP as storage for calendar entries. I am
> not suggesting mu4e should turn into a calendar client but I am suggesting
> that as calendar entries already appear as messages in the inbox mu4e
> should be able to properly index them and allow their refiling to tools
> such as OrgMode.

It'd be useful to do that... I think the aforementioned
'new-message-hook' should help here.

Anyway, thanks, it's good to keep such use cases in mind, so some future
mu can make them possible!

Thank you for listening and for all your work!

Best,

Charles.

Kind regards,
Dirk.

--
Dirk-Jan C. Binnema                  Helsinki, Finland
e:dj...@djcbsoftware.nl           w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

--
You received this message because you are subscribed to the Google Groups "mu-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.