Mu4e and the meaning of the "Deleted" IMAP flag

[Dima Kogan]

Hi.

I use mu/mu4e 0.9.9.5-dev6. When I delete a message in mu4e, two things
happen:

1. The message is moved to the Trash folder
2. The message gets a "Deleted" IMAP flag (T maildir flag)

Most of the time, this works, except sometimes I'm seeing that the
messages are expunged from the Trash, and thus disappear. After a few
months of living with this intermittent behavior (and keeping my mail in
git to fight these purges), I figured out the cause: my mail server
deletes all mail with the Deleted flag set, but only actually does this
when I visit their webmail interface, which I do on occasion. The server
is fastmail.fm, by the way.

I asked them about it, and they claim that the IMAP spec says that the
"Deleted" flag means that the mail should be deleted, thus they're right
to do it. Further, they said that it's uncommon and wrong to set the
Deleted flag on the mail in the Trash folder that is intended to be
kept.

I just looked at the IMAP spec. It says that any message with that flag
is deleted when an Expunge command. So I don't know who's right here.
Questions:

1. Does anybody know if mail servers are supposed to be sending the IMAP
Expunge command? Do most of them do it? If so, mu users should be
seeing disappearing mail in the Trash. I see at least one mailing
list post about mysteriously disappearing mail:

http://groups.google.com/group/mu-discuss/browse_frm/thread/cba4fe32958a5e1e/1cafd86fc81e9e53?lnk=gst&q=trashed#1cafd86fc81e9e53

I do know about the 'expunge' offlineimap setting, but I'd rather not
use it.

2. Is mu4e wrong to set the Deleted flag? Currently there's no setting
to prevent it from being added. Should there be? The flag is added
with the "+T" in mu4e-mark-execute-all() in mu4e-mark.el.

Thanks much.

dima

[Dirk-Jan C. Binnema]

Hi,

mu4e is not setting the 'deleted' flag, it's setting the 'trashed' flag;
as the maildir spec[1] says:

"Flag "T" (trashed): the user has moved this message to the trash;
the trash will be emptied by a later user action."

So, the issue here is that offlineimap interprets this as the "Deleted"
flag, and then the IMAP-server just uses that.

Now, maybe there is some option for offlineimap to change that
behaviour; alternatively you could just create a folder and move the
messages there instead of trashing them.

Best wishes,
Dirk.


[1] http://cr.yp.to/proto/maildir.html

--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

[Dima Kogan]

Hi Dirk.

The solution I'd like to see is a mu4e configuration setting to prevent
the Trashed maildir flag from being set. Would you consider merging a
patch that does this?

Thanks

dima

[Dirk-Jan C. Binnema]

On Mon, May 13 2013, Dima Kogan wrote:

> Dirk-Jan C. Binnema <dj...@djcbsoftware.nl> writes:

<snip>

>> mu4e is not setting the 'deleted' flag, it's setting the 'trashed' flag;
>> as the maildir spec[1] says:
>>
>> "Flag "T" (trashed): the user has moved this message to the trash;
>> the trash will be emptied by a later user action."
>>
>> So, the issue here is that offlineimap interprets this as the "Deleted"
>> flag, and then the IMAP-server just uses that.
>>
>> Now, maybe there is some option for offlineimap to change that
>> behaviour; alternatively you could just create a folder and move the
>> messages there instead of trashing them.
>>

>> [1] http://cr.yp.to/proto/maildir.html
>
>
> Hi Dirk.
>
> The solution I'd like to see is a mu4e configuration setting to prevent
> the Trashed maildir flag from being set. Would you consider merging a
> patch that does this?

Hmmm, I don't really see the point. Trashing without setting the T-flag
is the same as simply moving a message to a folder; why not just use
that?

Cheers,
Dirk.

[Dima Kogan]

>> The solution I'd like to see is a mu4e configuration setting to prevent
>> the Trashed maildir flag from being set. Would you consider merging a
>> patch that does this?
>
> Hmmm, I don't really see the point. Trashing without setting the T-flag
> is the same as simply moving a message to a folder; why not just use
> that?

From an interface viewpoint, currently I use the 'd' key to move mail to
the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
want. If my issue is very uncommon, then indeed I should just do that.

I have a suspicion, though, that this problem is common enough that a
setting would be useful. At the very least fastmail.fm users are
affected. If nobody else says they're affected on this thread, then
maybe it's not worth worrying about.

Thanks

[Joost Kremers]

I use fastmail.fm and I don't have a problem with the way that trashed mail is handled. Specifically, I'm wondering why you move mail to the trash and expect it to remain there. I mean, the trash mail folder is really what its name says: trash. Stuff you throw away. If you want to keep a mail, you shouldn't be putting it in the trash, right?

Or am I missing something?


--
Joost Kremers
Life has its moments

[Dima Kogan]

Maybe I've been doing it wrong all these years. My mail workflow is to
move all mail that I'm done with into the Trash, but to never actually
delete anything. I treat the "Trash" as an "archive", so to speak, and I
don't have any actual "Trash".

This has been working fine, in general. Some mail servers like to purge
your Trash for you, so I disable that.

Is this workflow very different from what others use? Do most people
have an "archive" and a "Trash"?

As for actually sending Expunge, I just tested two other mail servers I
have access to. Zimbra doesn't Expunge Deleted mail, at least not
immediately. Gmail apparently does, like fastmail.fm. So any gmail IMAP
users that use stock offlineimap and stock mu4e would see mail they
place into the Trash disappear.

dima

[James Ladan]

On Monday, 13 May 2013 15:52:04 UTC-7, Dima Kogan wrote:

From an interface viewpoint, currently I use the 'd' key to move mail to
the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
want. If my issue is very uncommon, then indeed I should just do that.

I have a suspicion, though, that this problem is common enough that a
setting would be useful. At the very least fastmail.fm users are
affected. If nobody else says they're affected on this thread, then
maybe it's not worth worrying about.

Gmail had the same issue for me. I had to move to trash instead of using 'd'.

[James Ladan]

On Monday, 13 May 2013 16:00:22 UTC-7, Joost Kremers wrote:

I use fastmail.fm and I don't have a problem with the way that trashed mail is handled. Specifically, I'm wondering why you move mail to the trash and expect it to remain there. I mean, the trash mail folder is really what its name says: trash. Stuff you throw away. If you want to keep a mail, you shouldn't be putting it in the trash, right?

Or am I missing something?

In the Gmail case, it automatically flushes things out after ~30 days, so I can trash to my heart's content and if I make a brain-dead move to trash, I can always recover it within that window. With the interaction between mu4e and offlineimap, the email is immediately purged and gone for good. Usually not so bad, but not what's expected for the normal Gmail workflow.

[Dirk-Jan C. Binnema]

It's not that I don't want to help you, but adding extra complexity here
doesn't seem warranted since mu4e allows you to work-around the
offlineimap behavior -- I'd be happy to include a note in the
documentation, and you can change the keybindings etc.

However, I think the best place to change this would be in offlineimap.

Best wishes,

[Mandar Mitra]

Dima Kogan wrote (Mon, May 13, 2013 at 04:15:20PM -0700):
> Is this workflow very different from what others use? Do most people
> have an "archive" and a "Trash"?

I can't speak for most people, but yes, "archive" and "trash" are
different for me (I use Gmail). Gmail keeps archived email in "All
Mail" and deleted mail in Trash. "Trash" seems an odd choice of name
for archive, no?

> As for actually sending Expunge, I just tested two other mail servers I
> have access to. Zimbra doesn't Expunge Deleted mail, at least not
> immediately. Gmail apparently does, like fastmail.fm. So any gmail IMAP
> users that use stock offlineimap and stock mu4e would see mail they
> place into the Trash disappear.

I use mutt + offlineimap and have bound 'd' (normally delete) to save
a message to Gmail's Trash. This way, I get behaviour that is consistent
with the web interface: "deleted" mail goes to Trash and remains there
for some days. Gmail takes care of actually removing it all together.

mandar.

[Alan Schmitt]

Dirk-Jan C. Binnema writes:

> Now, maybe there is some option for offlineimap to change that
> behaviour; alternatively you could just create a folder and move the
> messages there instead of trashing them.

I was having trouble with this a while back, so I put this in my
configuration:

,----
| (setq mu4e-maildir-shortcuts
| '( ...
| ("/Trash" . ?T)
| ...)
|
| (fset 'my-move-to-trash "mT")
| (define-key mu4e-headers-mode-map (kbd "d") 'my-move-to-trash)
| (define-key mu4e-view-mode-map (kbd "d") 'my-move-to-trash)
`----

Alan

[Steve Dowe]

Dirk-Jan C. Binnema writes:

> However, I think the best place to change this would be in offlineimap.

I am inclined to agree with this. The deletion behaviour is completely
mail server specific. The Dovecot IMAP server, for example, doesn't
automatically expunge trashed mail after a set period, whereas the
Courier IMAP server apparently does.

Mu sits "down the line" from IMAP behaviour. My experience with Mu,
offlineimap and Dovecot has been one of harmony - /almost/ no
inconsistencies at all... :D

--
Steve Dowe warp2.me/sd
Warp Universal warpuni.com

[Dima Kogan]

Dirk-Jan C. Binnema <dj...@djcbsoftware.nl> writes:

> On Tue, May 14 2013, Dima Kogan wrote:
>
>>>> The solution I'd like to see is a mu4e configuration setting to prevent
>>>> the Trashed maildir flag from being set. Would you consider merging a
>>>> patch that does this?
>>>
>>> Hmmm, I don't really see the point. Trashing without setting the T-flag
>>> is the same as simply moving a message to a folder; why not just use
>>> that?
>>
>> From an interface viewpoint, currently I use the 'd' key to move mail to
>> the trash. Indeed I can use 'm' instead, or tweak the lisp to do what I
>> want. If my issue is very uncommon, then indeed I should just do that.
>>
>> I have a suspicion, though, that this problem is common enough that a
>> setting would be useful. At the very least fastmail.fm users are
>> affected. If nobody else says they're affected on this thread, then
>> maybe it's not worth worrying about.
>
> It's not that I don't want to help you, but adding extra complexity here
> doesn't seem warranted since mu4e allows you to work-around the
> offlineimap behavior -- I'd be happy to include a note in the
> documentation, and you can change the keybindings etc.
>
> However, I think the best place to change this would be in offlineimap.

OK.

If you could include a documentation note that'd be great.

Thanks again.

dima