Draft messages and mbsync

[Tassilo Horn]

Hi all,

I wonder how drafts are to be managed sensibly with mu4e and mbsync. I
set `mu4e-drafts-folder' in my `mu4e-contexts' to different maildirs (by
mail account). What I observe so far is the following:

- When writing a mail for an extended period of time, many
instances/versions of the mail will show up in my drafts folder. A
quick tests suggest that a new instance will show up with every mbsync
run if the draft has been modified.

- When eventually sending off the mail, the drafts of that message seem
to stick around. Or possibly the latest instance might be deleted but
not the former instances.

- mbsync synchronizes the drafts folder as any other maildir and I can't
find any reference to handling drafts in a special way in its man
page.

- Some drafts are shown in mu4e but not in my mail providers web
interface. For example, currently mu4e shows

--8<---------------cut here---------------start------------->8---
09:08:21 DS Tassilo Horn Managing draft messages
09:08:21 DS Tassilo Horn \- ~ Managing draft messages
09:08:21 DS Tassilo Horn Draft messages and mbsync
2021-09-09 DS Tassilo Horn Re: master 163e305: Add possibility to override the default highlighting
--8<---------------cut here---------------end--------------->8---

but the last message "Re: master..." is not shown in the web
interface, see the attached screenshot.

[Malcolm Matalka]

> --
> You received this message because you are subscribed to the Google Groups "mu-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/mu-discuss/87sfycc1qu.fsf%40gnu.org.
>
> - The drafts maildir contents are:
>
> --8<---------------cut here---------------start------------->8---
> ./new
> ./.uidvalidity
> ./cur
> ./cur/1631257701.d25e89f020f4def9.thinkpad-t440p,U=476:2,DS
> ./cur/1631257701.d25e89f020f4def9.thinkpad-t440p,U=477:2,DS
> ./cur/#1631220410.8d58d570d206a639.thinkpad-t440p,U=474:2,DS#
> ./cur/1631257701.d25e89f020f4def9.thinkpad-t440p:2,DS
> ./cur/1631257701.d25e89f020f4def9.thinkpad-t440p,U=478:2,DS
> ./tmp
> ./.mbsyncstate
> --8<---------------cut here---------------end--------------->8---
>
> Aha, the "Re: master..." mail is the message with # at the beginning
> and end. Possibly mbsync recognizes that as auto-save file and omits
> it from sync which would explain that it doesn't show up on the web
> interface.
>
> The other 4 files are different versions of this message where the one
> without U= seems to be the most current version.
>
> So does anyone have a configuration for mu4e and/or mbsync which makes
> draft handling a bit more sensible? I'm actually not sure if I dislike
> having multiple versions of a single mail. Maybe I consider can that a
> feature. However, when sending off a message, I would like to have all
> draft versions be deleted.

I am running into the same issue that you are. I end up deleting them
by hand in the Gmail interface because I'm not sure what else to do.

>
> Bye,
> Tassilo

[Daniel Fleischer]

Tassilo Horn <ts...@gnu.org> writes:

> So does anyone have a configuration for mu4e and/or mbsync which makes
> draft handling a bit more sensible? I'm actually not sure if I dislike
> having multiple versions of a single mail. Maybe I consider can that a
> feature. However, when sending off a message, I would like to have all
> draft versions be deleted.

I had the same experience; moreover, once the email sent wasn't the most
up-to-date draft but another "temporary" draft which was half-written!

At this point I just decided to NOT sync the draft folder, i.e. remove
it from the mbsync configuration. So now there is only one draft to is
being saved. After sending the email the draft is removed correctly.

The only downside is not having to start writing an email on one device
and continuing on another.

Best,

--

Daniel Fleischer

[Tassilo Horn]

Daniel Fleischer <danf...@gmail.com> writes:

>> So does anyone have a configuration for mu4e and/or mbsync which
>> makes draft handling a bit more sensible? I'm actually not sure if I
>> dislike having multiple versions of a single mail. Maybe I consider
>> can that a feature. However, when sending off a message, I would
>> like to have all draft versions be deleted.
>
> I had the same experience; moreover, once the email sent wasn't the
> most up-to-date draft but another "temporary" draft which was
> half-written!

Yes, that's possible, I didn't test exactly which one gets deleted.

> At this point I just decided to NOT sync the draft folder, i.e. remove
> it from the mbsync configuration. So now there is only one draft to is
> being saved. After sending the email the draft is removed correctly.
>
> The only downside is not having to start writing an email on one
> device and continuing on another.

I very seldomly do that but enjoy the good feeling of having a backup of
some important mail which gets written over an extended period of time.

Actually, the duplicate drafts issue is not too bothersome. I just
regularily clean up my drafts folder and that's it.

Bye,
Tassilo

[Nicolas P. Rougier (gmail)]

I personally use:

(defun my/mu4e-compose-hook ()
(auto-save-mode -1))
(add-hook 'mu4e-compose-mode-hook #'my/mu4e-compose-hook)

and manually save drafts when needed.

Nicolas

[Tassilo Horn]

"Nicolas P. Rougier (gmail)" <nicolas...@gmail.com> writes:

Hi Nicolas,

> I personally use:
>
> (defun my/mu4e-compose-hook ()
> (auto-save-mode -1))
> (add-hook 'mu4e-compose-mode-hook #'my/mu4e-compose-hook)
>
> and manually save drafts when needed.

I get that behaviour also without auto-save. It's enough that mbsync
runs between two saves, no matter how those have been triggered.

I guess, I'll better ask on the mbsync list.

Bye,
Tassilo

[Nicolas P. Rougier (gmail)]

I remember having the behavior you described (many drafts of the
same mail) and I somehow managed to fix it. I thought the
auto-save was the solution but maybe I'm mistaken. But then, I
don't remember what else I changed. I'll search.

Nicolas


Tassilo Horn <ts...@gnu.org> writes:

[Benjamin Slade]

I've also been trying to deal with this issue recently. I'm wary of disabling auto-saving, so I've done two things:

(1) Keep mbsync from syncing them:

--8<---------------cut here---------------start------------->8---
;; based on: https://emacs.stackexchange.com/a/24430
(defun draft-auto-save-buffer-name-handler (operation &rest args)
"for `make-auto-save-file-name' set '.' in front of the file name; do nothing for other operations"
(if
(and buffer-file-name (eq operation 'make-auto-save-file-name))
(concat (file-name-directory buffer-file-name)
"."
(file-name-nondirectory buffer-file-name))
(let ((inhibit-file-name-handlers
(cons 'draft-auto-save-buffer-name-handler
(and (eq inhibit-file-name-operation operation)
inhibit-file-name-handlers)))
(inhibit-file-name-operation operation))
(apply operation args))))

(setq bms/mu4e-mailbox-list '("myunimail-maildir" "myothermail-maildir" "yetanothermail-maildir" "whatsthis-maildir" "etcetc-maildir"))

(setq bms-draft-dotting nil)

(defun bms/draft-dot ()
(when (null bms-draft-dotting)
(dolist (mbox bms/mu4e-mailbox-list)
(let ((draft (concat mbox "/Drafts/cur/")))
(add-to-list 'file-name-handler-alist `(,draft . draft-auto-save-buffer-name-handler))))
(setq bms-draft-dotting t)))

(add-hook 'mu4e-compose-mode-hook #'bms/draft-dot)
--8<---------------cut here---------------end--------------->8---

(=draft-auto-save-buffer-name-handler= seems to get clobbered by something else during init, so I just delay setting this until the first time I compose a mail to get round this.)

This prevents remote mailboxes from getting cluttered.


(2) When I know I don't have any drafts I need, run:

--8<---------------cut here---------------start------------->8---
(defun bms/delete-all-mu4e-drafts ()
(interactive)
(shell-command "mu find flag:draft --fields \"l\" | xargs rm"))
--8<---------------cut here---------------end--------------->8---

to clean things up.

--
'(Benjamin Slade ("he/him") ( https://lambda-y.net )
`(pgp_fp: ,(21BA 2AE1 28F6 DF36 110A 0E9C A320 BBE8 2B52 EE19))
"sent by mu4e 1.6.6 in GNU Emacs 28.0.50 on Arch Linux")

[Tassilo Horn]

Tassilo Horn <ts...@gnu.org> writes:

Hi all,

> I guess, I'll better ask on the mbsync list.

And so I did [1], and this is the solution: just run

mdconvert --alt path/to/Drafts

for all your draft maildirs. That will change the UID storage scheme
from the native one where UIDs are encoded in the filename (the U=XXX
part) to an alternative one where the message's UIDs are stored in a
maildir-local BerkeleyDB. In the latter case, the message filenames
won't be changed.

Seems to work like a charm.

Bye,
Tassilo

[1] https://sourceforge.net/p/isync/mailman/isync-devel/thread/87sfy6bf60.fsf%40gnu.org/#msg37350843

[Tim Cross]

Thanks Tassilo, that is a great bit of information. Trying it out now.

[Husain Alshehhi]

"Tassilo Horn" <ts...@gnu.org> writes:

> And so I did [1], and this is the solution: just run
>
> mdconvert --alt path/to/Drafts
>
> for all your draft maildirs. That will change the UID storage scheme
> from the native one where UIDs are encoded in the filename (the U=XXX
> part) to an alternative one where the message's UIDs are stored in a
> maildir-local BerkeleyDB. In the latter case, the message filenames
> won't be changed.
>
> Seems to work like a charm.

Thanks Tassilo. I am also having the same issue as the one you
described.

But does this mean that this problem of syncing the draft directory is
because of the storage scheme that mbsync uses, vs the storage scheme
that mu4e uses, with respect to the UID?

I am not very familiar with UID and mail storage, but it seems to me
that this can be resolved permanently if mu4e stores drafts in the same
convention that mbsync uses. Is that correct?

--
Husain

[Tassilo Horn]

Husain Alshehhi <hus...@alshehhi.io> writes:

Hi Husain,

>> And so I did [1], and this is the solution: just run
>>
>> mdconvert --alt path/to/Drafts
>>
>> for all your draft maildirs. That will change the UID storage scheme
>> from the native one where UIDs are encoded in the filename (the U=XXX
>> part) to an alternative one where the message's UIDs are stored in a
>> maildir-local BerkeleyDB. In the latter case, the message filenames
>> won't be changed.
>>
>> Seems to work like a charm.
>
> Thanks Tassilo. I am also having the same issue as the one you
> described.
>
> But does this mean that this problem of syncing the draft directory is
> because of the storage scheme that mbsync uses, vs the storage scheme
> that mu4e uses, with respect to the UID?

Yes, kind of. It seems mu4e simply has no UID storage scheme, i.e., it
doesn't generate a new UID for messages it stored itself but relies on
whichever tool is used for populating the maildirs (e.g., offlineimap,
mbsync, whatever).

The only two scenarios where mu4e stores new mail itself are drafts and
copying/moving messages and here mbsync advises to use

(setq mu4e-change-filenames-when-moving t)

which will reliably make mbsync generate a new UID for that mail.

> I am not very familiar with UID and mail storage, but it seems to me
> that this can be resolved permanently if mu4e stores drafts in the
> same convention that mbsync uses. Is that correct?

Oswald, the mbsync maintainer, mentioned that as another possibility,
yes. However, he also said that doing it wrong (doubly assigned UIDs)
can lead to data loss. And to do it right, mu4e would need to check if
mbsync is used to sync a maildir, and additionaly guess the storage
scheme for each maildir. And probably it would need to check some
mbsync state files in order to find out the next valid UID where those
can reside either in ~/.mbsync/ or the maildirs, depending on mbsync
configuration...

So I guess it's a slippery slope and not really worth it given that the
hint "use the alternative UID storage scheme for draft folders" works
just fine.

Bye,
Tassilo