Indexing of sent mail

[Yuri D'Elia]

There's this behavior I'm experiencing that I'd like to improve.
It's quite reliable for all sent mail, although I'm not 100% sure of
what's exactly going on:

- Send email
- The email appears instantly in my headers view and selecting it works
fine as well
- Background re-indexing kicks-in
- Email is still in the headers view, but selecting it causes an empty
buffer to display
- Refresh the headers view
- Email is gone
- Background re-indexing kicks-in
- Email is there again as it should be

So it seems that the message is somehow added to the index, but the
location of the message itself changes, resulting in the message to
disappear temporarily, until reindexed.

The only way I can think of this behavior is mbsync moving the actual
message from one filename to another (for example, to change flags as
the sync is happening and no reindexing has occured yet).

And yes, I'm running mbsync outside of mu4e. So a reindex while a sync
is in progress is definitely possible.

Back to the question: is there anything I'm missing that could improve
this phenomenon?

[Dirk-Jan C. Binnema]

Hi Yuri,

On Friday Jan 13 2017, Yuri D'Elia wrote:

> There's this behavior I'm experiencing that I'd like to improve.
> It's quite reliable for all sent mail, although I'm not 100% sure of
> what's exactly going on:
>
> - Send email
> - The email appears instantly in my headers view and selecting it works
> fine as well
> - Background re-indexing kicks-in
> - Email is still in the headers view, but selecting it causes an empty
> buffer to display
> - Refresh the headers view
> - Email is gone
> - Background re-indexing kicks-in
> - Email is there again as it should be
>
> So it seems that the message is somehow added to the index, but the
> location of the message itself changes, resulting in the message to
> disappear temporarily, until reindexed.
>
> The only way I can think of this behavior is mbsync moving the actual
> message from one filename to another (for example, to change flags as
> the sync is happening and no reindexing has occured yet).

If this is indeed the problem, it'd be a bit hard to fully solve -- mu4e
depends on the database being synchronized with that's in your maildir;
if you change the maildir outside mu4e, it will be out of sync until the
next indexing operation.

> And yes, I'm running mbsync outside of mu4e. So a reindex while a sync
> is in progress is definitely possible.

You might be able to improve things by running mbsync from mu4e, so it
knows when things have changed.

If you also want run it outside mu4e, you could have a scripts that only
does its thing if it notices that mu4e is not running (check for a 'mu
server' process).

Kind regards,
Dirk.

--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

[Yuri D'Elia]

On Sat, Jan 14 2017, Dirk-Jan C. Binnema wrote:
>> And yes, I'm running mbsync outside of mu4e. So a reindex while a sync
>> is in progress is definitely possible.
>
> You might be able to improve things by running mbsync from mu4e, so it
> knows when things have changed.

I actually moved it out to have more control of the syncing behavior, so
I'd like to keep it that way.

> If you also want run it outside mu4e, you could have a scripts that
> only does its thing if it notices that mu4e is not running (check for
> a 'mu server' process).

I could at least reduce the time this can happen by triggering mu4e to
update when mbsync is done.

Assuming mu4e is running the server, can I trigger a re-index remotely?
pkill -USR1 mu ? ;)

[Dirk-Jan C. Binnema]

On Saturday Jan 14 2017, Yuri D'Elia wrote:

> On Sat, Jan 14 2017, Dirk-Jan C. Binnema wrote:
>>> And yes, I'm running mbsync outside of mu4e. So a reindex while a sync
>>> is in progress is definitely possible.
>>
>> You might be able to improve things by running mbsync from mu4e, so it
>> knows when things have changed.
>
> I actually moved it out to have more control of the syncing behavior, so
> I'd like to keep it that way.

That's fine, but that interferes with mu4e doing its job.

>> If you also want run it outside mu4e, you could have a scripts that
>> only does its thing if it notices that mu4e is not running (check for
>> a 'mu server' process).
>
> I could at least reduce the time this can happen by triggering mu4e to
> update when mbsync is done.
>
> Assuming mu4e is running the server, can I trigger a re-index remotely?
> pkill -USR1 mu ? ;)

We don't really know for sure what causes the problem you saw, so we
need to be careful with conclusions.

With mu4e, one way trigger a re-index would be trigger your emacs/mu4e
to do so using some IPC-mechanism; for instance by registering a
DBus-method if you emacs supports that.

But in general, I'd recommend using mu4e's built-in mechanisms, it'll
cause the least amount of tears in the long run, I suspect.

[Yuri D'Elia]

On Sat, Jan 14 2017, Dirk-Jan C. Binnema wrote:
> With mu4e, one way trigger a re-index would be trigger your emacs/mu4e
> to do so using some IPC-mechanism; for instance by registering a
> DBus-method if you emacs supports that.

Didn't think of dbus, I'll see how painful this turns out to be.

> But in general, I'd recommend using mu4e's built-in mechanisms, it'll
> cause the least amount of tears in the long run, I suspect.

I fully accept the limitations involved here.
Now, let me sob in the corner alone while I read about the dbus interface...

[Yuri D'Elia]

On Sat, Jan 14 2017, Dirk-Jan C. Binnema wrote:
> We don't really know for sure what causes the problem you saw, so we
> need to be careful with conclusions.

I'm now refreshing via mu4e using a dbus hook, which turned out to be
simple enough:

(require 'dbus)

(defun my4e~dbus-update-index ()
(mu4e-update-index)
:ignore)

(dbus-register-method :session "nl.djcbsoftware.mu4e"
"/nl/djcbsoftware/mu4e" "nl.djcbsoftware.mu4e"
"updateIndex" 'my4e~dbus-update-index)

triggered, after sync, with:

dbus-send --session --type=method_call --dest="nl.djcbsoftware.mu4e" \
"/nl/djcbsoftware/mu4e" "nl.djcbsoftware.mu4e.updateIndex"

as such, I've disabled the built-in update timer (not needed anymore).
This is much more responsive, since as soon as the cron sync succeeds,
the index is refreshed and new mail shows instantly.

Not so much crying in the corner after all...

*BUT*

the problem still persists for sent mail, despite this.

Since the index is refreshed just after syncing, mu should always see
a consistent maildir state.

[Dirk-Jan C. Binnema]

On Monday Jan 23 2017, Yuri D'Elia wrote:

> On Sat, Jan 14 2017, Dirk-Jan C. Binnema wrote:
>> We don't really know for sure what causes the problem you saw, so we
>> need to be careful with conclusions.
>
> I'm now refreshing via mu4e using a dbus hook, which turned out to be
> simple enough:
>
> (require 'dbus)
>
> (defun my4e~dbus-update-index ()
> (mu4e-update-index)
> :ignore)
>
> (dbus-register-method :session "nl.djcbsoftware.mu4e"
> "/nl/djcbsoftware/mu4e" "nl.djcbsoftware.mu4e"
> "updateIndex" 'my4e~dbus-update-index)
>
> triggered, after sync, with:
>
> dbus-send --session --type=method_call --dest="nl.djcbsoftware.mu4e" \
> "/nl/djcbsoftware/mu4e" "nl.djcbsoftware.mu4e.updateIndex"
>
> as such, I've disabled the built-in update timer (not needed anymore).
> This is much more responsive, since as soon as the cron sync succeeds,
> the index is refreshed and new mail shows instantly.
>
> Not so much crying in the corner after all...

Ah, this is still using mu4e's internal mechanism, so this is a clean
way to do this. So indeed, no tears needed!

> *BUT*
>
> the problem still persists for sent mail, despite this.
>
> Since the index is refreshed just after syncing, mu should always see
> a consistent maildir state.

Hmm, would be interesting to see where mu4e _thinks_ the message is and
where it actually is. mu4e only changes messages filenames when you
change their flags or move them somewhere.

[Yuri D'Elia]

On Tue, Jan 24 2017, Dirk-Jan C. Binnema wrote:
>> Since the index is refreshed just after syncing, mu should always see
>> a consistent maildir state.
>
> Hmm, would be interesting to see where mu4e _thinks_ the message is and
> where it actually is. mu4e only changes messages filenames when you
> change their flags or move them somewhere.

Any pointer on how turn on some extra debugging in this sense?

[Dirk-Jan C. Binnema]

Well, you could try `M-x mu4e-toggle-logging' and check the :path for
your message.

You could also try to index without mu4e running, and check for the
filename of some specific message -- at what point does it change?

[Yuri D'Elia]

On Tue, Jan 24 2017, Dirk-Jan C. Binnema wrote:
>> dbus-send --session --type=method_call --dest="nl.djcbsoftware.mu4e" \
>> "/nl/djcbsoftware/mu4e" "nl.djcbsoftware.mu4e.updateIndex"
>>
>> as such, I've disabled the built-in update timer (not needed anymore).
>> This is much more responsive, since as soon as the cron sync succeeds,
>> the index is refreshed and new mail shows instantly.
>>
>> Not so much crying in the corner after all...
>
> Ah, this is still using mu4e's internal mechanism, so this is a clean
> way to do this. So indeed, no tears needed!

I noticed that I do get a duplicated headers view from time to time with
this though.

It seems to happen whenever the update is triggered and there's a
*mu4e-view* buffer active.

Could you point roughly where this could happen so I can debug it?

[Yuri D'Elia]

On Wed, Feb 15 2017, Yuri D'Elia wrote:
> I noticed that I do get a duplicated headers view from time to time with
> this though.
>
> It seems to happen whenever the update is triggered and there's a
> *mu4e-view* buffer active.

Turns out, it's completely unrelated to my changes.

This happens because (get-buffer-window buf 'visible) considers frames
in another virtual desktop "invisible", causing mu4e to create a new
headers view at every refresh.

I created a PR to fix this:

https://github.com/djcb/mu/pull/1037