The dreaded "Waiting for message..." message and what to do about it

[Ævar Arnfjörð Bjarmason]

I know why it happens, because an indexing is happening in the background.

I've now set mu4e-update-interval after flip-flopping between that and
(* 60 3) for a while.

It's nice to never get this message anymore, but now I have to
occasionally C-c C-u to manually mu4e-update-mail-and-index, which is
the new thing I'm waiting on.

Would it be possible to make the "mu server" just abort indexing when
it gets other commands, and opportunistically index things in the
background?

I've had half a mind to just write something that occasionally kills
the "mu server" emacs has started if it hasn't been active in a bit
and then runs "mu index" in the background, and if emacs tries to
start "mu server" it'll also involve "killall mu" to kill the existing
index job, which might opportunistically start up again.

I think I can easily hack that up, but before I do, any suggestion for
a better "native" implementation? Maybe "mu server" could abort the
indexing if it gets interactive commands and then resume if after some
timeout?

[Dirk-Jan C. Binnema]

People have so little patience these days :)

You should be able to kill the mu server without any problems; mu4e will
automagically restart it when needed.

You can also just increase the interval for getting mail. Or you could
do it completely outside mu4e, and do the whole mail-fetching process /
indexing using a script. Then you never have to wait again.

Kind regards,
Dirk.


--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

[Ævar Arnfjörð Bjarmason]

On Wed, Jul 20, 2016 at 9:36 PM, Dirk-Jan C. Binnema
<dj...@djcbsoftware.nl> wrote:
>
> On Wednesday Jul 20 2016, Ævar Arnfjörð Bjarmason wrote:
>
>> I know why it happens, because an indexing is happening in the background.
>>
>> I've now set mu4e-update-interval after flip-flopping between that and
>> (* 60 3) for a while.
>>
>> It's nice to never get this message anymore, but now I have to
>> occasionally C-c C-u to manually mu4e-update-mail-and-index, which is
>> the new thing I'm waiting on.
>>
>> Would it be possible to make the "mu server" just abort indexing when
>> it gets other commands, and opportunistically index things in the
>> background?
>>
>> I've had half a mind to just write something that occasionally kills
>> the "mu server" emacs has started if it hasn't been active in a bit
>> and then runs "mu index" in the background, and if emacs tries to
>> start "mu server" it'll also involve "killall mu" to kill the existing
>> index job, which might opportunistically start up again.
>>
>> I think I can easily hack that up, but before I do, any suggestion for
>> a better "native" implementation? Maybe "mu server" could abort the
>> indexing if it gets interactive commands and then resume if after some
>> timeout?
>
> People have so little patience these days :)

Indeed, although to be fair I have a huge mailbox and having opening
my next mail take ~10-30s due to reindexing is a drag.

It's not the time that it takes per-se, I spend plenty of time ~10-30s
away from my E-Mail, it's that it's happening *while* I'm
interactively browsing something.

> You should be able to kill the mu server without any problems; mu4e will
> automagically restart it when needed.

*nod*

> You can also just increase the interval for getting mail. Or you could
> do it completely outside mu4e, and do the whole mail-fetching process /
> indexing using a script. Then you never have to wait again.

The problem with doing it completely outside of mu4e is that mu4e
needs a write handle on the index for e.g. changing flags, which
conflicts with the indexing process, or is there some trick I'm
missing?

Now if I try to index in the background I'll get "Unable to get write
lock ... already locked", and the mu4e UI will be similarly blocked.

In any case, I've just implemented this proposed hack of mine, I sent
you a patch for a hook to run before the mu4e "mu server" process is
started: https://github.com/djcb/mu/pull/886

The hook I have is:

(add-hook
'mu4e~proc-start-hook
'(lambda ()
(message "Now running the 'killall mu' hook!")
(shell-command "killall mu")
(sleep-for 0 250)))

I.e. before I start "mu server" I "killall mu" and sleep for 250ms.
Then I wrote a mu-monitor.pl script I keep running in the background:
https://github.com/avar/dotemacs/blob/master/scripts/mu-monitor.pl

All this script does is start up, strace the mu process for 1 second,
and kill it if it looks inactive.

Then I have this running continuously in a screen session:

while sleep 30; do time mu index --maildir=~/Maildir/work
--max-msg-size=500000000; done

The end result is that now my mu4e is always responsive. If I e.g.
browse to the next E-Mail with "n" instead of waiting 10-30 seconds
like before it pops up in <1s.

I'm sure there's some race conditions here, in particular I've noticed
that I still get "Waiting for message..." if I open a message just as
mu is getting killed due to previous inactivity, but just re-pressing
<RET> re-opens it with a new process and fixes that, which is much
better than having to previously wait the 10-20 seconds before the
index finished.