Help debugging slow mu4e

[John Kitchin]

Hi,

I have been using mu4e for a long time, with great happiness, thanks for making it!

This summer I switched computers from a Macbook air to a Mac desktop, and I have noticed a lot of performance issues with mu4e, and I am not sure how to debug them. For example, there will often be many second delays in loading a message where I just see "Loading message...", and sometimes when jumping to my inbox, it takes many seconds to display the messages.

I am currently using version 1.4.13. 

There are ~61,000 messages in the store, but this worked fine with an older mu on my old macbook.

Any tips on how to go about figuring out what the pauses are?

Thanks,

[John Kitchin]

I should also note, it is not always slow; the pauses are often intermittent.

I use Catalina 10.15.6 on the Mac, and it has a 3 GHz-6 core processor with 16 GB of ram.

I am using Emacs version 27.1 

[Norman Tovey-Walsh]

John Kitchin <johnrk...@gmail.com> writes:
> I should also note, it is not always slow; the pauses are often
> intermittent.
>
> I use Catalina 10.15.6 on the Mac, and it has a 3 GHz-6 core processor with
> 16 GB of ram.
>
> I am using Emacs version 27.1

I have a very similar experience on my MacBook Pro running Catalina
10.15.5. I’ve never run mu4e on anything else, so I just assumed the
periodic slowness was a normal consequence of indexing or something.

Happy to help debug if I can (I’d love to have consistently snappy
performance!). I have wondered if the slowness is related to messages
with digital signatures that have to be verified, or if there’s some
network traffic involved (gravatar images maybe?)

But it’s hard to investigate because it’s intermittent (and not the sort
of thing I have any experience debugging in Emacs) and it hasn’t risen
to the top of my stack.

Be seeing you,
norm

--
Norman Tovey-Walsh <n...@nwalsh.com>
https://nwalsh.com/

> Things work out best for those who make the best of the way things work
> out.

[Joost Kremers]

On Tue, Aug 18 2020, John Kitchin wrote:
> This summer I switched computers from a Macbook air to a Mac
> desktop, and I
> have noticed a lot of performance issues with mu4e, and I am not
> sure how
> to debug them. For example, there will often be many second
> delays in
> loading a message where I just see "Loading message...", and
> sometimes when
> jumping to my inbox, it takes many seconds to display the
> messages.

I see this happening too, though not so often that it gets
annoying. I'm using a (fairly old) laptop (Core i5, 8GB RAM, HDD)
running Linux, with about 35000 messages in the store.

I don't recall it happening on my travel laptop, which has lower
specs (Pentium IIRC and 4GB RAM, SSD), but far fewer messages in
the store (about 3000).

I run mu/mu4e from git on both machines.

I get the impression that I see it primarily at start up, when
I've fired up Emacs and mu4e (usually the first things I do) but
other processes are still starting in the background, mainly cloud
sync and the update daemon. Since these processes would also be
accessing the disk a lot, I've always assumed that that's where
the problem lies: multiple processes requiring substantial disk
access at the same time.

--
Joost Kremers
Life has its moments

[Alexis]

John Kitchin <johnrk...@gmail.com> writes:

> For example, there will often be many second delays in
> loading a message where I just see "Loading message..."

On this particular issue, i've found that it only seems to happen
with messages which have .p7s digital signatures.


Alexis.

[Dirk-Jan C. Binnema]

Hi John,

For good or bad, I don't see any of these slowdowns (on Linux), anyway,
this may give some hints:
https://www.djcbsoftware.nl/code/mu/mu4e/Reading-messages.html#Opening-messages-is-slower-than-expected-_002d-why_003f

That should help a bit with diagnosing this.

Kind regards,
Dirk.

--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
gpg: 6987 9CED 1745 9375 0F14 DA98 11DD FEA9 DCC4 A036

[Christophe de Dinechin]

Hello,


I use four machines sync'd with the same accounts, 12 accounts, 133 mailboxes, roughly 580k messages. Two of the machines run Linux (a desktop and a laptop), two run macOS Catalina (a 16' new MacBook pro, a 15' 2017 MacBook pro).

I observed the same slowdowns on occasions on all configurations, so this is definitely not a mac-only thing. Every time, an update process was ongoing, which I could see by switching to hidden buffer " *mu4e-updates" (notice the leading space).

Sometimes, this would be a temporary slow-down during mail download (slow DSL), but in most cases. it was mu4e indexing. I ended up setting mu4e-index-lazy-check to t, and it helped quite a bit. I still occasionally have to wait for a message, but much less often than before.

mu4e-update-interval 421 ;; Roughly 7 minutes
mu4e-headers-skip-duplicates t ;; Don't record duplicates
mu4e-index-cleanup nil ;; Don't do a full cleanup check
mu4e-index-lazy-check t) ;; Don't consider up-to-date dirs

Downside: I now have to occasionally run mu index manually to get some cleanup done (usually some message not showing up where it should be).

Back to the symptoms: a full indexing takes ~250s, so clearly, if you index for 250s and refresh every 420s, there is a good chance you will be indexing ;-) The next step for me is to more clearly separate "archive" and "non-archive" boxes, and mark the archive mailboxes as "noindex".

By comparison, Apple Mail also sometimes shows "white" emails while it downloads them. There is no feedback, just an empty white window. I prefer mu4e being honest and telling me "please wait".


Hope this helps,
Christophe

PS: I'm curious: what is the command-line option for mu-index to do a lazy indexing, if any.

> --
> You received this message because you are subscribed to the Google Groups "mu-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/mu-discuss/87sgcfrwif.fsf%40djcbsoftware.nl.

[Dirk-Jan C. Binnema]

On Thursday Aug 27 2020, Christophe de Dinechin wrote:

> Hello,
>
>
> I use four machines sync'd with the same accounts, 12 accounts, 133
> mailboxes, roughly 580k messages. Two of the machines run Linux (a
> desktop and a laptop), two run macOS Catalina (a 16' new MacBook pro,
> a 15' 2017 MacBook pro).
>
> I observed the same slowdowns on occasions on all configurations, so
> this is definitely not a mac-only thing. Every time, an update process
> was ongoing, which I could see by switching to hidden buffer "
> *mu4e-updates" (notice the leading space).
>
> Sometimes, this would be a temporary slow-down during mail download (slow DSL), but in most cases. it was mu4e indexing. I ended up setting mu4e-index-lazy-check to t, and it helped quite a bit. I still occasionally have to wait for a message, but much less often than before.
>
> mu4e-update-interval 421 ;; Roughly 7 minutes
> mu4e-headers-skip-duplicates t ;; Don't record duplicates
> mu4e-index-cleanup nil ;; Don't do a full cleanup check
> mu4e-index-lazy-check t) ;; Don't consider up-to-date dirs
>
> Downside: I now have to occasionally run mu index manually to get some cleanup done (usually some message not showing up where it should be).
>

> Back to the symptoms: a full indexing takes ~250s, so clearly, if you
> index for 250s and refresh every 420s, there is a good chance you will
> be indexing ;-) The next step for me is to more clearly separate
> "archive" and "non-archive" boxes, and mark the archive mailboxes as
> "noindex".

To add to that, you really only need a 'full index' when setting up mu4e
(or upgrading to a version with a different database layout) -- so very
rarely.

For the 'normal' case, `mu index` should be very fast; in my case (88K
messages) it takes less than a second if there are no new messages, and
if there are some new messages, it doesn't take much longer; mu should
be able to index 100s of message per second -- and I guess most people
don't get 100s of messages each few minutes.

Getting those messages (with mbsync, offlineimap, ...) would take much
longer than that. But retrieval doesn't block mu4e.

Now, if `mu index` runs much slower than that, we should investigate!
First thing would be to check if `mu index' is slow or if this only
happens with mu4e.

> PS: I'm curious: what is the command-line option for mu-index to do a
> lazy indexing, if any.

`mu help index` is your friend --> '--lazy-check'

[Ben Thaler]

Dear Dirk, 

thank you for the hints! I will try to figure these things out.

--
You received this message because you are subscribed to the Google Groups "mu-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/mu-discuss/874kokbdhf.fsf%40djcbsoftware.nl.

[Christophe de Dinechin]

On 30 Aug 2020, at 13:05, Dirk-Jan C. Binnema <dj...@djcbsoftware.nl> wrote:

PS: I'm curious: what is the command-line option for mu-index to do a
lazy indexing, if any.

`mu help index` is your friend --> '--lazy-check'

Would it be possible for mu index -h to show that same message? That's what I used.

Thanks

Christophe

[John Kitchin]

I just wanted to follow up on this. I increased the RAM on my desktop to
64GB, and things have sped up pretty noticeably. I previously had 16GB,
but that was frequently hitting swap, which I guess could slow this
down. I still don't think it is as snappy as it used to be, but it is
back to functional.

Norman Tovey-Walsh <n...@nwalsh.com> writes:

> John Kitchin <johnrk...@gmail.com> writes:
>> I should also note, it is not always slow; the pauses are often
>> intermittent.
>>
>> I use Catalina 10.15.6 on the Mac, and it has a 3 GHz-6 core processor with
>> 16 GB of ram.
>>
>> I am using Emacs version 27.1
>
> I have a very similar experience on my MacBook Pro running Catalina
> 10.15.5. I’ve never run mu4e on anything else, so I just assumed the
> periodic slowness was a normal consequence of indexing or something.
>
> Happy to help debug if I can (I’d love to have consistently snappy
> performance!). I have wondered if the slowness is related to messages
> with digital signatures that have to be verified, or if there’s some
> network traffic involved (gravatar images maybe?)
>
> But it’s hard to investigate because it’s intermittent (and not the sort
> of thing I have any experience debugging in Emacs) and it hasn’t risen
> to the top of my stack.
>
> Be seeing you,
> norm


--

Professor John Kitchin
Doherty Hall A207F
Department of Chemical Engineering
Carnegie Mellon University
Pittsburgh, PA 15213
412-268-7803
@johnkitchin
http://kitchingroup.cheme.cmu.edu

[Eduardo Mercovich]

Hi John.

[...]

> I just wanted to follow up on this. I increased the RAM on my desktop

> to 64GB, and things have sped up pretty noticeably. [...] I still

> don't think it is as snappy as it used to be, but it is back to
> functional.

Don't know if it's related or not, but since a couple months ago my
emacs seems a tad slower too. Nothing serious, but noticeable.

Do you observe this in mu4e only, or the rest of the emacsverse?

I have GNU Emacs 27.1 (build 2, x86_64-pc-linux-gnu, GTK+ Version
3.24.5, cairo version 1.16.0) of 2020-09-03

Best...

--
Eduardo Mercovich

Donde se cruzan tus talentos
con las necesidades del mundo,
ahí está tu vocación.
(Anónimo)

[Norm Tovey-Walsh]

Eduardo Mercovich <eduardo....@gmail.com> writes:
> Don't know if it's related or not, but since a couple months ago my
> emacs seems a tad slower too. Nothing serious, but noticeable.

I’ve been seeing that for a while too. I most often notice it when I
type M-x and start to type something but the minibuffer isn’t ready. It
sometimes takes several seconds. Not sure what that’s about, or exactly
when it started, but it sure is annoying.

Be seeing you,
norm

--

Norman Tovey-Walsh <n...@nwalsh.com>
https://nwalsh.com/

> In every work of genius we recognize our own rejected thoughts; they
> come back to us with a certain alienated majesty.--Emerson

[Eduardo Mercovich]

Hi Norm.

>> Don't know if it's related or not, but since a couple months ago my

>> emacs seems a tad slower too. [...]
>
> I’ve been seeing that for a while too. [...]

What emacs and mu4e version do you have? And you John K.?

[Norm Tovey-Walsh]

Eduardo Mercovich <eduardo....@gmail.com> writes:
>>> Don't know if it's related or not, but since a couple months ago my
>>> emacs seems a tad slower too. [...]
>>
>> I’ve been seeing that for a while too. [...]
>
> What emacs and mu4e version do you have? And you John K.?

I don’t actually think this is mu4e related. I have my own build: GNU
Emacs 27.1 (build 2, x86_64-apple-darwin19.5.0, NS appkit-1894.50
Version 10.15.5 (Build 19F101)) of 2020-08-11

My mu4e is one I built from the 1.4.7 tag of g...@github.com:djcb/mu.git

Be seeing you,
norm

--
Norman Tovey-Walsh <n...@nwalsh.com>
https://nwalsh.com/

> My problem lies in reconciling my gross habits with my net
> income.--Errol Flynn