mu incredibly slow (index + display) and sometimes doesn't show message at all

[Nico Schottelius]

Hello everyone,

first of all thanks for writing this nice piece of software! It is nicely integrated into emacs and does a nice job with the xapian backend.

However, I have the problem that re-indexing takes > 30s and sometimes reading a message also 10-20s or an email is not even displayed after minutes of waiting.

My maildir details:

[19:47] wurzel:Maildir% find * -type f | wc -l

1225077

[19:48] wurzel:Maildir% du . | sort -g | tail -n 2

12803000 ./.notmuch

38959224 .

So roughly 26 GiB of emails in roughly 1.2 million files. I assume this should probably be pretty average for any maildir of ~20 years age.

What can i do to

a) speedup indexing

b) speedup displaying

c) ensuring a mails is eventually displayed?

Best regards,

Nico

Relevant packages:

[19:51] wurzel:Maildir% pacman -Q mu-git emacs xapian-core linux

mu-git v0.9.16.126.g5228e24-1

emacs 24.5-4

xapian-core 1:1.2.23-1

linux 4.5.4-1

[Eduardo Mercovich]

Hi Nico.

> [...] I have the problem that re-indexing takes > 30s and sometimes

> reading a message also 10-20s or an email is not even displayed after
> minutes of waiting.

> My maildir details: [...]

What machine (CPU and RAM) is this indexing being done?

I have 22.5 Gb in my Mailbox too, and while a small amount is not
indexed (old accounts), it takes only a few seconds. But it is a new
machine...

Best.


--
eduardo mercovich

Donde se cruzan tus talentos
con las necesidades del mundo,
ahí está tu vocación.

[Nico Schottelius]

Hy Eduardo,

I'm indexing on a thinkpad x1 carbon (2015). Here are some of the specs:

-  4 cores Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz
- 8 GiB RAM
- SSD: SAMSUNG MZHPV512HDGL-000L1

I know it might not be the most recent system, however even running an index without any change just after running an index before takes 14s:


[0:55] wurzel:~% date; time mu  index -v; date
Tue Jul  5 00:55:37 KST 2016
indexing messages under /home/users/nico/Maildir [/home/users/nico/.mu/xapian]
| processing mail; processed: 1224375; updated/new: 0, cleaned-up: 0
cleaning up messages [/home/users/nico/.mu/xapian]
- processing mail; processed: 1224404; updated/new: 0, cleaned-up: 0
elapsed: 4 second(s), ~ 306101 msg/s
\ processing mail; processed: 1224404; updated/new: 0, cleaned-up: 0
elapsed: 14 second(s), ~ 87457 msg/s
mu index -v  6.49s user 7.31s system 97% cpu 14.193 total
Tue Jul  5 00:55:51 KST 2016


Attached below are some more specs.

I'm btw. running ext4 on top of dm-crypt - not sure if this is might be a problem?

Best regards,

Nico



[0:50] wurzel:admininstrativ% free -m
              total        used        free      shared  buff/cache   available
Mem:           7889        2599         439         354        4849        4540
Swap:          4095           4        4091
[0:51] wurzel:admininstrativ%

[0:46] wurzel:admininstrativ% cat /proc/cpuinfo
processor    : 0
vendor_id    : GenuineIntel
cpu family    : 6
model        : 61
model name    : Intel(R) Core(TM) i7-5500U CPU @ 2.40GHz
stepping    : 4
microcode    : 0x16
cpu MHz        : 2400.093
cache size    : 4096 KB
physical id    : 0
siblings    : 4
core id        : 0
cpu cores    : 2
apicid        : 0
initial apicid    : 0
fpu        : yes
fpu_exception    : yes
cpuid level    : 20
wp        : yes
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch epb intel_pt tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx smap xsaveopt dtherm ida arat pln pts
bugs        :
bogomips    : 4791.73
clflush size    : 64
cache_alignment    : 64
address sizes    : 39 bits physical, 48 bits virtual
power management:

....

[00:52:19] wurzel:~# smartctl -a /dev/sda
smartctl 6.5 2016-05-07 r4318 [x86_64-linux-4.6.3-1-ARCH] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Device Model:     SAMSUNG MZHPV512HDGL-000L1
Serial Number:    S1WUNYAG204857
LU WWN Device Id: 5 002538 900000000
Firmware Version: BXW22L0Q
User Capacity:    512,110,190,592 bytes [512 GB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    Solid State Device
Device is:        Not in smartctl database [for details use: -P showall]
ATA Version is:   ACS-2, ATA8-ACS T13/1699-D revision 4c
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Jul  5 00:52:27 2016 KST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

[Eduardo Mercovich]

Hi Nico.

> I'm indexing on a thinkpad x1 carbon (2015). Here are some of the

> specs: [...]

Similar to mine, so it shouldn't take that long.

> I'm btw. running ext4 on top of dm-crypt - not sure if this is might
> be a problem?

No idea, but it certainly could if it has to decrypt thousands of small
files to read... Try to check if other big read operations may take time
on that same encrypted filesystem and let's see. :)

Best...

[Nico Schottelius]

Hey Eduardo,

usually things are a bit faster - or let's say it that way: reading 32k messages with mutt from the maildir takes roughly 2-3 seconds. As far as I know mutt reads all messages to be able to display the headers.

Is there any way I can debug the slowness of mu here? I was wondering if this is a problem of xapian or mu, because I see that notmuch is indeed much faster in indexing when there are no or almost no new mails:

[19:13] wurzel:~% date; notmuch new >/dev/null; date 

Sun Jul 10 19:13:16 CEST 2016

Sun Jul 10 19:13:17 CEST 2016

comparing this with mu looks rather slow:

[19:14] wurzel:~% date; mu index >/dev/null; date 

Sun Jul 10 19:14:58 CEST 2016

Sun Jul 10 19:15:25 CEST 2016

(I've repeated both index commands three times to verify the time difference)

What do you think about this? Expected results or is there a way to speedup mu?

[Eduardo Mercovich]

Hi Nico.

> usually things are a bit faster - or let's say it that way: reading
> 32k messages with mutt from the maildir takes roughly 2-3 seconds. As
> far as I know mutt reads all messages to be able to display the
> headers.

In the same filesystem and machine?

> Is there any way I can debug the slowness of mu here? [...]

It beats me... someone else have an idea here? :)

[Nico Schottelius]

Same machine, same folder, same files. Otherwise I wouldn't raise my concern about mu's slowness.

[Dirk-Jan C. Binnema]

Hi Nico,

On Thursday Jun 23 2016, Nico Schottelius wrote:

> Hello everyone,
>
> first of all thanks for writing this nice piece of software! It is nicely
> integrated into emacs and does a nice job with the xapian backend.
>
> However, I have the problem that re-indexing takes > 30s and sometimes
> reading a message also 10-20s or an email is not even displayed after
> minutes of waiting.

When the mu-server (the daemon part with which mu4e communicates) is
busy indexing, it cannot cough up messages, so that's probably the reason.

> My maildir details:
>
> [19:47] wurzel:Maildir% find * -type f | wc -l
> 1225077
>
> [19:48] wurzel:Maildir% du . | sort -g | tail -n 2
> 12803000 ./.notmuch
> 38959224 .
>
> So roughly 26 GiB of emails in roughly 1.2 million files. I assume this
> should probably be pretty average for any maildir of ~20 years age.

It's somwhat on the high side of mail corpa I have seen being used with
mu/mu4e, so it's interesting use case!

> What can i do to
>
> a) speedup indexing
> b) speedup displaying
> c) ensuring a mails is eventually displayed?

I'd check out the mu-index manpage and look for '.noupdate'; if a) is
faster, I suspect b) and c) will do so as well.

Cheers,
Dirk.


--
Dirk-Jan C. Binnema Helsinki, Finland
e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl
pgp: D09C E664 897D 7D39 5047 A178 E96A C7A1 017D DA3C

[Nico Schottelius]

Hello Dirk,

I'll give .noupdate a try these days. However do you have any idea why updating with the same backend as notmuch is significantly slower with mu?

I feel a bit uneasy about having to decide manually what directories to have a look at, because in theory mu should be able to figure this out by itself by stat'ing the directory times, or am I mistaken?

I was also wondering if xapian is actually the right choice... I was even wondering if a reasonable good import to postgres might actually be even faster?

Best regards,

Nico

e:d...@djcbsoftware.nl           w:www.djcbsoftware.nl

[Dirk-Jan C. Binnema]

On Wednesday Jul 20 2016, Nico Schottelius wrote:

> Hello Dirk,
>
> I'll give .noupdate a try these days. However do you have any idea why
> updating with the same backend as notmuch is significantly slower with mu?

Dunno - have you done any analysis/profiling?

> I feel a bit uneasy about having to decide manually what directories to
> have a look at, because in theory mu should be able to figure this out by
> itself by stat'ing the directory times, or am I mistaken?

There are some corner cases where that doesn't work, such as with
certain file systems or when message were modified (rather than
added/removed) - so we use the dir time-stamps to figure that out, and
if needed, re-index updated files.

Now, perhaps we could be a bit laxer and ignore those corner cases, but
I haven't bothered with that.

,----
| (~) % time mu index
| indexing messages under /home/djcb/Maildir [/home/djcb/.mu/xapian]
| | processing mail; processed: 72675; updated/new: 0, cleaned-up: 0
| cleaning up messages [/home/djcb/.mu/xapian]
| - processing mail; processed: 72695; updated/new: 0, cleaned-up: 0
| elapsed: 0 second(s)
| \ processing mail; processed: 72695; updated/new: 0, cleaned-up: 0
| elapsed: 1 second(s), ~ 72695 msg/s
| /home/djcb/Sources/mu/mu/mu index 0,55s user 0,38s system 80% cpu 1,165 total
`----

> I was also wondering if xapian is actually the right choice... I was even
> wondering if a reasonable good import to postgres might actually be even
> faster?

That's unlikely. But sure, you can give it a try.

Kind regards,

Dirk.


--
Dirk-Jan C. Binnema Helsinki, Finland

e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl

[Nico Schottelius]

Hello,

I've just tried the .noupdate option for almost *all* my top level maildirs, besides 4-5 of them.

Running mu index now results in 6-7s run time.

In comparison: running untuned notmuch results in <1s run time (when there is nothing to index).

I think .noupdate helps, but overall mu, even with this tuning looks to be at least 6 times to slow.

I haven't checked out mu vs. notmuch code yet, but I wonder if there is a specific reason for mu to be valid 6 times slower with the same backend as notmuch? (i.e. is mu more sane? does it do something substantially different?)

Test results are attached below.

Best regards,

Nico

[19:37] wurzel:Maildir% i=1; while [ $i -lt 4 ]; do  date; mu index; date; i=$((i+1)); done

Thu Jul 28 19:38:03 CEST 2016

indexing messages under /home/users/nico/Maildir [/home/users/nico/.mu/xapian]

| processing mail; processed: 79575; updated/new: 0, cleaned-up: 0

cleaning up messages [/home/users/nico/.mu/xapian]

\ processing mail; processed: 1225658; updated/new: 0, cleaned-up: 0

elapsed: 4 second(s), ~ 306414 msg/s

| processing mail; processed: 1225658; updated/new: 0, cleaned-up: 0

elapsed: 5 second(s), ~ 245131 msg/s

Thu Jul 28 19:38:09 CEST 2016

Thu Jul 28 19:38:09 CEST 2016

indexing messages under /home/users/nico/Maildir [/home/users/nico/.mu/xapian]

| processing mail; processed: 79575; updated/new: 0, cleaned-up: 0

cleaning up messages [/home/users/nico/.mu/xapian]

\ processing mail; processed: 1225658; updated/new: 0, cleaned-up: 0

elapsed: 4 second(s), ~ 306414 msg/s

| processing mail; processed: 1225658; updated/new: 0, cleaned-up: 0

elapsed: 5 second(s), ~ 245131 msg/s

Thu Jul 28 19:38:16 CEST 2016

Thu Jul 28 19:38:16 CEST 2016

indexing messages under /home/users/nico/Maildir [/home/users/nico/.mu/xapian]

| processing mail; processed: 79575; updated/new: 0, cleaned-up: 0

cleaning up messages [/home/users/nico/.mu/xapian]

\ processing mail; processed: 1225658; updated/new: 0, cleaned-up: 0

elapsed: 5 second(s), ~ 245131 msg/s

| processing mail; processed: 1225658; updated/new: 0, cleaned-up: 0

elapsed: 6 second(s), ~ 204276 msg/s

Thu Jul 28 19:38:22 CEST 2016

[19:38] wurzel:Maildir% i=1; while [ $i -lt 4 ]; do  date; notmuch new ; date; i=$((i+1)); done     

Thu Jul 28 19:39:07 CEST 2016

Note: Ignoring non-mail file: /home/users/nico/Maildir/behoerden/.noupdate

Note: Ignoring non-mail file: /home/users/nico/Maildir/bildung/.noupdate

Note: Ignoring non-mail file: /home/users/nico/Maildir/computer/.noupdate

Processed 710 total files in 7s (94 files/sec.).

Added 652 new messages to the database.

Thu Jul 28 19:39:16 CEST 2016

Thu Jul 28 19:39:16 CEST 2016

Note: Ignoring non-mail file: /home/users/nico/Maildir/behoerden/.noupdate

Note: Ignoring non-mail file: /home/users/nico/Maildir/bildung/.noupdate

Note: Ignoring non-mail file: /home/users/nico/Maildir/computer/.noupdate

Error reading file /home/users/nico/Maildir/drafts/cur/.#1469221924.abb1ee5e92439c78.wurzel:2,DS: No such file or directory

Processed 3 total files in almost no time.

No new mail.

Note: A fatal error was encountered: Something went wrong trying to read or write a file

Thu Jul 28 19:39:17 CEST 2016

Thu Jul 28 19:39:17 CEST 2016

Note: Ignoring non-mail file: /home/users/nico/Maildir/behoerden/.noupdate

Note: Ignoring non-mail file: /home/users/nico/Maildir/bildung/.noupdate

Note: Ignoring non-mail file: /home/users/nico/Maildir/computer/.noupdate

Error reading file /home/users/nico/Maildir/drafts/cur/.#1469221924.abb1ee5e92439c78.wurzel:2,DS: No such file or directory

Processed 3 total files in almost no time.

No new mail.

Note: A fatal error was encountered: Something went wrong trying to read or write a file

Thu Jul 28 19:39:17 CEST 2016

e:d...@djcbsoftware.nl           w:www.djcbsoftware.nl

[Dirk-Jan C. Binnema]

On Thursday Jul 28 2016, Nico Schottelius wrote:

> Hello,
>
> I've just tried the .noupdate option for almost *all* my top level
> maildirs, besides 4-5 of them.
>
> Running mu index now results in 6-7s run time.
>
> In comparison: running untuned notmuch results in <1s run time (when there
> is nothing to index).
>
> I think .noupdate helps, but overall mu, even with this tuning looks to be
> at least 6 times to slow.
>
> I haven't checked out mu vs. notmuch code yet, but I wonder if there is a
> specific reason for mu to be valid 6 times slower with the same backend as
> notmuch? (i.e. is mu more sane? does it do something substantially
> different?)

See the earlier message; I suspect the main reason is mu being rather
thorough with its scanning.

Anyway, there are some recent options in mu-git to influence that; check:
https://groups.google.com/forum/#!topic/mu-discuss/0Uyu0It1du4
for some more detail.

Cheers,

Dirk.

--
Dirk-Jan C. Binnema Helsinki, Finland

e:dj...@djcbsoftware.nl w:www.djcbsoftware.nl