mu indexing performance appears much slower than Notmuch

[Johnny Utahh]

I'm comparing Notmuch and mu indexing performance. My tests show mu to be much slower. Am I potentially mis-configuring or mis-using mu?

https://github.com/johnnyutahh/search-based-email#core_choice

[Ken Mankoff]

Did you add the .noindex file after initial sync to old/archive/unchanging folders? I set up annual archives so this would work and it makes mu snappy. 

  -k. 

Please excuse brevity. Sent from pocket computer with tiny non-haptic feedback keyboard. 

On Jul 12, 2015, at 17:53, Johnny Utahh <johnnny...@gmail.com> wrote:

I'm comparing Notmuch and mu indexing performance. My tests show mu to be much slower. Am I potentially mis-configuring or mis-using mu?

https://github.com/johnnyutahh/search-based-email#core_choice

--
You received this message because you are subscribed to the Google Groups "mu-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mu-discuss+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Johnny Utahh]

I haven't. That's helpful thanks. I'm not sure if it will reduce majority of the performance difference, but it's definitely something. Can '.noindex' simply be a zero-sized file, and are there any docs for this (.noindex)?

[Ken Mankoff]

On 2015-07-12 at 18:50, Johnny Utahh <johnnny...@gmail.com> wrote:
> I haven't. That's helpful thanks. I'm not sure if it will reduce majority
> of the performance difference, but it's definitely something.

It should, but you haven't given information about what performance issues you have, so it is hard to know.

> '.noindex' simply be a zero-sized file

Yes.

> , and are there any docs for this (.noindex)?

There is mention of it. Try google. BUT that is the wrong file. You want .noupdate, not .noindex.

-k.

[Johnny Utahh]

> It should, but you haven't given information about what performance issues you have, so it is hard to know. 

Performance issues were noted; from earlier post:
https://github.com/johnnyutahh/search-based-email#core_choice
https://gist.github.com/johnnyutahh/f4e3d2d3fb07de5fa146 (embedded link at above page)

Summary: Notmuch performs approximately 15 times faster than mu.

I'm guessing my "non-changing" maildir content is possibly less than half of my total maildir content. If the performance scales linearly, then at best that makes the performance different 7x instead of 15x. Maybe there's more tricks I can employ to reduce this difference even further? 

> There is mention of it. Try google. BUT that is the wrong file. You want .noupdate, not .noindex. 

Ok, thanks. '.noindex' was referenced in original post, but I'm assuming '.noindex' was incorrect (and has no other use/function), and I'll use '.noupdate' instead. 

[Ken Mankoff]

On 2015-07-13 at 11:13, Johnny Utahh <johnnny...@gmail.com> wrote:
>> It should, but you haven't given information about what performance
>> issues you have, so it is hard to know.
>
> Performance issues were noted; from earlier post:
> https://github.com/johnnyutahh/search-based-email#core_choice
> https://gist.github.com/johnnyutahh/f4e3d2d3fb07de5fa146 (embedded
> link at above page)

Sorry didn't follow that link.

> I'm guessing my "non-changing" maildir content is possibly less than
> half of my total maildir content. If the performance scales linearly,
> then at best that makes the performance different 7x instead of 15x.
> Maybe there's more tricks I can employ to reduce this difference even
> further?

You have a mailbox with 500,000 messages that needs to be frequently updated and can't archive the bulk of it? Then, no, mu probably isn't the right tool for this setup.

I have 200k messages but on Jan 1 put the 10k emails from 2014 into a 2014_archive folder, index it, and then add the .noupdate file. I can search and read them almost instantly, and they show up appropriately in all views including ongoing threads which include those messages. Since I get/keep about 10k messages per year, half-way through 2015 I am now indexing 5000 messages. This takes ~1 second. In December my updates will take ~2s.

$ time mu index
indexing messages under /Users/mankoff/Maildir [/Users/mankoff/.mu/xapian]
| processing mail; processed: 5775; updated/new: 0, cleaned-up: 0
cleaning up messages [/Users/mankoff/.mu/xapian]
- processing mail; processed: 118607; updated/new: 0, cleaned-up: 0
elapsed: 1 second(s), ~ 118607 msg/s
\ processing mail; processed: 118607; updated/new: 0, cleaned-up: 0
elapsed: 1 second(s), ~ 118607 msg/s
1.01 real 0.39 user 0.37 sys

>> There is mention of it. Try google. BUT that is the wrong file. You
>> want .noupdate, not .noindex.
>
> Ok, thanks. '.noindex' was referenced in original post, but I'm
> assuming '.noindex' was incorrect (and has no other use/function), and
> I'll use '.noupdate' instead.

Yes. .noindex was a typo on my part in first message. It has other uses.

-k.

[Joost Kremers]

On Mon, Jul 13 2015, Johnny Utahh <johnnny...@gmail.com> wrote:
> Ok, thanks. '.noindex' was referenced in original post, but I'm assuming
> '.noindex' was incorrect (and has no other use/function), and I'll use
> '.noupdate' instead.

They are both in use, see `man mu-index` for details. (Basically,
.noindex doesn't index at all, while .noupdate does index but does not
update.)

--
Joost Kremers
Life has its moments