Re: Purpose of indexing
Wayne Mery
> There has been some talk here about indexing in TB3 but I am unclear
> about its purpose. Of course, the question seems silly at first, but:
>
> - is this equivalent to rebuilding the msf files? If not, are those msf
> files still useful, and does compacting work as it used to in previous
> versions/
no, yes, and yes. These items are unchanged in version 3.
> - the usual purpose of indexing is to help do searches. How do you do
> searches in TB3?
In v3 there are 4 types of searches:
* filters, aka quick search ... not new to v3, but improved
* search messages, aka advanced search ... not new to v3, but improved
* on MS Windows, you can enable Windows' indexing of thunderbird
messages and search via MS Windows [1] ... not new to v3
* search all messages, aka gloda search ... new to v3
The last item, gloda search, is totally new [2]. Gloda does fulltext
indexing of messages in the background using message headers, and bodies
of message which are stored "offline" (on your PC and synced in the
background). The index (no relation to .msf files) is an sqlite database
whose data includes addresses, body text words, header information (eg
tags), and pointers to messages. By default it does this across all
accounts and all folders, except news accounts, and trash and junk folders.
> - if a user uses a different indexing system on his machine, as I do, is
> it possible to totally disable this feature which exacts a heavy penalty
> on performances, or to limit the CPU time indexing uses?
>
> Thanks
You can impact cpu by tuning what is being indexed, selectively or in
total. Assuming you have indexing turned on, deselections can be done by
folder (eg. computer archive logs) or by account. Change by account by
unchecking "Keep messages for this account on this computer" in Accounts
> Syncronization and Storage. Indexing by folder is best affected
installing the glodaquilla extension found on http://mesquilla.com/ and
at addons.mozilla.org. The author is quite interested in improving the
extension, so you are encouraged to engage him with ideas. Note that
these changes affect only the fulltext indexing process, not the message
header. Also, display of search results for imap will be slower because
of the need to stream messages from the server.
During initial indexing, you will see fairly continuous, high cpu usage
that lasts until the full message store has been scanned. CPU usage is
dynamically tuned based on your activity on the computer. For computers
with slow disk, like laptops in particular, this may take many hours.
Initial indexing may impact your response time either in Thunderbird or
in other applications, but it should not be so bad as to substantially
affect your work.
During normal operations (i.e. after initial indexing is finished),
there is a less aggressive scheduling algorithm as to when messages are
indexed. This obviously requires some CPU, but should not affect your
response time.
If you want to use gloda searching and are seeing an unacceptable
performance penalty, please file a bug and describe the situation in
detail. https://bugzilla.mozilla.org/enter_bug.cgi?product=Thunderbird
A common question is what happens to laptops on battery power. Indexing
presently is not battery power aware due to limitations in gecko (core
mozilla code). Indexing stops if you switch Thunderbird to offline mode
since that's the best approximation for a laptop likely being on
battery power. (But there's presently a bug identified where if
Thunderbird starts up in offline mode that we still index things.)
Gloda is designed to be extensible, bot in what it indexes, and what and
how results are displayed
For more detail about Gloda see [2].
[1] Options > Advanced > General > Allow Windows Search
[2]
http://www.visophyte.org/blog/2009/09/03/sos-your-facet-faceted-global-search-for-mozilla-thunderbird/
https://developer.mozilla.org/en/Thunderbird/gloda
https://wiki.mozilla.org/User:Andrew_Sutherland/MailNews/GlobalDatabase
https://wiki.mozilla.org/Thunderbird:Debugging_Gloda
--
http://wiki.mozilla.org/Thunderbird:Testing
http://www.spreadthunderbird.com/aff/165/
Gord McFee
> There has been some talk here about indexing in TB3 but I am unclear
> about its purpose. Of course, the question seems silly at first, but:
>
> - is this equivalent to rebuilding the msf files? If not, are those msf
> files still useful, and does compacting work as it used to in previous
> versions/
The msf files still serve as index files, like they always have. Of
course, every time you add to or delete from a group or folder, the .msf
file will change.
The new search engine is called GLODA and is explained here:
https://developer.mozilla.org/en/Thunderbird/gloda
> - the usual purpose of indexing is to help do searches. How do you do
> searches in TB3?
There is a window at the top right that says "Search all messages". You
perform the searches there. If you have large folders and/or many
folders of archived messages, the function is very useful. The first
time it indexes, it can take a while if you have a great deal of folders
and mail, but after that, it is quite fast.
And searches, for ,me at lerast, are very quivk and return excellent
results.
> - if a user uses a different indexing system on his machine, as I do, is
> it possible to totally disable this feature which exacts a heavy penalty
> on performances, or to limit the CPU time indexing uses?
You can disable it as Ron explained.
--
Best regards
Gord McFee
John Doue
> Am 16.12.2009 14:45, schrieb John Doue:
>> I have not looked seriously for it, but is there a submenu to search
>> emails in TB3? I would not believe an indexing feature exists without it.
>
> There is a search bar above the message list. You can choose "Message in
> all accounts" or something similar there, and it will do a global full
> text search.
>
>> With regards to the indexing itself, where are the index files, what are
>> they named?
>
> global-messages.sqlite somewhere in the profile.
Thanks.
It seems difficult to avoid those sqlite databases thoses days. Not that
they are user friendly when one needs to get under the hood!
--
John Doue
John Doue
regarding the laptop issue.
Obviously, when I have some time on hand, I will revisit TB3 at length
and will probably revise my initial sceptical opinion in view of the
added complexity.
--
John Doue
bslade
>
> The msf files still serve as index files, like they always have. Of
> course, every time you add to or delete from a group or folder, the .msf
> file will change.
>
Are you saying Gloda is redundant with the msf files? (they both do
mail body indexing?)
Asking the question another way, if I just need to search within a
specific inbox, does Gloda help me at all?
Ben Slade
Wayne Mery
> On Dec 16, 9:44 pm, Gord McFee<gord.mc...@rogers.com> wrote:
>>
>> The msf files still serve as index files, like they always have. Of
>> course, every time you add to or delete from a group or folder, the .msf
>> file will change.
>>
>
> Are you saying Gloda is redundant with the msf files? (they both do
> mail body indexing?)
.msf does not index message bodies. So if you do what is now a "Message
body filter", you are physically reading the entire message folder
store, for each search. Furthermore, msf is not an indexed
file/database. So if you search on subject "test" for example it reads
the entire msf file.
"Search All Messages" on the other hand does not need to read entire
folders, it uses gloda's indexed relational database. So gloda's index
is not redundant, it's a different type of search.
> Asking the question another way, if I just need to search within a
> specific inbox, does Gloda help me at all?
if you don't use "Search All Messages" then gloda's index is not needed
bslade
Just to be clear, if I use the quick search field on the main screen
choosing something other than "Search All Messages", I'm not using the
indexed search. Additionally, if I do a regular search (edit->Find-
>Search Messages->Search for messages in->my inbox), then I'm also not
using the Gloda indexes?
Thanks
Ben