IMAP search capabilities...

444 views
Skip to first unread message

Tanstaafl

unread,
Feb 28, 2012, 8:18:36 AM2/28/12
to tb-ent...@mozilla.org
Hi all,

There is a bug currently open for an option to run all searches on the
server:

https://bugzilla.mozilla.org/show_bug.cgi?id=564168

Thunderbird should have an option to *always* rely on the server for
handling the search/filtering, whether you are using the Advanced Search
or the Quickfilter Toolbar. Obviously this would only really be useful
for servers that provide server side indexing of emails (this includes
but is not limited to Dovecot, Cyrus and GMail), but this results in
*lightning fast* search results *even when all OFFLINE settings are
disabled (no caching of local mail)*. This would be *huge* for those of
use who use multiple computers with IMAP accounts with many multiple
Gigabytes of email.

I'm primarily interested in this ability being used in conjunction with
the Quickfilter Toolbar (I rarely if ever use the Advanced Search, but
use the QF toolbar filtering all the time), but obviously it would make
sense to make the option applicable to one or the other *or* both.

Obviously, this would need to be on a per account basis, since not all
IMAP servers support server side indexes (courier-imap being one)...

So, my question is, is there currently a way to do this through user.js
or some other magic? Or is my only hope that the above bug gets
implemented some day, and done so such that the option could be made to
apply to the Quickfilter Toolbar too?

Thanks,

Charles
_______________________________________________
tb-enterprise mailing list
tb-ent...@mozilla.org
https://mail.mozilla.org/listinfo/tb-enterprise

Patrick Ben Koetter

unread,
Feb 28, 2012, 10:00:25 AM2/28/12
to tb-ent...@mozilla.org
* Tanstaafl <tans...@libertytrek.org>:

> Hi all,
>
> There is a bug currently open for an option to run all searches on
> the server:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=564168
>
> Thunderbird should have an option to *always* rely on the server for
> handling the search/filtering, whether you are using the Advanced
> Search or the Quickfilter Toolbar. Obviously this would only really
> be useful for servers that provide server side indexing of emails
> (this includes but is not limited to Dovecot, Cyrus and GMail), but
> this results in *lightning fast* search results *even when all
> OFFLINE settings are disabled (no caching of local mail)*. This
> would be *huge* for those of use who use multiple computers with
> IMAP accounts with many multiple Gigabytes of email.


I agree. Local search doesn't work for us and server-side search is real fast
with server that implement such capabilities.


> I'm primarily interested in this ability being used in conjunction
> with the Quickfilter Toolbar (I rarely if ever use the Advanced
> Search, but use the QF toolbar filtering all the time), but
> obviously it would make sense to make the option applicable to one
> or the other *or* both.
>
> Obviously, this would need to be on a per account basis, since not
> all IMAP servers support server side indexes (courier-imap being
> one)...

ACK

> So, my question is, is there currently a way to do this through
> user.js or some other magic? Or is my only hope that the above bug
> gets implemented some day, and done so such that the option could be
> made to apply to the Quickfilter Toolbar too?

Another option might be to sponsor development - either as individual or as
group of sponsors. We would contribute money and ressources (IMAP server,
etc.) to develop such an enterprise feature.

p@rick


--
state of mind ()
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstraße 15 Telefon +49 89 3090 4664
81669 München Telefax +49 89 3090 4666

Amtsgericht München Partnerschaftsregister PR 563

Ben Bucksch

unread,
Feb 28, 2012, 10:02:38 AM2/28/12
to tb-ent...@mozilla.org
On 28.02.2012 14:18, Tanstaafl wrote:
> servers that provide server side indexing of emails (this includes but
> is not limited to Dovecot, Cyrus and GMail),
> not all IMAP servers support server side indexes (courier-imap being
> one)...

I have a local Cyrus (standing in the next room), and the server-side
search is dog-slow, and I am not aware of any index files on the server.

Ben

Patrick Ben Koetter

unread,
Feb 28, 2012, 10:04:49 AM2/28/12
to tb-ent...@mozilla.org
* Ben Bucksch <ben.b...@beonex.com>:

> On 28.02.2012 14:18, Tanstaafl wrote:
> >servers that provide server side indexing of emails (this includes
> >but is not limited to Dovecot, Cyrus and GMail),
> >not all IMAP servers support server side indexes (courier-imap
> >being one)...
>
> I have a local Cyrus (standing in the next room), and the
> server-side search is dog-slow, and I am not aware of any index
> files on the server.

Throw it a bone and use the time to get and install Dovecot. ;)

Seriously: The client could ask if it should proceed the search on the
server-side and users would be able to choose. That leaves room for fast and
slow searches on all ends.

p@rick

--
state of mind ()
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstraße 15 Telefon +49 89 3090 4664
81669 München Telefax +49 89 3090 4666

Amtsgericht München Partnerschaftsregister PR 563

_______________________________________________

Tanstaafl

unread,
Feb 28, 2012, 10:15:40 AM2/28/12
to tb-ent...@mozilla.org
On 2012-02-28 10:04 AM, Patrick Ben Koetter <p...@state-of-mind.de> wrote:
> Ben Bucksch<ben.b...@beonex.com>:
>> On 28.02.2012 14:18, Tanstaafl wrote:
>>> servers that provide server side indexing of emails (this includes
>>> but is not limited to Dovecot, Cyrus and GMail),
>>> not all IMAP servers support server side indexes (courier-imap
>>> being one)...

>> I have a local Cyrus (standing in the next room), and the
>> server-side search is dog-slow, and I am not aware of any index
>> files on the server.

> Throw it a bone and use the time to get and install Dovecot. ;)
>
> Seriously: The client could ask if it should proceed the search on the
> server-side and users would be able to choose. That leaves room for fast and
> slow searches on all ends.

Actually, I just saw a reply on the dovecot list from someone
complaining about a problem with server side searches and thunderbird,
and they had logs showing that Thunderbird wasn't *using* server side
search even when explicitly told to do so (enabling 'Run search on
server' in the Advanced Search window)...

Patrick - can you look into this on your end? Does server side search
actually work for you using dovecot and Thunderbird? Can you confirm
with logs that it send the appropriate commands?

Here is a link to the entire thread:

http://www.mail-archive.com/dov...@dovecot.org/msg43290.html

and more importantly, here is a link to Timo's last response saying this
is definitely a Thunderbird bug:

http://www.mail-archive.com/dov...@dovecot.org/msg43366.html

Marc Patermann

unread,
Feb 28, 2012, 10:25:46 AM2/28/12
to Ben Bucksch, tb-ent...@mozilla.org
Ben Bucksch schrieb (28.02.2012 16:02 Uhr):
> On 28.02.2012 14:18, Tanstaafl wrote:
>> servers that provide server side indexing of emails (this includes but
>> is not limited to Dovecot, Cyrus and GMail),
>> not all IMAP servers support server side indexes (courier-imap being
>> one)...
>
> I have a local Cyrus (standing in the next room), and the server-side
> search is dog-slow, and I am not aware of any index files on the server.
Is squatter turned on?
http://linux.die.net/man/8/squatter


Marc

David Bienvenu

unread,
Feb 28, 2012, 10:32:02 AM2/28/12
to tb-ent...@mozilla.org
On 2/28/2012 7:00 AM, Patrick Ben Koetter wrote:
> *

> Another option might be to sponsor development - either as individual or as
> group of sponsors. We would contribute money and ressources (IMAP server,
> etc.) to develop such an enterprise feature.
Yes, patches are welcome. It's not a big priority for us, since not that many users have access to servers with advanced search, and we've invested a lot in our client side
search. But I realize there are advantages to server-side search for users with servers that support it well.

- David

Ben Bucksch

unread,
Feb 28, 2012, 10:32:29 AM2/28/12
to Marc Patermann, tb-ent...@mozilla.org
On 28.02.2012 16:25, Marc Patermann wrote:
> Ben Bucksch schrieb (28.02.2012 16:02 Uhr):
>> On 28.02.2012 14:18, Tanstaafl wrote:
>>> servers that provide server side indexing of emails (this includes
>>> but is not limited to Dovecot, Cyrus and GMail),
>>> not all IMAP servers support server side indexes (courier-imap being
>>> one)...
>>
>> I have a local Cyrus (standing in the next room), and the server-side
>> search is dog-slow, and I am not aware of any index files on the server.
> Is squatter turned on?
> http://linux.die.net/man/8/squatter

Nope, this is a standard install of Cyrus 2.2.

I'm just saying that you can't say that all users on Cyrus have this
ability. I would think only a minority does.

Does the server somehow announce this ability via a CAPABILITY string?

Ben

Tanstaafl

unread,
Feb 28, 2012, 10:48:12 AM2/28/12
to tb-ent...@mozilla.org
On 2012-02-28 10:32 AM, David Bienvenu <dbie...@mozilla.com> wrote:
> On 2/28/2012 7:00 AM, Patrick Ben Koetter wrote:
>> Another option might be to sponsor development - either as
>> individual or as group of sponsors. We would contribute money and
>> ressources (IMAP server, etc.) to develop such an enterprise
>> feature.

Would that Mozilla would provide some kind of bounty system to make
things like this easier...

> Yes, patches are welcome. It's not a big priority for us, since not that
> many users have access to servers with advanced search,

Not true at all, since GMail has supported this since forever, as has
dovecot. So, anyone using gmail, or any public service that uses dovecot
(I'm sure there are lots - ie, I know dreamhost does).

The bigger question is, is there a reliabel way (something in the IMAP
CAPABILITY response) that could be used to determine t his
automagically... I'll go ask Timo...

Marc Patermann

unread,
Feb 28, 2012, 10:54:19 AM2/28/12
to Ben Bucksch, tb-ent...@mozilla.org
Ben,

Ben Bucksch schrieb (28.02.2012 16:32 Uhr):
> On 28.02.2012 16:25, Marc Patermann wrote:
>> Ben Bucksch schrieb (28.02.2012 16:02 Uhr):
>>> On 28.02.2012 14:18, Tanstaafl wrote:
>>>> servers that provide server side indexing of emails (this includes
>>>> but is not limited to Dovecot, Cyrus and GMail),
>>>> not all IMAP servers support server side indexes (courier-imap being
>>>> one)...
>>>
>>> I have a local Cyrus (standing in the next room), and the server-side
>>> search is dog-slow, and I am not aware of any index files on the server.
>> Is squatter turned on?
>> http://linux.die.net/man/8/squatter
>
> Nope,

So, here is your problem. :)

> this is a standard install of Cyrus 2.2.

Define "standard".

> I'm just saying that you can't say that all users on Cyrus have this
> ability. I would think only a minority does.

What is your point?
This is the enterprise list. You cannot expect that all software out of
the box fits all your needs, in most case you have to
configure/tune/customize until it fits you.
If you're missing server side search on your IMAP server, turn it on.
If you miss recent features of IMAPd, update.
If not, leave it as that.

This thread - as I understand it - is about using fast server side
search in TB, if it is available.


Marc

Patrick Ben Koetter

unread,
Feb 28, 2012, 11:04:35 AM2/28/12
to tb-ent...@mozilla.org
* Tanstaafl <tans...@libertytrek.org>:

> On 2012-02-28 10:04 AM, Patrick Ben Koetter <p...@state-of-mind.de> wrote:
> >Ben Bucksch<ben.b...@beonex.com>:
> >>On 28.02.2012 14:18, Tanstaafl wrote:
>
> Actually, I just saw a reply on the dovecot list from someone
> complaining about a problem with server side searches and
> thunderbird, and they had logs showing that Thunderbird wasn't
> *using* server side search even when explicitly told to do so
> (enabling 'Run search on server' in the Advanced Search window)...

I've been lurking too ...


> Patrick - can you look into this on your end? Does server side
> search actually work for you using dovecot and Thunderbird? Can you
> confirm with logs that it send the appropriate commands?

A quick check:

- With TB and in "Advanced Search window" I don't see anything in the Dovecot
mail log.
- With telnet to Dovecot I do see fts_solr log lines.

p@rick


--
state of mind ()
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstraße 15 Telefon +49 89 3090 4664
81669 München Telefax +49 89 3090 4666

Amtsgericht München Partnerschaftsregister PR 563

_______________________________________________

Ben Bucksch

unread,
Feb 28, 2012, 11:13:09 AM2/28/12
to Marc Patermann, tb-ent...@mozilla.org
On 28.02.2012 16:54, Marc Patermann wrote:
>> this is a standard install of Cyrus 2.2.
> Define "standard".

1. apt-get install cyrus-imapd-2.2
2. create user accounts

> What is your point?

That the original claim "cyrus has fast, index-based search" is wrong,
if only a fraction of users do. I read it so that all users on Cyrus
servers have it, which would have been great.

> This is the enterprise list.

...which asked for a new feature in Thunderbird. How many users will be
able to use it, and how many can't, is very relevant to that and to how
the feature will look like.

FWIW, I support the original proposal to allow purely server-based
search, with a custom setting.

I'd like it even better when Thunderbird automatically knew whether it
needs a local search or can use server-side search. That's why I asked
about the CAPABILITY string. That detection is important to make such a
feature useful in general.

Ben

Tanstaafl

unread,
Feb 28, 2012, 11:13:40 AM2/28/12
to tb-ent...@mozilla.org
On 2012-02-28 11:04 AM, Patrick Ben Koetter <p...@state-of-mind.de> wrote:
> * Tanstaafl<tans...@libertytrek.org>:

>> Patrick - can you look into this on your end? Does server side
>> search actually work for you using dovecot and Thunderbird? Can you
>> confirm with logs that it send the appropriate commands?
>
> A quick check:
>
> - With TB and in "Advanced Search window" I don't see anything in the Dovecot
> mail log.
> - With telnet to Dovecot I do see fts_solr log lines.

Ok, so, it appears server side search is broken in Thunderbird.

Since you have access to a system running dovecot (I don't currently, my
client is still using courier, although we will be switching soon to
dovecot) and can get proper logs exhibiting the problem, would you
hopefully have time to nail down a decent bug description and open one
for Thunderbird?

I don't have any experience with a sniffer, or I'd test one of my gmail
accounts and see if it does the same thing...

Patrick Ben Koetter

unread,
Feb 28, 2012, 11:16:03 AM2/28/12
to tb-ent...@mozilla.org
* Tanstaafl <tans...@libertytrek.org>:

> On 2012-02-28 11:04 AM, Patrick Ben Koetter <p...@state-of-mind.de> wrote:
> >* Tanstaafl<tans...@libertytrek.org>:
> >>Patrick - can you look into this on your end? Does server side
> >>search actually work for you using dovecot and Thunderbird? Can you
> >>confirm with logs that it send the appropriate commands?
> >
> >A quick check:
> >
> >- With TB and in "Advanced Search window" I don't see anything in the Dovecot
> > mail log.
> >- With telnet to Dovecot I do see fts_solr log lines.
>
> Ok, so, it appears server side search is broken in Thunderbird.
>
> Since you have access to a system running dovecot (I don't
> currently, my client is still using courier, although we will be
> switching soon to dovecot) and can get proper logs exhibiting the
> problem, would you hopefully have time to nail down a decent bug
> description and open one for Thunderbird?

I will try to do that. My time is very limited the next two weeks.

> I don't have any experience with a sniffer, or I'd test one of my
> gmail accounts and see if it does the same thing...

You have access to gmail log files? ;)

p@rick

--
state of mind ()
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstraße 15 Telefon +49 89 3090 4664
81669 München Telefax +49 89 3090 4666

Amtsgericht München Partnerschaftsregister PR 563

_______________________________________________

Tanstaafl

unread,
Feb 28, 2012, 11:20:45 AM2/28/12
to tb-ent...@mozilla.org
Expanding on my earlier reply...

On 2012-02-28 10:32 AM, David Bienvenu <dbie...@mozilla.com> wrote:

> On 2/28/2012 7:00 AM, Patrick Ben Koetter wrote:
>> Another option might be to sponsor development - either as
>> individual or as group of sponsors. We would contribute money and
>> ressources (IMAP server, etc.) to develop such an enterprise
>> feature.

I would love it if Mozilla would implement some kind of bounty system,
and I don't see why you guys wouldn't love it too. It would be a very
easy way for you to deflect enhancement requests, and more importantly,
be a vehicle that would potentially drive many improvements in
Thunderbird going forward. Maybe the Firefox guys (since they have so
many more resources) could be prodded to do this, so that us corporate
Thunderbird users wo0uld have a way to 'vote' on issues with our
pocketbooks, and there would be no more reason for you guys to resist
fixing things that will provide *enormous* benefits to a *large* number
of users - especially things that shouldn't be all that hard to fix (you
do already support it via the Advanced Search window, although that
appears to be broken at the moment)...

> But I realize there are advantages to server-side search for users
> with servers that support it well.

Not just 'advantages', there are many situations where Local Search is
simply not a viable option, even for home users (but this is the
enterprise list, so discussing things in that context should not be
subjected to boilerplate comebacks about enterprise users not being your
target market, no?), and poor/nonexistent server side search support
basically tells us you don't care about a large portion of your user base.

Tanstaafl

unread,
Feb 28, 2012, 11:27:22 AM2/28/12
to tb-ent...@mozilla.org
On 2012-02-28 11:13 AM, Ben Bucksch <ben.b...@beonex.com> wrote:
> On 28.02.2012 16:54, Marc Patermann wrote:
>>> this is a standard install of Cyrus 2.2.
>> Define "standard".

> 1. apt-get install cyrus-imapd-2.2
> 2. create user accounts

>> What is your point?

> That the original claim "cyrus has fast, index-based search" is wrong,
> if only a fraction of users do. I read it so that all users on Cyrus
> servers have it, which would have been great.

Just because you didn't enable it (it is easy to do) doesn't mean that
no one else does.

In fact, I'd wager that the vast majority of people running Cyrus *do*
enable it. In fact, I'd wager that that is one of the main reasons they
choose Cyrus (and now one of the main reasons people are choosing
dovecot, which is far easier to install/configure).

>> This is the enterprise list.

> ...which asked for a new feature in Thunderbird.

Well... to be precise, the core feature already exists (though
apparently currently broken) - we're just talking about (what should
hopefully be) a fairly simple modification for how/when it is applied.

> How many users will be able to use it, and how many can't, is very
> relevant to that and to how the feature will look like.

Good point...

> FWIW, I support the original proposal to allow purely server-based
> search, with a custom setting.

Great to hear... :)

> I'd like it even better when Thunderbird automatically knew whether it
> needs a local search or can use server-side search. That's why I asked
> about the CAPABILITY string. That detection is important to make such a
> feature useful in general.

Absolutely - I'm asking Timo on the dovecot list (who should know better
than most), and will report back here with his answer...

David Bienvenu

unread,
Feb 28, 2012, 11:34:20 AM2/28/12
to tb-ent...@mozilla.org
On 2/28/2012 7:48 AM, Tanstaafl wrote:
>
> Not true at all, since GMail has supported this since forever, as has dovecot. So, anyone using gmail, or any public service that uses dovecot (I'm sure there are lots -
> ie, I know dreamhost does).
I don't see the gmail IMAP server advertising any kind of cross-folder search. Am I missing something? Implementing global search by searching every imap folder one by one
is unlikely to be faster than using our own local search.

The primary non-global searches that Thunderbird does are on header fields (the quick filter widget) and since that just uses the in memory summary file, it's sufficiently
fast for the vast majority of users.

So I'm curious where in the UI you see the fast server-side support would be exposed, other than the advanced search dialog. Or are you proposing changing the searches
exposed by the UI to take advantage of fast server-side searches, e.g., add body searches to the quick filter widget?

- David

Ben Bucksch

unread,
Feb 28, 2012, 12:15:44 PM2/28/12
to tb-ent...@mozilla.org
On 28.02.2012 17:20, Tanstaafl wrote:
> I would love it if Mozilla would implement some kind of bounty system,
> and I don't see why you guys wouldn't love it too. It would be a very
> easy way for you to deflect enhancement requests, and more
> importantly, be a vehicle that would potentially drive many
> improvements in Thunderbird going forward.

We've been over this repeatedly. It basically chokes with the
discrepancy between the price of competent developers ($1000 per day),
effort to make changes in TB (typically days of work even for small
changes), and the willingness of users to pay (typically $10, or $50
tops, and typically only 2-3 users per feature). Do the math.

If you are an enterprise and have a serious need for something, and can
create the corresponding budget, there are ways for you to get features.
I am for hire, for example.

Ben

Marc Patermann

unread,
Feb 28, 2012, 12:36:05 PM2/28/12
to Ben Bucksch, tb-ent...@mozilla.org
Ben Bucksch schrieb (28.02.2012 17:13 Uhr):
> On 28.02.2012 16:54, Marc Patermann wrote:
>>> this is a standard install of Cyrus 2.2.
>> Define "standard".
>
> 1. apt-get install cyrus-imapd-2.2
> 2. create user accounts
So blame Debian/Ubuntu/... Either for shipping 2.2. in 2012.

Everything about changing the "defaults" was said before.


Marc

Tanstaafl

unread,
Feb 28, 2012, 1:14:10 PM2/28/12
to tb-ent...@mozilla.org
On 2012-02-28 11:34 AM, David Bienvenu <dbie...@mozilla.com> wrote:
> On 2/28/2012 7:48 AM, Tanstaafl wrote:
>>
>> Not true at all, since GMail has supported this since forever, as has
>> dovecot. So, anyone using gmail, or any public service that uses
>> dovecot (I'm sure there are lots - ie, I know dreamhost does).

> I don't see the gmail IMAP server advertising any kind of cross-folder
> search. Am I missing something?

Yes...

I wasn't saying it *advertised* it (because I didn't know if there was a
standard way that this would be done or not, although I now have an
answer on this from Timo that I'll be posting in a minute), I was saying
that it *works*.

I don't currently have access to a gmail account with a lot of messages,
but I can get access to one that has 8GB in it in an hour or two...

Hmmm... I went to test this, and apparently Thunderbird doesn't support
BODY searches in the Advanced Search Window?!?!

Anyway, I'll post a follow-up with a few different comparisons in a
little while...

> Implementing global search by searching every imap folder one by one
> is unlikely to be faster than using our own local search.

I don't have first-hand data to support such a claim, but there have
been numerous reports from others on the dovecot list praising how much
faster searches are than any clients local search, but admittedly
nothing that I have seen directly compares TB's current GLODA
capabilities (with a large, fully synced mail store) with a purely
server side search, I'm asking on the dovecot list for people who use
Thunderbird with dovecot to do some tests... dunno if anyone will or how
long it might be though)...

> So I'm curious where in the UI you see the fast server-side support
> would be exposed, other than the advanced search dialog.

An option in Tools > Account > Server Settings > Advanced:

[ ] Run all searches on server
[ ] Use server for Quickfilter Toolbar searches

Since this is a server thing, there is no reason to for it to be
anything other than a 'permanent' server setting.

> Or are you proposing changing the searches exposed by the UI to take
> advantage of fast server-side searches,

Not quite sure what you mean here...

I'd like to be able to tell Thunderbird *once* to always perform
server-side searches on account X (so, as I described above, this would
be a per account option), whether I'm using the Quickfilter toolbar or
GLODA/Advanced Search window - and like I said, I'd be ok with this
being something I had to set manually with user.js, but I'd prefer it to
be an option in the GUI. I'm still hurting from some of the last manual
changes I made a long time ago (at the suggestion of people in the
forums, some of whom were devs) that caused me tons of frustration when
they started misbehaving a year or so later (presumably from other
changes in Thunderbird that changed how these modifications affected
things).

> e.g., add body searches to the quick filter widget?

? Currently, the Quickfilter toolbar already *has* the ability to search
in bodies, but the Advanced Search Window does *not* - so, did you get
that backwards? Or am I just not understanding something?

Tanstaafl

unread,
Feb 28, 2012, 1:20:16 PM2/28/12
to tb-ent...@mozilla.org
On 2012-02-28 12:15 PM, Ben Bucksch <ben.b...@beonex.com> wrote:
> On 28.02.2012 17:20, Tanstaafl wrote:
>> I would love it if Mozilla would implement some kind of bounty
>> system, and I don't see why you guys wouldn't love it too. It would
>> be a very easy way for you to deflect enhancement requests, and
>> more importantly, be a vehicle that would potentially drive many
>> improvements in Thunderbird going forward.

> We've been over this repeatedly.

We have? I don't recall it...

> It basically chokes with the discrepancy between the price of
> competent developers ($1000 per day), effort to make changes in TB
> (typically days of work even for small changes), and the willingness
> of users to pay (typically $10, or $50 tops, and typically only 2-3
> users per feature). Do the math.

But this is just an assumption, Ben (or, has Mozilla tried it before?),
so doing the math is kind of meaningless. The fact is, how do you *know*
this is how it would work until you've tried it? You might be right - or
you might be surprised.

Also, if those 2-3 users happened to be *corporate* users, each pledging
$1,000, that would also put a different light on the numbers.

Is there an existing Bounty system that is already functioning, that
Mozilla could formally join/become a member of, so they wouldn't have to
reinvent the wheel?

> If you are an enterprise and have a serious need for something, and can
> create the corresponding budget, there are ways for you to get features.
> I am for hire, for example.

But it is much easier for me to point my boss to an officially
sanctioned page on mozilla.com (or .org), where he can submit his pledge
himself in a formal manner, than it is to convince him to start
discussions with some one individual.

Ben Bucksch

unread,
Feb 28, 2012, 1:48:21 PM2/28/12
to tb-ent...@mozilla.org
On 28.02.2012 19:20, Tanstaafl wrote:
> The fact is, how do you *know* this is how it would work until you've
> tried it?

I have tried it, repeatedly. It was a disaster every time. Lots of
excitement, but hardly anybody was actually putting up.

As Patrick mentioned, you can pool your resources on this very topic.
People willing *can* reply right here, or in private.

Ben

Tanstaafl

unread,
Feb 28, 2012, 2:45:43 PM2/28/12
to tb-ent...@mozilla.org

Ok, here are my original question and his answers...

To summarize - according to Timo, "All IMAP servers are *required* to
support SEARCH command...", and they can optionally advertise server
side FTS (full text search) indexes using SEARCH=FUZZY. He said that
neither Cyrus nor GMail currently advertise this, but they both support
it (well, Cyrus *can* support it if it is enabled), but he also said
that the RFC defining this spec is only about a year old.

So, there are standards that apply... why not support them! :)

If there is at least agreement in general that adding this support would
be a 'good thing', I'll go see if a bug exists to cover it, and if not ,
will create one.

On 2012-02-28 11:38 AM, Timo Sirainen <t...@iki.fi> wrote:
> On 28.2.2012, at 18.33, Charles Marcus wrote:
>> This question is a result of an ongoing discussion on the mozilla
>> enterprise list...
>>
>> Is there a standard/reliable way for an IMAP client to determine
>> that an IMAP server supports server side search

> All IMAP servers are required to support SEARCH command. Some crappy
> ones don't, but I think all widely used ones do.

>> (with indexes)?

> No way to know that.
>
> Well, okay, actually if server advertises FUZZY extension you can be
> quite certain that it supports indexed server side searches. And that
> reminds me, I should hide that extension when FTS isn't enabled in
> Dovecot.. (I don't know if there are any other servers besides
> Dovecot implementing FUZZY.)

First follow-up from him:

On 2012-02-28 11:51 AM, Timo Sirainen <t...@iki.fi> wrote:
> On 28.2.2012, at 18.38, Timo Sirainen wrote:
>> Well, okay, actually if server advertises FUZZY extension you can
>> be quite certain that it supports indexed server side searches.

> I meant SEARCH=FUZZY

>> And that reminds me, I should hide that extension when FTS isn't
>> enabled in Dovecot..

> v2.1.2 will no longer advertise it unless fts=solr or fts=lucene:
> http://hg.dovecot.org/dovecot-2.1/rev/bdc881838b00

and the last follow-up to my follow-up:

On 2012-02-28 1:31 PM, Timo Sirainen <t...@iki.fi> wrote:
> On 28.2.2012, at 20.29, Charles Marcus wrote:
>> Off the top of your head, do you know if Cyrus or GMail (I guess
>> the two other most popular IMAP servers that support server side
>> indexes) advertise SEARCH=FUZZY?

> Neither. Probably no servers besides Dovecot. But it is less than a
> year old RFC..

Tanstaafl

unread,
Feb 28, 2012, 3:31:43 PM2/28/12
to tb-ent...@mozilla.org
On 2012-02-28 1:48 PM, Ben Bucksch <ben.b...@beonex.com> wrote:
> On 28.02.2012 19:20, Tanstaafl wrote:
>> The fact is, how do you *know* this is how it would work until you've
>> tried it?

> I have tried it, repeatedly. It was a disaster every time. Lots of
> excitement, but hardly anybody was actually putting up.

Interesting... as I said, you could be right, but I honestly don't
recall anything like this ever being done by Mozilla... when was this?
We've been using both Firefox/Thunderbird since about version 0.8 or 0.9...

That said, if a system is already there, why take it down? What does it
hurt to just leave it up? Then whenever anyone complains about their pet
feature/bug not being implemented/fixed, you can simply point them to
the Bounty page and tell them to put up or shut up. That would be far
better (and more professional) than the usual 'where's your patch'...

> As Patrick mentioned, you can pool your resources on this very topic.
> People willing *can* reply right here, or in private.

Well, yes, people like Patrick and I can, but more than likely we are
not the ones who will write the checks. Hopefully you will acknowledge
that to corporate CEO's (they *are* the ones who write the checks),
there is simply no comparison between:

a) a few email back and forths with some random unidentified geeks (no
offense intended) on a mail list, and

b) a formal process with formal procedures outlining contractual
obligations along with a web page where they can see a target
bounty amount for how much will be required and see how much has
already been pledged (they could then make a snap decision on
whether or not to fund the outstanding balance)

Patrick Ben Koetter

unread,
Feb 28, 2012, 3:56:25 PM2/28/12
to tb-ent...@mozilla.org
* Tanstaafl <tans...@libertytrek.org>:

> On 2012-02-28 1:48 PM, Ben Bucksch <ben.b...@beonex.com> wrote:
> >On 28.02.2012 19:20, Tanstaafl wrote:
> >>The fact is, how do you *know* this is how it would work until you've
> >>tried it?
>
> >I have tried it, repeatedly. It was a disaster every time. Lots of
> >excitement, but hardly anybody was actually putting up.
>

<snip />

Why not use a crowd funding platform?


> >As Patrick mentioned, you can pool your resources on this very topic.
> >People willing *can* reply right here, or in private.
>
> Well, yes, people like Patrick and I can, but more than likely we
> are not the ones who will write the checks. Hopefully you will

Actually I do and I will write one for Ben once he is done with SPECIAL-USE.
We ordered that as TB Feature (and Dovecot too) and it is planned to be
delivered for TB 13.


> acknowledge that to corporate CEO's (they *are* the ones who write
> the checks), there is simply no comparison between:
>
> a) a few email back and forths with some random unidentified geeks (no
> offense intended) on a mail list, and
>
> b) a formal process with formal procedures outlining contractual
> obligations along with a web page where they can see a target
> bounty amount for how much will be required and see how much has
> already been pledged (they could then make a snap decision on
> whether or not to fund the outstanding balance)

It's all about risk reduction!

The ones with the money don't know the ones that do the job. You need to add
something that represents values a buyer recognizes as trustful.

p@rick

--
state of mind ()

http://www.state-of-mind.de

Franziskanerstraße 15 Telefon +49 89 3090 4664
81669 München Telefax +49 89 3090 4666

Amtsgericht München Partnerschaftsregister PR 563

_______________________________________________

Dave Ewart

unread,
Feb 29, 2012, 4:27:27 AM2/29/12
to tb-ent...@mozilla.org
On Tuesday, 28.02.2012 at 08:18 -0500, Tanstaafl wrote:

> Hi all,
>
> There is a bug currently open for an option to run all searches on
> the server:
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=564168
>
> Thunderbird should have an option to *always* rely on the server for
> handling the search/filtering,

Absolutely. And, until it does, I modify omni.jar to include this patch
to "always search on server":

diff -U 5 -r content.orig/messenger/SearchDialog.xul content/messenger/SearchDialog.xul
--- content.orig/messenger/SearchDialog.xul 2010-08-02 18:04:58.000000000 +0100
+++ content/messenger/SearchDialog.xul 2010-09-13 14:51:13.000000000 +0100
@@ -79,10 +79,11 @@
<spacer flex="10"/>
<button label="&resetButton.label;" oncommand="onResetSearch(event);" accesskey="&resetButton.accesskey;"/>
</hbox>
<hbox align="center">
<checkbox id="checkSearchOnline"
+ checked="true"
label="&searchOnServer.label;"
accesskey="&searchOnServer.accesskey;"
oncommand="updateSearchLocalSystem();"/>
</hbox>
</vbox>

since that's all it needs.

This is a faff, because I need to unpack omni.jar (now named 'omni.ja'),
apply the above patch, then repack omni.jar.

This allows all users to get "search on server" all the time, by
default.

Dave.

--
Dave Ewart
da...@ceu.ox.ac.uk
Computing Manager, Cancer Epidemiology Unit
University of Oxford / Cancer Research UK
N 51.7516, W 1.2152

signature.asc

Patrick Ben Koetter

unread,
Feb 29, 2012, 4:30:56 AM2/29/12
to tb-ent...@mozilla.org
* Dave Ewart <da...@ceu.ox.ac.uk>:

> On Tuesday, 28.02.2012 at 08:18 -0500, Tanstaafl wrote:
>
> > Hi all,
> >
> > There is a bug currently open for an option to run all searches on
> > the server:
> >
> > https://bugzilla.mozilla.org/show_bug.cgi?id=564168
> >
> > Thunderbird should have an option to *always* rely on the server for
> > handling the search/filtering,
>
> Absolutely. And, until it does, I modify omni.jar to include this patch
> to "always search on server":

What does it take (time/money) to make this configurable in .js per account?

p@rick


--
state of mind ()

Digitale Kommunikation

signature.asc

Tanstaafl

unread,
Feb 29, 2012, 8:29:56 AM2/29/12
to tb-ent...@mozilla.org
Interesting - but does this work for the Quickfilter Toolbar too?

I can't remember the last time I used the Advanced Search window...

Tanstaafl

unread,
Feb 29, 2012, 8:31:00 AM2/29/12
to tb-ent...@mozilla.org
On 2012-02-29 4:30 AM, Patrick Ben Koetter <p...@state-of-mind.de> wrote:
> * Dave Ewart<da...@ceu.ox.ac.uk>:
>> On Tuesday, 28.02.2012 at 08:18 -0500, Tanstaafl wrote:
>>
>>> Hi all,
>>>
>>> There is a bug currently open for an option to run all searches on
>>> the server:
>>>
>>> https://bugzilla.mozilla.org/show_bug.cgi?id=564168
>>>
>>> Thunderbird should have an option to *always* rely on the server for
>>> handling the search/filtering,
>>
>> Absolutely. And, until it does, I modify omni.jar to include this patch
>> to "always search on server":
>
> What does it take (time/money) to make this configurable in .js per account?

Ah, yeah, this being a global change would be a problem too for many
people (but not for me)...

Dave Ewart

unread,
Feb 29, 2012, 9:21:52 AM2/29/12
to tb-ent...@mozilla.org
On Wednesday, 29.02.2012 at 08:29 -0500, Tanstaafl wrote:

> Interesting - but does this work for the Quickfilter Toolbar too?

Probably not, since that's not what I was trying to achieve, but I
haven't tested it.

signature.asc

Ben Bucksch

unread,
Feb 29, 2012, 10:53:16 AM2/29/12
to tb-ent...@mozilla.org
On 29.02.2012 14:29, Tanstaafl wrote:
> Interesting - but does this work for the Quickfilter Toolbar too?

No, it wouldn't.
But as bienvenu said, the quickfilter normally searches metadata like
subject and sender, and only in the current folder, and that information
is available locally anyway (needed for the threadpane display) and thus
faster as a local search.

As for gloda, that DB has a very high-level API. Using server-based
search instead of gloda would be a huge change, which I don't know
whether it's feasible. It's probably easier to rewrite the search UI.

Ben

Ben Bucksch

unread,
Feb 29, 2012, 10:56:30 AM2/29/12
to tb-ent...@mozilla.org
On 29.02.2012 10:30, Patrick Ben Koetter wrote:
> What does it take (time/money) to make this configurable in .js per account?

That depends mostly on the review, tests (if any) etc. required. The
change itself should be fairly simple, even if per account.

Tanstaafl

unread,
Feb 29, 2012, 12:05:58 PM2/29/12
to tb-ent...@mozilla.org
On 2012-02-29 10:53 AM, Ben Bucksch <ben.b...@beonex.com> wrote:
> On 29.02.2012 14:29, Tanstaafl wrote:
>> Interesting - but does this work for the Quickfilter Toolbar too?

> No, it wouldn't.

Bummer...

> But as bienvenu said, the quickfilter normally searches metadata like
> subject and sender,

Yes, but I can't tell you how many times I've had people bring our
Courier-IMAP server to its knees by clicking the 'Body' criteria on a
folder with 10GB+ of email in it... ;)

> and only in the current folder,

Yes, but I have a series of questions I regularly polled our users on,
and one of these was about using the Advanced Search vs the Quickfilter
- turned out that most *never* use the Advanced Search (and yes, they
all know what it is and how to get to it because I train them on it
regularly), and a few power users only use it very occasionally - and
after I showed them how to 'pin' the quickfilter, they said they
probably wouldn't use the Advanced Search again (or if they did, only
once in a blue moon).

> and that information is available locally anyway (needed for the
> threadpane display) and thus faster as a local search.

Understood - but like I said above, I would like to be able to toggle
the 'Body' criteria on and have it use the indexes on a server that
provides them (we are currently on Courier-IMAP which doesn't provide
them, but will be migrating to Dovecot in the next couple of months,
which does).

> As for gloda, that DB has a very high-level API. Using server-based
> search instead of gloda would be a huge change, which I don't know
> whether it's feasible. It's probably easier to rewrite the search UI.

Well, I wasn't talking about a permanent *change*, not sure where you
got that idea.

What I'm talking about is providing the ability for the Quickfilter
Toolbar searchbox to hook into the *existing* ability to perform server
side searches (the option in the Advanced Search window to 'Run search
on server'). So, if an option is made available to 'always perform
server side searches for this account' is enabled, the Quickfilter
Toolbar searchbox would automatically make use of it too (I wouldn't be
against providing a separate option for it too, but why?)

While ianap, I do understand that what seems to me like an almost
trivial thing like this (hooking into existing functionality) may not
really be trivial, so if there are technical reasons why that is so, by
all means, elaborate...

As for GLODA - I have said many times in the past when discussing GLODA
(and mentioning the fact that I don't and probably will never use it
because I only use IMAP, have many accounts on different servers that
have many multiple GB of email in them and access these from at least 4
different computers along with my phone, so keeping local copies of all
of that mail on all of these different systems is jut not something I
want to do) - I absolutely understand the need for GLODA, and think it
is a great tool - for those who use POP for their email and/or don't
have (lots of) huge IMAP stores. It just isn't the right tool for *me*
(and many others I know)... but my Mom is another story... ;)

Ben Bucksch

unread,
Feb 29, 2012, 5:43:32 PM2/29/12
to tb-ent...@mozilla.org
On 29.02.2012 18:05, Tanstaafl wrote:
> On 2012-02-29 10:53 AM, Ben Bucksch <ben.b...@beonex.com> wrote:
>> On 29.02.2012 14:29, Tanstaafl wrote:
>>> Interesting - but does this work for the Quickfilter Toolbar too?
>
>> No, it wouldn't.
>
> Bummer...
>
>> But as bienvenu said, the quickfilter normally searches metadata like
>> subject and sender,
>
> Yes, but I can't tell you how many times I've had people bring our
> Courier-IMAP server to its knees by clicking the 'Body' criteria on a
> folder with 10GB+ of email in it... ;)

Outch.
What is happening? TB doesn't have the bodies, fetches and caches them
all, and then searches locally? Or TB does a server-based search?

I thought Courier doesn't have indexed full-text search? It seems to me
it would make the problem only worse. Server-side search without index
needs to read all emails from disk every time the user does a search.
With TB local search, the server needs to read all emails and push them
over the net, but only once.

> (we are currently on Courier-IMAP which doesn't provide them, but will
> be migrating to Dovecot in the next couple of months, which does).

Ah, I see. But you see how that one server feature changes everything.
That's why it's so important to be able to detect this server capability.

(I bet admins would *hate* users who enabled the "use server-side
search" pref on a server without indexing.)

>
> most [people] *never* use the Advanced Search (and yes, they all know

> what it is and how to get to it because I train them on it regularly),
> and a few power users only use it very occasionally

Doesn't surprise me, same with me. The Advanced Search is so cumbersome
that it takes up to 1 minute to formulate a query. The whole point of
search is to find what I need in a matter of seconds.

>
>> As for gloda, that DB has a very high-level API. Using server-based
>> search instead of gloda would be a huge change, which I don't know
>> whether it's feasible. It's probably easier to rewrite the search UI.
>
> Well, I wasn't talking about a permanent *change*, not sure where you
> got that idea.

I meant "change" in the sense of "how do we need to /change/ the TB
source code to optionally allow to use server-side search instead of
gloda", e.g. for the "search all messages..." textfield.

> What I'm talking about is providing the ability for the Quickfilter
> Toolbar searchbox to hook into the *existing* ability to perform
> server side searches (the option in the Advanced Search window to 'Run
> search on server').

Quickfilter, gloda and Advanced Search all have different UI, both for
query and result display. You can't just use the Advanced Search and
show the result in the normal thread pane (as Quicksearch does), at
least not just like that, the features work in a totally different way
in the source code.

> While ianap, I do understand that what seems to me like an almost
> trivial thing like this (hooking into existing functionality) may not
> really be trivial, so if there are technical reasons why that is so,
> by all means, elaborate...

done above :)

> I absolutely understand the need for GLODA, and think it is a great
> tool - for those who use POP for their email and/or don't have (lots
> of) huge IMAP stores

FWIW, gloda works fine even with big IMAP stores. I use gloda, even
though the server is just a room away and exclusively for me.

Ben

Tanstaafl

unread,
Mar 1, 2012, 7:15:23 AM3/1/12
to tb-ent...@mozilla.org
On 2012-02-29 5:43 PM, Ben Bucksch <ben.b...@beonex.com> wrote:

> On 29.02.2012 18:05, Tanstaafl wrote:
>> Yes, but I can't tell you how many times I've had people bring our
>> Courier-IMAP server to its knees by clicking the 'Body' criteria on a
>> folder with 10GB+ of email in it... ;)

> Outch.
> What is happening? TB doesn't have the bodies, fetches and caches them
> all, and then searches locally? Or TB does a server-based search?

Hmmm... very interesting question, thanks - I hadn't really considered
the implications there. It definitely isn't downloading everything
locally, so... maybe it *is* already falling back to a server side
search, and simply switching to dovecot would solve the problem?

Any chance someone who knows/can read the code can see if this is the
case - ie, does a quickfilter search fall back to server side search
when filtering on folders that are not offline enabled/sync'd? Maybe it
is not really something TB is doing intentionally, but just the way it
works?

I'll find this out myself soon enough when we switch to dovecot, but it
would be nice to know what to expect...

> I thought Courier doesn't have indexed full-text search? It seems to me
> it would make the problem only worse. Server-side search without index
> needs to read all emails from disk every time the user does a search.
> With TB local search, the server needs to read all emails and push them
> over the net, but only once.
>
>> (we are currently on Courier-IMAP which doesn't provide them, but will
>> be migrating to Dovecot in the next couple of months, which does).
>
> Ah, I see. But you see how that one server feature changes everything.
> That's why it's so important to be able to detect this server capability.

Detecting isn't necessary for the option to be made available - the Sys
Admin will *know* if the server supports it or now.

However, yes, I absolutely agree that *if* this can be reliably detected
automatically, that would be best.

> (I bet admins would *hate* users who enabled the "use server-side
> search" pref on a server without indexing.)

Yes, but I would think there would be some way Thunderbird could detect
when this happens and pop-up a warning to the user, something like:

"You have enabled 'Always enable server side searches' on XYZ IMAP
server, but apparently this server doesn't provide server side indexes
because this search is taking too long - it is recommended that you
disable this preference now." with the default behavior set to 'Disable'.

>>> As for gloda, that DB has a very high-level API. Using server-based
>>> search instead of gloda would be a huge change, which I don't know
>>> whether it's feasible. It's probably easier to rewrite the search UI.
>>
>> Well, I wasn't talking about a permanent *change*, not sure where you
>> got that idea.
>
> I meant "change" in the sense of "how do we need to /change/ the TB
> source code to optionally allow to use server-side search instead of
> gloda", e.g. for the "search all messages..." textfield.

Ah, ok - but all I'm interested in is the Quickfilter functionality,
because that is all I ever use - I don't use the Global Search either,
but I don't see why this would require *any* UI change (except maybe a
little visual indicator when the pref is enabled) - if the pref is
eabled, the searchbox simply uses the different search algorithm

>> What I'm talking about is providing the ability for the Quickfilter
>> Toolbar searchbox to hook into the *existing* ability to perform
>> server side searches (the option in the Advanced Search window to 'Run
>> search on server').

> Quickfilter, gloda and Advanced Search all have different UI, both for
> query and result display. You can't just use the Advanced Search and
> show the result in the normal thread pane (as Quicksearch does),

Well, the code for sending the search to the server is obviously already
written, right? Hopefully you guys are writing code in such a way that
it can be re-used when desired, right? What is that called? Abstracting
out things that can/should be reusable into 'functions', then calling
those functions when/where needed? If you aren't, then I'd have to ask,
'why the hell not??' ;)

> FWIW, gloda works fine even with big IMAP stores. I use gloda, even
> though the server is just a room away and exclusively for me.

Sure, and I have said as much myself - but this is true *only* when
those big IMAP stores are fully sync'd locally in offline mode, which,
in the modern world where everything is in the cloud, this is, simply
put, sub-optimal.

Personally, I think Thunderbird should be headed in the direction of
assuming that, sometime (sonner rather than later - 2 years? 5?), *if*
people are still using email heavily, they will be using IMAP for
everything, and no one will want huge local stores of everything on
every device just so they can search it easily - although I can
certainly see the argument for some kind of 'back-up all my IMAP email'
feature, so that you could have a single Thunderbird client dedicated to
backing up all of your IMAP mail stores locally, so that if GMail (or
whoever) lost all your mail, you would be able to restore it yourself
and not rely on them for backing it up.

Tanstaafl

unread,
Mar 1, 2012, 7:21:56 AM3/1/12
to tb-ent...@mozilla.org
On 2012-03-01 7:15 AM, Tanstaafl <tans...@libertytrek.org> wrote:
> Any chance someone who knows/can read the code can see if this is the
> case - ie, does a quickfilter search fall back to server side search
> when filtering on folders that are not offline enabled/sync'd

Let me reword this to be more precise:

What happens (or is supposed to happen) when the criteria for a
quickfilter search includes a criteria (ie, the 'Body' criteria is
clicked when filtering a folder that is not in offline mode and is not
synced locally) the data for which is not currently in the offline store)?

Patrick Ben Koetter

unread,
Mar 1, 2012, 7:25:54 AM3/1/12
to tb-ent...@mozilla.org
* Tanstaafl <tans...@libertytrek.org>:

> On 2012-02-29 5:43 PM, Ben Bucksch <ben.b...@beonex.com> wrote:
> >On 29.02.2012 18:05, Tanstaafl wrote:

<snip />

> Detecting isn't necessary for the option to be made available - the
> Sys Admin will *know* if the server supports it or now.

Maybe a very naive question, but isn't that something client and server should
negotiate?

IMAP4 provides a mechanism for a client to ask the server to search for
messages meeting a variety of criteria. This mechanism avoids requiring
clients to download every message in the mailbox in order to perform these
searches.
-- <http://en.wikipedia.org/wiki/Imap#Server-side_searches>

The way I see it:

The server announces its extensions. If the server announces SEARCH, the
client can choose it. If the server does not announce it, the client cannot
choose it.

I'd like to be able to tell my users client to always search on the server as
we are not going to sync GBs of mails to our client workstations for obvious
reasons (traffic + storage x roaming user = expensive).

Anyone using a server with unsatisfying server-side search performace should
turn the extension off or alter the banner. Courier, just to name one, can do
that.

p@rick


--
state of mind ()
Digitale Kommunikation

http://www.state-of-mind.de

Franziskanerstraße 15 Telefon +49 89 3090 4664
81669 München Telefax +49 89 3090 4666

Amtsgericht München Partnerschaftsregister PR 563

_______________________________________________

Nikolay Shopik

unread,
Mar 1, 2012, 7:53:28 AM3/1/12
to Patrick Ben Koetter, tb-ent...@mozilla.org
Patrick,

Search is integral part of RFC3501 so every server must support it, only
difference is wherever this indexed or not. But I don't know methods of
detecting indexed vs non-indexed searches on serer.

On 01/03/12 16:25, Patrick Ben Koetter wrote:
> The server announces its extensions. If the server announces SEARCH, the
> client can choose it. If the server does not announce it, the client cannot
> choose it.

Tanstaafl

unread,
Mar 1, 2012, 7:57:30 AM3/1/12
to tb-ent...@mozilla.org
On 2012-03-01 7:25 AM, Patrick Ben Koetter <p...@state-of-mind.de> wrote:
> The server announces its extensions. If the server announces SEARCH, the
> client can choose it. If the server does not announce it, the client cannot
> choose it.

According to Timo, *all* IMAP servers are *required* to support SEARCH,
even if they do not explicitly announce it.

What should be explicity tested for is the existence of server side
*indexes*, and Timo said the testing for SEARCH=FUZZY is the proper way
to do that, and apparently dovecot does this.

So, since it is best to stick with the standards, if this is to be
implemented, this is how the auto-negotiate test should work (ie, if the
server announces SEARCH=FUZZY, this new option would be enabled by
default - if it doesn't, it isn't, but the user or admin could enable it
manually if they wanted to if they know that the server supports it even
if it doesn't announce it).

Also, Thunderbird could enable it for server types that don't announce
it but do support it (like gmail).

> Anyone using a server with unsatisfying server-side search performace should
> turn the extension off or alter the banner. Courier, just to name one, can do
> that.

Yes, but as I said earlier, Thunderbird could also do some basic math,
and when the user is experiencing serious delays when searching with
this option enabled, Thunderbird should inform the user, and offer to
disable it.

Tanstaafl

unread,
Mar 1, 2012, 8:01:26 AM3/1/12
to tb-ent...@mozilla.org
On 2012-02-28 11:16 AM, Patrick Ben Koetter <p...@state-of-mind.de> wrote:
>> I don't have any experience with a sniffer, or I'd test one of my
>> gmail accounts and see if it does the same thing...

> You have access to gmail log files?;)

No, thats why I mentioned the sniffer (you'd have to use a sniffer
*because* we don't have access to the logs)...

Tanstaafl

unread,
Mar 1, 2012, 8:03:01 AM3/1/12
to tb-ent...@mozilla.org
On 2012-03-01 7:53 AM, Nikolay Shopik <sho...@inblock.ru> wrote:
> Search is integral part of RFC3501 so every server must support it, only
> difference is wherever this indexed or not. But I don't know methods of
> detecting indexed vs non-indexed searches on serer.

Timo said it is 'SEARCH=FUZZY, but that no other server (besides recent
versions of dovecot) advertise it, since it is a relatively new standard
(a year or so old)...

Ludovic Hirlimann

unread,
Mar 1, 2012, 8:36:36 AM3/1/12
to tb-ent...@mozilla.org
Why don't you attach the patch to the bug ?

--
@lhirlimann on twitter
https://wiki.mozilla.org/Thunderbird:Testing

my photos http://www.flickr.com/photos/lhirlimann/collections/


Dave Ewart

unread,
Mar 1, 2012, 9:03:07 AM3/1/12
to tb-ent...@mozilla.org
On Thursday, 01.03.2012 at 14:36 +0100, Ludovic Hirlimann wrote:

> > Absolutely. And, until it does, I modify omni.jar to include this patch
> > to "always search on server":
> >
> > diff -U 5 -r content.orig/messenger/SearchDialog.xul content/messenger/SearchDialog.xul
> > --- content.orig/messenger/SearchDialog.xul 2010-08-02 18:04:58.000000000 +0100
> > +++ content/messenger/SearchDialog.xul 2010-09-13 14:51:13.000000000 +0100
> > @@ -79,10 +79,11 @@
> > <spacer flex="10"/>
> > <button label="&resetButton.label;" oncommand="onResetSearch(event);" accesskey="&resetButton.accesskey;"/>
> > </hbox>
> > <hbox align="center">
> > <checkbox id="checkSearchOnline"
> > + checked="true"
> > label="&searchOnServer.label;"
> > accesskey="&searchOnServer.accesskey;"
> > oncommand="updateSearchLocalSystem();"/>
> > </hbox>
> > </vbox>
> >
> > since that's all it needs.
> >
> > This is a faff, because I need to unpack omni.jar (now named 'omni.ja'),
> > apply the above patch, then repack omni.jar.
> >
> Why don't you attach the patch to the bug ?

Well, for a start the patch is utterly trivial (just adding a single
line) and it's also not very flexible: it changes a setting from ALWAYS
OFF to ALWAYS ON, when the best code modification should result in this
setting being configurable via Options or via detected client/server
behaviour (as discussed elsewhere in this thread).

I'm happy for anyone else to submit the patch, modified or extended as
required, of course. :-)

signature.asc

Ben Bucksch

unread,
Mar 1, 2012, 11:11:53 AM3/1/12
to tb-ent...@mozilla.org
On 01.03.2012 13:15, Tanstaafl wrote:
> On 2012-02-29 5:43 PM, Ben Bucksch <ben.b...@beonex.com> wrote:
>> On 29.02.2012 18:05, Tanstaafl wrote:
>>> Yes, but I can't tell you how many times I've had people bring our
>>> Courier-IMAP server to its knees by clicking the 'Body' criteria on a
>>> folder with 10GB+ of email in it... ;)
>> Outch.
>> What is happening? TB doesn't have the bodies, fetches and caches them
>> all, and then searches locally? Or TB does a server-based search?
>
> Hmmm... very interesting question, thanks - I hadn't really considered
> the implications there. It definitely isn't downloading everything
> locally, so... maybe it *is* already falling back to a server side search

I doubt it, but you can trivially find out by using Wireshark.

>
> Ah, ok - but all I'm interested in is the Quickfilter functionality,
> because that is all I ever use - I don't use the Global Search either

OK, good to know (you were earlier talking about gloda).

I think you'd be best served for now to just disable body search from
quicksearch somehow. Don't ask me how, I don't know whether or how it's
possible, but it's surely a lot easier than hooking it up to server search.

> I don't see why this would require *any* UI change (except maybe a
> little visual indicator when the pref is enabled) - if the pref is
> eabled, the searchbox simply uses the different search algorithm

But it requires a big code change. The application is far more than the UI.

> Hopefully you guys are writing code in such a way that it can be
> re-used when desired, right?

In theory possible, but in reality it's not simple at all, but a lot of
work to hook up the different parts.

> If you aren't, then I'd have to ask, 'why the hell not??' ;)

Sorry, but you can't discuss these topics unless you are 1) a developer
and 2) know the relevant Thunderbird code. In fact, I can't discuss this
further without having spent time to look into it.

Ben

Kent James

unread,
Mar 1, 2012, 11:53:17 AM3/1/12
to tb-ent...@mozilla.org
On 3/1/2012 4:15 AM, Tanstaafl wrote:
> Any chance someone who knows/can read the code can see if this is the
> case - ie, does a quickfilter search fall back to server side search
> when filtering on folders that are not offline enabled/sync'd? Maybe
> it is not really something TB is doing intentionally, but just the way
> it works?
I did some work on that code a few years back in bug 511131. From my
comment 6, here is at least the attempted philosophy:

"Since I now specify online or offline for the search, plus am being
careful to use the correct validity table, I wanted the actual search
code to honor my choice when possible, yet just make it work otherwise.
So I scan through the validity tables and decide whether a combination
of virtual, user, and view terms can be done with the user's choice. If
so, I use the user's choice. If not, I use whatever will work."

Also in comment 7:

"This patch allows the user to select online search, and honors that
request if it can. But it will silently switch to offline if the
validity tables say that's the only way it will work."

That bug contains a lot of discussion of issues. For example, Bienvenu
sets the tone with "Quick search should bias towards offline search
whenever possible" in
https://bugzilla.mozilla.org/show_bug.cgi?id=511131#c11 I think that
the underlying issue is that online search has usually been slower than
offline search, so it is rarely the best choice, and most people using
it don't really understand that.

One thing that makes this complex is that advanced search can be cross
server, and servers have different capabilities.

rkent

Reply all
Reply to author
Forward
0 new messages