filter-mgmt bugs

[Wayne Mery]

aceman and others are doing deep triage of filter bugs (Tools|Filters,
not quick filter bar) and the results so far are impressive. It has
evolved to include identifying bugs which affect *how* a user manages
filters. We tag such bugs in the whiteboard with [filter-mgmt] [1].
Anyone can tag a qualifying bug.

The purpose of this posting is to refine the definition of filter-mgmt.
Feedback is welcomed.

The purpose of filter-mgmt tagging:
a) easily identify and queried specific filter bugs
b) so anyone who cares how the users are impacted can help fix them

This started as a private conversation. The current state of the
definition can be summarized as:
* it is narrow - we want only a few bugs, distinct from the hundreds of
bugs that merely have "filter" in the summary or component
* severity=ENH is not an automatic "in", and not a criteria of inclusion
* junk and spam bugs are not part of this effort (and they have their
own meta bugs [2] to identify them)
* bug need not be in the filter component
* bug may (but doesn't have to) involve the following dialogs
- Message Filters
- Filter Rules
- Filter Log

We continue exploring the definition by jumping off from
https://bugzilla.mozilla.org/show_bug.cgi?id=297852#c15 ...


(In reply to :aceman (away 27.-2.) from comment #15)
> Wayne, I think you defined filter-mgmt as being about UI to do with
creation
> and editing of filters. See your email of 22.10.2011.
> I think comment 0 and comment 11 actually say this bug is requesting
to add
> the UI for the existing backend features.
> Is that not the case?

Yeah, I did not fully express my thinking on scope. And I should have
stated what I think we don't want to include, with examples. Note: the
following is strictly my opinion.

By management of filters I mean
* how the user interacts with the UI - create, edit, manipulate - but
not the types of rules and the criteria themselves, nor the specific
operations that the rules represent (and whether they work or not)
* places where thunderbird itself affect the list of filters and the
rules, for example bugs which affect the accuracy of filters, like Bug
454589 - if filter move target folder is deleted ... filter should get
disabled

So IMO not qualified:
- new filter types, eg Bug 59365 - Filters should have triggers
- rule tweaks, eg Bug 601939 - filter size is imprecise with regard to
usage of KB (rounds to KB, can't express bytes exactly)

An open question is whether we include bugs which immensely enhance
productivity, such as reducing the number of filters needed, as in
bug 297852 - Enable Boolean expressions in Filter UI. But I am a little
leery of going down that path.



[1] [filter-mgmt] bugs:
https://bugzilla.mozilla.org/buglist.cgi?type1-0-0=substring&order=Importance&list_id=1587150&field0-0-0=short_desc&type0-0-1=substring&field0-0-1=keywords&type1-0-1=allwordssubstr&resolution=---&classification=Client%20Software&classification=Components&status_whiteboard_type=allwordssubstr&query_format=advanced&status_whiteboard=filter&type0-0-0=anywordssubstr&field1-0-0=short_desc&product=MailNews%20Core&product=Thunderbird&field1-0-1=short_desc

[2] Bug 11035 - spam blocking filters features, Bug 228674 - Plan for
Spam, Bug 531787 - (junktracker) Junk filter and UI problems

--
contribute ... http://wiki.mozilla.org/Thunderbird:Testing
evangelize ... http://www.spreadthunderbird.com/aff/165/

[John_David_Galt]

Wayne Mery <vseer...@lehigh.edu> wrote:
> aceman and others are doing deep triage of filter bugs (Tools|Filters,
> not quick filter bar) and the results so far are impressive. It has
> evolved to include identifying bugs which affect *how* a user manages
> filters. We tag such bugs in the whiteboard with [filter-mgmt] [1].
> Anyone can tag a qualifying bug.
>
> The purpose of this posting is to refine the definition of filter-mgmt.
> Feedback is welcomed.

This subject is important to me because I use a huge list (500+
filters).
Roughly 100 of those are about filtering out unwanted stuff (for
instance,
anything with "URGENT" in the subject line is probably a spam unless
it
comes from colleagues who actually write me about urgent problems).
The rest are about sorting all the mail from the hundreds of mailing
lists
I'm on into appropriate folders, either by list or by topic.

The most common "filter-mgmt" task I do is to add new filters to the
list,
followed by editing existing filters (for instance, because a sender
of ads
I want has a new domain name). Both of these are now quite tedious.
When I add a new filter, I first create it with "new", then I have to
sit there
for 10+ minutes hitting "Move Down" over and over again, but not so
fast
that all those clicks either overload Thunderbird's buffer or cause TB
to
slow down even more than it already is. When editing a filter, all
those
"Move Downs" are not needed, but scrolling down to it with the down
arrow is.

So I'd like to see the filter-mgmt interface include the kind of basic
tools
already included in every text editor. Specifically, I want:
1. The ability to resize the "Message Filters" dialog, so I can see
50+
filters on a screen (just as I now see 50+ lines in text editors).
2. "Page Up" and "Page Down" abilities in the filter list (which
would
jump a distance I can set as a preference, not just the 10 lines or so
I
can now jump by clicking in the scroll-bar).
3. The ability to jump to arbitrary places in the list of filters by
dragging
the scroll-bar slider, just as I can in most other Windows apps and
even
in Thunderbird's own folder-list, message-list, and message-body
panes.
4. The ability to search for strings in the list of filters, with or
without
limiting the search to particular fields within a filter definition.
5. The ability to "Cut", "Copy", and "Paste" whole filter
definitions, both
within a filter list and among different lists (say, the one for Inbox
and
the one for the newsgroups server), preferably using the same keys I
use for those functions in a text editor.

[Michael Scharf]

I am working in a totally different area on filters and I am not
familiar with the Thunderbird filter code. It is very good to define a
filter 'engine' independently of the UI. This engine should have a
clear filter language and maybe a concrete syntax that can be used to
build up the "abstract syntax tree" of the filter expressions. This
would allow for different UIs to parse the filter expressions and
generate new filter expressions.

If I look into the condition if the msgFilterRules.dat is seems
relatively simple.

I wonder how well things are specified. When I look at it it seems
that
(to or cc,contains,something)
has 3 parts:
1. the mail field
2. operation
3. argument

I wonder how well the mail field selection language is specified:
to or cc
all addresses
my-header

Is it possible to say
my-field1 or cc or my-field2
?

And what the semantics of the operation is
contains
doens't contain
ends with

It seems that the semantics of contians has changed recently (form
considering
all user fields to only consider the first one (https://
bugzilla.mozilla.org/show_bug.cgi?id=697096))

Whatever you do in the UI, it is *very* important to have a well-
tested filter engine...



Michael

[Axel Grude]

Subject: Re: filter-mgmt bugs
Date: Fri, 28 Oct 2011 11:20:00 -0700 (PDT)
From: jdg
To:

> Wayne Mery <vseer...@lehigh.edu> wrote:
>> aceman and others are doing deep triage of filter bugs (Tools|Filters,
>> not quick filter bar) and the results so far are impressive. It has
>> evolved to include identifying bugs which affect *how* a user manages
>> filters. We tag such bugs in the whiteboard with [filter-mgmt] [1].
>> Anyone can tag a qualifying bug.
>>
>> The purpose of this posting is to refine the definition of filter-mgmt.
>> Feedback is welcomed.
>

Wayne,

I have just added a screenshot to

https://bugzilla.mozilla.org/show_bug.cgi?id=297852

with a suggestion on building a fairly minmal UI. This is based on the idea that we
can "nest" filters and this way just re-use the existing interface to build more
complex conditions.

I* would be interested in feedback from the Thunderbird core team, and added bwinton ? UI.

> This subject is important to me because I use a huge list (500+
> filters).
> Roughly 100 of those are about filtering out unwanted stuff (for
> instance,
> anything with "URGENT" in the subject line is probably a spam unless
> it
> comes from colleagues who actually write me about urgent problems).
> The rest are about sorting all the mail from the hundreds of mailing
> lists
> I'm on into appropriate folders, either by list or by topic.

You might be interested in the quickFilters addon as it helps in defining filters
while (and after) you move mails to the folders.

> So I'd like to see the filter-mgmt interface include the kind of basic
> tools

> 2. "Page Up" and "Page Down" abilities in the filter list (which
> would
> jump a distance I can set as a preference, not just the 10 lines or so
> I
> can now jump by clicking in the scroll-bar).
> 3. The ability to jump to arbitrary places in the list of filters by
> dragging
> the scroll-bar slider, just as I can in most other Windows apps and
> even
> in Thunderbird's own folder-list, message-list, and message-body
> panes.

I have added the Search box since, so I hope that these navigation improvements are
not so important anymore. You can move filters while you have the search box active I
use this regularly in order to group related filters together.

> 4. The ability to search for strings in the list of filters, with or
> without
> limiting the search to particular fields within a filter definition.

Yes - we need a way to determine other search fields than just Name, but this is also
a UI problem. The Filter DIalog at current is already overloaded so we will need to
move towards a bigger design with a toolbar, such as Firefox Bookmarks / History.

> 5. The ability to "Cut", "Copy", and "Paste" whole filter
> definitions, both
> within a filter list and among different lists (say, the one for Inbox
> and
> the one for the newsgroups server), preferably using the same keys I
> use for those functions in a text editor.

This is something I will be working on over the next 2 months for my Addon
quickFilters. The Filter List will definitely have to undergo a better more radical UI
redesign including an integrated toolbar.

Axel

[Wayne]

On 3/3/2013 4:07 AM, Axel Grude wrote:
> Subject: Re: filter-mgmt bugs
> Date: Fri, 28 Oct 2011 11:20:00 -0700 (PDT)
> From: jdg
> To:
>> Wayne Mery <vseer...@lehigh.edu> wrote:
>>> aceman and others are doing deep triage of filter bugs (Tools|Filters,
>>> not quick filter bar) and the results so far are impressive. It has
>>> evolved to include identifying bugs which affect *how* a user manages
>>> filters. We tag such bugs in the whiteboard with [filter-mgmt] [1].
>>> Anyone can tag a qualifying bug.
>>>
>>> The purpose of this posting is to refine the definition of filter-mgmt.
>>> Feedback is welcomed.
>>
>
> Wayne,
>
> I have just added a screenshot to
>
> https://bugzilla.mozilla.org/show_bug.cgi?id=297852
>
> with a suggestion on building a fairly minmal UI. This is based on the
> idea that we can "nest" filters and this way just re-use the existing
> interface to build more complex conditions.
>
> I* would be interested in feedback from the Thunderbird core team, and
> added bwinton ? UI.

I like it! Although I don't have much basis for providing constructive
comment.

[ishikawa]

I am curious about this filter extension because
I am trying to sort out many ML and other occasional mailings into a large
number of folders, and yes, sometimes I feel that a general boolean
combinations of
conditions may be very welcome.

So this addition, when I think about it, would be useful for the
case of "Match any of the following" and in the nested folder we would have
"Match all of the following".

E.g.

A || B || C || D

and let us say, we would like to have B constructed as

b1 && b2 && b3 && b4
(where b1, b2, b3, ... are primitive terms [or whatever are called in TB.])
then B can be built using this nested folder, and in that filter, it would
be checked as match all of the following.

(Sure we can certainly use the nested folder for short hand notation for
A || B || C || D
where B is
b1 || b2 || b3 || b4 and "match any of the following" are checked for both
the original and nested filters.)

Of course, we should be allowed to nest deeper than 1 level.

I like the idea.

The general boolean expression (introducing new syntax, etc.) seems to be
too general without much gain.

On the other hand, since I saw TB crashed three times in about last two
weeks, and was surprised to find to my chagrin that it even failed to send
the crashreport for one of the crashes, I feel we may want to put this off
and put it on a wish-list for now, and fix whatever are causing the latest
instability first :-(

[Axel Grude]

> On (2013年03月03日 18:07), Axel Grude wrote:
>> Subject: Re: filter-mgmt bugs
>> Date: Fri, 28 Oct 2011 11:20:00 -0700 (PDT)
>> From: jdg
>> To:
>>> Wayne Mery <vseer...@lehigh.edu> wrote:
>>>> aceman and others are doing deep triage of filter bugs (Tools|Filters,
>>>> not quick filter bar) and the results so far are impressive. It has
>>>> evolved to include identifying bugs which affect *how* a user manages
>>>> filters. We tag such bugs in the whiteboard with [filter-mgmt] [1].
>>>> Anyone can tag a qualifying bug.
>>>>
>>>> The purpose of this posting is to refine the definition of filter-mgmt.
>>>> Feedback is welcomed.
>>>

<snip>

>>> So I'd like to see the filter-mgmt interface include the kind of basic
>>> tools
>>> 2. "Page Up" and "Page Down" abilities in the filter list (which
>>> would
>>> jump a distance I can set as a preference, not just the 10 lines or so
>>> I
>>> can now jump by clicking in the scroll-bar).
>>> 3. The ability to jump to arbitrary places in the list of filters by
>>> dragging
>>> the scroll-bar slider, just as I can in most other Windows apps and
>>> even
>>> in Thunderbird's own folder-list, message-list, and message-body
>>> panes.
>>
>> I have added the Search box since, so I hope that these navigation
>> improvements are not so important anymore. You can move filters while you
>> have the search box active I use this regularly in order to group related
>> filters together.

the list needs positional drag-and-drop support, badly.

>>
>>> 4. The ability to search for strings in the list of filters, with or
>>> without
>>> limiting the search to particular fields within a filter definition.
>>
>> Yes - we need a way to determine other search fields than just Name, but
>> this is also a UI problem. The Filter DIalog at current is already
>> overloaded so we will need to move towards a bigger design with a toolbar,
>> such as Firefox Bookmarks / History.

keep a look out for quickFilters, there will be UI changes, hopefully pointing the way
towards integration into Tb core at a later stage!

>>
>>> 5. The ability to "Cut", "Copy", and "Paste" whole filter
>>> definitions, both
>>> within a filter list and among different lists (say, the one for Inbox
>>> and
>>> the one for the newsgroups server), preferably using the same keys I
>>> use for those functions in a text editor.
>>
>> This is something I will be working on over the next 2 months for my Addon
>> quickFilters. The Filter List will definitely have to undergo a better more
>> radical UI redesign including an integrated toolbar.
>>
>> Axel
>>
>>
>
> I am curious about this filter extension because
> I am trying to sort out many ML and other occasional mailings into a large
> number of folders, and yes, sometimes I feel that a general boolean
> combinations of
> conditions may be very welcome.

Actually, quickFilters 1.8 has just been reviewed and it supports cut / copy / paste
across accounts.

https://addons.mozilla.org/en-US/thunderbird/addon/quickfilters/

One area that I will be working on next is the transposition of target paths so that
it can be used for migrating filters across folders / partitions / etc.

Of course, if you have a large number of folders you also use QuickFolders, alright?

>
<snip>

> (Sure we can certainly use the nested folder for short hand notation for
> A || B || C || D
> where B is
> b1 || b2 || b3 || b4 and "match any of the following" are checked for both
> the original and nested filters.)
>
> Of course, we should be allowed to nest deeper than 1 level.

I am not sure about the term "folder" what I proposed in the bug was nesting the
filters themselves, no folders involved. If we allow filters (or rather their filter
condition part only) to be nested within filters we can build "bracketed expressions"
relatively easily without building an overly complicated UI:

A && ((B && C) || D)

with nesting this is easy to achieve.

1. First build the B && C filter (condition),

2. then build the D filter (condition) - these two part can already be achieved using
the existing UI.

3. Then build the 2nd ((xx) || x) filter selecting any, (B&&C) , D.

4. Finally build the final filter by selecting all, A, ((xx)||x).

This would be possible "out of the box" if there was a filter condition "fulfills
existing filter condition". Of course it would only be really useful if you could
build it "from the outside in", meaning you start with A, the select "Fulfills nested
filter" and build more conditions in a separate filter properties window. This would
also have to have no action portion (as it is purely a condition) and a name for
easier readability in the resulting filter.

>
> I like the idea.
>
> The general boolean expression (introducing new syntax, etc.) seems to be
> too general without much gain.

I think the syntax as such already exists (bracketed boolean expressions), the
difficult part is building a UI that everyone understands.

I think this is what

https://bugzilla.mozilla.org/show_bug.cgi?id=297852

should be about. As far as I am aware the filtering "engine" is already able to
process such filters (I might be wrong).

>
> On the other hand, since I saw TB crashed three times in about last two
> weeks, and was surprised to find to my chagrin that it even failed to send
> the crashreport for one of the crashes, I feel we may want to put this off
> and put it on a wish-list for now, and fix whatever are causing the latest
> instability first :-(
>

At the moment it is TB22 beta testing week, so all efforts go into testing this latest
build, indeed.

Ax

[Jens Müller]

Am 29.10.2011 02:20, schrieb Michael Scharf:
> I am working in a totally different area on filters and I am not
> familiar with the Thunderbird filter code. It is very good to define a
> filter 'engine' independently of the UI. This engine should have a
> clear filter language and maybe a concrete syntax that can be used to
> build up the "abstract syntax tree" of the filter expressions. This
> would allow for different UIs to parse the filter expressions and
> generate new filter expressions.

It might be a good idea to re-use Sieve[1] for that purpose. That would
also facilitate to use Thunderbird to control server-side filtering,
e.g. with Manage Sieve[2].

-- Jens



[1] http://en.wikipedia.org/wiki/Sieve_%28mail_filtering_language%29
[2] http://tools.ietf.org/html/rfc5804