filterTiddlers core enhancement testing

[JayFresh - http://jayfresh.wordpress.com]

Hi,

I've been working on an enhancement to the core filterTiddlers method,
to support a complex filter syntax. As part of this, I've been
encouraged to write a testing TiddlyWiki to show that the function
works in the way I expected it to.

As a learning exercise (I've never done unit testing, for instance),
having seen the work of others* (Udo, Clint, Paul), I decided to put
together a simple testing framework in a TiddlyWiki.

Here's the TiddlyWiki:
http://jonny.jonathan.googlepages.com/filterTiddlersTesting.html

The testing plugin (which is pretty tiny) is here:
http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/verticals/filterTiddlersTesting/js/filterTiddlersTestingPlugin.js

I'm not trying to re-do other people's work, but I thought this might
be of interest.



J.

*http://groups.google.com/group/TiddlyWikiDev/browse_thread/thread/
2f3b62f938882c89/b03de124b94b56e5?lnk=gst&q=twUnit#b03de124b94b56e5

[JayFresh - http://jayfresh.wordpress.com]

On the subject of unit testing a patch for the core.

The extension to store.filterTiddlers now includes the ability to
limit the number of tiddlers returned in a filter and you can sort the
results. Both of these can apply to either the whole results array or
an individual step in the filtering.

Full unit testing here: http://jonny.jonathan.googlepages.com/filterTiddlersTesting.html
(go to "Run the tests")

Trac ticket: http://trac.tiddlywiki.org/ticket/454



J.

On Feb 15, 4:29 pm, "JayFresh - http://jayfresh.wordpress.com"

<jonny.jonat...@googlemail.com> wrote:
> Hi,
>
> I've been working on an enhancement to the corefilterTiddlersmethod,
> to support a complex filter syntax. As part of this, I've been
> encouraged to write a testing TiddlyWiki to show that the function
> works in the way I expected it to.
>
> As a learning exercise (I've never done unit testing, for instance),
> having seen the work of others* (Udo, Clint, Paul), I decided to put
> together a simple testing framework in a TiddlyWiki.
>
> Here's the TiddlyWiki:http://jonny.jonathan.googlepages.com/filterTiddlersTesting.html
>

> The testing plugin (which is pretty tiny) is here:http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/verticals...

[JayFresh - http://jayfresh.wordpress.com]

> The extension to store.filterTiddlersnow includes the ability to

> limit the number of tiddlers returned in a filter and you can sort the
> results. Both of these can apply to either the whole results array or
> an individual step in the filtering.

You can also return all by using the wildcard "*" in the filter.

Here's some examples of filters:

[[Tiddler Title]]
[tag[myTag]]
[tag[myTag]sort[modified]]
*
[tag[myTag]sort[-modified]limit[5]]
* [-tag[yourTag]]


J.

[chris...@gmail.com]

On Mar 26, 5:08 pm, "JayFresh - http://jayfresh.wordpress.com"

<jonny.jonat...@googlemail.com> wrote:

> [[Tiddler Title]]
> [tag[myTag]]
> [tag[myTag]sort[modified]]
> *
> [tag[myTag]sort[-modified]limit[5]]
> * [-tag[yourTag]]

I've implemented similar things in TiddlyWeb, but it looks like we
might want to do some finagling to come to some agreement on a common
syntax and semantics.

In TiddlyWeb filters, an empty filter defaults to all tiddlers in the
bag. Compound filters are spelt individually, as follows:

[[Tiddler Tittle]] [tag[systemConfig]] [sort[-modified]] [count[5]]

I ended up with that based on my analysis of the filtering code that
was already in the core and the description I was able to get out of
Martin. At that time I said, "this doesn't sound quite the same as
what Js doing".

In addition to -tag I also have !tag (meaning all those tiddlers that
do not have the indicate tag).

Instead of 'limit' I've used 'count' because a) I didn't already know
what you're up to, b) I thought limit wasn't specific enough and could
be used later to mean something along the lines of 'limit by.

I plan to add some kind of pattern matching.

It's fairly easy for me to change the syntax of the filters in
TiddlyWeb, a regular expression is used to parse the string[1] to a
stack of functions which are processed later, so as long as the
functions remain the same, it is just the parsing that needs to
change. I'd prefer to avoid changing it often, though, so if we can
reach some kind of consensus on how they should be written OR decide
that TiddlyWeb and TiddlyWiki tiddler filters should not converge that
would be cool.

[1] See compose_from_string in
http://trac.tiddlywiki.org/browser/Trunk/contributors/ChrisDent/experimental/TiddlyWeb/tiddlyweb/filter.py#L14

[Eric Shulman]

> I've been working on an enhancement to the core filterTiddlers method,
> to support a complex filter syntax.

see also:
http://www.TiddlyTools.com/#MatchTagsPlugin

which extends the basic [tag[tagname]] syntax so you can use *full
boolean expressions* in place of a simple tagname. This includes
using combination of and/or/not with nested parentheses as needed, to
create complex filtering conditionals.

While I'd love to see this syntax incorporated into the core, I had a
discussion with Jeremy in which he expressed some reservations about
that. Although I don't see the problem in the same way as he does,
I'm OK with keeping the extended syntax as a plugin enhancement that
can be installed for those who want/need it.

The only thing I'd really want core support for is to refactor the
following code in the TiddlyWiki.prototype.filterTiddlers() function:
--------------------------
switch(match[2]) {
case "tag":
this.forEachTiddler(function(title,tiddler) {
if(tiddler.isTagged(match[3]))
results.pushUnique(tiddler);
});
break;
--------------------------
by moving it to a separate function, like this:
--------------------------
TiddlyWiki.prototype.matchTags(tagexpression) {
var results=[];
this.forEachTiddler(function(title,tiddler) {
if(tiddler.isTagged(tagexpression))
results.pushUnique(tiddler);
});
return results;
}
--------------------------
then, the filterTiddlers() code would be simplified to:
--------------------------
switch(match[2]) {
case "tag":
results = this.matchTags(match[3]);
break;
--------------------------
and my plugin could easily redefine or hijack the store.matchTags()
function to extend the syntax without having to *replace* the entire
core formatter.

thanks,
-e

[JayFresh - http://jayfresh.wordpress.com]

Eric,

> The only thing I'd really want core support for is to refactor the
> following code in the TiddlyWiki.prototype.filterTiddlers() function:
> --------------------------
> switch(match[2]) {
> case "tag":
> this.forEachTiddler(function(title,tiddler) {
> if(tiddler.isTagged(match[3]))
> results.pushUnique(tiddler);
> });
> break;

[...]

> and my plugin could easily redefine or hijack the store.matchTags()
> function to extend the syntax without having to *replace* the entire
> core formatter.

This sounds reasonable. Can you think of a way that we could bring
your boolean expression handling into this function? As it is, the
filter syntax I'm designing might be able to support boolean queries
e.g. [tag[one]tag[two]] is equivalent to [tag[one AND two]], and
[tag[one]-tag[two]] is equivalent to [tag[one NOT two]]...


J.


J.

[JayFresh - http://jayfresh.wordpress.com]

Chris,

> In TiddlyWeb filters, an empty filter defaults to all tiddlers in the
> bag. Compound filters are spelt individually, as follows:
>
> [[Tiddler Tittle]] [tag[systemConfig]] [sort[-modified]] [count[5]]

So, am I correct in interpreting that as: the tiddler "Tiddler
Tittle", plus everything tagged with "systemConfig", all sorted by
modified in reverse order and limited to five results?

That's what you'd get if you ran it through my filterTiddlers method
too (if you changed "count" to "limit").

> I ended up with that based on my analysis of the filtering code that
> was already in the core

It's sufficiently basic for us to reconcile our differences and keep
everything backwards-compatible, I believe.

> In addition to -tag I also have !tag

I wanted this, but if I explain the model for filtering that I've
taken (below) you'll see why I couldn't use it.

> Instead of 'limit' I've used 'count'

What sort of "limit by..." did you have in mind? That's an interesting
possibility.

To describe my filtering model then: you start with an empty bag, if
you want to call it that. You add in handfulls of tiddlers or you take
them away. As you're adding handfulls in, you can sort and limit them
before dropping them in. At the end, you can sort or limit everything
if you like.



J.

[Eric Shulman]

> This sounds reasonable. Can you think of a way that we could bring
> your boolean expression handling into this function?

This is precisely what MatchTagsPlugin is currently doing!

The code in MatchTagsPlugin already refactors filterTiddlers() to use
store.matchTags() (which is also defined by the plugin), so that it
uses the enhanced boolean expression handling.

-e

[chris...@gmail.com]

On Mar 27, 5:03 pm, "JayFresh - http://jayfresh.wordpress.com"

<jonny.jonat...@googlemail.com> wrote:
> Chris,
>
> > In TiddlyWeb filters, an empty filter defaults to all tiddlers in the
> > bag. Compound filters are spelt individually, as follows:
>
> >   [[Tiddler Tittle]] [tag[systemConfig]] [sort[-modified]] [count[5]]
>
> So, am I correct in interpreting that as: the tiddler "Tiddler
> Tittle", plus everything tagged with "systemConfig", all sorted by
> modified in reverse order and limited to five results?

Yes.

You can play with this on live data by going to URLs like

http://peermore.com:8080/bags/TiddlyWeb/tiddlers?PlayAround%20[tag[systemConfig]]%20PlayAround%20[-tag[systemConfig]]

(that particular filter string is a bit redundant, but its like that
to demonstrate how redundant bits work)

> It's sufficiently basic for us to reconcile our differences and keep
> everything backwards-compatible, I believe.

I agree.

> > Instead of 'limit' I've used 'count'
>
> What sort of "limit by..." did you have in mind? That's an interesting
> possibility.

I'm not sure, sort takes a field argument. It seems like perhaps there
could be [limit[<field>=<arg>]].

Dunno. I think it is more that count means count, whereas limit means
lots of things.

> To describe my filtering model then: you start with an empty bag, if
> you want to call it that. You add in handfulls of tiddlers or you take
> them away. As you're adding handfulls in, you can sort and limit them
> before dropping them in. At the end, you can sort or limit everything
> if you like.

I think this is the root of the confusion. What you're describing is
not a filter, what's built in the TiddlyWiki core is not a filter, and
what I made in TiddlyWeb (trying to integrate the other two models I
encountered) is not a filter.

A filter is like a sieve. You start with everything, and you make
statements about what you do or don't want. A filter filters things
out.

What we're creating instead is a query string or a select string.

Which is fine and useful, once we've agreed that's the case. Early on
I had assumed that each step in the filter process was "filtering" the
results of the previous step. Conversation with Martin got me past
that.

What I want out of a filter is to be able to satisfy some use cases
that I can list. I think that might be a good place for us to start
working these things out. Here's some of them:

* return the N most recently modfied tiddlers in the bag
* get all the tiddlers which are tagged X
* get all the tiddlers modified in March of 2007
* get all the tiddlers whose last modifier was J
* get all the tiddlers that were modified somewhere in their history
by J
* any of the above modified by a list of tags (union or intersection),
to support things like
** N most recently modified tiddlers in the bag, tagged programming
and tagged javascript
* the above, in the final step, sorted by some field ascending or
descending

I have a concern that much of the power we're actually building into
filters is not that necessary. It is cool, but there aren't necessary
use cases to support them. In other words, its YAGNI. For example I'm
not sure I understand why it would be important to sort tiddlers as
they are being added to the list. Isn't the final sort the important
thing?

[Martin Budden]

Chris is correct, what we are creating is not a filter but a
compositor or a composer. I guess we should rename it.

There is a real use case behind the sort and count operations -
replacing the current generateRss code with a filter, but that might
be such a specialist use case that the operations are not used for
anything else.

Martin

[JayFresh - http://jayfresh.wordpress.com]

> > > Instead of 'limit' I've used 'count'
>
> > What sort of "limit by..." did you have in mind? That's an interesting
> > possibility.
>
> I'm not sure, sort takes a field argument. It seems like perhaps there
> could be [limit[<field>=<arg>]].

Ok, I haven't allowed for arguments yet.

>
> A filter is like a sieve. You start with everything, and you make
> statements about what you do or don't want. A filter filters things
> out.
>
> What we're creating instead is a query string or a select string.

I see what you mean. I'd say that the function does indeed filter
tiddlers, given that a single application of filterTiddlers "sieves"
from all available tiddlers. Subsequent filterTiddlers operations on
the returned tiddlers would sieve them further. The arguments passed
to filterTiddlers calls are what we could say are misnamed, as they
are currently termed "filter strings", but they are more like
selectors or compositors you and Martin have referred to.

> What I want out of a filter is to be able to satisfy some use cases
> that I can list. I think that might be a good place for us to start
> working these things out. Here's some of them:
>
> * return the N most recently modfied tiddlers in the bag
> * get all the tiddlers which are tagged X
> * get all the tiddlers modified in March of 2007
> * get all the tiddlers whose last modifier was J
> * get all the tiddlers that were modified somewhere in their history
> by J
> * any of the above modified by a list of tags (union or intersection),
> to support things like
> ** N most recently modified tiddlers in the bag, tagged programming
> and tagged javascript
> * the above, in the final step, sorted by some field ascending or
> descending

I can't support all of those yet, as the algebra in my syntax is
incomplete. I can't do "tagged with X AND Y", for instance.

> I have a concern that much of the power we're actually building into
> filters is not that necessary. It is cool, but there aren't necessary
> use cases to support them. In other words, its YAGNI. For example I'm
> not sure I understand why it would be important to sort tiddlers as
> they are being added to the list. Isn't the final sort the important
> thing?

Take this example: "last 5 tiddlers tagged with 'myBlog' reverse-
sorted by modified order plus last 5 tiddlers tagged with 'yourBlog'
reverse-sorted by modified".


J.

[JayFresh - http://jayfresh.wordpress.com]

The discussion about the filterTiddlers patch has shifted to a
discussion of the most suitable filter syntax to support, so the
subject has been changed.

On the table so far, we have:
* Eric's suggestion of support for BOOLEAN logic (encapsulated in the
MatchTagsPlugin)
* Chris Dent's TiddlyWeb implementation
* My (Jon Lister) patch

After some discussion with Martin today, we've put together what we
see as the differences between the approaches out there right now,
what we could do from here and the backwards-compatibility
considerations that exist. We've made no effort made here to express a
preference and it would be useful for others to amend these lists as
necessary until we all agree we're at.

A. Difference between the implemented syntaxes
1. Empty filter string means either everything or nothing returned
2. Existence of a NOT syntax or an ALL syntax
3. The "handfulls" concept, which is collecting sub-sets of tiddlers
before adding them to the results
4. Support or not for attributes to operators in the filter strings
e.g. limit[modified=JonLister]
5. More general limit or more specific count syntax
6. Full BOOLEAN logic support or incomplete support

B. What we could do and consequences
1. Support BOOLEAN syntax natively e.g. tag[this] AND tag[that AND
the_other]; this would require the DefaultTiddlers tiddler to go back
to being a list and not have filter strings in it, as the " "
character is currently interpreted as OR
2. Introduce NOT or AND syntax into JonLister's patch
3. Stick with a very simple filter syntax and handle special cases,
such as RSS saving (where we want everything except tiddlers tagged
with excludeLists) with a special function; this would probably mean
that TiddlyWeb and core TiddlyWiki diverge in filter string syntax
4. Adopt Chris Dent's syntax after the addition of support for
"handfulls"

C. Backwards-compatibility considerations
1. Handling DefaultTiddlers needs to support lists of tiddlers



J.

[FND]

> After some discussion with Martin today, we've put together what we
> see as the differences between the approaches out there right now,
> what we could do from here and the backwards-compatibility
> considerations that exist.

Great summary! (Disclaimer: As far as I can tell, not being intimately
familiar with the technical details... )

From a user's perspective, I'm very much in favor of supporting boolean
constructions.
That's because many people right now are using complex ForEachTiddler
macro calls for rather mundane issues.
Much of this could be simplified with such "filter queries", which would
also enable regular users to harness TiddlyWiki's data-processing
capabilities - which in turn could lead to more innovation.
(Disclaimer: I might be completely missing the point here... )

With regards to backwards compatibility, I believe this would be worth
risking to break some DefaultTiddlers. Any errors should be immediately
apparent to the respective authors, and very easy to fix.
Obviously, we'd need to mention this in the release notes.

-- F.

[Simon Baird]

I'm not sure how useful it is, but I have some neat code to convert arbitrary boolean tag expressions to a eval-able javascript string.

http://svn.tiddlywiki.org/Trunk/contributors/SimonBaird/mgtd3/framework/MgtdAppFramework.js

see parseTagExpr (a string method)

also see:
matchesTagExpr (a tiddler method)

and filterByTagExpr (an array method)

So you can to this for instance:

var filtered = arrayOfAllTiddlers.filterByTagExpr( "Tag1 || (Tag2 && !Tag3)"); // any arbitrarily complicated boolean expression

--
simon...@gmail.com

[Eric Shulman]

> I'm not sure how useful it is, but I have some neat code to convert
> arbitrary boolean tag expressions to a eval-able javascript string.

MatchTagsPlugin does something similar... it takes a "boolean tag
expression" using and/or/not (or && || !), and creates a regex-based
JS conditional that can be eval()ed. It then loops through the
document's tiddlers using store.forEachTiddler(), eval's the
conditional against each tiddler, resulting in an array of those
tiddlers for which the conditional evaled to TRUE.

-e

[chris...@gmail.com]

On Fri, 28 Mar 2008, JayFresh - http://jayfresh.wordpress.com wrote:

> > I have a concern that much of the power we're actually building
into
> > filters is not that necessary. It is cool, but there aren't
necessary
> > use cases to support them. In other words, its YAGNI. For example
I'm
> > not sure I understand why it would be important to sort tiddlers
as
> > they are being added to the list. Isn't the final sort the
important
> > thing?
>
> Take this example: "last 5 tiddlers tagged with 'myBlog' reverse-
> sorted by modified order plus last 5 tiddlers tagged with
'yourBlog'
> reverse-sorted by modified".

Okay, I get that, but it remains unclear (other than trying to
optimize the
calling code somewhat) why the sorting is happening in the adding to
the
results stage rather than in finalization of two filters that get
summed.

Your query described above (with a slight extension) could also be
stated
as (in tiddlyweb's filter syntax):

// psuedo-code
// get top 5 most recent tiddlers from two blogs
tiddlers = allTiddlers.filter('[tag[myBlog]] [sort[-modified]]
[count[5]]') +
allTiddlers.filter('[tag[yourBlog]] [sort[-modified]]
[count[5]]');
// interleave the results by mod time and choose 5 of the ten
results = tiddlers.filter('[sort[-modified]] [count[5]]');

Which leads me to think that a more functional approach for these
queries is
what's warranted so that you can use the results of a filter as the
argument
for another filter.

And so it is as I go down that slippery slope that I start to think
I'm
architecture astronauting and thinking about fun code rather than
useful ways
of dealing with tiddlers.

The difference between the TiddlyWeb syntax and the Jon syntax is that
sort
and count work on the "current results" of the previous strings in the
filter,
not on the stuff that is being brought in. That latter is more
powerful and
flexible, but seems like it could get out of hand in terms of
readability
rather quickly. If I understand correctly, the above code in Jon
filter could
be:

[[tag[myBlog]sort[-modified]limit[5]] [tag[yourBlog]sort[-
modified]limit[5]] \
sort[-modified]limit[5]]

Now we're starting to look like scheme or something.

Just to be clear, I'm not fighting against this stuff. I'm pointing
out
concerns that pop into my head. It's always a bummer when those
concerns come
out later. Filters are pretty critical to TiddlyWeb, so I'm excited
about
getting them as useful as possible.

In the TiddlyWeb environment complex filters aren't really required as
you can
filter the same bag[1] for content multiple times in one recipe. And
then, if
you're feeling sassy, you can filter the final results of the recipe
by
putting a filter string on the recipe itself.

[1] It's interesting how easily that phrase falls off the tongue, so
maybe
filter _is_ the right word?

--
Chris Dent http://burningchrome.com/
[...]

[chris...@gmail.com]

On Mar 28, 3:33 pm, "JayFresh - http://jayfresh.wordpress.com"

<jonny.jonat...@googlemail.com> wrote:

> A. Difference between the implemented syntaxes
> 1. Empty filter string means either everything or nothing returned
> 2. Existence of a NOT syntax or an ALL syntax
> 3. The "handfulls" concept, which is collecting sub-sets of tiddlers
> before adding them to the results
> 4. Support or not for attributes to operators in the filter strings
> e.g. limit[modified=JonLister]
> 5. More general limit or more specific count syntax
> 6. Full BOOLEAN logic support or incomplete support

Just to be clear: I've thought about item 4, but not actually
implemented it yet.

[Eric Shulman]

> > > I have a concern that much of the power we're actually building

> architecture astronauting and thinking about fun code rather than

It seems to me that this entire syntax is getting needlessly
complex... I'd like to think of the entire exercise as extending the
current "space-separated list of tiddlers" syntax, e.g.,:

[[TiddlerA]] [[TiddlerB]] [[TiddlerC]]

in which each term operates *independently* of the others and simply
concatenates another tiddler to the resulting list.

The extension for 'filtering' would be along those same lines: each
term operates independently of the others, rather than being an order-
dependent 'progressive filter' of the previously computed partial
result. Thus:

[[TiddlerA]] [tag[...]] [[TiddlerC]]

results in: TiddlerA + some list of tag-filtered tiddlers + TiddlerC

> Take this example: "last 5 tiddlers tagged with 'myBlog' reverse-
> sorted by modified order plus last 5 tiddlers tagged with
> 'yourBlog' reverse-sorted by modified".

Unlike the simple concatentation to build the list of tiddlers, the
"sort" and "count" operators would be new syntax that *is* applied as
order-dependent functions, by using separate terms within the list.
Thus:

[tag[myBlog]] [sort[-modified]] [count[5]] [tag[yourBlog] [sort[-
modified]] [count[5]]

returns 10 tiddlers, the first 5 being tagged with 'myBlog' and the
next five being tagged with 'yourBlog', each being separately sorted
in most-recent-first order, while

[tag[myBlog]] [count[5]] [tag[yourBlog] [count[5]] [sort[-modified]]
would produce the same list of tiddlers, but the results would be
sorted together, interleaving 'myBlog' and 'yourBlog' tiddlers.

In addition, to produce complex 'filters', using a boolean syntax (as
I've done with MatchTagsPlugin), makes the construction of the
conditonal logic relatively straightforward:

[tag[myBlog OR yourBlog AND NOT excludeLists]] [sort[-modified]]
[count[10]]

returns 10 suitably tagged tiddlers sorted most-recent-first.

In the language of the current discussion, each term in the list is a
separate 'bag', and the entire list result is simply a concatenation
of those 'bags'

-e

[Eric Shulman]

> Unlike the simple concatentation to build the list of tiddlers, the
> "sort" and "count" operators would be new syntax that *is* applied as
> order-dependent functions, by using separate terms within the list.

Upon re-reading this suggestion, I see a flaw in my analysis:

> [tag[myBlog]] [sort[-modified]] [count[5]] [tag[yourBlog] [sort[-
> modified]] [count[5]]

Doesn't do as I described:

> returns 10 tiddlers, the first 5 being tagged with 'myBlog' and the
> next five being tagged with 'yourBlog', each being separately sorted
> in most-recent-first order

rather, because the [sort[...]] and [count[...]] operators apply are
'progressively applied' to the entire list, it would result in only
the 5 most recent tiddlers with either tag, which is equivalent to the
shorter syntax:
[tag[myBlog]] [tag[yourBlog] [sort[-modified]] [count[5]]

In order to produce the results that were intended, the sort and count
operators would have to be applied to each term independently of each
other. This could be achieved by enclosing the operators within the
given terms, like this:

[tag[myBlog]sort[-modified]count[5]] [tag[yourBlog]sort[-
modified]count[5]]

Each term is then evaluated separately and simply concatenated to
produce the final list.

However, there are still cases where the 'progressive' aspect of sort/
count modifiers would be useful... for example, when combining tag
filtered terms with explicitly named tiddlers, where the results
should be sorted/counted as a complete set. To achieve this, the sort/
count operators could also be specified as separate terms applied to
the entire list, like this:
TiddlerA [tag[myBlog]] TiddlerB [tag[yourBlog]] [sort[-modified]]
[count[10]]

Note that in each of the above use-cases, the independent [tag[...]]
terms are combined with an implicit "OR" operator to build the final
results. To support "AND" or "NOT" operators, it has been suggested
that the [tag[...]] syntax could include a "+" (and) or "-" (not)
prefix, which effectively makes them 'progressively applied' as well.
For example
[tag[myBlog]] [+tag[yourBlog] [sort[-modified]] [count[10]]
selects tiddlers that are tagged with *both* 'myBlog' and 'yourBlog',
while
[tag[myBlog]] [-tag[yourBlog] [sort[-modified]] [count[10]]
selects tiddlers that are tagged with 'myBlog' but NOT 'yourBlog'

However, the same results could be achieved without 'progressive'
application by using the boolean expression syntax that I previously
suggested, within a single term, like this:
[tag[myBlog OR yourBlog]sort[-modified]count[10]]

In addition, with boolean expressions (using AND/OR/NOT plus nested
parentheses), you can achieve a full range of complex combination
filtering, such as:
[tag[settings AND NOT (systemConfig OR systemTheme)]]
which would simply not be possible by using the simple "+" or "-"
prefixes.

In conclusion: it seems that the syntax, in order to be really robust,
does need to be a bit more complicated than the independent
concatenation I was suggesting in my previous posting. Through use of
boolean expressions, with sort/count modifiers (either contained in a
single term or applied progressively) the overall syntax could be
consistently applied to support almost any possible use case that can
be imagined (and for most cases would probably not be nearly as
complex as the examples I've given above)

-e

[JayFresh - http://jayfresh.wordpress.com]

Hello,

I know it's been a while since we last discussed this, but I have been
looking at this area again.

Eric's last post summarised very well how a filter syntax can emerge
from an extension to the current "space-separated tiddler list"
syntax.

I'll pick up by commenting on the last paragraphs of Eric's post where
the thinking is more exploratory.

[talking about building up filter results through 'progressive'
opertions]

> Note that in each of the above use-cases, the independent [tag[...]]
> terms are combined with an implicit "OR" operator to build the final
> results. To support "AND" or "NOT" operators, it has been suggested
> that the [tag[...]] syntax could include a "+" (and) or "-" (not)
> prefix, which effectively makes them 'progressively applied' as well.
> For example
> [tag[myBlog]] [+tag[yourBlog] [sort[-modified]] [count[10]]
> selects tiddlers that are tagged with *both* 'myBlog' and 'yourBlog',
> while
> [tag[myBlog]] [-tag[yourBlog] [sort[-modified]] [count[10]]
> selects tiddlers that are tagged with 'myBlog' but NOT 'yourBlog'
>
> However, the same results could be achieved without 'progressive'
> application by using the boolean expression syntax that I previously
> suggested, within a single term, like this:
> [tag[myBlog OR yourBlog]sort[-modified]count[10]]

The '+' and '-' additions to the [tag[...]] syntax can be applied as
part of each "bag" (used in the sense of an intermediary part of the
results). We can almost do the tag-related parts of the above in my
current implementation like so (not the use of "or" not "and" in the
first example):

[tag[myBlog]] [tag[yourBlog]]
selects tiddlers that are tagged with "myBlog" or "yourBlog"
[tag[myBlog]-tag[yourBlog]]
selects tiddlers that are tagged with "myBlog" but not "yourBlog"

> In addition, with boolean expressions (using AND/OR/NOT plus nested
> parentheses), you can achieve a full range of complex combination
> filtering, such as:
> [tag[settings AND NOT (systemConfig OR systemTheme)]]
> which would simply not be possible by using the simple "+" or "-"
> prefixes.

The problem is that we don't have support for "and" in the bags, but
as Eric's suggested, that could be implemented using [+tag[...]] -
currently, that behaves exactly the same as [tag[...]]. I can see a
need to want to filter for tiddlers tagged with multiple tiddlers
(i.e. "and" constructions).

As an aside, you can convert Eric's quite complex boolean filter
above to this syntax like so:

[tag[settings AND NOT (systemConfig OR systemTheme)]]

is:
[tag[settings]-tag[systemConfig]-tag[systemTheme]]

Hmm... I don't think you could convert this, even using something like
suggested [+tag[...]] to support "and":
[tag[settings AND NOT (systemConfig AND systemTheme]]


J.

[JayFresh - http://jayfresh.wordpress.com]

Hi,

After some discussion with Chris and Jeremy today, we're suggesting
three amendments to bring us closer to a converged syntax and
semantics for TiddlyWeb and TiddlyWiki:

1) Chris will support "composite sub-expressions" e.g.
[tag[blah]sort[modified]]

2) Jon will change TiddlyWiki.prototype.filterTiddlers to take an
optional argument that switches the return value when the input filter
is null: if false (or null), a null filter returns no tiddlers (for
backwards-compatibility); if true, a null filter returns all the
tiddlers (the behaviour exhibited on Chris' TiddlyWeb)

3) A new core filterTiddlers function will be created, which
TiddlyWiki.prototype.filterTiddlers will wrap, which will take an
array and return an array, allowing expressions to be chained.

A big part of this discussion has been about full support for Boolean
expressions. We think that with the addition of (3), Unions are now
supported.


J.

On Apr 16, 1:48 pm, "JayFresh - http://jayfresh.wordpress.com"

[Eric Shulman]

> 3) A new core filterTiddlers function will be created, which
> TiddlyWiki.prototype.filterTiddlers will wrap, which will take an
> array and return an array, allowing expressions to be chained.
>
> A big part of this discussion has been about full support for Boolean
> expressions. We think that with the addition of (3), Unions are now
> supported.

I have already implemented full boolean expression handling via
http://www.TiddlyTools.com/#MatchTagsPlugin
which hijacks the core's story.prototype.getTaggedTiddlers() function.

This hijack allows the current [tag[foo]] syntax to be transparently
extended so that complex expressions such as:

[tag[foo OR (bar AND baz) OR NOT (mumble AND (frotz OR gronk))]]

can be processed when passed to the current (TW2.4) core
filterTiddlers() function which, in turn, calls upon
getTaggedTiddlers() to actually retrieve the desired array of
tiddlers.

Please make sure that the proposed changes to the core function won't
break the enhanced syntax provided by MatchTagsPlugin.

thanks,
-e

[jayfresh]

Eric,

> I have already implemented full boolean expression handling via
>    http://www.TiddlyTools.com/#MatchTagsPlugin
> which hijacks the core's story.prototype.getTaggedTiddlers() function.

...

>
> Please make sure that the proposed changes to the core function won't
> break the enhanced syntax provided by MatchTagsPlugin.
>

Looking at the code for MatchTagsPlugin, it seems to override
TiddlyWiki.prototype.filterTiddlers: note the revision history -
2008.05.14 [1.3.4] instead of hijacking getTaggedTiddlers(), added
tweak of filterTiddlers() prototype to replace getTaggedTiddlers()
with getMatchingTiddler() so that core use of getTaggedTiddlers() does
not perform boolean processing of tiddler titles such as To Be or not
To Be. Also, improved "filter error" messages in getMatchingTiddlers()
to report tag expression in addition to actual eval error.

What's your opinion about the best route now so I don't suggest a
breaking change?


J.

[Eric Shulman]

> Looking at the code for MatchTagsPlugin, it seems to override
> TiddlyWiki.prototype.filterTiddlers: note the revision history -
>

> What's your opinion about the best route now so I don't suggest a
> breaking change?

The new code doesn't 'override' the whole filterTiddlers() function.
Instead, it uses a new technique I've developed to directly *tweak*
the core function by using a regexp replace() to substitute all
occurences of "getTaggedTiddlers" with "getMatchingTiddlers":
-------------
var fn=TiddlyWiki.prototype.filterTiddlers;
fn=fn.toString().replace(/getTaggedTiddlers/g,"getMatchingTiddlers");
eval("TiddlyWiki.prototype.filterTiddlers="+fn);
--------------
Note that the results returned by getMatchingTiddlers() are 100%
compatible with those returned by getTaggedTiddlers()... Thus, as long
as your proposed changes are still using 'getTaggedTiddlers', the
above code tweak should continue to work.

-e

[Jonathan Lister]

Ah, that's very clever. Ok, so my code at the moment would break that as I don't use getTaggedTiddlers any more. Having had a quick look at things, I think I can move back to using getTaggedTiddlers without any trouble, and it will make my plugin a bit shorter. The performance impact amounts to an extra sort function every time a tag sub-element of the filter expression is processed, but I think that isn't so bad.

J.

--
http://www.jaybyjayfresh.co.uk

[Eric Shulman]

> Ah, that's very clever. Ok, so my code at the moment would break that as I
> don't use getTaggedTiddlers any more. Having had a quick look at things, I
> think I can move back to using getTaggedTiddlers without any trouble, and it

As a general rule-of-thumb, you should always try to use a core-
provided 'access function' whenever practical. This ensures that any
changes in the core will be automatically applied to your code,
without having to replicate those changes locally.

It also helps to ensure that your code will 'play well' with plugins
that extend the logic and data handling of various core functions.
For example, IncludePlugin modifies several common core functions that
access the tiddler 'store' data, such as getTaggedTiddlers(), so that
they will be able to access tiddlers that have been loaded into the
*separate* store objects created on-the-fly by IncludePlugin. Thus,
by using getTaggedTiddlers() in your filterTiddlers() code, it will
also have access to 'included' tiddlers, even though your code has no
special knowledge of IncludePlugin's implementation.


-e

[Jonathan Lister]

As a general rule-of-thumb, you should always try to use a core-
provided 'access function' whenever practical.  This ensures that any
changes in the core will be automatically applied to your code,
without having to replicate those changes locally.

It also helps to ensure that your code will 'play well' with plugins
that extend the logic and data handling of various core functions.

Very good point..!

J.

 

For example, IncludePlugin modifies several common core functions that
access the tiddler 'store' data, such as getTaggedTiddlers(), so that
they will be able to access tiddlers that have been loaded into the
*separate* store objects created on-the-fly by IncludePlugin.  Thus,
by using getTaggedTiddlers() in your filterTiddlers() code, it will
also have access to 'included' tiddlers, even though your code has no
special knowledge of IncludePlugin's implementation.

-e

--
http://www.jaybyjayfresh.co.uk

[Jonathan Lister]

Eric,

I've updated the filterTiddlersPlugin to use the core store.getTaggedTiddlers as we discussed, although I have run into another problem: the MatchTagsPlugin appears to be being called before filterTiddlersPlugin, and so my plugin overwrites yours. If my version of filterTiddlers in injected into the core, clearly this problem will go away, but I wanted to alert you to this as it makes it difficult to test!

Here is the revised filterTiddlersPlugin code:

http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/plugins/filterTiddlersPlugin.js

J.

[Eric Shulman]

On Jun 17, 8:29 am, "Jonathan Lister" <jnthnl...@googlemail.com>
wrote:

> Eric,
> I've updated the filterTiddlersPlugin to use the core
> store.getTaggedTiddlers as we discussed, although I have run into another
> problem: the MatchTagsPlugin appears to be being called before
> filterTiddlersPlugin, and so my plugin overwrites yours. If my version of
> filterTiddlers in injected into the core, clearly this problem will go away,
> but I wanted to alert you to this as it makes it difficult to test!

How about renaming "filterTiddlersPlugin" to
"FilterTiddlersPlugin" (note capital "F")? Not only does this ensure
that your plugin is loaded before MatchTagsPlugin. In addition, a
plugin *is* just a tiddler, so why not give it a name that is a proper
WikiWord...

-e

[Paul Downey]

>
> How about renaming "filterTiddlersPlugin" to
> "FilterTiddlersPlugin" (note capital "F")? Not only does this ensure
> that your plugin is loaded before MatchTagsPlugin.

The alphabetic ordering of how plugins are loaded keeps tripping me up!

> In addition, a plugin *is* just a tiddler, so why not give it a name
> that is a proper WikiWord...

I ended up renaming a "LogMessagePlugin" to "_LogMessagePlugin"
to game the sort order, and keep it as a WikiWord ..

I'm not happy with this hack, and would prefer to have some more
reliable way of ordering the loading of plugins.

Didn't someone have a proposal for dependency chaining?

Paul
--
http://blog.whatfettle.com

[jnthnlstr]

Latest update:

1) FilterTiddlersPlugin includes the change mentioned below, so as not
to break any existing plugins that expect the use of
getTaggedTiddlers.
2) The "|" character is supported as a means of successively filtering
the results i.e. AND logic. For example, "[!tag[systemConfig]] | [!
tag[excludeLists]]" returns everything not tagged with systemConfig or
excludeLists. This would be the set of tiddlers you would normally
output in your RSS feed. This was not possible in the previous
version, as using the filter expression "[!tag[systemConfig]] [!
tag[excludeLists]]" would results in anything not tagged systemConfig,
merged with anything not tagged excludeLists. The same applies to the
expression "[!tag[systemConfig]!tag[excludeLists]]".

Source at: http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/plugins/FilterTiddlersPlugin.js


J.

On Jun 16, 3:40 pm, "Jonathan Lister" <jnthnl...@googlemail.com>
wrote:

[jnthnlstr]

This is an attempt to document the FilterTiddlers function on the
community wiki (I have built on what was already there):

http://tiddlywiki.org/wiki/Dev:FilterTiddlers



J.

On Jul 7, 6:11 pm, jnthnlstr <jnthnl...@googlemail.com> wrote:
> Latest update:
>
> 1) FilterTiddlersPlugin includes the change mentioned below, so as not
> to break any existing plugins that expect the use of
> getTaggedTiddlers.
> 2) The "|" character is supported as a means of successively filtering
> the results i.e. AND logic. For example, "[!tag[systemConfig]] | [!
> tag[excludeLists]]" returns everything not tagged with systemConfig or
> excludeLists. This would be the set of tiddlers you would normally
> output in your RSS feed. This was not possible in the previous
> version, as using the filter expression "[!tag[systemConfig]] [!
> tag[excludeLists]]" would results in anything not tagged systemConfig,
> merged with anything not tagged excludeLists. The same applies to the
> expression "[!tag[systemConfig]!tag[excludeLists]]".
>

> Source at:http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/plugins/F...

[jnthnlstr]

Hello.

Two things:

1) after discussing with Chris and Martin, have come to the conclusion
that using "|" to separate steps in the filter, which essentially
evaluate to a logical AND, isn't great because although it is the UNIX
symbol for a pipe (which makes sense in this context), it is mixing
spaces - in JavaScript "||" means OR, which is very close to "|" and
therefore confusing.

However, whilst "&&" is AND in JavaScript, I am against using this to
split the steps of a filter, just because I feel it is not very
compact. Are my sensibilities misaligned? Any advice/opinions?

2) the implementation of filterTiddlers splits the filter string into
pieces using a simple String.split(), which is going to cause problems
if whatever character we settle on from point 1 appears in a Tiddler
title.

Tomorrow, Martin and I are going to have a go at fixing this using a
regular expression. Aha, regex joy.


J.

[jnthnlstr]

Chaps and chappesses,

Trac ticket for this patch:
http://trac.tiddlywiki.org/ticket/699

Question of "|" not resolved, any opinions welcome.

Updated implementation not to use String.split("|") to split up the
filter string into steps; instead, uses string examination to check
that the delimiter is outside the bracketed steps, which allows you to
use the delimiter inside a tiddler name or tag.


J.

On Jul 8, 5:42 pm, jnthnlstr <jnthnl...@googlemail.com> wrote:
> Hello.
>
> Two things:
>
> 1) after discussing with Chris and Martin, have come to the conclusion

> that using "|" to separate steps in thefilter, which essentially

> evaluate to a logical AND, isn't great because although it is the UNIX

> symbol for apipe(which makes sense in this context), it is mixing

> spaces - in JavaScript "||" means OR, which is very close to "|" and
> therefore confusing.
>
> However, whilst "&&" is AND in JavaScript, I am against using this to

> split the steps of afilter, just because I feel it is not very

> compact. Are my sensibilities misaligned? Any advice/opinions?
>

> 2) the implementation of filterTiddlers splits thefilterstring into

> pieces using a simple String.split(), which is going to cause problems
> if whatever character we settle on from point 1 appears in a Tiddler
> title.
>
> Tomorrow, Martin and I are going to have a go at fixing this using a
> regular expression. Aha, regex joy.
>
> J.
>
> On Jul 7, 7:02 pm, jnthnlstr <jnthnl...@googlemail.com> wrote:
>
> > This is an attempt to document the FilterTiddlers function on the
> > community wiki (I have built on what was already there):
>
> >http://tiddlywiki.org/wiki/Dev:FilterTiddlers
>
> > J.
>
> > On Jul 7, 6:11 pm, jnthnlstr <jnthnl...@googlemail.com> wrote:
>
> > > Latest update:
>
> > > 1) FilterTiddlersPlugin includes the change mentioned below, so as not
> > > to break any existing plugins that expect the use of
> > > getTaggedTiddlers.
> > > 2) The "|" character is supported as a means of successively filtering
> > > the results i.e. AND logic. For example, "[!tag[systemConfig]] | [!
> > > tag[excludeLists]]" returns everything not tagged with systemConfig or
> > > excludeLists. This would be the set of tiddlers you would normally
> > > output in your RSS feed. This was not possible in the previous

> > > version, as using thefilterexpression "[!tag[systemConfig]] [!

> > > tag[excludeLists]]" would results in anything not tagged systemConfig,
> > > merged with anything not tagged excludeLists. The same applies to the
> > > expression "[!tag[systemConfig]!tag[excludeLists]]".
>
> > > Source at:http://svn.tiddlywiki.org/Trunk/contributors/JonathanLister/plugins/F...
>
> > > J.
>
> > > On Jun 16, 3:40 pm, "Jonathan Lister" <jnthnl...@googlemail.com>
> > > wrote:
>
> > > > Ah, that's very clever. Ok, so my code at the moment would break that as I
> > > > don't use getTaggedTiddlers any more. Having had a quick look at things, I
> > > > think I can move back to using getTaggedTiddlers without any trouble, and it
> > > > will make my plugin a bit shorter. The performance impact amounts to an

> > > > extra sort function every time a tag sub-element of thefilterexpression is

[chris...@gmail.com]

The short version of the posting is:

Do you think you could write an EBNF for your current form of filter
strings? It would be very useful for other implementors and perhaps
help clarify some ambiguities.

The longer version:

Does your implementation protect against and/or support things like
this:

[[tag[foobar]] | [tag[boom]]] | [tag[systemConfig] [limit[10]]

My read on the current code at your linked patch is that that will
split on the first '|' prior to splitting on the second '|', possibly
causing a breakdown in the nestedness of the first parenthetical
statement. But perhaps such nestedness is not supported or desired. I
don't know.

More abstract version:

I really think the filter string syntax needs to be described at a
more abstract level, rather than in code. As we continue to work on it
in code, the grammar feels like ambiguities are being discovered,
piecemeal. If we just wrote a grammar, we'd know where they are _now_.
Major questions present right now are related to intersections and
precedence.

More self deprecating version:

It could be I'm just in this mess because I'm trying to tokenize
filter strings rather than use regular expressions, but I'm inclined
to think that my efforts are exposing weakness that would exist in
either technique. More thought required.

[jnthnlstr]

Chris,

I've had a go at writing an EBNF for the filter syntax - take a look
here:

http://tiddlywiki.org/wiki/Dev:FilterTiddlers

To answer your specific question,

[[tag[foobar]] | [tag[boom]]] | [tag[systemConfig] [limit[10]]

is invalid, as the '|' character expects only fully-formed filter
steps on either side of it. It is, if you like, the sign to use the
results of one filter as the input set for the next.



J.

[chris...@gmail.com]

On Jul 14, 9:23 am, jnthnlstr <jnthnl...@googlemail.com> wrote:
> Chris,
>
> I've had a go at writing an EBNF for the filter syntax - take a look
> here:
>
> http://tiddlywiki.org/wiki/Dev:FilterTiddlers

Thanks, that does clarify some things. For example, base on that
grammar, it doesn't look like there's any longer support for a
bareword (without [ ]) indicating a Tiddler.

Also, unless I'm misreading it, your grammar won't generate the string
in this example:

store.filterTiddlers("[tag[mish]] [tag[mash]]") // two filter
elements

To me, that looks like two filters, each containing a filterStep of
one filterElement. To get that result using the grammar I have to add
a step to the top of the grammar: the identification of filters inside
a filter_string. Adding that step is where ambiguity about | enters
into the show.

BTW: I feel like I'm being a jerk here, but I don't mean to be. I'm
earnestly trying to find the steady state. If I seem like I'm being a
jerk, sorry about that. It's the language.

[Jonathan Lister]

Thanks for reviewing Chris, take a look at the revised grammar.

http://tiddlywiki.org/wiki/Dev:FilterTiddlers

You're right about not allowing bare tiddler titles at the moment; it's simpler for writing the grammar out, although for backwards-compatibility we'll probably need to include it (so that existing DefaultTiddlers filter work).



J.

--
http://www.jaybyjayfresh.com