TiddlyWeb Issues: filters

[chris...@gmail.com]

I've been wanting to release a 1.0 version of TiddlyWeb for some time
now, but there continue to be a few issues which may or may not need
to be addressed before such a thing can happen:

* filters
* server side validation
* something I can't remember now (feel free to point things out)

In this message I'd like to discuss filters.

In the original design discussions surrounding what eventually became
TiddlyWeb, filters were created as a syntax for selecting a subset of
tiddlers from a bag listed in a recipe. A recipe would list several
bags, and filters would limit the tiddlers pulled from the bags.

A TiddlyWeb evolved, it became clear that a filter could be used on
any collection of tiddlers: a bag, the list of tiddlers made from a
recipe, etc. So that's how things are now.

The name "filter" and the syntax used comes from TiddlyWiki filters,
used in the <<list>> macro.

However the implementation is not exactly the same and therein lies
the rub, the issue, and the question. The name and similarity in
syntax leads people to think that the implementation and results will
be the same as when using TiddlyWiki filters. It's not. This issue has
been known for some time but nothing done about it in the hope that
some filter use cases would accumulate and from those a decision could
be made about how best to "fix" filters in TiddlyWeb.

TiddlyWeb has had a fairly significant number of downloads, and I've
had reports from a fair few people about what they are up to, but the
filter use cases have not accumulated. Please take this message as a
(verbose) invitation to provide them.

There seems to be a few different ways forward (some of these might be
combined):

1. Carry on with filters as they are now and let them evolve as
requirements are revealed.
2. Rename TiddlyWeb filters to something else, to distinguish them
from TiddlyWiki filters.
3. Make TiddlyWeb filters act exactly the same as TiddlyWiki filters.
4. Create something new and better that is specific for the TiddlyWeb
setting.

4 is of interest to me (as opposed to 3) because I think the bags with
filters in recipes use case is not the same as the <<list>> macro use
case in TiddlyWiki. The former is kind of a selector, while the latter
is effectively a filter.

3 is not as easy as you might think at first glance. It would be
possible to port the TiddlyWiki filter Javascript to Python and carry
on from there, but it would be a deep chance to how filters currently
work in TiddlyWeb. I feel this way of working is a good thing (in fact
I think TiddlyWiki would benefit from using it). Last I checked, in
TiddlyWiki a regular expression parses the filter string and while
this filter string is being parsed a results list of filtered/found
tiddlers is created and extended. In TiddlyWeb the filter string is
parsed to a list of filter functions. After the parsing those filter
functions are applied to the collection of tiddlers being filtered.

This separation of parsing and filtering means that the filtering
mechanism is easy to extend without having to invade the core filter
code. This a patently good thing.

The couple of times I've given it a brief effort I've found myself too
stupid or too discouraged to port the TiddlyWiki filter handling to
TiddlyWeb while still preserving the function compilation process that
TiddlyWeb uses, so I've stalled out. Assistance most welcome!

2 is growing in my mind as the right thing to do. Filters as currently
implemented meld several concepts that are quite distinct into one
mechanism and I think is probably bad for future easy development.
Filters are used to both get a limited set of tiddlers as well as to
sort tiddlers. Better to have things which limit and things which
sort: limiters and sorters. There may be others things as well.

1 might be acceptable except that there are some significant problems
with the current implementation. The most notable is that in one
filter string, two filters result in a union of the found tiddlers,
not the intersection. So if you:

/bags/system/tiddlers?filter=[tag[systemConfig]][tag[foobar]]

you get all the tiddlers that are tagged systemConfig and all the
tiddlers that are tagged foobar. This doesn't seem aligned with the
intent, especially since it is possible to get that result, in a
recipe, by listing the same bag twice:

/bags/system/tiddlers?filter=[tag[foobar]]
/bags/system/tiddlers?filter=[tag[systemConfig]]

My conclusion from this is that some kind of change needs to happen.
My inclination is to experiment with something new (thus option 4) but
I'm willing to consider option 3 if we can come up with a way to code
it cleanly.

I'd:

* rather not make this decision unilaterally
* prefer to make the decision before a 1.0 so that the big changes
that will happen still fall under the (vaguely meaningless) umbrella
of "alpha" or "beta"

Thank you for your input.

[Eric Shulman]

> 1 might be acceptable except that there are some significant problems
> with the current implementation. The most notable is that in one
> filter string, two filters result in a union of the found tiddlers,
> not the intersection. So if you:
>
>    /bags/system/tiddlers?filter=[tag[systemConfig]][tag[foobar]]
>
> you get all the tiddlers that are tagged systemConfig and all the
> tiddlers that are tagged foobar. This doesn't seem aligned with the
> intent, especially since it is possible to get that result, in a
> recipe, by listing the same bag twice:
>
>    /bags/system/tiddlers?filter=[tag[foobar]]
>    /bags/system/tiddlers?filter=[tag[systemConfig]]

I've never liked the TW filter syntax provided by the core. It uses a
unique syntax and semantics that, as you've observed, don't
necessarily jibe with what people expect.

I (and many others) prefer using the well-defined syntax of Boolean
logic instead. That's why I wrote:
http://www.TiddlyTools.com/#MatchTagsPlugin
which extends the TW filter syntax so that instead of using a simple
tag value, you can use of any combination tag values joined with
boolean operators (AND, OR, NOT) and nested parentheses, e.g.:
[tag[foobar and systemConfig]]
OR
[tag[(foobar or (mumble and systemConfig)) and not (frotz or
gronk)]]

Although it's not the same approach as TiddlyWeb is currently using,
it does offer the advantage of using a syntax that is both well-known
and sufficiently robust to support a full range of complex tiddler
filtering expressions for virtually any use-case.

Note: this extended support doesn't prevent use of the existing filter
syntax, so you can still use [limit[nn]] and [sort[fieldname]] along
with a boolean [tag[...]] expression, like this:

[tag[foobar and systemConfig]][limit[5]][sort[-modified]]

-e
Eric Shulman
TiddlyTools / ELS Design Studios

[FND]

> My conclusion from this is that some kind of change needs to happen.
> My inclination is to experiment with something new

I think some unencumbered exploration in this area would be a good thing.
As we've seen, trying to mimic TiddlyWiki's behavior doesn't get us very
far - especially given that the current implementation seems incomplete.
So if you were to come up with an "optimal" solution without having to
worry about compatibility, we could compare the two approaches, perhaps
even revisiting the TiddlyWiki core's implementation.

-- F.

[Jeremy Ruston]

For those that are interested, the motivation for the design of the
TiddlyWiki filter syntax was to make it compatible with the existing
syntax for the DefaultTiddlers tiddler. Previously, DefaultTiddlers
had been defined as a whitespace-delimited list of tiddler titles to
be opened at startup, with the wiki-ish convention that tiddler titles
containing spaces would be quoted with square brackets. Hence, you get
DefaultTiddlers that look like this:

----
MyFirstTiddler
[[Another Tiddler]]
----

The use-case I wanted to support was things like being able to specify
in DefaultTiddlers things like "show me the last 10 tiddlers to have
been modified", which frequently cropped up from users interested in
using TiddlyWiki in a blog-like way. Given such an extended syntax, I
could see other places (like the <<list>> macro) where filters might
also be handy.

The easiest way to do this without introducing a breaking change to
the way that DefaultTiddlers was parsed, hence the design that is now
documented on tiddlywiki.org:

http://tiddlywiki.org/index.php?title=Dev:FilterTiddlers

Returning to Chris' original question, being an eternal optimist, I
still believe that (3) is feasible, but if we can't manage that in
time for 1.0, I'd have thought that (2) would be the sensible option.

Cheers

Jerm

--
Jeremy Ruston
mailto:jer...@osmosoft.com
http://www.tiddlywiki.com

[Martin Budden]

In my view, either the TiddlyWeb filters are exactly the same as the
TiddlyWiki ones, or they are not. So the options 1,2, and 4 are just
variations are a theme. "3 or not 3?", that is the question.

I also think that something similar to TiddlyWiki syntax, but not
exactly the same is worse than something completely different. So I
think I'd go for a different syntax, and having made that decision,
I'd just adopt a Boolean syntax as Eric suggests.

Martin


2009/4/29 Jeremy Ruston <jeremy...@gmail.com>:

[chris...@gmail.com]

On Apr 29, 11:31 am, Jeremy Ruston <jeremy.rus...@gmail.com> wrote:
> Returning to Chris' original question, being an eternal optimist, I
> still believe that (3) is feasible, but if we can't manage that in
> time for 1.0, I'd have thought that (2) would be the sensible option.

As Martin so aptly put it, it really boils down to 3 or not 3.

So that rephrases the question as: are what are currently called
filters in TiddlyWeb performing the same function as what are called
filters in TiddlyWiki?

I'm fairly certain the answer is no; the desired selecting/limiting/
sorting functionality desired in TiddlyWeb is both semantically and
lexically different from TiddlyWiki because of the primary position of
recipes and bags in the context.

However I'm willing to be convinced that this is because I have an
antipathy to the TiddlyWiki filter code.

[Martin Budden]

I agree. There is superficial similarity between filters in TiddlyWiki
and TiddlyWeb, but they are performing different functions.

Martin

2009/4/29 cd...@peermore.com <chris...@gmail.com>:

[Jeremy Ruston]

> I'm fairly certain the answer is no; the desired selecting/limiting/
> sorting functionality desired in TiddlyWeb is both semantically and
> lexically different from TiddlyWiki because of the primary position of
> recipes and bags in the context.
>
> However I'm willing to be convinced that this is because I have an
> antipathy to the TiddlyWiki filter code.

I'd also given thought to the difference between selection and
filtering. My simplistic way of fitting them together was to invent
the idea of a implicit selector at the start of a filter list. Thus, a
construction like "[tag[blog]sort[modified]limit[10]]" would be
treated as "[select[*]tag[blog]sort[modified]limit[10]]".

Cheers

Jerm

[chris...@gmail.com]

On Apr 29, 12:46 pm, Jeremy Ruston <jeremy.rus...@gmail.com> wrote:
> I'd also given thought to the difference between selection and
> filtering. My simplistic way of fitting them together was to invent
> the idea of a implicit selector at the start of a filter list. Thus, a
> construction like "[tag[blog]sort[modified]limit[10]]" would be
> treated as "[select[*]tag[blog]sort[modified]limit[10]]".

Yeah, that's perniciously hard to reliably parse for both a person (at
least this) and a computer, and it couples into the same engine three
very different pieces of functionality. That coupling is what you
might call anathema to the TiddlyWebWay™.

As things currently stand filters in TiddlyWeb are though of as a
single query parameter on a URL (?filter=<the filter string>). Better
I think would be (at least) three:

* select or filter for things like <fieldname>:<fieldspec (include
simple regular expressions)>
* sort
* limit or count (can't decide if this would be just a number of would
have some kind of spec to)

Then you can parse the query string, and have specific bits of code
that select, sort or limit do what they do best, rather than a
monolith that is confusing.

Set handling could be done as follows (I'm making this up as I go, so
please suggest something else if this seems wack):

* Unions: use multiple select query paramters, each parameter does
another select on the full set of tiddlers are the collection being
filtered/selected from: eg select=tag:blog&select=tag:coment gets all
tiddlers tagged blog plus all tiddlers tagged comment.
* Intersections: multiple selection statements in one select paramter:
select=tag:blog%20tag:comment gets only those tiddlers that are tagged
both tag and comment.

(One reason I like this idea is that I find it easy to imagine the
solution: it is still based on generating functions from parsing the
query string: tag:blog leads to a particular function, whether it is
doing an intersection or union is based on which set of tiddlers are
fed to it.)

Obviously someone can come up with some case that this syntax will not
satisfy but we can do that all day and night, with any syntax. No
syntax will be perfect. Sometimes you just have to say, "you can't do
that with this, you'll need to do your own thing."

Some of the shortcomings in this syntax can be overcome by having the
same bag listed in one recipe multiple times and filtering the output
of the entire recipe (this is already the common way to create a
recent changes syndication feed).

Comments?

[Martin Budden]

There's a slight problem with the syntax, namely if you read

select=tag:blog&select=tag:coment

quickly. the "&" sticks in the mind and it gets parsed as

select=tag:blog&tag:coment

that is as an intersection rather than a union. What I'm trying to say
is I'm used to applying the boolean before the selection rather than
after it and so a cursory glance is misread. I don't think I'm
atypical in that respect.

Martin


2009/4/29 cd...@peermore.com <chris...@gmail.com>:

[chris...@gmail.com]

On Apr 29, 4:33 pm, Martin Budden <mjbud...@gmail.com> wrote:
> There's a slight problem with the syntax, namely if you read
>
> select=tag:blog&select=tag:coment
>
> quickly. the "&" sticks in the mind and it gets parsed as
>
> select=tag:blog&tag:coment

Then use ';'. It is a better cgi parameter separator anyway because it
doesn't need to be HTML escaped.

select=tag:blog;select=tag:comment

[Jeremy Ruston]

> Yeah, that's perniciously hard to reliably parse for both a person (at
> least this) and a computer, and it couples into the same engine three
> very different pieces of functionality. That coupling is what you
> might call anathema to the TiddlyWebWay™.

I think that part of the dissonance is that the TiddlyWiki filter
syntax is intentionally wiki-ish, while the TiddlyWeb design is being
drawn more towards URI-ishness. And URI-ness intentionally beats
wiki-ness for TiddlyWeb, I think. Originally, I was attracted to a
common syntax for the client and the server because it makes it easier
to offload client-side work to the server, but, I guess the ability to
convert between the two different syntaxes would be OK.

I'm wondering if we exploit the idea of the same bag being listed in
the same recipe multiple times we can't make the syntax substantially
simpler? The best syntax would surely be almost no syntax at all :)

Cheers

Jeremy

[Paul Downey]

On 29 Apr 2009, at 17:55, Jeremy Ruston wrote:

>> Yeah, that's perniciously hard to reliably parse for both a person
>> (at
>> least this) and a computer, and it couples into the same engine three
>> very different pieces of functionality. That coupling is what you
>> might call anathema to the TiddlyWebWay™.
>
> I think that part of the dissonance is that the TiddlyWiki filter
> syntax is intentionally wiki-ish, while the TiddlyWeb design is being
> drawn more towards URI-ishness. And URI-ness intentionally beats
> wiki-ness for TiddlyWeb, I think.

I thought that was true of TiddlyWiki filters, given they're also
useful for permaview fragids?

I gained a lot of utility using a newline to combine multiple filters:

http://whatfettle.com/2008/07/SegmentedFilterPlugin/

but using ';' as a union operator seems fine and more fragid friendly.

I'm not sure intersection adds that much beyond optimisation and
syntactic sugar.

> Originally, I was attracted to a
> common syntax for the client and the server because it makes it easier
> to offload client-side work to the server, but, I guess the ability to
> convert between the two different syntaxes would be OK.

I'd push for a common syntax, or at least seeing if one can be a
subset of the other.

Paul (psd)
--
http://blog.whatfettle.com

[Jeremy Ruston]

>> I think that part of the dissonance is that the TiddlyWiki filter
>> syntax is intentionally wiki-ish, while the TiddlyWeb design is being
>> drawn more towards URI-ishness. And URI-ness intentionally beats
>> wiki-ness for TiddlyWeb, I think.
>
> I thought that was true of TiddlyWiki filters, given they're also
> useful for permaview fragids?

Yes, I think at the time I was so thrilled that I'd figured out a way
to get permaviews working at all that I didn't pay enough attention to
the hideousness of the URLs. All that %5B%5C stuff is not pretty at
all.

On a related point, I carefully made the TiddlyWiki client code only
use the bit of the URI after the '#' fragment identifier, because the
specs seemed to imply that the fragment was for the client, and the
query string was for the server. I was particularly thinking that
people might need to host TiddlyWiki documents on URIs that they
couldn't control, and therefore might include random query strings.

Now that we've got TiddlyWeb, it has occurred to me that it might be
neat to give the client code (when run in serverless mode) the ability
to parse query strings itself. It would then be possible to give
someone a static URI for a permalink to a particular tiddler in a
TiddlyWiki document, and for that URI to survive the TiddlyWiki
document being upgraded into an instance of TiddlyWeb.

Cheers

Jerm

[chris...@gmail.com]

On Apr 29, 1:29 pm, "cd...@peermore.com" <chris.d...@gmail.com> wrote:

> * select or filter for things like <fieldname>:<fieldspec (include
> simple regular expressions)>
> * sort
> * limit or count (can't decide if this would be just a number of would
> have some kind of spec to)

I've started a 'filter2' branch on github to start implementing
something along these lines. To get started I've written a short spec:
http://github.com/tiddlyweb/tiddlyweb/blob/filters2/FILTERS2

Please comment, redirect, or otherwise engage yourself as you feel.

[chris...@gmail.com]

On May 5, 12:34 pm, "cd...@peermore.com" <chris.d...@gmail.com> wrote:
> I've started a 'filter2' branch on github to start implementing
> something along these lines. To get started I've written a short spec:http://github.com/tiddlyweb/tiddlyweb/blob/filters2/FILTERS2
>
> Please comment, redirect, or otherwise engage yourself as you feel.

An initial prototype of the new filters is implemented in the branch
referenced above. The old style filters have been removed. The work
has exposed a couple of issues worth thinking about before moving on:

* Dealing with the presumably common case of selecting multiple
tiddlers by title from the same bag. See a commit message about that
which suggests some alternatives.

http://github.com/tiddlyweb/tiddlyweb/commit/b4f14b1303d6bc390927b6457e1cf8d91b8072d9

* The question of whether or not to preserve the old filter syntax.
This is possible but I'd really really really rather not because:
** We're still in beta mode so now is the right time to make
significant changes for the better.
** Leaving behind backwards compatibility code makes for enormous
cruft and complexity in the code causing gaps in the test coverage and
corner cases galore.
** The new code is much more concise, flexible, powerful, extensible
than the previous.
** However, the new code will require existing recipes that have
filters to be updated (note, though, that recipes that don't use
filters have the same form whichever filter setup is used).

Your comments appreciated.

[chris...@gmail.com]

On May 8, 11:15 am, "cd...@peermore.com" <chris.d...@gmail.com> wrote:
> * Dealing with the presumably common case of selecting multiple
> tiddlers by title from the same bag. See a commit message about that
> which suggests some alternatives.

After lots of productive conversation with FND, I've just checked in
what might be a workable solution, the addition of an mselect
"command" where 'm' means 'multi'. We consider 'u' for 'union' but
thought 'multi' was a bit less obscure.

http://github.com/tiddlyweb/tiddlyweb/commit/33a2ff6cb5391043e60d8f61665d43e8e7c018b0

The syntax looks like this:

mselect=title:foo,tag:bar,title:zoom;select=tag:boom;sort=modified

This will return a list of tiddlers where:

* mselect yields: a tiddler titled 'foo', a tiddler titled 'zoom' and
all tiddlers with tag 'bar'
* then that set is narrowed to only those which are tagged 'boom'
* then that set is ordered by modifed field

The ',' separator in mselect is just a place holder for now. It's not
clear what the correct separator should be.

Does this seem workable to people?

This model works well in the code because mselect simple dispatches to
the same code that select uses, but accumulates the results.

[Martin Budden]

I'd drop the old syntax, for the reasons you state.

Martin

2009/5/8 cd...@peermore.com <chris...@gmail.com>:

[chris...@gmail.com]

On May 8, 2:54 pm, "cd...@peermore.com" <chris.d...@gmail.com> wrote:
> After lots of productive conversation with FND, I've just checked in
> what might be a workable solution, the addition of an mselect
> "command" where 'm' means 'multi'. We consider 'u' for 'union' but
> thought 'multi' was a bit less obscure.

And after still more conversation, we decided to move the
implementation to a plugin and remove it from the TiddlyWeb core.
Besides removing something that might be wrong and might change out of
the core, it also nicely demonstrates one way to extend the filter
syntax:

http://github.com/tiddlyweb/tiddlyweb-plugins/blob/master/mselect/mselect.py