A Potential Extension to Filter Syntax

[Jed Carty]

The quick version is that I want to propose an extension to the normal filter syntax to allow defining variables and have multiple filters in one filter string. filters are separated by commas (,) and variables are assigned, similar to how the set widget defines them, using ->. One filter can be assigned to a variable that is then used in later filters in the filter string.

The long version is:

As part of my WikiDB project I needed a way to make more complex filters using only one string than you can using tiddlywiki so I added some extensions to the filter parser I wrote for the database that may be worth considering for the core.

The database is written in Julia so I don't have any demo for this.

The extensions let you define variables and multiple filters in the same string.

The string

[tag[a]] -> 1, [tag[b]] -> 2, [tag<1>tag<2>]

Is roughly equivalent to

<$list filter='[tag[a]]' variable=1>
  <$list filter='[tag[b]]' variable=2>
    <$list filter='[tag<1>tag<2>]'>

    </$list>
  </$list>
</$list>

but in my version the output is all concatenated into one list so it could be sorted or have the duplicated removed, instead of a bunch of lists inside the inner widget.

You could also have multiple filters at the output

[tag[a]] -> 1, [tag[b]] -> 2, [tag<1>tag<2>], [field:foo[bar]]

which is the displayed output from this, but in one list:

<$list filter='[tag[a]]' variable=1>
  <$list filter='[tag[b]]' variable=2>
    <$list filter='[tag<1>tag<2>]'>

    </$list>
  </$list>
</$list>
<$list fliter='[field:foo[bar]]>

</$list>

Writing this in Julia meant that I was able to start from scratch and make everything exactly how I wanted it, I am not sure how difficult it would be to implement this for tiddlywiki. We would have to be very careful because of how important filters are.

[PMario]

On Tuesday, September 15, 2020 at 7:24:57 PM UTC+2, Jed Carty wrote:

Writing this in Julia meant that I was able to start from scratch and make everything exactly how I wanted it,

I am not sure how difficult it would be to implement this for tiddlywiki. We would have to be very careful because of how important filters are.

You are right. Lists and filters are "the main" functionality in TW.  The internal memory structure of the "filter compiler" is recursive, indirect and mind-boggingly complex.

I think the "recursive" nature is a problem here, which may make it difficult to store internal variables. ...

Most filter results are indexed now. So may be we can use the index somehow.

have fun!

mario

[TonyM]

Jed,

I think this a worthy goal, bas you say, if not done perfectly it could really confuse things, filters are sophisticated as they stand, and if such a possibility is available they could be made extremely sophisticated.

This would solve a number of issues commonly raised so its a great idea.

Perhaps a plugin or macro first or only?

• Having considered this path before perhaps a macro or plugin that you 
• pass an "advanced filter" to that pre-processes additional syntax like you "->"
• Then generates the nested lists version with the filters and varnames applied

I was thinking something similar such as "(filterrun)outputvaname (filterrun2)outputvaname2  (filterrun3)outputvaname3" and a template parameter

Programatically generates

<$list filter='filterrun' variable=outputvaname>
  <$list filter='filterrun2' variable=outputvaname2>
    <$list filter='filterrun3' variable=outputvaname3>
             {{||TEMPLATE}}
    </$list>
  </$list>
</$list>

In which template can return contain

<<outputvaname>> <<outputvaname2>> <<outputvaname3>><br>

Or call other macros with the values

<$macrocall$name-othermacro p1=<<outputvaname>> p2=<<outputvaname2>> p3=<<outputvaname3>>/>

Ideal this would nest as many times as there are "subfilters"

Subfilters?

That made me think, what about extending the subfilter operator here?

subfilter:varname[subfilter] or subfilter:varname<subfilter>

Or perhaps thinking even bigger?

Be it through extending filters or other means, 

• looking to address all the kinds of queries we may be able to do, in something such as SQL, would be valuable in the long run.
• Perhaps even a different set of Queries that return json files with the results there in?
  
• It would be great if we could treat related tiddlers as rows in a table, fields and columns and use actual SQL or similar.

Regards

Tones

[Jed Carty]

Replacing the filter with nested list widgets defeats the purpose of having it all in one list at the output so that you can manipulate the result. And it also doesn't let you make independent filters because they would all be nested and would duplicate all the filters that come afterward.

In a document database where you don't necessarily have structured data SQL seems like a step backwards, the point of the database I made was that it uses tiddlywiki filters, switching to SQL seems like a step backwards considering there are plenty of options if you want to use SQL.

I don't understand the JSON part, we can already do that easily with tiddlywiki and it would be a trivial addition to whatever consumes the output of the database.