Transclude a variable number of filters

[Swicky]

I'm trying to use an unknown number of filters in a filter search. I can, so far, hardcode transclusions of two or more tiddlers. So, if tiddler A has [all[]] in it, and tiddler B has +[tag[asdf]] in it, transcluding into filter search shows everything that's tagged asdf.. but I want to do this for any number of filters.

At the moment, I'm thinking of two ideas, but I'm not quite sure about them:

• Use tagged tiddlers - I could make something like a "new journal here" button, which would make a new tiddler tagged "filter" (or something). Then, just transclude the text field of every tiddler tagged "filter".
• Use a field - I could also make a completely different "new" button, that adds a filter to a field, and transclude the entire field.

Of course, I might be thinking about this entirely wrong. Maybe Tiddlywiki is the wrong tool for this job. If it is, then I need something else that can work in the wiki. Any ideas?

[TonyM]

Swicky,

I am sure it would be possible to apply  an unknown number of filters in a filter search.

I think there would be more value in explaining what you want to do, than asking questions halfway through your current attempt to answer these questions.

There are a number of ways to concatenate text of any type into a variable which you can then use in a filter and other places. Of course the order is important in filters.

Similarly you can use nested <$list widgets to separate the "problem" into a number of filter statements rather than trying to force many into one. The trick is to use the variable=variablename parameter of one list widget as the "input" "[<variablename>... to the next List widget, unless you always refer to current Tiddler and do not use the variable parameter.

The reality however is, what are these filter meant to achieve?, because once experienced with filter's there is a lot that can be done with "smart" filters". Perhaps you do not need to concatenate these filters as you think. 

Why are your filters unknown in number and what are they trying to find.

Regards

Tony

[h0p3]

Hey TonyM,

You are always gracious. Thank you. My 12yo daughter didn't see your post until it was bedtime, and I don't want to leave you hanging. She is trying to build a tool for me as part of her CS subject in school. She's trying to make a GUI search tool which can construct arbitrarily long filters. We're still brainstorming and just trying to make a prototype. The gist of it can be found here:

https://philosopher.life/#Search%20of%20h0p3:%5B%5BSearch%20of%20h0p3%5D%5D

The goal is to have a graphical tool for incrementally constructing an arbitrary number of atomic and complex sentences (built from atomic and other complex sentences using drop-down menus that include the previously constructed sentences). The final complex sentence will contain the main connective and will be the final filter for the search. Writing several complex filters back-to-back is labor intensive (some people who use my wiki cannot write them at all, but they know enough First Order Logic they could use a graphical tool), and I want to abstract away as much as I can.

Performance matters to me here too. My wiki is fairly large (now pushing ~22MB of pure text in ~7k tiddlers), and I think it will only continue to grow year after year. There are certain kinds of filter-based tools which do not work on my wiki; it's simply too big. Presumably, this tool will also be used to wrap other search tools which I otherwise couldn't afford to use without complex filters first narrowing down the search space.

There may be multiple ways to do this. We may even be wrong about what is possible or worthwhile here. We could certainly benefit from your expertise. Please advise us. What are your thoughts and what direction do you think we should go?

Sincerely,

h0p3

[TonyM]

h0p3/Swicky

It sounds like a great challenge, but I am sure I do not understand all the details. A few acronyms have me a little confused. Never the less I think I get the Gist.

To be clear are you searching a large collection of tiddlers containing text, tags and fields that you want to list as the result of a sophisticated compound search consisting of multiple filters?

I do feel that TiddlyWikis filters are in someways specific to the TiddlyWiki's platform requirements. Rather than let this complexity overwhelm, If I understand what you are trying to do, I would build a set of search primitives from TiddlyWiki filters that generate a set of tiddlers, then add these abstracted primitives together rather than drop down into the TiddlyWiki filters syntax, in your large compound "search criteria". 

Designed carefully every search primitive could be invoked regardless of what proceeds or follows it. In which case you could specify the primitives and their order by tagging a tiddler lets call it a "Query tiddler" with the set of primitives, and the desired order (using the tag pill drag and drop ordering) and have the result "drop out at the end" or Not. This would allow you to look at the intermediate results ie Primative1 generates list A, Primative2 to uses this list to generate list B etc...

I feel a form of loop could be built that takes as its input the previous list generated by the previous primitive. If constructed correctly each step will result in a much smaller subset to test/filter and as a result the performance should be good.

In keeping with the original post I would then use the resulting list to transclude the content.

Perhaps I am way off here, but this is my brave attempt to respond to what looks like a deeply conceptual experiment, the type I love, but are best constructed in my own mind, rather than trying to understand prior work.

Regards

tony

[TonyM]

Post Post (not Post Script :)

Using the standard TOC method, you could build a hierarchy of primitives, new here for a sub primitive, If you pick any leaf in this hierarchy and use Bimlas's kin operator (Coming for TW 5.1.18) you could use the strait line of tiddlers from the root tiddler to the current leaf, as the list of primitives for a given query. I would use a TOC Tabbed Internal for this. Imagine if you clicked on each node/leaf in the hierarchy you would see the list of tiddlers (or not) that that meet your compound search defined as from the root. 

Noting here that each node would be named according to its "filter nature" with fields containing the translation to tiddlywiki filter code eg "tagged fred" means +[tag[fred]] 

Inspiring.

Tony

[Jed Carty]

I have put some thought into a tool like this but I always got distracted before I got anywhere with it.

I made some notes about how to connect logic statements filters to that may be useful here https://ooktech.com/jed/ExampleWikis/FilterLogicExamples/

I think I will need to update it when the next version of tiddlywiki comes out.

To help answer the original question, I think that a set of tiddlers with the filters, either tagged or with a consistent prefix, list, and a wikify widget would be the best way to do it.

Each filter step would create a tiddler with names like $:/filters/step1,  $:/filters/step2, $:/filters/step3 ...

And then you have a list widget and wikify widget that lists them all like this:

<$wikify name=FullFilter text="""
<$list filter='[prefix[$:/filters/]]'>
<$view field='text'/>
</$list>
"""
>

The filter is: <<FullFilter>>

Which gives:

<$list filter=<<FullFilter>>>

</$list>
</$wikify>

there are a lot of details between this and what it sounds like your final output is supposed to be, but hopefully that gets you started on the part you are asking about anyway.

One thing to note is that you have to have + or - at the start of all but the initial filter if you want the filter added to affect the output of the previous filter.

[h0p3]

TonyM and Jed, thank you for your help!

You have given us several approaches and ideas to think about. We are indebted to you.

We're working on it. It may be a while (this is no small task for us). If we run into more trouble, we'll let you know.

It was https://ooktech.com/jed/ExampleWikis/FilterLogicExamples/ and my heavy use of Tiddlytool that caused me to think this GUI would be useful in my context. I am curious to know what you will be updating in 5.1.18.

[Evan Balster]

Hello, h0p3 —

In case the wikify approach proves to be a little too complex, I will suggest you might be able to construct and run filters a little more concisely with my Formula plugin!  I wrote it to make more advanced calculations easier to do in TiddlyWiki.

I don't have a bunch of filter pieces in that wiki, but we can fake it with this formula:

"[title[" & join("]] [title[", [tag[Syntax]]) & "]]"

You can test this on the landing page of my wiki.  This expression makes a "title" filter step for every Syntax-tagged tiddler in my wiki and joins them all together, producing a filter string.  We could put that into a variable with the $formula-vars widget, but it's also possible to construct and run the filter in one expression:

join("<br>", datum("(= [title[" & join("]] [title[", [tag[Syntax]]) & "]] =)"))

In your case you'll probably want to be applying the filters in series rather than adding their results together, so you'll probably want a plus or minus as suggested by Jed.  You might want to enclose the results of the final filter in [[brackets]] so the list widget can understand them.  So your final formula might be something like this:

"[[" & join("]] [[", datum("(=" & join("+", [your_filter_construction_filter_here[]]) & "=)")) & "]]"

If I dust the plugin off for more development, I might have to add some new functions to make this sort of thing less hacky...

[Evan Balster]

Sorry, I've realized there's a simpler and more efficient solution than what I suggested there.  It's:

<$list filter=<<formula """ join("+", [your_filter_construction_filter_here[]]) """>> >
{{!!title}}<br/>
</$list>

Testable in Formulas wiki:

<$list filter=<<formula """ "[title[" & join("]] [title[", [tag[Syntax]]) & "]]" """>> >
{{!!title}}<br/>
</$list>

[h0p3]

Sorry that I'm insanely late getting to this. Thank you Evan. I love your Formula plugin, which is also one of the inspirations for this tool I'm after. My daughter and I are looking into it!