Broken Concat / Embedded Widget Calls / JSON Tiddler Filter

[D X]

I imagine I'm missing something obvious, but I'm unable to concat together a working filter to search JSON tiddlers for a specified field value on TW 5.1.18. Concat() is a usual $1$$2$$3$, etc. macro. Here are a few variations that don't work for me:

1. Non-$macrocall, single brace field ref, quote-delimited:

<$set name=search value=<<concat "'" '[tag[data_page]search:text:literal["father": "' {!!parent_page} '"]]' "'" >> > 

search: <<search>>

<$list filter=<<search>> >

* list: {{!!title}}

</$list>

</$set>

Output 1:

search: '[tag[data_page]search:text:literal["father": "{!!parent_page}"]]'

• list: [tag[data_page]search:text:literal["father": "{!!parent_page}"]]

2. Non-$macrocall, double brace field ref, quote-delimited:

<$set name=search value=<<concat "'" '[tag[data_page]search:text:literal["father": "' {{!!parent_page}} '"]]' "'" >> > 

search: <<search>>

<$list filter=<<search>> >

* list: {{!!title}}

</$list>

</$set>

Output 2:

search: '[tag[data_page]search:text:literal["father": "Kyle"]]'

• list: [tag[data_page]search:text:literal["father": ""]]

3. Non-$macrocall, double brace field ref, non-quote-delimited:

<$set name=search value=<<concat  '[tag[data_page]search:text:literal["father": "' {{!!parent_page}} '"]]' >> > 

search: <<search>>

<$list filter=<<search>> >

* list: {{!!title}}

</$list>

</$set>

Output 3:

search: [tag[data_page]search:text:literal["father": "Kyle"]]

--LIST DOESN'T RETURN RESULTS--

4. $macrocall, double brace field ref, non-quote-delimited:

<$set name=search value=<$macrocall concat  1='[tag[data_page]search:text:literal["father": "' 2={{!!parent_page}} 3='"]]' /> > 

search: <<search>>

<$list filter=<<search>> >

* list: {{!!title}}

</$list>

</$set>

Output 4:

<$set name=search value=<$macrocall concat 1='[tag[data_page]search:text:literal["father": "' 2=Kyle 3='"]]' </$macrocall> > search: 

--LIST RETURNS EVERY TIDDLER--

Seems like the quote delimiting is broken here, but I don't see the issue. What's more, trying to close the <$macrocall> tag with /> seems to be closing the <$set> macro as well.

And there are a few other permutations of the above, none of which work. =/ Help welcome and thanks in advance!

[Mark S.]

Wow, I didn't realize search had changed so much. I really need to pay attention to those release notes.

The five laws of concatenation, which I just made up:

Wikitext can't eat Wikitext

Wikitext can't eat Widgets

Widgets can't eat Widgets

Widgets CAN eat Wikitext.

Wikify can sometimes bend laws 1 through 3

The following version may be too distant from what you wanted, but might serve as a start. It did work in my quick test:

\define ppfilter(pp) [tag[data_page]search:text:literal["father": "{{!!parent_page}}"]]

<$wikify name=search text=<<ppfilter>>>
Variable: <<search>>
<$list filter=<<search>> >

* Search Result: {{!!title}}
</$list>
</$wikify>

HTH

-- Mark

On Monday, January 7, 2019 at 8:17:10 PM UTC-8, D X wrote:

I imagine I'm missing something obvious, but I'm unable to construct a working filter to search JSON tiddlers for a specified field value on TW 5.1.18. Concat() is a usual $1$$2$$3$, etc. macro. Here are a few variations that don't work for me:

1. Non-$macrocall, single brace field ref, quote-delimited:

<$set name=search value=<<concat "'" '[tag[data_page]search:text:literal["father": "' {!!parent_page} '"]]' "'" >> > 

Variable: <<search>>

<$list filter=<<search>> >

* Search Result: {{!!title}}

</$list>

</$set>

Output 1:

[D X]

Thanks, Mark S.!

I'm not sure what 'eat' means in these rules, but your example has managed to get me to something working. Interestingly, I had experimented with <$wikify> before posting, but it couldn't get it to work right either and had assumed it was a dead-end. For those curious (I spent a few hours searching and experimenting before posting), here's how Mark's solution works for the example:

<$wikify name=search text=<<concat [tag[data_page]search:text:literal['"'father'": ''"' {{!!parent_page}} '"']] >> > 

search: <<search>>

<$list filter=<<search>> >

* list: {{!!title}}

</$list>

</$wikify>

Output:

search: [tag[data_page]search:text:literal["father": "Kyle"]]

• list: Angie - Data

• list: Kyle - Data

[Mark S.]

Eat == "use inside of"

[D X]

I see. 

New problem: I now need to expand this filter to include both the "father" and "mother" JSON indexes (and later on do some sorting), but no matter how I apply the quotes it seems like <$wikify> is running out of steam when I try to append another run to this filter string.

[Mark S.]

There are different ways that you might expand it. This one looks for the contents of parent_page as either mother or father. It takes advantage of another new feature in 5.1.19 -- the subfilter operator.

-- Mark

\define concat(a b c d e f g h) $a$$b$$c$$d$$e$$f$$g$$h$
<$wikify name=fatherFilter text="""<<concat [tag[data_page]search:text:literal['"'father'": ''"' {{!!parent_page}} '"']] >>""" >
<$wikify name=motherFilter text="""<<concat [tag[data_page]search:text:literal['"'mother'": ''"' {{!!parent_page}} '"']] >>""" >
Father search: <<fatherFilter>> <br/>Mother Search: <<motherFilter>>

<$list filter="[subfilter<fatherFilter>] [subfilter<motherFilter>]">

* list: {{!!title}}
</$list>
</$wikify>

</$wikify>

[D X]

Thanks, Mark. I haven't tried subfilters. I'll experiment with that tonight.

[Mohammad]

D X,

 This is a very good discussion both on search and using subfilter, could give a short example of that json tiddler and application of Mark script to see how things work?

--Mohammad

[Mohammad]

Mark,

 I want to document this for further reference, could you give some more explanation of the rules?

I know I can nest widgets like

<$tiddler ....

 <$list ....

or I can use wikitext inside widgets like

<$set value={{}}

I appreciate to explain me in more details.

Best

Mohammad

[Mark S.]

You can use wikitext (markup with angle brackets ("<<"), or curly braces ("{{") ) inside of widgets or other HTML tags like:

<$vars mine={{!!stuff}}>

However, AFAIK, you can't use wikitext links (square brackets) as widget attributes.

You can't use wikitext inside of wikitext like

<<mymacro {{!!stuff}}>> or like [[mylink|{{!!stuff}}]]

and you can't use widgets as attributes inside other widgets like

<$vars mine=<$view field="stuff/> />

Sometimes it's possible to use wikitext curly braces inside of wikitext angle braces (macros) if you process the text through the Wikify widget, but to be honest I just poke around at it until something works ;-)

-- Mark

[D X]

To wrap this thread up, while I love TiddlyWiki and don't see myself changing due to its elegance and flexibility, I also at times find aspects of it (several underlying this thread) frustrating:

• Hard-to-Intuit / Understand Syntax / Parsing Rules: Do I reference something in a given context with double braces ({{}})? Single braces ({})? Double angle brackets (<<>>)? Single angle brackets (<>)? Dollar signs? Parens and dollar signs? Or will the kind of reference I'm trying to make not work at all in my current context? And how will some given wikitext / macro call / string be mangled by TW? Who knows? I haven't really developed a predictive mental model of TW's behavior in 6+ months of working with it.
• Lack of String Manipulation: I expect built-in ways to do string concatenation, searching, and splitting that nest and work in most contexts (e.g., not hand-crafted and unreliable macros).
• Limitations of Fields: It's hard to leverage the elegant database structure of TW when field names are constrained by some arcane naming constraints. (I realize not entirely TW's fault.)
• Lack of JSON Index Operator Parity with Fields: And, to wit, it's not possible to fully move to JSON data-driven Tiddlers (instead of fields) when filters have so much built-in field searching and sorting operator support that isn't supported for indexes.
• Documentation and examples at https://tiddlywiki.com/ are often either too simple for me (e.g., operator examples rarely show complex / dynamic values populated by fields and variables like we see in this thread, which is where I usually need help) or too terse, i.e., the non-example doc pages often have sentence-fragment-like descriptions of an operator's purpose or behavior.

That said, after all the help from Mark (thank you!), I suspected that my filter string concatenation might work better if I weren't trying to assemble it from string literals in the wikitext (which is a bit counterintuitive from traditional programming), so I made my filter string components into fields that I could reference in my concat macro. This helped, although ultimately TW would only process the first 2 of these and the oddly the last one had to go back to being a literal:

<$wikify name=search text=<<concat {{'Basics Template'!!search_prefix}} {{!!parent_page}} {{'Basics Template'!!search_middle}} {{!!parent_page}} '"]]+[sort[]nsort[birth_date]]' >> >

<$list filter=<<search>> >

In which search_prefix = [tag[data_page]search:text:literal["father": "

And search_middle = "]][tag[data_page]search:text:literal["mother": "

By the way, a more compact (although I suspect less performant) way to accomplish this filter / search is via a regex search filter similar to this:

[tag[data_page]search:text:regexp[(\"mother\")|(\"father\"): \" {{!!parent_page}} \"]]

Ultimately, however, even though I have mostly solved the search problem in this thread, I am finding JSON data tiddler filter operators too limited (e.g., no sorting) relative to the fairly robust field filter operators in TW, so I'm changing back to a mostly field-driven data model (with the exception of complex field names that must refer to Tiddler names, e.g., field names that must contain capital letters and spaces, which TW cannot persist).

However, the help has been much appreciated!

[@TiddlyTweeter]

D X

FWIW, there is a long thread here that has useful posts about syntax and a couple of crib sheets ... https://groups.google.com/d/msg/tiddlywiki/3DyHxqqFQbg/1JP6_DzeAAAJ

Best, J.

[Mohammad]

Thank Mark.

 I am collecting these notes for future reference.

Thank you for you time and help.

--Mohammad

[Mark S.]

I don't understand the "Lack of String Manipulation" either. Writing filters or macros for these standard tools would not be that difficult. I can only assume there is some philosophical reluctance. Perhaps because filters are supposed to work on Titles, not strings, and changing a Title's case, for instance, makes it into a new tiddler. But there are already existing filters that break this premise, so ... dunno.

-- Mark

[D X]

Hi Mark,

Every language I've worked in (I know TW is not a language per se, although it does have many language-like features like loops, conditionals, references, substitution, and look-up) has first-class (i.e., usable anywhere and arbitrarily nestable) support for string manipulation and construction via non-macro functions like concat(), substr(), instr(), and length(). This thread is a good example of how hard it is to construct a longer arbitrary string with arbitrary variable values concatenated into it (ultimately, we couldn't concat complex strings past a certain length).

This thread is a good example of how macro-based string manipulation with embedded single and double quotes doesn't scale to longer strings or exhibit easily predictable behavior. You yourself wrote this just a couple posts above: :D

... but to be honest I just poke around at it until something works ;-)