Evaluate Wiki text and store the result as literal text

[TonyM]

Folks,

As my tiddlywiki familularity grows and I contribute to support people in the forum the issue of "Evaluating Wiki text and storing the result as literal text" continues to be an issue.

That is taking text references (to tiddlers text or other fields) or variables or macro results and returning the actual values rather than deferring this until the tiddler is rendered into html. We also need to do this with the result of various widgets including the list widget that generates multiple line outputs.

Please share if you have any elegant ways to do this, or the only ways you know how to do this, however I think this need is what drives the misuse of the wikify Widget, and confusion on the part of new and experienced users.

The macro definitions and various widgets available seem to have different subtleties and as a result there is no simple way to communicate how to do this. 

With your suggestions I would like to do one or more of the following;

• Support users more effectively
• Improve the documentation
• Provide a supporting macro
• Suggest a solution (if applicable)
• Potentially request a new widget

Your contribution would be highly desired.

Regards

Tony

[PMario]

Hi Tony,

That's a very good suggestion. ... Even experienced users have problems here. ...

I wanted to throw out a fast explanation, but it's not that simple. I did create a tiddler that contains all the possibilities, but it doesn't show why 1 example does concatenate text parameters and others don't.

So I'll need to split them into several tiddlers and we will need the "advanced" preview pane, to actually see, what's going on. ... The end result will be a relative deep dive into how TW parsing and rendering works. ...

So I'm not sure, if it should be part of tiddlywiki.com ... I think it should go into tiddlywiki.com/dev ..

-mario

[PMario]

Hi,

I'll try to post a video soon.

-m

[TonyM]

Mario,

Thanks for your support.

A Quick example

\define sira-office-csv()
<$list filter="[object-type[office]has[state-cover-office]]" variable=office>
<$view tiddler=<<office>>  field=state-cover-office format=text/>,<$view tiddler=<<office>> field=office-address format=text/><br>
</$list>
\end

<$button message="tm-copy-to-clipboard" param=<<sira-office-csv>> >
Copy to clipboard
</$button>

In the above example the macro definition is copied to the clipboard, not its result.

I have tried a number of alternate methods all of which cause grief

The best I have come up with today is;

\define sira-office-csv()
<$list filter="[object-type[office]has[state-cover-office]]" variable=office><$view tiddler=<<office>>  field=state-cover-office format=text/>,<$view tiddler=<<office>> field=office-address format=text/></$list>
\end
<$wikify name=csv text=<<sira-office-csv>> >
<<csv>>

<$button message="tm-copy-to-clipboard" param=<<csv>> >
Copy to clipboard csv wikify
</$button>
</$wikify>

However I cant seem to may this work as the output either does not have line breaks or has two line breaks.

Regards

Tony

[TonyM]

Mario,

I look forward to your video.

All,

My hunch is we need an "evaluate" widget (name open to suggestions).

• One which returns a text representation of the wikitext in a variable.
• Perhaps with a choice of line delimiter eg <cr>, <cr><lf>, <lf> or just space or a provided delimiter.

So my previous Quick example would be written as such

\define sira-office-csv()
<$list filter="[object-type[office]has[state-cover-office]]" variable=office>
<$view tiddler=<<office>>  field=state-cover-office format=text/>,<$view tiddler=<<office>> field=office-address format=text/><br>
<

/$list>
\end
<$evaluate name=csv text=<<sira-office-csv>> delimiter="<cr><lf>">

<$button message="tm-copy-to-clipboard" param=<<csv>> >
Copy to clipboard

</$button>
</$evaluate>

Alternatively the text widget could be extended

Regards

Tony

[TonyM]

A Possible additional feature of the "evaluate widget" or using another method

Allow the output format to be rendered in alternate ways such as block text, code block etc...

Regards

Tony

[Mat]

[OP]

In deed. I believe I have read explanations but I don't succeed in internalizing what makes it work in some cases and not in others so I resort to trial an error.

One recent tip, that I got from Xavier, was to use the SetWidget with the select=0 paramter. Apparently this evaluates the filter and assigns the result to the set variable. This specific parameter could definitely do with some more documentation. (To name one thing; if =0 is the result... then what is e.g =1?)

My spontaneous guess is that your "EvaluateWidget" is the wikifywidget. Regardless, another syntax might be to change the macrocall syntax from <<>> into some other enclosing symbols that indicate "evaluate". Or some suffix in the call like <<sira:evaluated>> (this is pretty sweet because of the similarity to the "parameter:value" syntax in macroalls)

<:-)

[Paul Marrington]

I like the "Improve the documentation" item. I find the tobibeer page on the filter and list invaluable, but I keep spending inordinate amounts of time getting variable, parameters and references to transclude or not transclude at the correct time. I realise that it is my own ignorance of the parsing, but I find myself trying all the different methods until one accidentally works. by this I mean using $macrocall instead of <<>> based on where I need variables or transclusions to take place. Using $set, $wikify and $text. Or wrapping macros in other macros so that something can be set then called with $()$. Invariably I end up with a complex that when I look at it and refactor out all the unnecessary combinations I return to what appears to be the original solution, but that now works. I am a software developer with over 40 years experience and I am drowning. How must it be for the fiction book writer, research assistant or graphics artist?

I have managed to create a saver that uses diff-match-patch to reduce the average save to a few hundred bytes. I have combined this with PWA so that I can run TiddlyWiki as a stand-alone app on desktops and tablets that updates when on-line but work perfectly well (and very quickly off-line). At last, I (almost) have the environment I desire. I will publish my results once I have the kinks out - if I can ever work out when to use $$, $()$, <<>>, [[]] or {{}} with or without quotes :)

---
regards,
Paul Marrington

ph: (0412) 409446

pa...@marrington.net

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/882948ee-330c-4192-a869-f2153c52a331%40googlegroups.com.

[TonyM]

Paul,

Thanks for your support on the Original Thread OT.

On Thursday, July 25, 2019 at 11:35:18 AM UTC+10, Paul Marrington wrote:

.. but I keep spending inordinate amounts of time getting variable, parameters and references to transclude or not transclude at the correct time. I realise that it is my own ignorance of the parsing, but I find myself trying all the different methods until one accidentally works. by this I mean using $macrocall instead of <<>> based on where I need variables or transclusions to take place. Using $set, $wikify and $text. Or wrapping macros in other macros so that something can be set then called with $()$. Invariably I end up with a complex that when I look at it and refactor out all the unnecessary combinations I return to what appears to be the original solution, but that now works. I am a software developer with over 40 years experience and I am drowning. How must it be for the fiction book writer, research assistant or graphics artist?

I think this is a pervasive experience.  Often worsened by generating an "imposter syndrome" in people who thought they were coders.

I have managed to create a saver that uses diff-match-patch to reduce the average save to a few hundred bytes. I have combined this with PWA so that I can run TiddlyWiki as a stand-alone app on desktops and tablets that updates when on-line but work perfectly well (and very quickly off-line). At last, I (almost) have the environment I desire. I will publish my results once I have the kinks out - if I can ever work out when to use $$, $()$, <<>>, [[]] or {{}} with or without quotes :)

Paul I am very interested in this, especially with PWA because I found a lot of resistance when I was arguing for its value conceptually, if not to the letter of the PWA standard.

• So if you need help with Kinks please post here and I will be watching. 
• There may be some key techniques available so you need not publish a separate edition to do this. 

What do you mean "updates when on-line"? do you mean gives you access to changed tiddlers in the online copy? Presumably those not replaced buy the remotely changed tiddlers.

Regards

Tony

[TonyM]

Paul,

Re:

 I will publish my results once I have the kinks out - if I can ever work out when to use $$, $()$, <<>>, [[]] or {{}} with or without quotes :)

See here for a Start

 I have a reviewed and Improved version somewhere, which I will add when I locate it.

Regards

Tony

[TonyM]

A key use case for Evaluate Wiki text and store the result as literal text

Is to use the templates such as 

$:/core/templates/exporters/JsonFile

• To generate a result and be able to save this as evaluated into a tiddler.
  
• It would then be easy to pack tiddlers into a JSON tiddler with a filter, 
• Or let uses drag and drop the JSON to their own wiki.
• and reimport/extract them again later.

Other such that generate static HTML and more will provide some nice options.

Regards

Tony

[Mat]

@Paul

[...] refactor out all the unnecessary combinations I return to what appears to be the original solution, but that now works.

This made me laugh out loud from identifying with it.

[saver] I will publish my results once I have the kinks out

 

Very much looking forward to this. Would this be a plugin or is it an external thing?

If I'm guessing right about what a "diff-match-patch" saver is, it sounds like the most economical type of saver. So if it is not a matter of pride, you could publish it because it would likely gain a lot of interest and others could come with suggestions.

<:-)

[Jeremy Ruston]

A key use case for Evaluate Wiki text and store the result as literal text

To clarify that that is precisely what the wikify widget does at the moment. I understand that there’s confusion around its usage, but I’m finding this thread hard to follow because it is structured as a vague proposal for something new, rather than an exploration to understand what we already have. It feels like a lot of work: to deal with this thread I have to try to both explain how the wikify widget works, and also explain the problems with the proposal that is being made, and the misapprehensions around it, which requires me to understand the proposal, which is tricky because it’s so vague.

Perhaps it would be better to pick up the suggestion of focusing the discussion on improving the documentation and examples of the wikify widget. I think we need to get to the point where the participants in this discussion all understand what we’ve got before we consider changing it.

Best wishes

Jeremy

-- 
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/1e62e5c5-af44-4f48-a61c-fed15f79228e%40googlegroups.com.

[PMario]

On Thursday, July 25, 2019 at 11:58:17 AM UTC+2, Mat wrote:

@Paul

[...] refactor out all the unnecessary combinations I return to what appears to be the original solution, but that now works.

This made me laugh out loud from identifying with it.

+1

-mario

[TonyM]

Jeremy,

With respect you misread my OT "Please share if you have any elegant ways to do this, or the only ways you know how to do this, however I think this need is what drives the misuse of the wikify Widget, and confusion on the part of new and experienced users."

This is a call for information from those who have established methods like yourself. I was keen to get the experts opinion to try and clear this up once and for all. Not only for myself but even naive users. And yes, update the Wikify documentation.

See also my post on a use case such as "evaluate and save tiddlers generated with templates" such as  $:/core/templates/exporters/JsonFile and if I understand correctly this in not necessarily the Wikify widgets Job.

Although not on hand,  I recently had an example where I used the wikify widget for this purpose and it returned un-evaluated wiki text. Others have supported me both here and elsewhere because something in the facilities available there is more than one counter intuitive fact. I am speaking for the community as a whole and plan to take the steps to remedy this through documentation etc...

Thus this is a request to the development savvy community members to provide the inside techniques.

Regards

Tony

To unsubscribe from this group and stop receiving emails from it, send an email to tiddly...@googlegroups.com.

[PMario]

Hi Tony,

IMO that's a case, where wikify has to be used.

\define sira-office-csv()
<$list filter="[object-type[office]has[state-cover-office]]" variable=office emptyMessage="no matching tiddlers - check the filter!"><$view tiddler=<<office>>  field=state-cover-office format=text/>, <$view tiddler=<<office>> field=office-address format=text/><br>
</$list>
\end

\define test()
<$wikify name=xx text=<<sira-office-csv>>>
<$action-sendmessage $message="tm-copy-to-clipboard" $param=<<xx>>/>
</$wikify>
\end

<$button actions=<<test>> >
Copy to clipboard
</$button>

Have fun!

mario

[@TiddlyTweeter]

Jeremy Ruston wrote:

 I understand that there’s confusion around its usage, but I’m finding this thread hard to follow 

Me too. 

IMO its, so far, not helpful. Its confusing.

TT

[@TiddlyTweeter]

With respect TonyM

Your previous is too convoluted, I think. What is the question?

TT

[@TiddlyTweeter]

Ciao PMario

I think what is lacking may be DEMONSTRATION?

What I mean is that the talk here is down in the macro world. But WHAT is the OUTCOME?

I think demonstrable application matters to comprehension by humans like me.

My two cents worth :-).

TT

[PMario]

On Friday, July 26, 2019 at 1:41:21 PM UTC+2, @TiddlyTweeter wrote:

...

I think what is lacking may be DEMONSTRATION?

That's exactly, what I'm thinking about at the moment. .. I'm preparing several videos _but_ I need to start at the beginning, with the simple stuff. Like "concatenating, the content of 2 tiddlers to create a clickable link".

 

What I mean is that the talk here is down in the macro world. But WHAT is the OUTCOME?

A step by step intro, that tries to explain, why things are as they are atm. There needs to be some TW history too.

 

I think demonstrable application matters to comprehension by humans like me.

Exactly that. The TWs I'll use for the videos will be available online.

-m

[@TiddlyTweeter]

This is a side of TW I want (need) to understand unambiguously. Just FYI, when I read what folk say about it I'm still mostly confused (it reads often like an argument in Latin to me) .

 A demo I'm sure will help a lot!

Best wishes

TT

[TonyM]

With all the great evidence, advice etc using the wikify widget do as per the original post is fragile. Tiddlywiki is fragile, or obscures things for designers to achive what they want.

For me this fragility always appears in part of my code that is very hard to extract a simple demonstration case from. There are a few transclusions deep, a number of variables with a range of text references involved.

I don't claim to be a rocket scientist or brain surgeon but I know computers well and if after 3+ years if deep focus in tw5 and the help of smart people here, this all remains fragile then something is very wrong. Fortunatly since others seem to have the same problems it is not me who is mad.

My guess is those who understand how to combat this fragility have a level of expertise or tacit knowledge that even they do not know how to communicate it to others, especially less expirenced people.

Experts can often fix the issue in specific cases detailed in this forum, if the user can explain the details. Unfortunatly if someone else's issue is slightly different such solutions fail. This is the fragility.

If you teach a man to fish.....

[Jeremy Ruston]

Hi Tony

> If you teach a man to fish…..

This, sadly, resonates with me. I’ve felt strongly for many years that the original sin of computer programmers is that we put software out into the world whose primary impact is to make people who use it feel stupid. My goal has thus been to build software that empowers people, making them feel and act smarter.

Of course, those goals can conflict with one another, particularly in the context of open source communities where hackability and power are highly prized.

The complexity of TiddlyWiki is exacerbated by the fact that it is being invented on the fly: it is not a clone of an existing product, like many open source tools.

Anyhow, the complexity of the wikify widget is of a time honoured sort. It is analogous to the “eval” function of JavaScript or BASIC, exposing the full functionality of the underlying interpreter for programmatic access. There’s a sort of recursion going on that is apt to melt anyones brain.

I have a theory that the specific difficulty with the wikify widget is that people often try to visualise the output like this:

<$wikify name="myvar" text=<<mytext>>>
<<myvar>>
</$wikify>

The trouble with that example is the construction <<myvar>>. Rather than just displaying the text of the specified variable, it actually wikifies that text. If you want to see exactly what has been output by the wikify widget then it’s important to display it without wikifying it. For example:

<$wikify name="myvar" text=<<mytext>>>
<$text text=<<myvar>>/>
</$wikify>

With that technique you can reach into a troublesome wikify widget to visualise what it is producing.

Further complexity comes from a corollary of the observation that the wikify widget is seldom required: those situations in which is it useful tend to be subtle, and the resulting code tends to be hard to read.

For example, Mark S. has several times demonstrated a cunning technique where the wikify widget is used to generate a filter string. It provides a means to introduce widget logic into filters, opening up possibilities that couldn’t be achieved any other way. It’s an incredibly powerful technique, but it can be very hard to read and understand without a very good understanding of the underlying primitives.

Best wishes

Jeremy

[TonyM]

Jeremy

Thanks for understanding the problem the tips etc...

The desire to solve it remains. I feel we need to step back and observe why people try and use it. Then provide a method in its place without the complexity. Perhaps almost over simplified.

This why I originally suggested an evaluate widget. The idea being to evaluate the values contained in text references and variables as they would appear on screen and store them in a variable as those values, not the original text. The key is to have the values evaluated and cast in stone.

The cause of the difficulty may be complex. Is there a simple answer buy stepping away from the details?

Tony

[Jeremy Ruston]

Hi Tony

> The desire to solve it remains. I feel we need to step back and observe why people try and use it. Then provide a method in its place without the complexity. Perhaps almost over simplified.

It’s already pretty much the simplest thing that it could possibly be.

> This why I originally suggested an evaluate widget. The idea being to evaluate the values contained in text references and variables as they would appear on screen and store them in a variable as those values, not the original text. The key is to have the values evaluated and cast in stone.

Again, the problem here is that that is exactly what the wikify widget already does.

> The cause of the difficulty may be complex. Is there a simple answer buy stepping away from the details?

It would be helpful if you could outline some cases where the wikify widget does not do what you expect?

Best wishes

Jeremy

>
> Tony
>
> --
> You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/79643967-f8b8-4993-b239-4211e2000865%40googlegroups.com.

[Mat]

@Tony, here's a little demo that could be improved to illustrate how things work:

Put this in a tiddler and create some 2-3 other new tiddlers without changing their titles so they keep their "New Tiddler" names.

<$set name=a filter="[prefix[New]]">
<$button set=!!set setTo=<<a>>>
set
</$button>
</$set>
<$edit-text field=set size=55 />

<$set name=a filter="[prefix[New]]" select=0 >
<$button set=!!set2 setTo=<<a>>>
set select=0
</$button>
</$set>
<$edit-text field=set2 size=55 />

<$wikify name=a text="[prefix[New]]">
<$button set=!!wikify setTo=<<a>>>
wikify filterstring
</$button>
</$wikify>
<$edit-text field=wikify size=55/>

<$wikify name=a text={{{ [prefix[New]] }}}>
<$button set=!!wikify2 setTo=<<a>>>
wikify evaluated filter
</$button>
</$wikify>
<$edit-text field=wikify2 size=55/>

I am frankly surprised by the last one - i.e why doesn't

<$wikify name=a text={{{ one two }}}>
<$text text=<<a>>/>
</$wikify>

give "one two" but only "one" ?

<:-)

[Jeremy Ruston]

Hi Mat

On 30 Jul 2019, at 14:06, Mat <matia...@gmail.com> wrote:

I am frankly surprised by the last one - i.e why doesn't

<$wikify name=a text={{{ one two }}}>
<$text text=<<a>>/>
</$wikify>

give "one two" but only "one" ?

An attribute value quoted with triple braces is interpreted as a filter that is evaluated, and then the first result is used as the value of the attribute. So the above example is equivalent to:

<$wikify name=a text=“one">


<$text text=<<a>>/>
</$wikify>

Best wishes

Jeremy

[Mat]

Thanks Jeremy.  Ah, of course!

That said; my moment of confusion was to assume that the triple braces should work as encapsulations just like for variables or transclusions... which also has to do with the format of the ouput of things! Maybe a general tactic to avoid such confusions could be to have the user specify the output format explicitly... Perhaps some kind of filter operator "output[text]" or "output[rawhtml]" etc? Just an idea.

<:-)

[Jeremy Ruston]

Hi Mat

On 30 Jul 2019, at 15:17, Mat <matia...@gmail.com> wrote:

That said; my moment of confusion was to assume that the triple braces should work as encapsulations just like for variables or transclusions... which also has to do with the format of the ouput of things!

Can you expand on that? What do you mean by “encapsulation” here?

Maybe a general tactic to avoid such confusions could be to have the user specify the output format explicitly... Perhaps some kind of filter operator "output[text]" or "output[rawhtml]" etc? Just an idea.

The result of a filter is always text.

Best wishes

Jeremy

[Mat]

Jeremy Ruston wrote:

Can you expand on that? What do you mean by “encapsulation” here?

 

My interpretation of how "encapsulating delimiters" work:

When, say, <<...>> or {{...}} are used we don't need delimiting quotation marks "...", because these brackets are encapsulator delimiting the evaluated expression and effectively making the content therein be evaluated before its output is e.g applied to something, e.g x={{...}}. (I know macros are not evaluate in that sense but merely text substitutions... but I'm not 100% sure of the practical consequences from this. They do typically appear to be evaluated.)

For {{{...}}}, my spontaneous feeling, which I know is incorrect but I have to make an effort to resist, is that this behaves the same way, i.e that the expression is evaluated but still "contained" as a unit so that 

$set name=x value={{{a b c}}}  really would give the same result as $set name=x filter="a b c".

What I assume really happens is that it is more like filter=a b c and drops off those other items silently (...maybe the surrounding widget should alert about unknown parameters?)

>Maybe a general tactic to avoid such confusions could be to have the user specify the output format explicitly... Perhaps some kind of filter operator "output[text]" or "output[rawhtml]" etc? Just an idea.

The result of a filter is always text.

You mean as in "it is always a string of characters", right? Yes, (of course) but what I mean is for the user to specify what the output is parsed as, i.e [[apple]] vs apple might be the output from "[tag[fruit]output[raw]]" vs "[tag[fruit]output[wikitext]]". Or maybe a widget parameter makes more sense as in ouput=raw or output=wikitext because then another could be output=unprocessed which gives the filter itself as output. I realize this is a totally different way of handling things and not realistic to implement in TW5 but is there any fundamentally wrong with this line of thought?

<:-)

P.S @Tony - sorry, I have definitely hijacked your thread but I have a feeling you are also interested in the reasoning.

[TonyM]

Mat

Your contributions are welcome.

I am awaiting a medical diagnostic in hospital so can't review your code yet.

Tony

[TonyM]

Jeremy,

I Will place examples in This tread soon but as stated previously they tend to occur in less than strait forward situations.

One example I can explain easily is all the save and export templates. I would love to be able to capture what they generate and save them in a tiddler or clipboard. For example a json defined by a filter evaluated and stored in a variable or text field.

Regards
Tony

[TonyM]

Ways I have possibly incorrectly tried to use wikify

I would like to "evaluate" the following and store as text in another tiddler

• Use define macroname() {{!!fieldname}} <<varname>> $parmname$ to concatenate the content of these with their values at the time and save into a  fieldname literally.
• The Advanced search allows you to export tiddlers as a JSON file I want to save this json in a tiddler (literal text)
• I would like to store a static HTML tiddler in a tiddler (In effect a snapshot of the result)
• Tiddlers can include HTML objects and display them, I would like to be able to store this in a tiddler as literal text and thus interrogate the content.
• I would like to generate full HTML files just as the tiddler wiki save mechanism does and save the result in a tiddler for later export to a website - a webpage generator.
  
• I would like to take any tiddler that generates content from a range of widgets and wiki text save and export it as a single html file such as publishing a static report.
• I would like to copy to clipboard any of the above evaluated content.

All of these should be possible but my current understanding or tiddlywiki itself makes designing these solutions very difficult.

Regards

Tony