[TW5/TWX] Replacing macros with WikiText functions

[Evan Balster]

Hey, all —

A while ago I noticed Jeremy expressing some regrets about how TiddlyWiki has become dependent on text substitution macros, and suggesting that TWX will do something differently:

Jeremy Ruston:

Thus, there are lots of quite prominent features that were introduced early on and that I would now handle differently (eg, the fact there is a difference between transclusion and the view widget, or the presence of text substitution macros which I now think was a mistake).

Having developed a lot of things in TW and reviewed the widget architecture with an eye for performance, I have to agree.  People across the experience gamut seem to routinely stumble on the order-of-operations involved in parsing widgets and expanding macros.  I'm also speaking as a compiler-designer:  I've made the mistake of letting macros stand in for functions in a language I've designed, and it was a problem for both maintainability and performance.

I view TiddlyWiki as a functional programming system, where each widget is a node with some parameters (attributes, parse tree / children) and a result (DOM tree).  Macros are the de-facto stand in for programmable subroutines, but they are unstructured and must be re-parsed whenever they appear.  Transcluded tiddlers may also behave like subroutines, and while they're structured they don't support explicit parameters.

The solution I anticipate would be to introduce a "wiki-function" entity which is pre-parsed and whose invocation does nothing more than set a few variables and instantiate its parse tree into widgets.  Perhaps something like this, widget-wise, with a WikiText syntax to get rid of the boilerplate:

<$function $name="tablerows" filter>
    <$list filter=<<filter>> >
        <tr>
            <th>{{!!title}}</th>
            <td>{{!!field1}}</td>
            <td>{{!!field2}}</td>
        </tr>
    </$list>
</$function>


<table>
    <tr>
        <th>Title</th>
        <th>field1</th>
        <th>field2</th>
    </tr>
<tr><th colspan=3>Important Stuff</th></tr>
    <$call $name="tablerows" filter="[tag[Tag1]tag[Important]]" />
<tr><th colspan=3>Less-Important Stuff</th></tr>
    <$call $name="tablerows" filter="[tag[Tag1]!tag[Important]]" />
</table>

Now, this does look basically the same as a macro call.  The differences are crucial, though: lower parsing and refresh overhead, only <<one syntax>> for variables, and no gotchas with quoting, whitespace, order-of-operations or open markup.  As far as I'm concerned, the similarity in usage could make transitioning trivial in many cases and the more consistent behavior could make TiddlyWiki "programming" much easier to learn.

[PMario]

Hi Evan,

I'm in favour of your proposal. But I would like it in a more backwards compatible way, if possible.

On Wednesday, January 3, 2018 at 8:01:05 AM UTC+1, Evan Balster wrote:

Now, this does look basically the same as a macro call. 

What's the problem, with making it exactly look like the existing syntax and just implement the performance improvement?

<$function $name="tablerows" filter>

\define tablerows(filter:<default>)




</$function>

\end




Doesn't make a difference for me. We just would need to change the macro parser. ... right?

 

The differences are crucial, though: lower parsing and refresh overhead, only <<one syntax>> for variables, and no gotchas with quoting, whitespace, order-of-operations or open markup. 

IMO those are implementation details, that could be implemented in a backwards compatible way. ... right?

 

As far as I'm concerned, the similarity in usage could make transitioning trivial in many cases and the more consistent behavior could make TiddlyWiki "programming" much easier to learn.

I personally think, that

\define xx(filter)

\end

is much easier to read and less to type than:

<$function $name="xx" filter>
</$function>

just some thoughts
mario

[Jeremy Ruston]

Hi Evan

Apologies, the TWX discussion is spread all over the place, but I think I may have mentioned before that in TWX the idea is to make macro invocations and transclusions be the same thing. In other words, macro invocations would become parameterised transclusions, which simply means that the parameters for the macrocall/transclusion would be available as local variables within the transclusion.

A related part of my thinking is that the tiddler store would become a widget; the tiddlers that it defines would be available within the scope of the widget. The ancestor cascade would take care of the shadow tiddler mechanism.

So, “global macros” would be transclusions of tiddlers defined in the main store, and “local macros” would be transclusions of tiddlers defined within the same tiddler.

Best wishes

Jeremy.

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/a6e13075-d639-4af5-b3c5-3883745688cd%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Evan Balster]

Hey, all —

Jeremy:  in TWX the idea is to make macro invocations and transclusions be the same thing.  In other words, macro invocations would become parameterised transclusions, which simply means that the parameters for the macrocall/transclusion would be available as local variables within the transclusion.

This sounds pretty good to me, and assuming some kind of parse-tree caching is involved, this would work just like the "wiki-functions" I describe under the hood.

My reason for describing an inline declaration syntax is that it's handy to define macros without adding another tiddler that clutters the namespace.  On that note...

Jeremy:  A related part of my thinking is that the tiddler store would become a widget; the tiddlers that it defines would be available within the scope of the widget. The ancestor cascade would take care of the shadow tiddler mechanism.

This sounds brilliant... it promises to address concerns like namespacing, combining local and remote wiki-stores...  Good stuff.

As pertains to the current discussion, I wonder if it would be possible for a "macro"-heavy tiddler to have its own sub-tiddler store for those subroutines?

Mario:  What's the problem, with making it exactly look like the existing syntax and just implement the performance improvement?

That's basically what I'm getting at:  It would be possible to make the syntax for defining these exactly the same as (or substantially similar to) defining macros.  However, there would be some changes and incompatible behaviors, so it would probably be better for the pragma to have a different name like \function.

Mario:  IMO those are implementation details, that could be implemented in a backwards compatible way. ... right?

Not exactly.  $parameter$ and $(variable)$ substitution have the effect of inserting text into the macro prior to parsing.  That behavior has to change for the performance improvement to take effect, because we want to parse in advance and any change in the raw text could change the structure of the parse tree.

However, it's my experience that most parameter and variable substitutions in macros end up being either direct insertion into WikiText, or attribute assignment in the form attrib="""$param""" — in both of these cases, the <<variable>> behavior will have the desired effect.  Most of the remaining cases just force a move to widgets.

Some examples:

\define make_list(filter) <$list filter="$filter$"/>
\function make_list(filter) <$list filter=<<filter>>/>

\define keycap(text) <kbd>$text$</kbd>
\function keycap(text) <kbd><<text>></kbd>

Macros that would not be directly convertible to wiki-functions would be anything where parameters need to be inserted into other pieces of syntax, or where parameters need to insert "open markup" such as // which is closed elsewhere:

\define get_date(tiddler) {{$tiddler$!!date}}

\define apply_tag(tag, text) <$tag$>$text$</$tag$>

\define apply_markup(markup, text) $markup$$text$$markup$  <!-- where markup is something like // or ``` -->

Note that all these restrictions apply to transclusions as they already exist in TiddlyWiki, as well as the new <<__param__>> syntax.

[Jeremy Ruston]

Hi Evan

Indeed, there would be an equivalent to \define that would define a tiddler for the scope of the remainder of the tiddler in which it is defined.

Turning to another aspect of TWX, my idea is that it should be written in C, and compiled to WebAssembly. There are several motivations:

* To encourage an efficient architecture

* To get the best performance in browsers and Node.js

* To be able to run TWX on the bare metal of e.g. a Raspberry Pi, being its own operating system

* To avoid TWX ending up being a functional rewrite of TW5. There’s no point in making an incompatible thing that still has the same functional limitations as TW5

I’m imagining a tiny C kernel of less than 10,000 lines of code, with the vast majority of the system being in userland (ie written in wikitext/filters/actions). An analogous situation would be FORTH interpreters, which typically have a tiny assembly language kernel.

A key implementation insight for me is that although the fundamental entity in userland is a tiddler, the kernel needs to think of the field (or name/value pair) as the fundamental primitive. Accordingly, the tiddler model is different in TWX: it is now an ordered bundle of named field values. For example:

title: HelloThere

text: Welcome to TiddlyWiki

tag: Welcome

tag: Another Tag

You can see in that example the difference from TW5: there can be multiple fields with the same name. Together, they can be treated like a list.

Some actions operate on the first occurrence of a field (eg using the first title as the primary title, and the others as aliases).

That simple change gets rid of a lot of gubbins in the current boot kernel concerned with tiddler field modules and stringifying lists.

https://github.com/Jermolene/TiddlyWiki5/blob/3f5e8aed3dd176842533a37967723b6d974eea96/boot/boot.js#L868-L892

Best wishes

Jeremy.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/bba9e126-74be-4b2a-9b45-ad1dcb5f5fe2%40googlegroups.com.

[BJ]

I have a plugin that defines parametrized tiddlers as components, there is a demo here:

http://components.tiddlyspot.com/

The demo overrides the implementation of the tabs macro as an example.

BJ

On Wednesday, January 3, 2018 at 8:01:05 AM UTC+1, Evan Balster wrote:

[PMario]

On Wednesday, January 3, 2018 at 7:55:16 PM UTC+1, Jeremy Ruston wrote:

Turning to another aspect of TWX, my idea is that it should be written in C, and compiled to WebAssembly. There are several motivations:

IMO creating a new kernel should use a language that was designed to create kernels :) ... I'd love to go with rust here.

Just a thought

-m

[Jeremy Ruston]

IMO creating a new kernel should use a language that was designed to create kernels :) ... I'd love to go with rust here.

Indeed, Rust would probably be a better choice these days.

Best wishes

Jeremy.

Just a thought

-m

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/2e218423-1c94-4f75-ad6d-784824a9b972%40googlegroups.com.

[Evan Balster]

Hey, all —

I have a plugin that defines parametrized tiddlers as components, there is a demo here:

Neat!  So under the hood this is functionally equivalent to <$vars><$transclude/></$vars> right?

What I'd really like to see in the future is a "parse-tree cache" for more efficient transclusions.

...Turning to another aspect of TWX, my idea is that it should be written in C, and compiled to WebAssembly...

...I'd love to go with rust here...

Hmmm...  I've got thoughts on this but I'd rather start a new thread so this one doesn't veer off-topic.

[BJ]

On Thursday, January 4, 2018 at 7:48:00 PM UTC+1, Evan Balster wrote:

Hey, all —

I have a plugin that defines parametrized tiddlers as components, there is a demo here:

Neat!  So under the hood this is functionally equivalent to <$vars><$transclude/></$vars> right?

that correct -its to avoid the parsing of macros, but also to make wikitext components to be more like equivalents to widgets.

What I'd really like to see in the future is a "parse-tree cache" for more efficient transclusions.

there is caching in the core - see 'getCacheForTiddler'

[Evan Balster]

there is caching in the core - see 'getCacheForTiddler' 

Haha, I almost bit my tongue when I sent that E-mail — I was aware of the cache mechanism but hadn't seen that it was implemented for parse trees as well.

Trouble is, transclusions don't use the parse tree cache.  They call parseTextReference, which parses the transcluded tiddler with the current ruleset.  Part of the trouble with implementing a parse-caching optimization is that the context where the tiddler is transcluded might define different parse rules, resulting in a different parse tree for that context.  As noted by other users, however, this "feature" can be unexpected or undesirable.

[PMario]

On Thursday, January 4, 2018 at 7:48:00 PM UTC+1, Evan Balster wrote:

What I'd really like to see in the future is a "parse-tree cache" for more efficient transclusions.

We "kind of" have one, that is used to find backlinks. But it's only 1 level deep.

I did implement an alias cache for my uni-link plugin: https://github.com/wikilabs/plugins/blob/master/wikilabs/uni-link/tiddlers/wiki-methodes.js

We would need to make the stuff more generic, so I wouldn't need to do a new parse run, just to get alias links.

have fun!
mario

[BJ]

On Thursday, January 4, 2018 at 8:56:29 PM UTC+1, Evan Balster wrote:

there is caching in the core - see 'getCacheForTiddler' 

Haha, I almost bit my tongue when I sent that E-mail — I was aware of the cache mechanism but hadn't seen that it was implemented for parse trees as well.

Trouble is, transclusions don't use the parse tree cache.  They call parseTextReference, which parses the transcluded tiddler with the current ruleset.  Part of the trouble with implementing a parse-caching optimization is that the context where the tiddler is transcluded might define different parse rules, resulting in a different parse tree for that context.  As noted by other users, however, this "feature" can be unexpected or undesirable.

if the transclusion is for a 'text' then  parseTextReference calls parseTiddler which uses the caches.

[BJ]

On Thursday, January 4, 2018 at 9:40:09 PM UTC+1, BJ wrote:

On Thursday, January 4, 2018 at 8:56:29 PM UTC+1, Evan Balster wrote:

there is caching in the core - see 'getCacheForTiddler' 

Haha, I almost bit my tongue when I sent that E-mail — I was aware of the cache mechanism but hadn't seen that it was implemented for parse trees as well.

Trouble is, transclusions don't use the parse tree cache.  They call parseTextReference, which parses the transcluded tiddler with the current ruleset.  Part of the trouble with implementing a parse-caching optimization is that the context where the tiddler is transcluded might define different parse rules, resulting in a different parse tree for that context.  As noted by other users, however, this "feature" can be unexpected or undesirable.

There is no 'context for transclusions' in tiddlywiki, (actually there is the inline/block distiction) the parsing is determined by the content of the tiddler (includings the pragmas).

[Mat]

Forgive a mere mortal, and if this is irrelevant then just disregard it but;

IMO defining multiple macros, or macros in a tiddler containing other stuff, is not true to tiddler philosophy. There mere fact that a macro is defined indicates that it should be an individual tiddler.

Adhering to tiddler philosophy here would also open up for a much more friendly UI for creating macros, as I've noted elsewhere. Briefly; an individual macro-tiddler could show a special edit template with separate fields for the macro title and another for parameters. Why should a user have to learn special syntax for macros when they are basically just encompassing "shells"?

Currently, in practice one typically needs to define multiple macros within the same tiddler. This could be easily solved, in accordance with what I just wrote, by adding new macros comparable to how we currently add new fields. In actuality, what is created are separate macro tiddlers but they are presented (transcluded) as if part of the current edit view.

<:-)

[Måns Mårtensson]

Hi @Mat

You are getting close to something I thought about sometimes - wouldnt it be nice to have a (create new) button presenning boiler plate templates for creating tiddlers within tiddlers when you need to construct intricate lists, lists within lists, macros and "cascading macros". 

When you are done filling out the boiler plate fields (select boxes and/or text boxes) a tiddler is created (silently) and all you are left with is a transclusion of the tiddler containing the macro or list eg {{list tiddler name}}...

This way you could get workflows where "code blocks" are put aside and don't clutter your (parent)tiddler too much.

Maybe an idea for a couple of new plugins from Mat Von Twaddle :) 

Cheers Måns

fre. 5. jan. 2018 kl. 03.59 skrev Mat <matia...@gmail.com>:

Forgive a mere mortal, and if this is irrelevant then just disregard it but;

IMO defining multiple macros, or macros in a tiddler containing other stuff, is not true to tiddler philosophy. There mere fact that a macro is defined indicates that it should be an (individual) tiddler.

Adhering to tiddler philosophy here would also open up for a much more friendly UI for creating macros, as I've noted elsewhere. Briefly; an (individual) macro tiddler could for example have the type macro and thus a specific macro edit template with separate fields for the macro title and another for parameters. Why should a user have to learn special syntax for macros when they are basically just encompassing "shells".

In practice one currently typically needs to define multiple macros within the same tiddler. This could be easily solved, in accordance with what I just wrote, by adding new macros comparable to how we currently add new fields. In actuality, what is created are separate macro tiddlers but they are presented (transcluded) as if part of the current edit view.

<:-)

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/188e4f82-e542-4360-a95e-1085b840c1cd%40googlegroups.com.

[Jeremy Ruston]

Hi Mat

Forgive a mere mortal, and if this is irrelevant then just disregard it but;

Not at all. In this case, I think that your proposal here could actually be explored in TW5, but it’s nonetheless interesting if discussion of TWX prompts rethinking of how we use TW5.

IMO defining multiple macros, or macros in a tiddler containing other stuff, is not true to tiddler philosophy. There mere fact that a macro is defined indicates that it should be an (individual) tiddler.

I actually find “local macros” (ie macros defined and used within the same tiddler) to be an essential tool for organising the content within a tiddler, and so I would want to see them as part of TWX. The current thinking about shifting the tiddler store to be a stack of tiddler store widgets gives us a natural way to implement them.

Adhering to tiddler philosophy here would also open up for a much more friendly UI for creating macros, as I've noted elsewhere. Briefly; an (individual) macro tiddler could for example have the type macro and thus a specific macro edit template with separate fields for the macro title and another for parameters. Why should a user have to learn special syntax for macros when they are basically just encompassing "shells”.

I think you’re asking for a custom edit template to be used for tiddlers defining macros. As you say, that approach doesn’t mesh well with tiddlers that contain multiple macros, or macros along with their invocations.

We could instead have an editor toolbar button that brings up a wizard type UI for creating macros, and then inserts the test at the end.

Best wishes

Jeremy.

In practice one currently typically needs to define multiple macros within the same tiddler. This could be easily solved, in accordance with what I just wrote, by adding new macros comparable to how we currently add new fields. In actuality, what is created are separate macro tiddlers but they are presented (transcluded) as if part of the current edit view.

<:-)

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/188e4f82-e542-4360-a95e-1085b840c1cd%40googlegroups.com.

[Evan Balster]

BJ: if the transclusion is for a 'text' then  parseTextReference calls parseTiddler which uses the caches. 

BJ: There is no 'context for transclusions' in tiddlywiki, (actually there is the inline/block distiction) the parsing is determined by the content of the tiddler (includings the pragmas).

Ah!  I had totally misunderstood, and didn't review the code well enough.  Thanks for setting me straight.

Good work all on going back in time and solving this problem for me.  ;)

Mat: IMO defining multiple macros, or macros in a tiddler containing other stuff, is not true to tiddler philosophy. There mere fact that a macro is defined indicates that it should be an individual tiddler.

Jeremy: I actually find “local macros” (ie macros defined and used within the same tiddler) to be an essential tool for organising the content within a tiddler, and so I would want to see them as part of TWX. The current thinking about shifting the tiddler store to be a stack of tiddler store widgets gives us a natural way to implement them.

So, I agree with both of these sentiments.  I think it makes sense for macros to exist as separate objects (IE, outside tiddler markup) but I also think it's useful for them to be encapsulated within the tiddler where they're used.

Allowing a Tiddler to encapsulate its own tiddler-store, separate from the global namespace, would be a natural way to have macros as separate bodies of text that are nevertheless contained in the tiddler where they're used.  I'm finding myself suddenly keen on the idea that a field or similar entity could be a tiddler-store in TWX...  This would also be a great way to keep bulk data from bogging down the wiki store.