Proposing a plaintext parse & render mode for attributes & more

[Evan Balster]

Hey, all —

Currently TiddlyWiki implements plain-text wikification by applying WikiText parsing to generate a DOM, then extracting and concatenating any text-widgets in the DOM while discarding everything else.  While this functionality is very useful, it has some tricky drawbacks — wikification involves (often) unnecessary work processing irrelevant parse rules, and anything resembling an <element> or //wikitext markup will be stripped out, when this might be very undesirable (eg, when encountering JavaScript or C++ comments).

I'd like to propose an alternative mode where only those parse rules which are applicable to plaintext are considered — that is, widgets, transclusions (including lists), variables and a few other items.  That way, we can write things like this (a method stub generator):

<$list variable=class filter="[tag[Class]]">
<$list filter="[tag[Method][tag<class>]]">
// {{!!summary}}
{{!!return_type}} {{!!name}}({{!!parameters}}) {
}

</$list>
</$list>

I'd like to propose, further, that this functionality be available for attributes.  This will eliminate several situations where it's necessary to use macros or $wikify, replacing these mechanisms with a more efficient, briefer, pre-parsable expression.  As noted by Jeremy, one of the most common user mistakes involves expecting wiki syntax inside quoted strings (or imported via macro/transclusion) to work in an attribute...  Things like title="my name is {{!!name}}".  I see very few reasons (aside from compatibility) why they shouldn't work as expected!

A conservative approach could mandate a double-equals syntax or an alternative quotation style for wikified attributes:

<div style=="size: {{!!size}}px;"> 

...Though if I'm to be honest I would prefer to see all attributes parsed by default with an alternative syntax for "direct" values.  Perhaps I'm too radical.  :)

I don't think wikification needs to change much to implement this — another parse mode, with a smaller ruleset.  The tricker part of the implementation is getting widgets to parse imported text according to the proper rules.  To that end, I suggest that the namespace variable could be used to determine parsing behavior for imported text.  In the future this might also be leveraged to keep inappropriate elements out of <math> and <svg> namespaces...

With these changes, I believe TiddlyWiki's logic would become more powerful and, crucially, more consistent, helping newcomers to learn it with fewer "gotchas" and making it simpler for experts like me to build complicated constructs.  I also think it's an essential piece of the puzzle for evolving TiddlyWiki beyond macro dependency.

Interested to hear others' thoughts.

[Jeremy Ruston]

Hi Evan

I think that this ancient ticket might be roughly equivalent to what you’re after:

https://github.com/Jermolene/TiddlyWiki5/issues/345

The proposal still reflects my thinking in this area.

Best wishes

Jeremy.

--
You received this message because you are subscribed to the Google Groups "TiddlyWikiDev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywikide...@googlegroups.com.
To post to this group, send email to tiddly...@googlegroups.com.
Visit this group at https://groups.google.com/group/tiddlywikidev.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/7e867e78-da41-466a-b9ab-754439100af2%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[BJ]

For reference, I have a plugin that does some of this:

http://bjtools.tiddlyspot.com/#Flexitype%20Demos

cheers

BJ

[@TiddlyTweeter]

FWIW, I've been avidly using BJ's Flexitype plugin--especially the version that allows sequences of Regular Expressions to be run before the main parser kicks in. It's perfect for many use cases where you need develop a specialised markup (e.g. screenplays). Or cases where you want to extract strings from huge texts.

I found it stable and incredibly useful.

Best wishes
Josiah

[Evan Balster]

Vocabulary as a general concept seems like it fits the bill.  At that point, "smart" attributes would just be treated as wikitext with the "plaintext" vocabulary.

I have to confess I'd forgotten about typed blocks when I wrote the OP, and by running at parse-time they solve some of the stickier problems I mention in the OP.  They provide a simple way to insert "raw" plaintext into wikified plaintext, and to manipulate the parse settings before widgets are instantiated...  This puts some questions on my mind:

• Is it sensible to add vocabulary as another option for typed blocks?
• Would typed blocks would inherit parse settings the markup does not specify?  (vocabulary & render mode)
• Could widgets detect the parse settings that produced them and employ these when parsing imported text?
• Could the wikiparser cache support caching parse-trees for each vocabulary?
  

[Jeremy Ruston]

Hi Evan

Vocabulary as a general concept seems like it fits the bill.  At that point, "smart" attributes would just be treated as wikitext with the "plaintext" vocabulary.

The trouble there is that the plaintext vocabulary wouldn't permit widgets, which would rule out using the list widget. So we would need to be able to select the output mode between HTML and plain text (that's a setting that we already support for transclusion).

I have to confess I'd forgotten about typed blocks when I wrote the OP, and by running at parse-time they solve some of the stickier problems I mention in the OP.  They provide a simple way to insert "raw" plaintext into wikified plaintext, and to manipulate the parse settings before widgets are instantiated...  This puts some questions on my mind:

• Is it sensible to add vocabulary as another option for typed blocks?

I have been thinking of the vocabulary as a component of the mime type, not a separate field. The $tw.Tiddler object could explode and cache the components. But I think that some of your thoughts might imply the vocabulary being a separate setting.

• Would typed blocks would inherit parse settings the markup does not specify?  (vocabulary & render mode)

At the moment typed blocks are equivalent to transcluding a tiddler with the specified type and text fields, and parsing pragmas like rules are not inherited. I think we'll need to revisit that.

• Could widgets detect the parse settings that produced them and employ these when parsing imported text?

Widgets aren't necessarily produced directly from a parse tree, and so there might not be any parse settings. Again, we might want to revisit that; if so, variables should be the mechanism for the inheritance.

• Could the wikiparser cache support caching parse-trees for each vocabulary?
  

It would have to, yes (just as we currently separately cache the inline vs. block parsing of a tiddler).

Best wishes

Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/482bb2ac-1419-4ea5-a73c-d0f0f89afeb8%40googlegroups.com.

[Evan Balster]

Hey, Jeremy —

 

The trouble there is that the plaintext vocabulary wouldn't permit widgets, which would rule out using the list widget. So we would need to be able to select the output mode between HTML and plain text (that's a setting that we already support for transclusion).

 A "wikified plaintext" vocabulary then.  I imagine that it's necessary to support at least widgets, transclusion, variables and typed blocks for any vocabulary that supports "computation" (and honestly I believe these features should be enabled whenever possible).

The parse rules for <$widgets> and <elements> may need to be separately enabled...

Widgets aren't necessarily produced directly from a parse tree, and so there might not be any parse settings. Again, we might want to revisit that; if so, variables should be the mechanism for the inheritance.

So rendered/transcluded tiddlers and typed blocks should both be wrapped in some kind of widget that sets appropriate parse-control variables.  Simple enough, and I imagine there's a mechanism 

It would have to [support caching for each vocabulary], yes (just as we currently separately cache the inline vs. block parsing of a tiddler).

Here's a thought:  Block and inline are essentially vocabularies already.  That is, they're sub-divisions of the content type.  Can we just build on that mechanism? 

[Jeremy Ruston]

Hi Evan

The trouble there is that the plaintext vocabulary wouldn't permit widgets, which would rule out using the list widget. So we would need to be able to select the output mode between HTML and plain text (that's a setting that we already support for transclusion).

 A "wikified plaintext" vocabulary then.  I imagine that it's necessary to support at least widgets, transclusion, variables and typed blocks for any vocabulary that supports "computation" (and honestly I believe these features should be enabled whenever possible).

Sorry for the confusion, I meant something else. There are two ways that we turn the DOM tree generated by the widget tree into text: using either innerHTML or textContent. The macrocall widget uses innerHTML when the $output parameter is “text/html” and textContent when the $output parameter is “text/plain”. The wikify widget extends things with an “output” parameter that can be “text” or “html”, or a few special values documented on tw.com.

It may be adequate to always use textContent for widget attributes; one can still use an external wikify widget to get the effect of outerHTML.

The parse rules for <$widgets> and <elements> may need to be separately enabled…

Yes it sounds like that would be useful.

It would have to [support caching for each vocabulary], yes (just as we currently separately cache the inline vs. block parsing of a tiddler).

Here's a thought:  Block and inline are essentially vocabularies already.  That is, they're sub-divisions of the content type.  Can we just build on that mechanism? 

Indeed, the current implementation of block vs. inline will need substantial extension. Indeed, we also need to think through how the block vs. inline logic is handled. I’m still hoping we can move most of the logic from the parse stage to the rendering stage.

Best wishes

Jeremy

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywikidev/92bfa8eb-e0a4-4ab5-b61c-18455fff8916%40googlegroups.com.