TW parser to actually "replace" text

[Tony K]

(function(){

/*jslint node: true, browser: true */
/*global $tw: false */
"use strict";

exports.name = "myrule";
exports.types = {block: true};

exports.init = function(parser) {
    this.parser = parser;
    // Regexp to match
    this.matchRegExp = /-{3,}\r?(?:\n|$)/mg;
};

exports.parse = function() {
    // Move past the match
    this.parser.pos = this.matchRegExp.lastIndex;
    return [{type: "element", tag: "hr"}];
};

})();

this is the code of horiz rule of TW parser

I am using it to try to learn.

anyway when you enter --- it substitute that with hr but not in the original tiddler

how can i modify the code so that the actual --- is replaced by hr in the tiddler??

thank you 

[TiddlyTweeter]

You mean "a one way ticket"? Where the content of a tiddler is permanently changed, Once, forever?

Yes?

[Tony K]

yes sir

[TonyM]

TonyK

I have already seen spliting of the text field into lines which you could selectively append into a variable, subsequently saving a new text field. You could then process such lines to change something within each and this only with list filters and widgets. What makes this possible is generating a new temporary tiddler then copying it back. If built correctly you could go only half way and clone with changes generating a new version or the next step committing changes to the source.

Another approach may be to use a mechanisium like the draft method currently used for editing

Ultimately a replace function needs to be more than strings but able to interrogate the content. The recent xml plugin can interrogate and extract from xml and html perhaps have a look at that.

Regards
Tony

[TiddlyTweeter]

Is the aim to have these tiddlers be, basically, at end, in HTML. I.e you use WikiText to write but on edit save you want the HTML render to be saved not the WikiText?

Or is it a more complex mix?

[Tony K]

no sir, it is just a test to learn how to do it i am not looking to replace by html code or change format 

thank you 

[TiddlyTweeter]

Ok. Tips...

... The standard parsers dynamically create HTML. The render level is accessible and save able.

Both Mark S. & Jed Carty have used that to create savers of HTML-ised vers of tids in past.

... Its possible to create parsers made of raw regexes that run prior to the standard parsers. Its useful for ad-hoc needs. 

BJ did most work on this. See, for instance, http://flexibility.tiddlyspot.com/

BJ's Flexitype plugin helps understand parsers too: http://bjtools.tiddlyspot.com/#Flexitype%20Demos

HTH

TT

[PMario]

Hi Tony,

TW core has a so-called "hook" mechanism, that could be used here. It is described in the TW dev edition, with a minimal example, that just shows the basic mechanism. 

See the th-saving-tiddler hook info. ... You could do a javascript regexp search and replace in the hook. 

BUT as a user of such an editor, I personally would consider a behaviour like this as a very offensive action. ... If you just want to learn about the inner workings of TW core, I think it's a good example, with a little bit of risk involved, to keep it interesting.

I did use the th-importing-tiddler hook with my bundler plugin, to log imported tiddlers into a log tiddler. ... This can be a "real world" example how you can implement it as a plugin.

To do hooks right, you also need to understand when they are executed, so you'll also need to have a closer look at TW core code.

th-saving-tiddler is executed here: https://github.com/Jermolene/TiddlyWiki5/blob/d1441d29fde17fd382658d7b541cab513d86a936/core/modules/widgets/navigator.js#L323

th-importing-tiddler is executed here: https://github.com/Jermolene/TiddlyWiki5/blob/d1441d29fde17fd382658d7b541cab513d86a936/core/modules/widgets/navigator.js#L559

That's 1 way to do it.

have fun!

mario

[PMario]

Hi,

Referring to my last post, I do think, the parser isn't the right place to do stuff like this, because the parser does something completely different.

TW parser converts wiki text syntax into a so called AST Abstract Syntax Tree, which is an internal structure, that is kept in core memory only and is not available for end users.

It is available for plugin authors. The flow is like this

TW wikitext -> Parse Tree (AST) -> Widget Tree -> HTML

What you want is: TW wikitext -> AST -> TW wikitext

So the AST could be modified and "partially" converted back to wikitext. ... BUT...  since TW AST doesn't contain text structure info eg: info about white-text (spaces / line-breaks) and comments it would be a rewrite of the tiddler source. ... So you will loose information.

If you want to learn, how to modify TW wikitext, that way, without breaking it you'll need to understand how the relink-pluign works. ... Be aware this plugin is highly sophisticated and has been a lot of work! So you'll need to be prepared to "dig down the rabbit-hole", which needs some time!

That's an other way to do it ;)

have fun!

mario

[Saq Imtiaz]

+1 for everything P Mario said.

[PMario]

On Tuesday, May 12, 2020 at 4:13:04 PM UTC+2, PMario wrote:

...

What you want is: TW wikitext -> AST -> TW wikitext

Having something like this, with lossless conversion would be unbelievably cool.

It would allow us to store tiddler content in AST format, instead of wikitext, which probably can speed up tiddler rendering.

It would allow us to do things like: TW wikitext -> AST -> markdown (prose) text ... AND vice versa

BUT

It would mean that we rewrite a big part of the existing wikitext -> AST conversion and write a NEW  AST -> wikitext / md "converter".

... @Tony K ... trying to understand the underlying mechanisms has a lot of potential ;)

just dreaming!

-mario

[Tony K]

Hey PMario

thank you for your time writing all this to me

what I want to do, in simpler form, is replace all occurrences of a word (say Word1 ) with [[Word1]]

so, in other terms, converting a "freelink" to a "backlink"

I could do it in any language in 3 minutes top can't understand why it is so complicated in TW :S 

[Saq Imtiaz]

@TonyK it's not complicated, you have just been looking in the wrong place. A parser should not be altering source text.

Look at the saving tiddler hook that @pmario has pointed out to you.

Personally I wouldn't want to use any code that altered my source text, so a big disclaimer/warning in the plugin is probably the way to go :)

[Joshua Fontany]

I was just going to attempt this description, but I also +1 everything PMario just wrote.

It is very important to understand that the internal structure is a series of parsed trees:

TW wikitext -> Parse Tree (AST) -> Widget Tree -> HTML

I think a visual depiction of this would be incredibly useful for the community. Even the Parse-tree for a single tiddler in the Story List is very complex nested code, so even a micro-example of say, a $list with simple links to 3 other tiddlers (one of which has a `caption` field) would be a long example.

Best,

Joshua F

[PMario]

On Tuesday, May 12, 2020 at 6:42:55 PM UTC+2, Tony K wrote:

...

what I want to do, in simpler form, is replace all occurrences of a word (say Word1 ) with [[Word1]]

Hmmm, That is a completely different question.

 

so, in other terms, converting a "freelink" to a "backlink"

OK

So I'll try to rephrase it:

• There should be a list of existing tiddlers
• The occurrence of every tiddler title in the tiddler text, should be replaced by [[tiddler title]]
• The action should be global

I would prefer:

• There should be a list of existing tiddlers
• It should be possible to configure, which existing tiddler title should be "replaced"
• If configured, the occurrence of every tiddler title in the tiddler text, should be replaced by [[tiddler title]]
• The action should be global

This makes it more controllable, but not really flexible. ... IMO a global search and replace can be an extremely "destructive" functionality.

I personally would like to have much, much -- more control over the "find and replace" process. I would like, to be able to control it on a per tiddler level.

So for me an editor button would be the right thing to go with.

TW action-widgets know the tm-edit-text-operation. I think, the implementation of edit-toolbar buttons would be a good starting point. I think it would give users much more fine grained control, over the whole "review process".

Just my thoughts!

I could do it in any language in 3 minutes top can't understand why it is so complicated in TW :S 

hmm. ... Don't get me wrong. Then it would be the right thing to do it that way. ... right?

I did need much more time to write this response ;)

have fun!

mario

[Tony K]

PMario

all my apologies, I think you miss-understood me when I said "I could do it in any language in 3 minutes" I was referring to programming languages (Java / Python etc..) 

I am sorry if I offended you in any way 

[PMario]

On Wednesday, May 13, 2020 at 8:21:55 PM UTC+2, Tony K wrote:

..

I am sorry if I offended you in any way 

No offence taken. ...

I think, it is a good idea to have a possibility to switch from "feelinks" to "real links" in a simple way. ... As always I personally want to be "in control". That's why I think, global search and replace is a little bit of an overkill.

Eg: If I would import content from a "non TiddlyWiki source", freelinks is a valid option to get "initial" linking. ... Later in the process of "reviewing", I think "free linking" should be replaced by "real linking". ... Core processing is faster and imo much more precise, from an authors point of view.

--------- brainstorming

TW freelinks plubin already contains functions, that let us search for existing tiddlers and provides a function to create a regexp to identify tiddler titles in a given tiddler text.

IMO that's already half way of the functionality, we need to create the missing "replace in wikitext" function. ...

If a tiddler is in edit mode, the user has maximum control, because there is the Cancel button, which restores the existing content, if freelink -> real-link doesn't work as expected.

I think: Even if I need to edit 100+ tiddlers, a "high quality" review should be doable in an appropriate amount of time of 2h. "low quality" should be much faster.

just my thoughts

mario

[TonyM]

Folks,

To further this discussion I had a thought, perhaps a companion to freelinks in an editor toolbar button, to search the existing text and search for titles and an optionally replace with a [[hard link]]. Freelinks would let you discover the links, edit mode would allow you to elect which become permanent.

Personally I have always considered freelinks as an exploratory tool, something which makes more sense for the designer/author than the user. Users should be given what they need explicitly. the owner or designer uses it to discover. Turning off freelinks when not in discover mode.

Regards

Tony