Announcing Tiddlywiki and regexp Examples: Documentation wiki

[Mohammad]

Announcement: RegExp in Tiddlywiki

Date: Aug 25, 2019

Status: In progress

Purpose: Learn regexp in Tiddlywiki

Based on the great works of some Tiddlywikians a documentation wiki dedicated to learn and practice regular expression in Tiddlywiki has been created.

Mostly Mark S and Tiddlytweer (Josiah) were involved but other people also joined (twMat, TonyM, Hans, ...)

The wiki is organized to simple extended. I will add help how to add new pattern example.

Wiki: http://tw-regexp.tiddlyspot.com/

I highly welcome any feedback and invite interested people for collaboration, mostly in preparing documentation tiddlers.

Submitting use cases.

Mark S, and  TiddlyTweeter themselves are  collaborating authors of the above wiki, but highly appreciated their time and talent.

Ref:

1. https://groups.google.com/d/msg/tiddlywiki/VFJS9eB9oV4/S6_UO3eyAAAJ
   
2. https://groups.google.com/d/msg/tiddlywiki/43x2tbA4ALE/jLWRTPqjAQAJ
   
3. Announced wiki: http://tw-regexp.tiddlyspot.com/

Cheers

Mohammad

[Mohammad]

Announcement: RegExp in Tiddlywiki

Date: Aug 25, 2019

Status: In progress

Updated to Release 0.5

This update uses tag to collect all tiddlers describing a regexp pattern. These tiddlers have two fields a below and tagged with pattern

- pattern 

- description

A demo uses description and pattern and let you try different patterns

A link provided from demo tiddler and can go to documentation tiddler.

See new update:

http://tw-regexp.tiddlyspot.com/

--Mohammad

[Mohammad]

Announcement: RegExp in Tiddlywiki

Date: Aug 26, 2019

Status: In progress

Updated to Release 0.6

• 0.6 - Aug 26, 2019

1. Experimentation tiddler was added to let freely play with regexp patterns
2. Documentation improved
3. Some patterns were improved.

[@TiddlyTweeter]

A very good resource Mohammad! Of great long-term value. I hope it gets used! 

I'd ask Jeremy to link to it from docs!

One comment ...

• It should be mentioned that * should not be used instead + as the latter needs to have at least one character so does not matches empty string while * matches empty string as it indicates zero or more symbols.

This is not quite right. * has as much utility as +. Its just that it can match the "empty string". But often that is what you want to happen. 

If you use "+" always it can block the rest of a match. So it depends on the match context which one you use.

The "qualifiers" have a range of effects ...

* = match 0 or more times, as many times as possible ("greedy")

*? = match 0 or more times, as few times as possible ("lazy")

+ = match 1 or more times, as many times as possible ("greedy")

+? = match 1 or more times, as few times as possible ("lazy")

? = match 0 or one time

{5} = match exactly 5 times

{5,9} = match 5 to 9 times

TT

[@TiddlyTweeter]

Ciao Mohammad

I think it would be worth, in time, adding ...

• A Tiddler for Quick Reference to Regex Syntax (you could use the tiddler I wrote for Tiddler Commander regex section as a starting point?)

• A tiddler for the TW Regex (& related) Operators that summarises (1) how to use them; (2) notes on things the main docs don't always fully explain (field context; scope, importance of flag states; need for variable for "[...]")

Thoughts

TT

[Mohammad]

Thanks TT,

 I will use these comments on next update of tw-regexp 

Cheers

Mohammad

[@TiddlyTweeter]

M,

I will write a "Tips Note" for use of regex for your site.

There are some common problems that are puzzling until you understand them.

This I'll do later as I'm still understanding where users are having problems.

But they include ...

• When to "escape" via "\" and when not to.
• Why does "\b" (word boundary) not always work?
• What is the importance of "anchors" (\b, ^, $)? And when NOT to use them.
• Methods to span lines in text fields.

Let me know if you noticed others!

J.

[@TiddlyTweeter]

Ciao Mohammad

For the regex section of Tiddler Commander (a slightly hacked version) I wrote a custom macro that loads both "match" & "replace", "flags", "field" etc ... 

I wanted to add loading of example data  from a Tiddler (non-destructive; i.e. its copied to a temporary Tiddler to work on and discard) but still haven't finished that.

You might find it useful? At least as a proof. 

Part of the issue we are documenting is how to "match". 

We doing well now on that, but still weak on "replace". 

? ? IMO users may want to use regex when they actually want to CHANGE something ? ?

code ...

\define snr-regex(match:" ", replace:"""-""", field:"text", case:"i", flags:"g", description:"A Regex")
<blockquote>

; $description$
: field: ''`$field$`''&nbsp;&nbsp; flags: ''`$case$`''&nbsp;''`$flags$`''
: match: ''`$match$`''
: replace: ''`$replace$`''
<$button tooltip="load match and replace patterns">
   <!-- load match and replace regex via inline macro -------------------------------------------->
   load
   <$action-setfield $tiddler="$:/state/commander/snr/select-tiddler/field" text=$field$         />
   <$action-setfield $tiddler="$:/state/commander/snr/gm"                   text=$flags$         />
   <$action-setfield $tiddler="$:/state/commander/snr/whole-words"          text=""              />
   <$action-setfield $tiddler="$:/state/commander/snr/case-sensitive"       text="""$case$"""    />
   <$action-setfield $tiddler="$:/state/commander/snr/replace-text"         text="""$match$"""   />
   <$action-setfield $tiddler="$:/state/commander/snr/replace-text-with"    text="""$replace$""" />
</$button>
</blockquote>
\end

\define snr-regex-apply(match, replace)
<$button  tooltip="apply this pattern">
   <!-- apply match & replace regex -->
   apply / done
   <$action-setfield $tiddler="$:/state/commander/snr/replace-text"      text=""""""     />
   <$action-setfield $tiddler="$:/state/commander/snr/replace-text-with" text=""""""     />
</$button>
\end

It looks like this ...

TT

[Mohammad]

Great TT,

 Please prepare Tips Note, I am sure it can help people not familiar with regexp!

--Mohammad

[Mohammad]

Good,

 I will see how to adopt it for tw-regexp.

--Mohammad

[Mohammad]

Announcement: RegExp in Tiddlywiki

Date: Aug 29, 2019

Status: In progress

Revision: 0.7

Purpose: Learn regexp in Tiddlywiki

A new update has been published.

 

url: http://tw-regexp.tiddlyspot.com/

Please send your feedback.

• 0.7 - Aug 29, 2019

1. Regexp Search Box lets to search in any tiddler fields
2. More examples
3. Tobias Beer was acknowledged for his great script search in field

[HansWobbe]

Mohammad:

Great organizational work, indeed!

Thanks for this and your many other contributions.

Cheers,

Hans

[Mohammad]

Thanks Hans,

 I am happy you found it useful.

Cheers

Mohammad

[@TiddlyTweeter]

Note on "word" boundaries: \b & its limits

"\b" = "word-boundary" is an anchor that is complementary to "\w".

\b is shorthand for "before or after the characters [a-zA-Z0-9_]". 

In regular expressions "cat", "cat9", "cat_9" and "999" are all "words". 

"Über" is NOT a word in regex because \b won't match accented letters. 
On "Über" the regex (\b\w+\b) will return "ber".

So if you work in languages other than English this is good to know to save hassle.

TT

[Mohammad]

Thanks TT for the note!

[@TiddlyTweeter]

Note on "word" boundaries: \b in character class will not work!

Inside a Character Class, like so: [\b], no longer means "word-boundary", it means "match a backspace". This is why putting short-hands for character classes inside character classes is not good practice.  [\w] will work as expected, [\b] will not. It can get confusing!

TT

[Mohammad]

Announcement: RegExp in Tiddlywiki

Date: Aug30, 2019

Status: In progress

Revision: 0.8

Purpose: Learn regexp in Tiddlywiki

Wiki: http://tw-regexp.tiddlyspot.com/

• 0.8 - Aug 30, 2019

1. Regexp Search in Fields improved
2. A new search-regexp macro has been added which can be called anywhere.
3. RegExp Experimentation with a Specified Field was added to let freely play with regexp patterns in a user defined field
4. Search in Titles with Predefined Rexexp Patterns improved for documentation.
5. Several sample patterns are added

Your feedback are highly welcome.

Cheers

Mohammad

[TonyM]

Another useful CSS operating on the text field if possible?

Search a macro tiddler for macros of the form

\define macro-name()

then list all "macro-names"

Which can then be used to display the "current value" of that macro

Ie begins \define
one or more spaces
capture the name
end of name when "(" is detected.
Regards
Tony

Regards

Tony

[@TiddlyTweeter]

Ciao TonyM & Mohammad

Search a macro tiddler for macros of the form
\define macro-name()
then list all "macro-names"
Which can then be used to display the "current value" of that macro

Creating a regular expression to match what you want is quite easy.

My problem is maybe I don't know enough about TW operators to finalise it.

In the regex you can "capture" the name of a defined macro into a numbered group.

I just can't find any examples or documentation of how to return that in a listing, rather than a list of titles the macros are in.

Any ideas welcome.

TT

[Mohammad Rahmani]

Hi Tony

 I have already published a macro to find all macros and show them in a useful format!

Have a look at Yazd wiki!

By the way, your suggestion is useful in creating regexp example and hopefully Mark, Josiah will create one!

Best wishes

Mohammad

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/a52cac8d-d1bc-407d-9144-de6f8788b9aa%40googlegroups.com.

[coda coder]

On Saturday, August 31, 2019 at 1:40:41 PM UTC-5, Mohammad wrote:

Hi Tony

 I have already published a macro to find all macros and show them in a useful format!

Have a look at Yazd wiki!

link?

[Mohammad Rahmani]

This is a link to Yazd and show-macro

https://tw-scripts.github.io/Yazd/#macro%2Fshow-macro

Show macro uses the powerful find macro which itself can be used for partial transclusion.

Best wishes

Mohammad

--

You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/fb215b8e-0f18-466a-9c01-0d2a8e500438%40googlegroups.com.

[Mark S.]

There are no core tools for extracting sections identified by regular expressions directly. Instead you

have to apply various levels of splitting, identifying, extracting (sometimes using nth operator). The actual

separator gets thrown away -- the opposite of what would be convenient for extracting text.

So, if there were macros in tiddler test1, then this works (in my test) :

<$list filter="[[test1]get[text]splitregexp[\n]regexp[^\\define]splitregexp[^\\define]!is[blank]]" variable="splitme">
{{{ [<splitme>splitregexp[\(.*?\)]first[]] }}}<br/></$list>

So stuff gets split into lines. Then matching lines are identified. Then split. Compressed. Used in {{{ }}} where they are split again by (parameter), but this time only the part before the split is saved.

[TonyM]

Folks,

I will raise this again as I think it relevant and want to share my own methodology.

For every macro tiddler I write I document each macro in its own fieldname as follows on the macro tiddler in question

fieldname: macro-macroname

Fieldcontent: `<<macroname requiredparam [defaultparam]>> syntax notes on using this macro

However I would like a richer set of documentation for macros and I am seriously considering moving them into their own tiddlers so each can also have descriptions, be independently enabled/disabled and distributed.

For this reason I was keen to search for all "\define" and would consider doing the same macro-macro name for every tiddlywiki system macro (in my own copy) or a tiddler bundle you can install. The advantage is in advanced search filter or system you can quickly find where the macro is defined and return the syntax. 

I have attached an image of a control Panel info Tab I created to support my own macros but would like to see a common standard developed.

It would be better to have separate tiddlers under a namespace to document system macros so we do not generate overwritten shadow tiddlers.

Mark,

I tried to make your example work on tiddlywiki.com on tiddlers tagged with $:/tags/Macro but get a lot of false positives.

Regards

Tony

[Mark S.]

Hi Tony,

Here's the list it generated for me (I had to alphabetize them in order to get rid of google line break anomalies) . Which ones do you think are false positives?

 .advancedsearch-tab
 .attr
 .button
 .clink
 .combokey
 .controlpanel-tab
 .deprecated-since
 .dlink
 .dlink-ex
 .em
 .example
 .example-tab
 .field
 .flink
 .from-version
 .info-tab
 .inline-operator-example
 .js-macro-link
 .key
 .keycombo
 .link
 .lorem
 .macro-examples
 .mlink
 .mlink2
 .more-tab
 .mtitle
 .node-only-operator
 .note
 .olink
 .olink2
 .op
 .operator-example
 .operator-examples
 .otitle
 .param
 .place
 .preamble
 .s-matching-is-case-sensitive
 .sidebar-tab
 .state-prefix
 .strong
 .tab
 .tag
 .this-is-operator-example
 .this-is-static-link-variable
 .this-is-toolbar-config-variable
 .tid
 .tip
 .toc-lorem
 .toc-tab
 .using-days-of-week
 .value
 .var
 .variable-examples
 .vlink
 .vlink2
 .vtitle
 .warning
 .wid
 .widget-examples
 .wlink
 .wlink2
 .word
 style-guide
 tw-code
 tw-code-link
 wikitext-example
 wikitext-example-without-html
.concat
.def

[TonyM]

Mark,

Was your result on an empty.html?

Thanks I have the answer

I did not get the above results.But trying again on tiddlywiki.com did with

<$list filter="[all[]get[text]splitregexp[\n]regexp[^\\define]splitregexp[^\\define]!is[blank]]" variable="splitme">
{{{ [<splitme>splitregexp[\(.*?\)]first[]] }}}<br/></$list>

However I would like a title and link to the containing tiddler as well.

<table>
<tr>
<th>Macro</th>
<th>Defined in</th>
<th>Global (Tagged ~$:/tags/Macro)</th>
</tr>
<$list filter="[all[]sort[]]" variable=tiddlername>
<$list filter="[<tiddlername>get[text]splitregexp[\n]regexp[^\\define]splitregexp[^\\define]!is[blank]]" variable="splitme">
<tr>
<td><$text text={{{ [<splitme>splitregexp[\(.*?\)]first[]] }}}/></td>
<td><$link to=<<tiddlername>> ><$text text=<<tiddlername>>/></$link></td>
<td><$text text={{{ [<tiddlername>tag[$:/tags/Macro]then[Yes]else[No]] }}}/></td>
</tr>
</$list>
</$list>
</table>

On tiddlywiki.com this also finds "\define" in documentation (often proceeded by '''/n)

Regards

Tony

[Mohammad Rahmani]

Hi Tony,

 Just to note that in TW 5.1.20 the below

<$link to=<<tiddlername>> ><$text text=<<tiddlername>>/></$link>

can be write as simple as

<$link/>

--Mohammad

Best wishes

Mohammad

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/a3a521d1-677d-4702-b4fc-fe2fdb24c1ca%40googlegroups.com.

[Mohammad Rahmani]

Tony, Mark,

 That would be great if a slider is used to simply click and display the macro content (body).

Best wishes

Mohammad

On Sun, Sep 1, 2019 at 6:52 AM TonyM <anthony...@gmail.com> wrote:

[@TiddlyTweeter]

Thanks Mark S.

VERY helpful illustration!

Has anyone ever asked on GH to add ability to use Regex capture groups directly in filters? 

They are very useful as you are able to define in a regex exactly what you need back. In effect they 

could do your solution in, basically, one move.

Just wondering

TT 

[@TiddlyTweeter]

TonyM

Another good illustration. Useful tool!

Make the table heading columns sortable and you'd have both lookup by macro name and lookup by Tiddler name :-) 

TT

TonyM wrote:

... a title and link to the containing tiddler as well.

[@TiddlyTweeter]

Mohammad

An issue that is now clear in this thread is the need for full filter/operator examples documenting. Not just regex.

I highly doubt most anyone would have Mark's skill at combining the regex operators to extract as he does.

Nor TonyM's solution to his desire to list macros and link.

It is all doable, but I would not say its in anyway obvious.

Observations

TT

[@TiddlyTweeter]

Mohammad & all

I just want to make the general point, again, that regular expressions are NOT just for "matching", they are for "search & replace" too.

Filters are dominantly orientated to matching, to search & find, not change.

I thought your regex tool in Tiddler Commander a good step towards that.

I hope we can bring in the text field and changing it soon. Otherwise we are just finding things, not changing things.

TT

[Mohammad Rahmani]

Hi Josiah,

Good suggestion!

 I think I should write a book on Tiddlywiki :-)

Best wishes

Mohammad

--
You received this message because you are subscribed to the Google Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tiddlywiki+...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/tiddlywiki/3615b3f4-fdb6-483a-9319-e98126578f45%40googlegroups.com.

[@TiddlyTweeter]

Mohammad wrote:

Hi Josiah,

Good suggestion!

 I think I should write a book on Tiddlywiki :-)

Great. I'd happily edit it. 

J.