Regex Character Classes not working in search-replace

[John Davis]

I'm using a macro and search-replace to convert a list of FirstName LastName to LastName, FirstName.  It works fine with the shortcut class \w, but if I replace that with [a-zA-Z0-9_] it breaks, when logically those should be the same. My actual goal is to replace \w with something like [^ ], because where it's breaking is last names that include apostrophes.

This is an example that can be inserted in a new Tiddler to demonstrate the issue; the first list is the way it's currently working (you can see Vincent D'Onofrio is out of order and not in LastName, FirstName format) and the second list is broken due to the character class issue:

\define compare-by-last-name()
[<currentTiddler>search-replace:i:regexp[((?:.*\s)|^)(\w+$)],[$2, $1]]
\end
\define compare-by-last-name-with-character-class()
[<currentTiddler>search-replace:i:regexp[((?:.*\s)|^)([a-zA-Z0-9_]+$)],[$2, $1]]
\end

<$set name="listOfNames" value="[[Charlie Cox]] [[Vincent D'Onofrio]] [[Deborah Ann Woll]] [[Elden Henson]] [[Royce Johnson]] [[Geoffrey Cantor]] [[Jay Ali]] [[Jon Bernthal]] [[Stephen Rider]] [[Wilson Bethel]] [[Ayelet Zurer]] [[Peter McRobbie]] [[Amy Rutberg]] [[Elodie Yung]] Awkwafina" >

<ul>
<$list filter="[enlist<listOfNames>sortsub<compare-by-last-name>]">
<li><$link><$text text={{{[<currentTiddler>search-replace:i:regexp[(.*)(\s\w+$)],[$2, $1]]}}}/></$link></li>
</$list>
</ul>


<ul>
<$list filter="[enlist<listOfNames>sortsub<compare-by-last-name-with-character-class>]">
<li><$link><$text text={{{[<currentTiddler>search-replace:i:regexp[(.*)(\s\w+$)],[$2, $1]]}}}/></$link></li>
</$list>
</ul>


</$set>

[Saq Imtiaz]

 It works fine with the shortcut class \w, but if I replace that with [a-zA-Z0-9_] it breaks, when logically those should be the same.

You cannot have the characters [ and ] inside a literal operand to a filter operator.

The workaround is to define the regular expression as a variable and then use the variable as the operand.

Pseudo code below.

\define myregexp() ((?:.*\s)|^)([a-zA-Z0-9_]+$)

\define compare-by-last-name-with-character-class()
[<currentTiddler>search-replace:i:regexp<myregexp>,[$2, $1]]
\end

[Eric Shulman]

On Saturday, August 7, 2021 at 12:24:28 AM UTC-7 john....@gmail.com wrote:

I'm using a macro and search-replace to convert a list of FirstName LastName to LastName, FirstName.  It works fine with the shortcut class \w, but if I replace that with [a-zA-Z0-9_] it breaks, when logically those should be the same.

The problem is that regex syntax uses square brackets (to specify character classes), and TiddlyWiki filter syntax also uses square brackets (to specify literal text operands), and the filter parser does not allow nesting of brackets.  Thus, when you use the character class regex syntax, it breaks the filter syntax due to nesting of square brackets.

This is noted here: https://tiddlywiki.com/#regexp%20Operator, where it says:

The filter syntax makes it impossible to directly specify a regular expression that contains square brackets. The solution is to store the expression in a variable. 

The workaround is to put your regex patterns into variables, and then use those variables in the filter, like this:

\define compare-by-last-name-with-character-class()

<$vars search="((?:.*\s)|^)([a-zA-Z0-9_]+$)" replace="$2, $1">

[<currentTiddler>search-replace:i:regexp<search>,<replace>]

</$vars>

\end

Note that is this particular use-case, it's not strictly necessary to put the "replace" pattern into a variable, since it doesn't actually use square brackets in the syntax.  However, I find that putting both the search pattern and the replacement pattern into variables makes the filter syntax more consistent and easier to read.

enjoy,

-e

[John Davis]

Awesome, thanks.  I had tried that but messed up the search-replace syntax, I didn't realize the angle brackets would replace the square brackets and was using both:

[<currentTiddler>search-replace:i:regexp[<myregexp>],[$2, $1]]

Much appreciated.