TSE: Duplicate entries cause syncfg2.mac not to work and not show the syntax highlight colors.

[knud van eeden]

Hello,

FYI

Using a not clean TSE 4.42.00

I wondered why the usual compilation using the TSE syncfg2.mac macro did not show the

syntax highlight colors as expected.

It showed that this is caused because there were one or more duplicates in the file.

Steps to reproduce:

1. Create and save this to a text file in your TSE synhi directory

    (e.g. c:\TSE\synhi\wireshark.txt)

wordset=0-9A-Z_a-z

HiliteIncompleteQuotes=TRUE

IgnoreCaseOfKeyWords=TRUE

HiliteIf0StyleComments=TRUE

Transparent=FALSE

CursorLineFlag=TransparentCursorLine

NumberFlag=Integer|C-Hex|ASM-Hex

MultiLineDelimited1=

MultiLineDelimited2=

MultiLineDelimited3=

TillEOL1=

TillEOLStartCol1=0

TillEOL2=

TillEOLStartCol2=0

TillEOL3=

TillEOLStartCol3=0

Quote1="

Quote2='

Quote3=

QuoteEscape1 =\

QuoteEscape2 =\

QuoteEscape3 =

Directive1=#

Directive2=

Directive3=

[KeyWords1]

server

client

[KeyWords2]

PSH

ACK

FIN

RST

SYN

URG

[KeyWords3]

<

<

[KeyWords4]

ABNORMAL

UNEXPECTED

[KeyWords5]

[KeyWords6]

[KeyWords7]

[KeyWords8]

[KeyWords9]

2. The important and relevant part here are the 2 occurrences of the '<' in [Keywords3].

That is thus an example of duplication, that is the same thing appearing more than once

in that file.

3. Then stay in that file and run the macro syncfg2

4. That should compile it successfully

5. Save the result also in the same TSE synhi directory, but now with the file extension .syn

    (e.g. c:\TSE\synhi\wireshark.syn

6. Quit and restart TSE

7. Now as usual associate that file extension with that syntax highlight definition

(run TSE macro syncfg and associate the extension 'wireshark' with wireshark)

8. Then load a file with extension .wireshark, 

    e.g. 

     foobar.wireshark

E.g. containing arbitrarily the words

PSH

ACK

FIN

RST

SYN

URG

ABNORMAL

UNEXPECTED

and check the syntax highlight colors. They should not show as expected.

9. Now to reproduce, remove one of the duplicate '<' in the txt file, save it, run syncfg2 again, save it as .syn, restart TSE, load foobar.wireshark. 

That should now show the syntax highlight colors as expected.

10. So maybe the TSE native macro 'syncfg2' should flag those duplicate characters and not compile successfully, which it instead does until now.

with friendly greetings

Knud van Eeden

[Carlo Hogeveen]

Knud,

About converting syntax hiliting definitions in a .txt file to a .syn file.
Syncfg2 implements a low-level function for the editor.
Syncfg2 is undocumented, so if we macro programmers want to use it, then we need to deduce its pre- and post-conditions ourselves.
Three of those pre-conditions are, that the text file does not contain multiple keywords on one line, that the text file is sorted, and that the text file does not contain duplicate lines.
This can be found in TSE 4.42 in Syncfg2’s parent function Syncfg in lines 879 to 887.
Because the editor implements enforcing these preconditions there, the editor does not encounter the problem you encountered.
From syncfg2’s perspective us macro programmers get garbage out when we put garbage in.
That seems fair.

Carlo

[knud van eeden]

FYI:

1. Already for example in November 2006 this is what Semware (Sammy) informed:

"

...

You can convert this to a .syn file by:

Creating a 4nt.txt file that contains the text between the "cut

here".  Make that 4nt.txt the current file, and run the syncfg2

macro.  It'll create a 4nt.syn file.

"

2. That one always should run syncfg instead is nowhere suggested or required or demanded until now further. 

3. Further has this (very quick) syncfg2 method been used by me to create and test hundreds of syntax highlight definitions for TSE and that worked OK.

4. That there have been duplicates some time thus during all this testing and creating those text file is quite likely is assumed. 

5. Not sure why that demand for only non-duplicates when using syncfg2 now suddenly was so visible (in e.g. 4.42.00).

with friendly greetings

Knud van Eeden

--

---
You received this message because you are subscribed to the Google Groups "SemWare TSE Pro text editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semware+unsub...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/semware/004201d84416%240d4785b0%2427d69110%24%40ecarlo.nl.

[knud van eeden]

Root cause should be the extra parameter "-k" added to the sort.s TSE macro used in syncfg2.si 

===

1. -OK with duplicates in:

     * syncfg2 in TSE v4.20.29 (from May 11 2005) works OK with duplicates (in the .txt file)

     ...          

     * syncfg2 in TSE v4.40.48 (from September 11 2008) works OK with duplicates (in the .txt file)

     ... 

     * syncfg2 in TSE v4.40.97 (from October 07 2016) works OK with duplicates (in the .txt file)

     ...

     * syncfg2 in TSE v4.41.03 (from October 27 2018) works OK with duplicates (in the .txt file)

===

2. -Not OK with duplicates:

     Starting with: 

     

     * syncfg2 in TSE v4.41.05 (from February 12 2019) does not work OK with duplicates (in the .txt file)

     ... 

     * syncfg2 in TSE v4.42.00 (from December 2021) does not work OK with duplicates (in the .txt file)

3. -The difference is that in the working version this is used:

     if not ExecMacro("sort")

4. -The ROOT CAUSE of the duplicate sensitivity and difference is that in the non-working versions this is used:

     if not ExecMacro("sort -i -k")

5. -Conclusion:

     Somehow this new parameters used in sort.s do not handle duplicates anymore like before.

     The added parameters to sort in the non-working version are thus "-i -k"

     According to mac\sort.s these parameters means:

                   when '-i'

                ignore_case = TRUE

            when '-k'

                kill_dups = TRUE

     Conclusion and root cause: So this parameter -k should remove duplicates, but it seemingly does not and or interferes with the old working OK

     situation.

with friendly greetings

Knud van Eeden

Knud van Eeden

[knud van eeden]

As proof in the pudding, if you remove in 4.42.00 inb the original syncfg2.si this

  "sort -i -k"

back to the old

  "sort"

and save and recompile that syncfg2.si

then run again syncfg2 with the .txt file, 

save the result as .syn in the synhi directory,

then restart TSE 

and load the file to syntax highlight it works OK as before.

with friendly greetings

Knud van Eeden

[Carlo Hogeveen]

Knud,
I concur with your new information that this is a new bug in syncfg2.
“sort -i -k” in itself works perfectly, and removes the double “<”.
However, when converting text to binary the new syncfg2:
- Counts symbols like "<".
- Sorts symbols, now also deleting duplicates with "-k".
- Writes the number of previously counted symbols to the .syn file.
Oops!
Carlo

[knud van eeden]

> sort -i -k” in itself works perfectly, and removes the double “<”.

Ves, fully agreed, I tested that already separately on for example the .txt file with deliberately inserted duplicate '>' and it removed these successfully. I used it also elsewhere successfully ad far as I know at this moment 

What I can say though is that in the preliminary quick testing on 4.42.00  using thus "sort" instead of "sort -i -k"

seemingly overruled all the side-effects and as a netto result gave again the expected colored syntax highlight even when duplicates thus present in the .txt file. 

The bottom line and conclusion is thus if you do not deliberately delete the duplicates using in sort that -k parameter but thus simply keep the duplicates then it works as before.

with friendly greetings

Knud van Eeden

Sent from Yahoo Mail on Samsung Galaxy Note 9 black 512 gigabytes

Carlo

--

---
You received this message because you are subscribed to the Google Groups "SemWare TSE Pro text editor" group.
To unsubscribe from this group and stop receiving emails from it, send an email to semware+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/semware/000c01d84519%2460d4d710%24227e8530%24%40ecarlo.nl.