How to use [ and ] in array key

[Alexandru]

I stumbeld into a big issue.
I define a simple array where the key contain brackets:

set key {test[10]}
array set a [list $key 100]

"array get a" returns the content of the array as list as expected:

{test[10]} 100

But when I try to use a filter, it's returning empty string:

array get a $key

It's like my array is write only but not read!
Is there a trick for this?

Many thank
Alexandru

[Alexandru]

After more tests and trials I see that also the simple "string match" does not work:

set s1 {test[10]}
set s2 $s1
string match $s1 $s2

returns 0

Why???

[Alexandru]

I see now in the help to "string match" that one can use \x to match special chars like bracket. I'll see how I can use this...

[Uwe Klein]

Am 15.01.21 um 11:15 schrieb Alexandru:

> set key {test[10]}
> array set a [list $key 100]

array get takes a _pattern_.

$key in that context is interpreted as a glob style pattern.
that glob style pattern does not match the exact pattern of $key

try:
array names a -exact $key

Uwe
would avoid using such "selftrapping" keys for array elements.

[Alexandru]

Yes and I really neet glob matching because I actually match things like {test[10],*}
In the manual for "string match it states, that one should use \x as an excape character for thing like *?[]\.
Thanks!

[Uwe Klein]

Am 15.01.21 um 11:46 schrieb Alexandru:

what about sanitizing your keys first?
proc sanitize string {
.......
}
set akey [sanitize $key]
array set [ list $akey $value]
..

this is a recurring problem, actually.
( you get similiar issues when variable/element names are used to name
widgets. ( like in some graphic variable browser :-))

Uwe

[Alexandru]

Uwe Klein schrieb am Freitag, 15. Januar 2021 um 12:02:51 UTC+1:
> Am 15.01.21 um 11:46 schrieb Alexandru:
> > Uwe Klein schrieb am Freitag, 15. Januar 2021 um 11:44:02 UTC+1:
> >> Am 15.01.21 um 11:15 schrieb Alexandru:
> >>> set key {test[10]}
> >>> array set a [list $key 100]
> >> array get takes a _pattern_.
> >>
> >> $key in that context is interpreted as a glob style pattern.
> >> that glob style pattern does not match the exact pattern of $key
> >>
> >> try:
> >> array names a -exact $key
> >>
> >> Uwe
> >> would avoid using such "selftrapping" keys for array elements.
> >
> > Yes and I really neet glob matching because I actually match things like {test[10],*}
> > In the manual for "string match it states, that one should use \x as an excape character for thing like *?[]\.
> > Thanks!
> >
> what about sanitizing your keys first?
> proc sanitize string {
> .......
> }
> set akey [sanitize $key]
> array set [ list $akey $value]
> ..

Yes, exactly, this is what I coded:

proc StringEscapeChars {s} {
return [string map {\[ \\[ \] \\] \* \\* \? \\?} $s]
}

[Uwe Klein]

Am 15.01.21 um 13:17 schrieb Alexandru:

> Yes, exactly, this is what I coded:
>
> proc StringEscapeChars {s} {
> return [string map {\[ \\[ \] \\] \* \\* \? \\?} $s]
> }
>

if your keys are essentially nothing more than another sub_array naming
scheme:

why don't you convert your
test[123]
style element names

to just

test,123
or
test,123,..

this would make your live so much easier.

uwe

[Alexandru]

To explain why, would be very complex. To change this in the code would make this specific isue easier, but would make lots of other things more complex. The the element style is not given by me, it's just the name of files on disk. Those file names are read by our app and used as identifiers all over the code. Of course one could try to map file names to some other "safe" keys, but that would make other thing more complex, as I wrote.

[Rich]

Because [] are meaningful to string match (this from the manpage):

[chars] Matches any character in the set given by chars. If a
sequence of the form x-y appears in chars, then any
character between x and y, inclusive, will match.
When used with -nocase, the end points of the range
are converted to lower case first. Whereas {[A-z]}
matches "_" when matching case-sensitively (since "_"
falls between the "Z" and "a"), with -nocase this is
considered like {[A-Za-z]} (and probably what was
meant in the first place).

They tell string match to look for a character 1 or a character 0.

You need to escape the characters that have special meaning to the
"glob style" matches, which is also defined in the manpage:

\x Matches the single character x. This provides a way
of avoiding the special interpretation of the charac-
ters *?[]\ in pattern.

[Rich]

And string map can handle the escaping for you:

set escaped [string map [list \[ \\[ \] \\]] $input]

Note the \] is not /technically/ necessary, but also does no harm.

[js]

I have handled data like yours before. The [string map] trick works
well, and you only need to use it when adding entries to your array. It
is simple enough. Other things you can do include transforming your file
names into a more manageable form, like using a hash function. Or you
can keep file names in a list or dictionary and assign a unique number
each time a new file name arrives.

[Rich]

js <jsin...@outlook.com> wrote:
> Or you can keep file names in a list or dictionary and assign a
> unique number each time a new file name arrives.

If you keep the names in a list (and don't delete any entries), then
you get a "unique number" assignment as a side-effect. The unique
number is the index position of the filename in the list.

[Alexandru]

Is there a simple way to find out which commands in Tcl have the option -glob (explicit or implicit)?
Until now I have come with lsearch -glob, string match, dict keys
But I have a hunch, that this is not all...

[Uwe Klein]

Am 17.01.21 um 11:00 schrieb Alexandru:

> Is there a simple way to find out which commands in Tcl have the option -glob (explicit or implicit)?
> Until now I have come with lsearch -glob, string match, dict keys
> But I have a hunch, that this is not all...
>

explicit: ( via zgrep -l -- '-glob[^a]' *gz)
array.n.gz
fileutil.n.gz
graph.n.gz
lsearch.n.gz
matrix1.n.gz
matrix.n.gz
snit.n.gz
struct_tree.n.gz
switch.n.gz
tar.n.gz
TclX.n.gz
websocket.n.gz
gzip: wish.n.gz: No such file or directory

Uwe

[Alexandru]

Thanks!
How about the implicit usage of -glob in other commands?
"string match" and "dict keys" are only two of them. Are there more?

[Alexandru]

I just dicovered that "lsearch -dictionary" also needs escaping chars.
I feel very unsecure about those hidden traps in Tcl...

[Rolf Ade]

Alexandru <alexandr...@meshparts.de> writes:

> Alexandru schrieb am Sonntag, 17. Januar 2021 um 13:05:27 UTC+1:
>> > Am 17.01.21 um 11:00 schrieb Alexandru:
>> > > Is there a simple way to find out which commands in Tcl have the option -glob (explicit or implicit)?
>

>> How about the implicit usage of -glob in other commands?
>> "string match" and "dict keys" are only two of them. Are there more?
>
> I just dicovered that "lsearch -dictionary" also needs escaping chars.
> I feel very unsecure about those hidden traps in Tcl...

I'm unsure about what exactly you talk in the case of lsearch
-dictionary. A short example would be helpful.

[Alexandru]

As described in the OP, the issue is the presence of [ and ].
Hier is the new example with lsearch:
lsearch -dictionary {[1] [2] [3]} {[1]}
which returns -1

[heinrichmartin]

On Monday, January 18, 2021 at 8:09:30 AM UTC+1, Alexandru wrote:
> As described in the OP, the issue is the presence of [ and ].
> Hier is the new example with lsearch:
> lsearch -dictionary {[1] [2] [3]} {[1]}
> which returns -1

Iirc, we had this in another thread on clt: -dictionary does not imply -exact, but it is only useful with that option.

[Rolf Ade]

Alexandru <alexandr...@meshparts.de> writes:
> Rolf Ade schrieb am Montag, 18. Januar 2021 um 01:56:48 UTC+1:
>> Alexandru <alexandr...@meshparts.de> writes:
>>
>> > Alexandru schrieb am Sonntag, 17. Januar 2021 um 13:05:27 UTC+1:
>> >> > Am 17.01.21 um 11:00 schrieb Alexandru:
>> >> > > Is there a simple way to find out which commands in Tcl have the option -glob (explicit or implicit)?
>> >
>> >> How about the implicit usage of -glob in other commands?
>> >> "string match" and "dict keys" are only two of them. Are there more?
>> >
>> > I just dicovered that "lsearch -dictionary" also needs escaping chars.
>> > I feel very unsecure about those hidden traps in Tcl...
>> I'm unsure about what exactly you talk in the case of lsearch
>> -dictionary. A short example would be helpful.
> As described in the OP, the issue is the presence of [ and ].

I followed the thread; therefor my question.

> Hier is the new example with lsearch:
> lsearch -dictionary {[1] [2] [3]} {[1]}
> which returns -1

I think this one is clearly pilot error.

As stated in the documentation, the contents description options
(-dictionary is one of them) "are only meaningful wenn used with the
-exact and -sorted option".

The documentation also spells out that lsearch by default - without
matching style option - work in -glob mode.

So,

lsearch -exact -sorted -dictionary {[1] [2] [3]} {[1]}

return the expected 0.

[Alexandru]

Thanks! I'll look into that.

[Alexandru]

A few questions arise for me after reading the manual multiple times.
This is one of them:

The manual states with respect to the -nocase option:
"-nocase Causes comparisons to be handled in a case-insensitive manner. Has no effect if combined with the -dictionary, -integer, or -real options. "

So according to the manual

lsearch -dictionary -nocase [list "integer" "real" "string"] "REAL"

should return -1 but it returns the correct index of "real" in the list, which is 1.

Only if I omit -nocase, it returns -1. So the manual is wrong(?)

[Rich]

Alexandru <alexandr...@meshparts.de> wrote:
> A few questions arise for me after reading the manual multiple times.
> This is one of them:
>
> The manual states with respect to the -nocase option:
> "-nocase Causes comparisons to be handled in a case-insensitive
> manner. Has no effect if combined with the -dictionary, -integer, or
> -real options. "
>
> So according to the manual
>
> lsearch -dictionary -nocase [list "integer" "real" "string"] "REAL"
>
> should return -1 but it returns the correct index of "real" in the
> list, which is 1.
>
> Only if I omit -nocase, it returns -1. So the manual is wrong(?)

Nope, first, reading the lsearch man page, it says this (in part) about
the -dictionary option for lsearch:

(see lsort for a fuller description)

So, when one looks at the lsort man page, under dictionary one finds:

-dictionary
Use dictionary-style comparison. This is the same as
-ascii except (a) case is ignored except as a tie-breaker

So the -dictionary option already implies -nocase. So using -nocase
woulld "have no effect" because -dictionary is already ignoring case.

[Alexandru]

I get it: In a very complicated, prgramatic way, the manual does not states, what the reader thinks it states, exept that it does...
No, seariously, I urge the author of this part of the manual, to rephrase that sentence, or extend it so that normal people can correctly interpret it.

[Christian Gollwitzer]

Am 18.01.21 um 22:15 schrieb Alexandru:

It's been pointed out many times before that the Tcl manual is quite
bad. The problem is that it is much harder to find people who enjoy
writing technical documentation than developers, and those are very
scarce for Tcl anway.
So, if you have a concrete proposal how to rewrite the manual, I guess
the TCT would happily incorporate it.

Christian

[Alexandru]

I'm also not part of the needed kind of pople, but in this case I think the logic is simple:

lsearch -dictionary -nocase [list "integer" "real" "string"] "REAL"

returns 1

lsearch -dictionary [list "integer" "real" "string"] "REAL"
return -1

Hence, the -nocase option has an effect on the result of "lsearch -dictionary".
Hence, simply remove the "-dictionary" from the list of options and you get:

"Causes comparisons to be handled in a case-insensitive manner. Has no effect if combined with the -integer or -real options. "

I will not write a TIP for this and I just hope, that somebody from the TCT will read this and just do it.

[Rich]

Which one are you suggesting be rephrased? If you mean the lsort
-dictionary one, it is already perfectly clear.

[nemethi]

Am 18.01.21 um 22:42 schrieb Alexandru:

> Christian Gollwitzer schrieb am Montag, 18. Januar 2021 um 22:34:33 UTC+1:
>> Am 18.01.21 um 22:15 schrieb Alexandru:
>>> Rich schrieb am Montag, 18. Januar 2021 um 22:09:40 UTC+1:
>>>> So, when one looks at the lsort man page, under dictionary one finds:
>>>>
>>>> -dictionary
>>>> Use dictionary-style comparison. This is the same as
>>>> -ascii except (a) case is ignored except as a tie-breaker
>>>>
>>>> So the -dictionary option already implies -nocase. So using -nocase
>>>> woulld "have no effect" because -dictionary is already ignoring case.
>>>
>>> I get it: In a very complicated, prgramatic way, the manual does not states, what the reader thinks it states, exept that it does...
>>> No, seariously, I urge the author of this part of the manual, to rephrase that sentence, or extend it so that normal people can correctly interpret it.
>> It's been pointed out many times before that the Tcl manual is quite
>> bad. The problem is that it is much harder to find people who enjoy
>> writing technical documentation than developers, and those are very
>> scarce for Tcl anway.
>> So, if you have a concrete proposal how to rewrite the manual, I guess
>> the TCT would happily incorporate it.
>>
>> Christian
>
> I'm also not part of the needed kind of pople, but in this case I think the logic is simple:
>
> lsearch -dictionary -nocase [list "integer" "real" "string"] "REAL"
> returns 1
>
> lsearch -dictionary [list "integer" "real" "string"] "REAL"
> return -1
>
> Hence, the -nocase option has an effect on the result of "lsearch -dictionary".

This is not quite correct. In the first lsearch invocation above, the
-dictionary option is ignored because it is only significant in the
presence of -sorted. Hence this lsearch invocation is equivalent to

lsearch -nocase [list "integer" "real" "string"] "REAL"

which returns 1, as expected. If you make the -dictionary option
significant by adding -sorted then you will see that -nocase has no
effect indeed: all the 3 lines below return -1:

lsearch -dictionary -sorted [list "integer" "real" "string"] "REAL"
lsearch -dictionary -sorted -nocase [list "integer" "real" "string"] "REAL"
lsearch -nocase -dictionary -sorted [list "integer" "real" "string"] "REAL"

> Hence, simply remove the "-dictionary" from the list of options and you get:
>
> "Causes comparisons to be handled in a case-insensitive manner. Has no effect if combined with the -integer or -real options."

In the light of the above this would not be the proper improvement in
the manual. A better version could read:

"-nocase

Causes comparisons to be handled in a case-insensitive manner.

Has no effect if combined with -dictionary (only in the presence of
-sorted), -integer, or -real."

>
> I will not write a TIP for this and I just hope, that somebody from the TCT will read this and just do it.
>

--
Csaba Nemethi https://www.nemethi.de mailto:csaba....@t-online.de

[Alexandru]

Thanks Csaba, very good explanation.
Which shows how complicated things are with lsort.
And I think this is only scratching on the surface of traps.
I almost agree with your sentence for the manual. I would reformulate like this:

"-nocase
Causes comparisons to be handled in a case-insensitive manner.

Has no effect if combined with -dictionary and -sorted (these two together), -integer or -real."

[nemethi]

Am 19.01.21 um 18:12 schrieb Alexandru:

OK, this formulation is clearer than mine, thanks!