Tcl 8.6 String Length/Trim On Empty String Alters Behavior

[jarre...@gmail.com]

Hi all,

I am working on some Tcl 8.4->8.6 porting activities at Mentor (Siemens)
and am seeing a strange change in behavior for the command
{string match $object ""} after performing the command
{string length [string trim ""]}. When the former is run first, the $object is
retained and not altered. However, when running the latter command first,
$object is shimmered and no longer the type that it held prior. This same
behavior can be caused by comparing $object to a unicode string as well.

Trey Jackson talked to Donal and Don Porter at the Tcl/Tk Conference; Don
thought this may be a bug in Tcl 8.6 and suggested running the script on a
build of Tcl with the COMPAT macro defined within generic/tclStringObj.C
set to 1. This had no change on the behavior though.

Below is the C++ code and a script that showcases this behavior with a simple
custom object. However, the commands must be ran in interactive mode; when it
is ran as a script, the error does not occur.

C++ code:

#include <tcl.h>
#include <iostream>
#include <string>
#include <string.h>
#include <stdlib.h>


using std::cout;
using std::endl;

struct Foo {
int i;
};

void Free_Tcl_Foo_InternalRep(Tcl_Obj *g )
{
cout << "Free was called" << endl;
Foo* f = reinterpret_cast<Foo *>(g->internalRep.otherValuePtr);
delete f;
}

int Set_Tcl_Foo_FromAny(Tcl_Interp * /*interp*/ , Tcl_Obj * /*objPtr*/)
{
// Not used.
abort();
return 1;
}

void Dup_Tcl_Foo_InternalRep(Tcl_Obj *srcPtr, Tcl_Obj *copyPtr)
{

copyPtr->typePtr = srcPtr->typePtr;

// Is this safe?
copyPtr->internalRep.otherValuePtr = srcPtr->internalRep.otherValuePtr;
}

void UpdateStringOf_Tcl_Foo(Tcl_Obj *p)
{
const char* name = "Cell_Iterator";
p->length = strlen(name);
p->bytes = (char*)(ckalloc((unsigned) p->length+1 ));
strcpy(p->bytes, name);
return;
}

Tcl_ObjType Tcl_Foo = {
"Foo_Value",
Free_Tcl_Foo_InternalRep,
Dup_Tcl_Foo_InternalRep,
UpdateStringOf_Tcl_Foo,
NULL
};

Tcl_Obj * Create_Tcl_Foo ()
{
Foo* f = new Foo;
Tcl_Obj *objPtr = Tcl_NewObj();

objPtr->bytes = NULL;

objPtr->internalRep.otherValuePtr = (void*)f;
objPtr->typePtr = &Tcl_Foo;

return objPtr;
}


int do_Foo(
ClientData c_data,
Tcl_Interp *interp,
int argc,
Tcl_Obj * CONST argv[]
)
{
const char* usage = "usage: do_foo cmd [ foo_obj ]";
if ( argc < 2 || argc > 4 ) {
cout << usage << endl;
return TCL_ERROR;
}
Tcl_Obj* the_foo = NULL;
if ( argc == 3 ) {
// second argument should be a foo
the_foo = argv[2];
if ( the_foo->typePtr != &Tcl_Foo ) {
cout << "Not a foo" << endl;
return TCL_ERROR;
}
}
std::string cmd = Tcl_GetStringFromObj( argv[1], NULL );
if ( argc == 2 && cmd == "make" ) {
the_foo = Create_Tcl_Foo();
Tcl_SetObjResult( interp, the_foo );
return TCL_OK;
} else if ( argc == 3 && cmd == "name" ) {
Tcl_Obj* result = Tcl_NewStringObj( "Harry", -1 );
Tcl_SetObjResult( interp, result );
return TCL_OK;
}
cout << "Invalid argument or command for do_foo" << endl;
cout << usage << endl;
return TCL_ERROR;
}

int Foo_Appinit( Tcl_Interp* interp ) {
Tcl_CreateObjCommand(interp, "do_foo", do_Foo, NULL, NULL);
return TCL_OK;
}

int main( int argc, char* argv[] ) {
Tcl_Main( argc, argv, &Foo_Appinit );
}



script:

proc try_it {} {
set blank [encoding convertto euc-jp ""]
set foo [ do_foo make ]
puts "About to do string match with \"\""
# This causes the shimmering to happen on the second call
string match $foo ""
puts [ do_foo name $foo ]
puts "About to do string match with Unicode string"
# This causes the shimmering to happen here.
string match $foo $blank
puts "about to call do_foo on $foo"
if { [ catch { puts [ do_foo name $foo ] } errMsg] } { puts $errMsg }
}
try_it
string length [string trim ""]
try_it


Thanks,
Jarrett

[Don Porter]

On 10/26/2017 04:24 PM, jarre...@gmail.com wrote:
> I am working on some Tcl 8.4->8.6 porting activities at Mentor (Siemens)
> and am seeing a strange change in behavior for the command
> {string match $object ""} after performing the command
> {string length [string trim ""]}.

I will follow up with more detail, but let's leap to the practical
advice.

string match $object ""

That will return true exactly when $object is an empty string. The
clearer way to code the same functionality is either

{$object eq {}}

within an expression or

string equal $object ""

as a command. Both of those alternatives are available in Tcl 8.4,
and do no suffer from the trouble you describe.

Without thinking about it too deeply, that looks like a transformation
that could be automated as a global search and replace.

--
| Don Porter Applied and Computational Mathematics Division |
| donald...@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|

[Don Porter]

On 10/26/2017 04:24 PM, jarre...@gmail.com wrote:

> I am working on some Tcl 8.4->8.6 porting activities at Mentor (Siemens)
> and am seeing a strange change in behavior for the command
> {string match $object ""}

Specifically the trouble is unwanted loss of an intrep in $object
when it is shimmered to become a "string" type.

This happens because [string match] has several type-specific
implementations. When it sees that one of its arguments already
has the "string" type (that is, a Tcl_UniChar array), it converts
the other and uses a matching routine tuned to that representation.

So this reduces the question to

How did the value "" come to have the "string" type?
Why are the answers different in Tcl 8.4 and Tcl 8.6?

...after performing the command

> {string length [string trim ""]}.

(To a good approximation...) In both Tcl 8.4 and Tcl 8.6, the command
[string length $arg] will convert $arg to the "string" Tcl_ObjType.
This is not so much because of the benefits of a Tcl_UniChar array
representation, but because the "string" Tcl_ObjType has a field where
the character length of the value can be stored. The idea is that if
you are taking the time to compute the string length of a value, best
to save that for future re-use.

Arguably, this strategy is not compelling for the empty string.

Moving on....

The question then turns to how the return value from [string trim]
relates to the argument value it takes. In Tcl 8.4, [string trim]
(essentially) creates a new value to return. In Tcl 8.6 though,
when possible and appropriate, [string trim] is bytecode compiled
and using the bytecoded version, if there's nothing to trim,
[string trim $arg] is quite happy to return $arg. That's the source
of the overall change in behavior you see. Note that the bytecode vs
not distinction is reflected in your report of differences between
interactive and scripted execution.

The sharing of literal values is playing a role in the background
as well. That's why changes to the "" value in

string length [string trim ""]

show up in

string match $obj ""

That should explain the "why". I have to pause now, but I'll post
again on the remaining related questions:

How might I prevent that?
How might I undo it?

[Don Porter]

On 10/27/2017 03:33 PM, Don Porter wrote:
> That should explain the "why".  I have to pause now, but I'll post
> again on the remaining related questions:
>
> How might I prevent that?
> How might I undo it?

The options here depend on the constraints.

If you are free to change the code to replace [string match]
with [string equal], that's the best answer.

If the code cannot change, your only option is to change the
functioning of the commands that sum up to the undesired effects.
They all seem to point to making modifications so the command
[::string] is replaced with another that behaves differently.
A more targeted approach might seek to overwrite or selectively
direct away from [::tcl::string::trim].

The tougher call would be considering whether in Tcl 8.7 the
practice of converting empty strings to the "string" intrep
should be kept in place. That causes trouble here, but may
well be useful for other reasons.

[Gerald Lester]

On 10/30/2017 12:16 PM, Don Porter wrote:
> On 10/27/2017 03:33 PM, Don Porter wrote:
>> That should explain the "why".  I have to pause now, but I'll post
>> again on the remaining related questions:
>>
>> How might I prevent that?
>> How might I undo it?
>
> The options here depend on the constraints.
>
> If you are free to change the code to replace [string match]
> with [string equal], that's the best answer.
>
> If the code cannot change, your only option is to change the
> functioning of the commands that sum up to the undesired effects.
> They all seem to point to making modifications so the command
> [::string] is replaced with another that behaves differently.
> A more targeted approach might seek to overwrite or selectively
> direct away from [::tcl::string::trim].
>
> The tougher call would be considering whether in Tcl 8.7 the
> practice of converting empty strings to the "string" intrep
> should be kept in place.  That causes trouble here, but may
> well be useful for other reasons.
>

Don,

I would think that anytime an "object" is passed into a [string] command
that one should expect it to shimmer over to a string. Likewise, if
passed to a "list" command then one should expect it to shimmer over to
a list.

In the cases where it (currently) does not shimmer, to me that is a side
effect that should not be relied on.

Likewise, in the cases where it (currently) does shimmer, to me that is
also a side effect and should not be relied on.

In other words, Tcl is not a strongly typed language and "objects"
should be expected to shimmer at any time --- and the times may change
between one bug fix version and the next, much less between one minor
version and the next.

Please correct me if the above is in anyway a misstatement.

--
+----------------------------------------------------------------------+
| Gerald W. Lester, President, KNG Consulting LLC |
| Email: Gerald...@kng-consulting.net |
+----------------------------------------------------------------------+

[Don Porter]

On 10/30/2017 01:36 PM, Gerald Lester wrote:
> I would think that anytime an "object" is passed into a [string] command
> that one should expect it to shimmer over to a string.  Likewise, if
> passed to a "list" command then one should expect it to shimmer over to
> a list.

Tcl's "Naming of Names" here is conspiring to mislead everyone.

The "string" Tcl_ObjType refers to one of several representations
of strings, the one where an array of Tcl_UniChar is stored and
a count of the number of chars (not bytes) is also remembered.

The judgment about when that's the right representation to keep
on hand has changed over the different releases of Tcl over the
years. It is emphatically *not* the case that every [string]
subcommand does so. Even if it did, this particular change
isn't just a shimmer, but a bringing together of several pieces
of a larger puzzle.

> In the cases where it (currently) does not shimmer, to me that is a side
> effect that should not be relied on.
>
> Likewise, in the cases where it (currently) does shimmer, to me that is
> also a side effect and should not be relied on.

This is correct. This is what we say again and again. People still
care. They care because the abstraction leaks and shimmer vs no
shimmer means real and measurable performance differences, and
those cannot always be ignored.

There's no "bug" in Tcl to be fixed here. There is a change in
behavior that in this use case has been disadvantageous. Offering
ways to navigate around it is one piece of advanced Tcl programming.

Please remember that the best possible fix here is to just stop
coding
[string match ...]
when
[string equal ...]
is what was meant.

I'd bet there's a good chance that the choice to use [string match]
in the first place came out of some earlier round of performance
chasing.

[phil...@gmail.com]

For us, the challenge is that our objects generally contain more state than it is possible to convert into a string so that we can shimmer back. It is more analogous to Tk widgets or TclOO objects etc.

The Tcl-ish way to handle it would be to create objects that have a global name (like .canvas or TclOO::something_or_the_other and then the string gets set to the name of the object - that way it is always a string and it stays a string and the object is an object and stays an object.

The problem for us then is one of cleanup - we don't have a way to destroy things that is nearly as convenient as the reference count feature on Tcl_Obj. Memory wise, we can't handle that.

What I guess I really want is a way to say "tell me when there are no strings that point to my global object name" so that I can clean it up. Does anyone know how to do that?

[Donal K. Fellows]

On 26/10/2017 21:20, jarre...@gmail.com wrote:
> Trey Jackson talked to Donal and Don Porter at the Tcl/Tk Conference;
> Don thought this may be a bug in Tcl 8.6 and suggested running the
> script on a build of Tcl with the COMPAT macro defined within

> generic/tclStringObj.C set to 1. However, this had no change on the
> behavior.
The change may have been in 8.5 (i.e., I don't remember exactly when it
was applied by Miguel Sofer) and was that Tcl internally tries to share
an object for the empty string much more aggressively than before. Given
the sheer quantity of empty string objects being used, that's a win for
most programs. (Code that uses the full Tcl command interface still gets
a plain unshared new empty Tcl_Obj* to work with as the interpreter
result.) A key part of that change is that empty strings have their own
special object type, enabling Tcl to internally skip some processing
steps when they're unneeded; it's the sort of thing that we don't
typically announce and is part of why you're not supposed to look at the
internal representations of objects when they're not yours. :-)

In general, I don't recommend relying on special typing of the empty
string; it could be something other than what you expect. You can use
[::tcl::unsupported::representation] to probe what is going on under the
covers from your script without requiring C code. Generally speaking,
the [encoding convertto] command produces a bytearray, whereas most
[string] subcommands operate on a whole range of different internal
representations, but *not* usually any that you define. If you have a
precious representation, your code will have to take care to not lose it
through operations like matching or comparisons; Tcl as it currently
stands does not take care to preserve such internal representations.

A case could be made for changing that, but that's into the domain of
the Hydra work which was mentioned at the Tcl Conference and which I'm
very much not an expert on.

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.

[Donal K. Fellows]

On 30/10/2017 21:41, phil...@gmail.com wrote:
> What I guess I really want is a way to say "tell me when there are no
> strings that point to my global object name" so that I can clean it
> up. Does anyone know how to do that?

I want that, yet I don't have a reliable way to do it yet. The big issue
is that I don't have a good way to intercept the handling of reference
count management for command names, but if I put that in (not something
that can be done at the moment outside of the guts of Tcl) then it
becomes possible to GC TclOO objects. I'd probably keep it for things
allocated via 'new' and only until they are first [rename]d.

There are some notes on ugly stuff done here in the comments relating to
the tclCmdNameType variable in tclObj.c which note that TclJava and TCom
do tricky stuff here. Including this telling note:

TODO: Provide a better API for those extensions so that they can
coexist...

It's exactly that (non-existent) API that we need here.

[phil...@gmail.com]

It has been suggested to me that the trace command is the closest thing that Tcl has to providing this functionality, though I see the reference count issue you mention - it seems that trace could provide something similar to the C++ std::auto_ptr - i.e. a known point of ownership which destroys the object when it goes out of scope. i.e. others are free to use it, but when its gone, its gone even if you still know where it used to be.

[phil...@gmail.com]

Another Mentor engineer showed me itcl::local - it seems to do roughly what we are looking for here - but I note that local isn't available in TclOO. How is itcl::local working under the covers? And why doesn't TclOO offer that ability - is there a problem with it?

[Donal K. Fellows]

I'd guess it is probably using an unset trace on an otherwise-unused
local variable. That's the simplest reliable way to get a callback on
the termination of a frame (it even works once you put coroutine
contexts in the mix).

Donal (as to why it isn't done, I got sidetracked into this tclquadcode
project that interests some people).

[Christian Gollwitzer]

Am 08.11.17 um 01:20 schrieb phil...@gmail.com:

> Another Mentor engineer showed me itcl::local - it seems to do roughly what we are looking for here - but I note that local isn't available in TclOO. How is itcl::local working under the covers? And why doesn't TclOO offer that ability - is there a problem with it?
>

It does roughly the same as my "autovar" RAII simulator:

============================================

namespace eval SmallUtils {
# shortened from a collection of small things
variable ns [namespace current]
proc autovar {var args} {
variable ns
upvar 1 $var v
set v [uplevel 1 $args]
trace add variable v unset [list ${ns}::autodestroy $v]
}

proc autodestroy {cmd args} {
# puts "RAII destructing $cmd"
rename $cmd ""
}
}


package require Tk
proc RAII_demo {} {
SmallUtils::autovar v label .l -text "This will disappear"
pack $v
update
after 1000
# at this point, v is destroyed and therefore the label deleted
}

RAII_demo
================================================

The demo uses a Tk widget, but you can similarly construct a TclOO
object or anything else that follows the usual OO conventions.

Christian

[Andreas Leitgeb]

I wonder if such an unset-trace would also spoil optimizations...

[Donal K. Fellows]

On 08/11/2017 08:44, Andreas Leitgeb wrote:
> I wonder if such an unset-trace would also spoil optimizations...

It forces the presence of rather more expensive data structures, which
slows things down quite a bit.

Donal.

[phil...@gmail.com]

Interesting - I was able to make the "trace on an otherwise unused local variable" work just fine.

It suggests a (perhaps hack-ish) work around that wouldn't be subject to trace's overhead and wouldn't be subject to shimmering.

If I create an otherwise unused Tcl_Obj based local variable of my own, and no one touches it with anything (string match, or lappend, or etc.), then it shouldn't shimmer into a string and I should be able to use it as a Tcl_Obj based ref-counted thing that tells me when to clean up my global objects, right? If no one knows its there, then they can't do evil things to it to cause it to shimmer.