RFC: Dramatic overhaul of lists, eval, and subst
Dan Smart
and I came to several conclusions, which I have attempted to explain in a
number of postings to that thread. This is an attempt to pull those
disparate threads together, and present a somewhat more cogent and coherent
picture.
My real aim here is to be able to manipulate scripts object safely. Part of
the promise of TCL was always that you could write new control structures,
and extend the language almost with out limit. While that is still true, it
is no longer possible to do so efficiently. If you need substitution to
occur on your constructed script, you have to convert that script to a
string, at which point bang go the object reps of everything in the
command.
In order to generally manipulate scripts in such a fashion, we have to
manipulate them as lists, unfortunately currently not all scripts are legal
lists, and when you have a script that is representable as a list, there is
no way to perform substitution on it (at least not object safely).
It has been suggested that my desire to have scripts be manipulable as
lists is a product of some desire to mutate TCL into LISP. Nothing could be
further from the truth, TCL is different from LISP, and in almost every
case that difference works to TCLs advantage. However, being able to safely
and efficiently manipualate scripts is an area where TCL currently loses
out.
I will present first the issues that I see, and then a possible solution.
(Note when I am refering to existing TCL commands I will bracket them thus:
[list], when I am refering to proposed new commands, I will bracket them
thus: <listx>)
Issues:
=======
1) The usage of "[list]" to mean "quote"
The current list command is a little schizophrenic. Most of the time it is
used to build lists, but sometimes it is used to protect a (possibly
virtual) string from substitution.
(Historical Aside)
I am interested to learn when this usage became idiomatic, were [list]s
always designed this way, our were they extended?
When lists were strings, the functions that created and modified lists had
to quote intra-element white space, "{", "}" and "\", in order to be able
to parse the list. It is not necessary to quote "$", ";", "[" or "]" if all
you are concerned with is parsing the list. The fact that these characters
are quoted implies to me that the usage of list to protect strings from
substitution when passed to [eval] was by design. No matter we live in a
different world now.
(End Aside)
There are very few reasons for wanting to treat a list as a string, and
indeed conventional wisdom is that you shouldn't (it causes shimmering).
There are even fewer reasons for then wanting to convert it back to a list.
So making the behaviour that supports such actions the default makes no
sense to me.
2) Confusion between "parsing" and "substitution"
I appreciate that the implementation may currently perform parsing and
substitution in parallel, but they are logically different phases.
The parser splits things into words, and is concerned only with those
characters that seperate words (whitespace), and those characters that
group characters into a word (ignoring seperators).
The substitution engine is then responsible for replacing some of those
words with other values. There is significant overlap between the sets of
characters that are special between these to phases, but they are different
phases. {} [] "" and \ are all used to group characters in the first phase,
{} [] \ and $ are all special to the second phase.
3) The syntax and semantics of '#'
# is partially parsed, and semi-preprocessed. By this I mean that when
encountered where a command is expected, it causes the rest of the line to
be skipped, and not even parsed. However, except when used at the top
level, '#' is first encountered by the parser as part of a collection of
charcters that have been grouped in some way, under these circumstances,
the parser has no way of knowing that '#' is special, and hence does parse
the characters that follow it.
4) No mechanism for object safe substitution.
[subst] always returns a string. Sometimes, in the event of a string
containing multiple substitutions, or non substituted text, it needs to.
Other times it does not. [eval] can also not be used, as the object safe
form performs no substitution.
The Possible Solution
=====================
1) Provide an object safe [subst].
If the thing to be substituted is replaced in it's entirety by a single
substitution return the Tcl_Obj that replaces it. Further if the thing to
be [subst]ed is a pure list, perform this algorithm on each of the
elements.
2) Add a quote flag to Tcl_Obj.
This should be a tri-state field: "unquoted", "neither", "quoted". Provide
commands <quote> and <unquote> which manipulate this flag. Objects start in
state "neither", <quote> moves the flag 1 step towards "quoted", <unquote>
moves it one step the other way. If an object is in state "quoted" then
substitution will never be performed on its value.
(Fix [subst] above to pay attention to this flag).
3) Fix lists.
As part of migrating the list commands to major/minor syntax, make the new
command <listx>, and make it respect the "quote" flag on the objects it
collects. In other words if the list is converted to a string, only those
elements whose quote flag is set, are quoted, which may therefore produce a
string that either cant be interpreted as a list, or would be interpreted
as a different list. <quote>ing or <unquote>ing a list affects the quote
state of the list as a whole, not the state of the elements. Thus we
provide a -q flag to the command which <quote>s any added/modified
elements.
The syntax of this new command would therefore be:
listx ?-q? command ?args?
where command is one of:
create ?arg arg ...?
append list ?value value ...?
insert list index ?value value ...?
range list first last
length list
replace list first last ?element element ...?
search ?mode? list pattern
sort ?options? list
apply list monadic-function
Then alias:
[list] to <listx -q create>.
[lappend] to <listx -q append>
[linsert] to <listx -q insert>
[lreplace] to <listx -q replace>
The remaing commands are purely interagatory, and can thus be aliased
directly.
Note the tri-state nature of the quote flag allows us to do:
% set arg1 {Hello World}
% set arg2 {This is Cool}
% set arg3 [listx -q create word1 \$word2 word3]
% set foo [listx -q create command $arg1 $arg2 [unquote $arg3]]
% puts $foo
{Hello World} {This is Cool} word1 \$word2 word3
(Hands up who spots {expand} :-)
4) Provide a mechanism to allow commands to quote their arguments.
Specifically a flag that prevents the command string from being
substituted. Thus once the command string has been parsed, the first word
is substituted if necessary, the result is looked up as normal for a
command, and only if the 'quote' flag is clear on the command object are
the remaining arguments substituted, otherwise they are passed unchanged to
the command.
This is a genralisation of something that will need to be done to fix '#'.
Fixing '#' doesn't require this change as it can be special cased, but it
provides some interesting functionality (particularly to those people
interested in writing TCL debuggers in TCL).
5) Fix '#'
This the tricky one. None of the changes proposed above have any affect on
existing scripts (assuming the new commands get suitably hidden). This
change could.
We make '#' go through the parser, (but not the substitution engine) either
by virtue of the flag above, or with a special case. This means that {} now
escape newlines within comments (\ already does), and probably that ;
terminates a comment.
(grr I've never liked ; as a statement seperator)
Note that all but contrived examples of code that would fall foul of this
change can be detected by one of the following grep's:
grep '^[ \t]*#.*[{};]' *.tcl
grep ';[ \t]*#.*[{};]' *.tcl
Note that this will find comments and comment blocks that contain matched
braces {}. However on the code base I have to hand, the number of lines
that this matches is so small as to make manual pruning trivial.
Note further that this change is not essential, it's absence merely
prohibits us from manipulating scripts that contain comments, which would
be a shame but...
6) Fix eval (and friends)
[eval] with pure list argument(s) should call [subst] (as fixed above) on
it's argument(s). As it can now find unquoted ';' and '\n' it needs to be
enhanced to split the list at such points and process the sub lists
seperately.
In Conclusion
=============
If we make the above changes we get the mechanics of a safe {expand} and
the toolset to wrap it up in whatever syntax tcls our fancy. This toolset
also allows us to return to the days when we could expand TCL in any way we
chose, while maintaining the efficiency that the move to objects has bought
us.
Dan "Entirely Serious for once" Smart
--
Dan Smart. C++ Programming and Mentoring.
cpp...@dansmart.com
Alexander Nosenko
> I will present first the issues that I see, and then a possible solution.
(I've posted some reply into the {}$ thread before reading your article,
sorry)
As I can see it, your proposal means that parser behaviour and command
results will depend on state "tags" carried by Tcl_objects (commands and
variables). Old vapours of GPSS are floating around ;-)
For the programmer, the most immediate result is that script execution will
depend on the variables passed to it in the most untrivial way. One can
invent a lot of clever tricks with state-tags, but how they will influence
the robustness of the program? Will we need gobs of state-tags-checking
list-structucture-verifying code in _every_ one-liner? Will it be possible
to define script behaviour precisely for _any_ value/tags of _any_ argument?
Now we have those troubles mostly with run-time-generated scripts, and there
are not so many of them (thank God). With objects-state tags we'll get a lot
of new possibilities.
Are neural nets more simple than state machines?
Regards,
Alexander Nosenko <n...@titul.ru>
Tom Krehbiel
Thank you for the summary, I have noticed with interest the {}$ thread but
I do not have enough time in the day to keep up with all the postings. This
also means my understanding of core tcl is rather shallow. With this said
I believe I understand the issues you present and agree with your solutions.
I have been increasingly frustrated with the "quoting" issues presented
by tcl. I enjoy using tcl but invariably shortly after I recommend that
some should use tcl, I start getting call concerning its lack of
consistency regarding quote handling.
To the TCT,
During the discussions concerning OO inclusion in the core I said I would
support the inclusion of Itcl if it was allowed to evolve. Dan comment are
are a good example of what I meant by evolution. Tcl is a good language but
it still need some work to become great. A major stumbling block to tcl's
greatness is its lack of numerical calculation commands (like those availibe
in NumPy). I understand that the original intent for tcl was that it should
be a "glue" language, but the success of tcl and other scripting languages
has demonstrated that scripting has evolved beyond this simple axiom.
I think that the tcl community needs to be careful to allow the Tcl core
to grow so that it is not limited to (or burdened by) its "glue" language
heritage.
Thanks again Dan,
Tom K.
Jeffrey Hobbs
...
> During the discussions concerning OO inclusion in the core I said I would
> support the inclusion of Itcl if it was allowed to evolve. Dan comment are
> are a good example of what I meant by evolution. Tcl is a good language but
> it still need some work to become great. A major stumbling block to tcl's
> greatness is its lack of numerical calculation commands (like those availibe
> in NumPy). I understand that the original intent for tcl was that it should
> be a "glue" language, but the success of tcl and other scripting languages
> has demonstrated that scripting has evolved beyond this simple axiom.
> I think that the tcl community needs to be careful to allow the Tcl core
> to grow so that it is not limited to (or burdened by) its "glue" language
> heritage.
Evolution is good, but everyone will argue about the pace. Some will
in fact want devolution. You have to be very careful about "growth",
because not all growth is positive.
Take a look at Perl. Perl6 is a "from-scratch" rewrite, because it
had "grown" to the point of unmanagability. Noone would say we are
near that point now, but we also need to make sure that we move
carefully, so as not to reach that situation.
Tcl has been incredibly successful as a glue language. I think one
of the most important things is to exercise that in a standard
distribution that ties together many useful extensions (like math
packages) to reach a broader audience. At the same time, we can
look at the internals and see what sort of evolution makes sense
for the masses there.
--
Jeffrey Hobbs The Tcl Guy
hobbs at ajubasolutions.com Ajuba Solutions (née Scriptics)
Dan Smart
<97012140...@indigo.maxnet.ru>:
>Dan Smart <cpp...@dansmart.com> wrote:
>
>> I will present first the issues that I see, and then a possible
>> solution.
>
>(I've posted some reply into the {}$ thread before reading your article,
>sorry)
>
>As I can see it, your proposal means that parser behaviour and command
>results will depend on state "tags" carried by Tcl_objects (commands and
>variables). Old vapours of GPSS are floating around ;-)
I'm not sure what GPSS is, but I don't think you understand. First, lets
not refer to the parser, the parser is the thing that parses a tcl script
into lists (not currently [list]s) of commands that are lists (not
currently [list]s) of words. The substitution engine is what I am concerned
with, and this is only of interest when we take a second pass through the
substitution engine. Currently the only way to take a second pass through
the substitution engine is if you are dealing with a string, as [list]s are
defined as quoting there elements to protect them from substitution, and
hence eval on a list shortcuts the substitution engine. This prevents us
from taking an object-safe second pass through the substitution engine. We
can't even use [subst] today as that is not object safe. Thus extending TCL
in TCL in arbritrary ways becomes difficult if not impossible. Writing
[invoke] with nice syntax in TCL is effectively impossible.
The goal of this proposal is to remedy that situation, and in doing so
rationalise the mechaisms for quoting things with in TCL.
>For the programmer, the most immediate result is that script execution
>will depend on the variables passed to it in the most untrivial way. One
>can invent a lot of clever tricks with state-tags, but how they will
>influence the robustness of the program? Will we need gobs of
>state-tags-checking list-structucture-verifying code in _every_
>one-liner? Will it be possible to define script behaviour precisely for
>_any_ value/tags of _any_ argument? Now we have those troubles mostly
>with run-time-generated scripts, and there are not so many of them
>(thank God). With objects-state tags we'll get a lot of new
>possibilities.
Only if a list of such objects is passed to eval, or a quoted string rep of
such an object is required. This in fact simplifies code, as you no longer
need to rememeber tricks involving [lrange] and [list] to protect strings
containg special characters, you just <quote> them.
>
>Are neural nets more simple than state machines?
>
That depends on the complexity of each, and the problem domain.
>Regards,
>Alexander Nosenko <n...@titul.ru>
>
Dan
Dan Smart
<39D3A8C1...@ajubasolutions.com>:
>Evolution is good, but everyone will argue about the pace. Some will
>in fact want devolution. You have to be very careful about "growth",
>because not all growth is positive.
>
Indeed, you must also be careful of second system effect (or perhaps in
this case 9th system effect :-). Evolution (regardless of Gould's theories)
has to be a slow process, as you need to make sure that you change in tune
with your environment (in this case existing code base), failure to do so
leads inevitably to extinction.
Hence my proposal is considerably less drastic than some of my original
thoughts.
>Take a look at Perl. Perl6 is a "from-scratch" rewrite, because it
>had "grown" to the point of unmanagability. Noone would say we are
>near that point now, but we also need to make sure that we move
>carefully, so as not to reach that situation.
>
To an extent we *have* rewritten TCL from scratch, just incrementally. I'm
prepared to bet that there is very little of the 7 series code base that
has survived into the latest revisions of the 8 series (the story with TK
is probably different).
>Tcl has been incredibly successful as a glue language. I think one
>of the most important things is to exercise that in a standard
>distribution that ties together many useful extensions (like math
>packages) to reach a broader audience. At the same time, we can
>look at the internals and see what sort of evolution makes sense
>for the masses there.
>
TCL is my favourite glue language. I have deployed it in numerous places,
and it is a minor but significant part of the project I am currently
working on, it is likely to play an increasingly important role going
forwards.
I have occasion to play with a significant number of languages, many of
them scripting languages. While some of the scripting languages are better
than TCL at some things, as a general purpose stick things together (and
optionally slap a GUI on the front) language, TCL cannot be beat, at least
not today.
Dan "Think of it as evolution in action" Smart.
lvi...@cas.org
According to Dan Smart <cpp...@dansmart.com>:
:While some of the scripting languages are better
:optionally slap a GUI on the front) language, TCL cannot be beat, at least
:not today.
Despite being quite a Tcl advocate, I know I would not go this far.
In my personal experience, Tcl still doesn't approach the usefulness of
the Bourne/Korn shell. And in fact, I still find it quicker and in most
cases easier (ie less keystrokes) to write awk scripts than to write Tcl
scripts. So each experience is different.
--
<URL: https://secure.paypal.com/refer/pal=lvirden%40yahoo.com>
<URL: mailto:lvi...@cas.org> <URL: http://www.purl.org/NET/lvirden/>
Even if explicitly stated to the contrary, nothing in this posting
should be construed as representing my employer's opinions.
I wear the China Blue Cheese, it does not wear me.
/ In my personal experience, Tcl still doesn't approach the usefulness of
/ the Bourne/Korn shell. And in fact, I still find it quicker and in most
/ cases easier (ie less keystrokes) to write awk scripts than to write Tcl
/ scripts. So each experience is different.
Being able to type rather quickly and not wanting to expend extra neurons
on memorising each of the 5000 forms of ${...} in ksh, I prefer drop into
tclsh for some of the trickier file manipulations. For a practiced typist,
fewer keystrokes can be slower to type if they're out among the less used
keys instead of letters and the common punctuation. In the same way, I can
usually type Pascal programs faster than C.
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
Sign up for WASHINGTON MUTUAL BANK's special
We Rob You While You Sleep Service TODAY!
=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=*=
CACS: Collective Against Consensual Sanity v0.123
pretty pretty <blink> http://www.tsoft.com/~wyrmwif/
All new and improved web pages! Bookmark yours today!
:)-free zone. </blink> Elect LUM World Dictator!
Dan Smart
>According to Dan Smart <cpp...@dansmart.com>:
>:While some of the scripting languages are better
>:than TCL at some things, as a general purpose stick things together
>:(and optionally slap a GUI on the front) language, TCL cannot be beat,
>:at least not today.
>
>Despite being quite a Tcl advocate, I know I would not go this far.
>In my personal experience, Tcl still doesn't approach the usefulness of
>the Bourne/Korn shell. And in fact, I still find it quicker and in most
>cases easier (ie less keystrokes) to write awk scripts than to write Tcl
>scripts. So each experience is different.
>
I almost invariably use tcl in preference to AWK. I use tcsh as my primary
shell (but mostly because it supports emacs key bindings, and smart
completion), but my environment is entirely generated by TCL scripts.
When I've had a chance to do some tkCon hacking I've been meaning to get
around to, I'll may ditch tcsh completely.
Try it, stoke up tkCon, and use it as a shell for a while, You may find you
like it.
(
As an aside, the distributed system I work on is built by a build
management system I wrote in TCL, started, configured, and monitored by a
system I wrote in tcl, my unix development enviroment is managed by tcl,
and shortly management of vendor software installs will succumb to the
gentle persuasion of a set of tcl scripts.
I've realised that using other tools may sometimes save time once off, but
as soon as I use it more than once, I regret having written a shell script
or what ever. And of course sticking a gui on a shell script aint going to
happen.
)
Dan
Donal K. Fellows
<cpp...@dansmart.com> writes
>As an aside, the distributed system I work on is built by a build
>management system I wrote in TCL, started, configured, and monitored by a
>system I wrote in tcl, my unix development enviroment is managed by tcl,
>and shortly management of vendor software installs will succumb to the
>gentle persuasion of a set of tcl scripts.
My desktop environment is managed by loads of scripts in assorted
languages, with the obligatory large quantities of junk, sealing wax,
little bits of string, and dented tin cans. Or am I getting mixed up
with my office there? :^)
>I've realised that using other tools may sometimes save time once off, but
>as soon as I use it more than once, I regret having written a shell script
>or what ever.
Tcl is very good when it comes to putting together a solution to a
problem that needs to work reliably in a fairly wide range of
environments, since you need very few external programs. And when you
have extensions like Tk about, the incentive to move to using other
languages is rather small.
>And of course sticking a gui on a shell script aint going to
>happen.
That last is not strictly true due to the existence of things like
dtksh. Not that I'm advocating you use that instead of Wish/Expect...
Donal (currently working on interesting ways to convert Wiki-like plain
text into HTML, XML, LaTeX, etc.)
--
Donal K. Fellows (at home)
--
FOOLED you! Absorb EGO SHATTERING impulse rays, polyester poltroon!!
(WARNING: There is precisely one error in this message.)
lvi...@cas.org
:like it.
I have tried it, I didn't find that I liked it, and so I don't do it.
But that's fine - I'm not advocating everyone do things the way I prefer.
I am, however, trying to provide counter-point personal experience to
let those in my same situation realize that not everyone is running off
to python/perl/tcl/rebol/rexx/...
lvi...@cas.org
:happen.
It's actually happened several times - dtksh is one of these. In fact,
the son of the creator of the Korn shell did a paper a few years ago
about a merged ksh/tcl/tk shell ...
lvi...@cas.org
According to Tom Krehbiel <tom.kr...@motorola.com>:
:I think that the tcl community needs to be careful to allow the Tcl core
:to grow so that it is not limited to (or burdened by) its "glue" language
:heritage.
I disagree that the _core_ of Tcl needs stronger numerical calculation
support. I would _agree_ however that stronger numerical calculation
support is needed.
It is my personal opinion that the Tcl core needs to be strong enough
to enable people to write 80% or more of any particular application -
and stronger so that extensions can be written to fit the other 20% or so.
Adding large amounts of code to the core for specialized applications -
whether it is numerical analysis, SAP, security, WWW, etc. only adds the
possibility of additional fragility and dependencies that could result
in a collapse of the core. I particularly would like to urge spinning off
ALL platform specific code OUT OF THE CORE and into specific extensions - even
if those extensions happen to be packaged in the same .zip, .sit, or .tar.gz
files.
By separating as much as possible, portability of code increases,
clearer realization of what parts of scripts are and are not portable,
etc.
Kevin Kenny
lvi...@cas.org wrote:
> I disagree that the _core_ of Tcl needs stronger numerical calculation
> support. I would _agree_ however that stronger numerical calculation
> support is needed.
>
> It is my personal opinion that the Tcl core needs to be strong enough
> to enable people to write 80% or more of any particular application -
> and stronger so that extensions can be written to fit the other 20% or so.
>
> Adding large amounts of code to the core for specialized applications -
> whether it is numerical analysis, SAP, security, WWW, etc. only adds the
> possibility of additional fragility and dependencies that could result
> in a collapse of the core.
Hear, hear!
Let me add another voice calling for stronger numerical support
*without* compromising the integrity of the core. What we need, though
is a roadmap for getting there from here.
Let's examine one straw-man:
* Virtually all the numeric codes already in existence require
single- and multi-dimensional arrays of integers and of single-
and double-precision floating point numbers. For the success of
numerical calculations in the Tcl core, such arrays must have
natural representations as Tcl objects. George Howlett has
done some work on Tcl representations of vectors and matrices.
* One problem with interconverting vectors with Tcl lists and
strings is type shimmering -- the fact that the Tcl library can
represent two types for an object but not three leads to
difficult cases where types are converted inside inner loops.
For this reason, the Tcl numeric community should consider
Feather a high priority for integration in the core.
* Once representations of vectors and matrices (in column-major
order, please!) are available, efforts can begin on adapting
available numeric codes to Tcl language bindings. Much of
this work initially could be semi-automated conversion of
Fortran calls to Tcl commands. This could be handled by a
package like SWIG, extended to support passing array
dimensions as explicit parameters. (A few years ago, I
made a start at such a thing, directed at wrapping CGI
scripts around available Fortran codes. It's still out
on the Web at http://camnet.ge.com/ftncgi/ and I'm
durned if I know whether it still works.)
* There are a number of reasonably high-quality public codes
available as candidates for inclusion. Comprehensive
libraries such as SLATEC might be one starting point. It
would be interesting, too, to adapt some of the more
complex libraries to use Tcl conventions. An adapter so
that an optimization package like MINPACK could do its
function (and Jacobian?) evaluations with Tcl callbacks
would be an incredibly powerful framework for setting up all
kinds of solutions.
* Tacking something like VTK (http://www.kitware.com/) onto
the back end would then add a terrific visualization
envitonment to the mix. Alas, VTK would have to be made TEA-
compatible; I've been out of touch with the VTK guys, so
I don't know if there's any intention to do so; last I
heard, they were still satisfied with Tcl/Tk 8.0.2 and saw
no reason to move forward.
The only item that requires patching the core is the Feather
support, to make it easier for extensions to add new types of
first-class objects and avoid shimmering. Once that's done,
and we put together Tcl bindings for the BLAS, then we can turn
hordes of developers loose with SWIG or something similar to
bring in the libraries.
Of course, this is only a straw-man proposal; I eagerly solicit
comments!
--
73 de ke9tv/2, Kevin KENNY GE Corporate R&D, Niskayuna, New York, USA
John Ousterhout
the following questions/comments were already answered there...
Dan Smart wrote:
>
> The debate over {expand}/[invoke] made me think in depth about such things,
> and I came to several conclusions, which I have attempted to explain in a
> number of postings to that thread. This is an attempt to pull those
> disparate threads together, and present a somewhat more cogent and coherent
> picture....
The description in your message is at a pretty abstract level; it
isn't obvious to me why each of the issues you mentioned is in
fact a problem. Could you supply a simple (very concrete) example
that motivates each of your concerns, something like "I wanted to
write a script that does XYZ, but the closest I could get was ABC
because of QRS"? Said another way, are the issues you've raised
theoretical problems or practical problems?
Also, without specific examples it was hard for me to understand how
your proposed changes would work or whether they would have unwanted
ramifications.
> 1) The usage of "[list]" to mean "quote"
> The current list command is a little schizophrenic. Most of the time it is
> used to build lists, but sometimes it is used to protect a (possibly
> virtual) string from substitution.
>
> (Historical Aside)
> I am interested to learn when this usage became idiomatic, were [list]s
> always designed this way, our were they extended?
> When lists were strings, the functions that created and modified lists had
> to quote intra-element white space, "{", "}" and "\", in order to be able
> to parse the list. It is not necessary to quote "$", ";", "[" or "]" if all
> you are concerned with is parsing the list. The fact that these characters
> are quoted implies to me that the usage of list to protect strings from
> substitution when passed to [eval] was by design. No matter we live in a
> different world now.
> (End Aside)
It's been a long time since lists evolved into their current form
and my memory is sketchy. As I recall, when I first designed lists I
didn't notice the similarities between lists and commands and thus
didn't include all of the quoting features needed to use lists for
manipulating commands. Once I realized this, I added the automatic
quoting for "$", ";", etc. so that lists could be used to manipulate
commands.
> 2) Confusion between "parsing" and "substitution"
> I appreciate that the implementation may currently perform parsing and
> substitution in parallel, but they are logically different phases.
>
> The parser splits things into words, and is concerned only with those
> characters that seperate words (whitespace), and those characters that
> group characters into a word (ignoring seperators).
>
> The substitution engine is then responsible for replacing some of those
> words with other values. There is significant overlap between the sets of
> characters that are special between these to phases, but they are different
> phases. {} [] "" and \ are all used to group characters in the first phase,
> {} [] \ and $ are all special to the second phase.
I have occasionally tried to make this distinction, but in fact the
two phases are so closely tied together that it is very difficult to
separate them logically. By the time you have described one phase you
have mostly described the other phase as well.
Donal K. Fellows
Kevin Kenny <ken...@crd.ge.com> wrote:
> * One problem with interconverting vectors with Tcl lists and
> strings is type shimmering -- the fact that the Tcl library can
> represent two types for an object but not three leads to
> difficult cases where types are converted inside inner loops.
> For this reason, the Tcl numeric community should consider
> Feather a high priority for integration in the core.
Tcl doesn't have support for more than one (non-UTF8) type at a time
because of a trade-off between speed of execution and memory usage.
While it would be possible to extend Tcl_Objs with a second
internalRep-like field, this would expand a significant proportion of
the structures present in *every Tcl application* by 50% (out from 24
bytes to 36 bytes on a 32-bit architecture.) The code to manage these
values would also become significantly more complex; maybe the result
would be faster, but the number of bugs present would undoubtedly
increase by a lot too. And finally, the gain from doing this wouldn't
be too spectacular in the case where you have updates from the PoV of
a non-string rep, since then you may well have additional non-string
reps to discard in addition to the string rep.
All in all, the current implementation is good enough for 99% of its
uses, and the remaining 1% would add a lot more work (as well as being
incompatible with lots of existing code in extensions.)
FWIW, I just reckon there ought to be a nice easy way to link Tcl to
things like C and FORTRAN for this sort of thing (SWIG and/or Ffidl.)
Why try to be all things to all people when we can enlist others to
help out instead?
Donal.
--
"[He] would have needed to sell not only his own soul, but have somehow gotten
in on the ground floor of an Amway-like pryamid scheme delivering the souls
of kindergarten students to Satan by the truckload like so many boxes of Girl
Scout Cookies." -- John S. Novak, III <j...@concentric.net>
Tom Krehbiel
1) I agree with people that would like to see "tcl" (i.e. the core) stay
small and that other "stuff" should be put in packages. The problem with
this is that to a large extent the other "stuff" is what make tcl useful. If
the other"stuff" was distributed with "tcl" and the whole mess was called
"tcl" then I would not have a problem with leaving as much "stuff" as
possible out of the core to keep it clean. However, without a distribution
policy that includes many of the major packages, it is a nightmare trying
to distribute and application. If I use any of the major packages I am as
likely as not going to run into an installation problem because the sysadmin
controlling the target location will have a different combination of version
and packages. A solution here seems to be a "sumo" style distribution
that is approved by the TCT.
2) A second and even more importent consequence of not having a clear
vision of what "tcl" really means (i.e. core plus what??) is that the
development community doesn't know how to proceed. The development
of many resonable extentions requires the existence of other extentions
(i.e. packages are coupled) without guidence about what direction the
language is going the result at best is confusion and wasted time. This
has been particularly apparent with regard to the state of OO in tcl.
3) I think that tcl is indesperate need of a good numerical package but
a developer would be a fool to create one unless the OO issue is resolved.
A package like NumPy is by its very nature OO because the entities
that are being manipulated are not simple data types.
If you interperate the above to mean that I don't like tcl, you would be
wrong, I just want to make it better.
--Tom K.
Kevin Kenny
> Tcl doesn't have support for more than one (non-UTF8) type at a time
> because of a trade-off between speed of execution and memory usage.
> While it would be possible to extend Tcl_Objs with a second
> internalRep-like field, this would expand a significant proportion of
> the structures present in *every Tcl application* by 50% (out from 24
> bytes to 36 bytes on a 32-bit architecture.) The code to manage these
> values would also become significantly more complex; maybe the result
> would be faster, but the number of bugs present would undoubtedly
> increase by a lot too. And finally, the gain from doing this wouldn't
> be too spectacular in the case where you have updates from the PoV of
> a non-string rep, since then you may well have additional non-string
> reps to discard in addition to the string rep.
OK, I gave a *bit* of a wrong impression here. Feather doesn't
multiply internal representations; instead, it lets an internal
rep have multiple *interfaces* associated with it. With Feather,
the Tcl_* functions that operate on list objects get delegated
to a 'container' interface, which any Tcl_Obj type is free to
implement. If an object's internal rep doesn't support the
'container' interface, then a call to Tcl_ListObj* will result
in coercing the object to a 'list', which is the default
container. In this way, an object that represents a structure,
a vector, or another data type can function as a list and
not shimmer.
Paul's paper from the Tcl2K conference in Austin:
http://www.usenix.org/publications/library/proceedings/tcl2k/duffin.html
discusses all this.
Donal K. Fellows
<tom.kr...@motorola.com> writes
>2) A second and even more importent consequence of not having a clear
>vision of what "tcl" really means (i.e. core plus what??) is that the
>development community doesn't know how to proceed. The development
>of many resonable extentions requires the existence of other extentions
>(i.e. packages are coupled) without guidence about what direction the
>language is going the result at best is confusion and wasted time. This
>has been particularly apparent with regard to the state of OO in tcl.
There are some extremely different ideas about what OO in Tcl should
look like. About the only thing you'll get reasonably rapid agreement
on is that methods should be called something like this:
$object $methodname arg1 arg2 ...
After that, you're getting onto contentious ground... :^(
>3) I think that tcl is indesperate need of a good numerical package but
>a developer would be a fool to create one unless the OO issue is resolved.
>A package like NumPy is by its very nature OO because the entities
>that are being manipulated are not simple data types.
There's so many different groups wanting extended numerical/mathematical
packages, and they all want something different. HPC people want native
complex numbers, native matrices and tensors, and interoperability with
FORTRAN. Financial people want exact control over precision and
rounding, but aren't too bothered about trigonometric functions[*].
Security hackers want fixed-field arithmetic. And so on.
I don't think that supporting all of these in the core (or even at the
SUMO level) is very practical. This sort of code is often hard to get
right and there aren't that many people working on the core. It is much
better to provide a generic interface and let people put their own
functionality in there behind it. But that is *exactly* what Tcl is all
about; not providing advanced mathematical and numerical handling is
*precisely* within the spirit of the language. Not that we can't
discuss what the common interface would look like. :^)
For the record, I have no need for this sort of stuff in my own code.
Networking and Tk are much more useful to me, and regexps are a good
thing too.
>If you interperate the above to mean that I don't like tcl, you would be
>wrong, I just want to make it better.
I don't interpret it that way at all. I also want to make Tcl better,
but I don't want to lose what makes Tcl good now either. It's all a
matter of balancing Gain against the Gain/Pain ratio...
Donal.
[* Banking is complex enough without square root of minus one! ]
Andreas Leitgeb
> ...
sorry for my long pause, last month I was too busy to care about it.
meanwhile I have read the background-story about this proposal (Dan sent
it to me, as I haven't found it on my newsserver) and I have a few comments
to say:
the issues are reasonable.
- for preserving lists, it would be enough to quote blanks and
braces. when converting lists to strings, more than that is done.
It seems to have been list's second job to prevent substitution
in eval from its beginning, and this was/is reasonable for most
uses.
- it is not possible to build up scripts, that contain both object-
references and stuff-to-be-subst'ed-later.
- subst with an argument that gets expanded as a whole, should
return the object, not a string.
Anyway, using subst to "prepare" scripts is inherently dangerous,
as [subst {$a {$b}}] will substitute both $a and $b. there should
also be an option to subst that respects braces (and backslashes)
for deferred (that is: non) substitution.
the depicted solution is (imho) not ok, because:
- scripts cannot be object-save, because scripts are things that can
be saved to a file and loaded back at a later time, and such a thing
cannot contain object references.
Instead it would be like a normal list, each element of which would
have an extra tag stating that element's quoting-style. so far so
good, but such a "script" couldn't then be passed to something
that expects a script (such as a callback or a proc/if/uplevel/...-body)
Otoh, [unquote $list] would imply, that the flag is attached to
the object itself, not to the listelement that holds this object
(delicate difference), and thus such a tagged object could be
passed to any procedure causing all kinds of strange behaviour ...
- Where you imply to subsume the {expand}-syntax, you're wrong:
% set a {x $y [z]}; some_proc {expand}$a
here, the args to some_proc are supposed to be {x} {$y} {[z]}, not
"x", value of a, result of z.
From Dan's proposal I've got a more general idea about {tag}-syntax, but
that needs some input, before it is mature for a proposal:
Both Dan's proposal and the {expand}- proposal actually try to
get around one common obstacle, which is to allow applying tags to
some of the arguments, but without having to explicitly "untag" the
other arguments and without imposing limits on the arguments itself.
in his solution, Dan suggests to use [unquote $arg], but that fails,
because it would limit the untagged valid objects to those that
had not before that one flag set.
in a line like [listx -q create $arg1 [unquote $arg2]],
the $arg1 may have been [unquote]'d in some other context, and
lead to a behaviour not expected in this context.
A solution to this would make Dan's proposal easier to implement,
and it would also make {expand}-syntax possible, though only with
a special command, that handles the {expand}-tag (which I originally
wanted to avoid, but that's a different story)
Dan Smart
comprehensive answer as you were hoping for, but I am snowed under with
work. The reason that I have responded at all, is I want to keep this idea
alive, I think it could be a really big win.
ous...@scriptics.com (John Ousterhout) wrote in
<39D8DCC8...@scriptics.com>:
>I missed the earlier debate over {expand}/[invoke] so apologies if
>the following questions/comments were already answered there...
>
>Dan Smart wrote:
>>
>> The debate over {expand}/[invoke] made me think in depth about such
>> things, and I came to several conclusions, which I have attempted to
>> explain in a number of postings to that thread. This is an attempt to
>> pull those disparate threads together, and present a somewhat more
>> cogent and coherent picture....
>
>The description in your message is at a pretty abstract level; it
>isn't obvious to me why each of the issues you mentioned is in
>fact a problem. Could you supply a simple (very concrete) example
>that motivates each of your concerns, something like "I wanted to
>write a script that does XYZ, but the closest I could get was ABC
>because of QRS"? Said another way, are the issues you've raised
>theoretical problems or practical problems?
>
I want to be able to extend TCL in TCL efficiently. Prior to the
objectification of TCL, commands added to TCL written in TCL were
effectively no less efficient than manualy inlining the relevant code, this
is no longer the case. All the mechanisms for manipulating scripts, and
executing them either (a) convert things to strings thus destroying
objects, or (b) prevent substitution.
For instance, I have a comms library that builds self describing messages.
It exposes methods that allow creation of TCL objects of specified types.
It does this so that I can build a list, and from that list build a
message. I have a library layered on top of this that builds commands that
will be executed when an event occurs. Parts of these commands need to be
substituted when the event occurs, this destroys my carefully crafted typed
objects.
Thus the two main desires:
(1)
Making [subst] object safe. This is just a win. It would even bve
acceptable if I had to pass a flag to force object safe substitution.
Having an object safe subst that worked on lists would also help.
(2)
Seperating <quote> from [list]. This permits me to quote sections of lists,
and pass the whole to eval. Thus allowing object safe evaluation with
substitution.
The second desire (which was split between my fixes 2 and 3) is fairly
complex, as it requires a way of marking objects as being non-
substitutable, and manipulating that state in intuitive ways, we need not
only to flag something as 'quoted' but also as explicitly 'unquoted', hence
the tri-state nature of the flag.
We need the ability to 'unquote' as otherwise simple operations have to be
done in multiple steps:
% set l [listx create -q exec rm [unquote \$file]]
becomes
% set l [listx create -q exec rm]
% listx append l \$file
My proposed fix (4) is far more specialised, and is required for only two
things: a clean fix for '#' (which is not essential), and commands that can
see how they were invoked - very useful for writing debuggers, and other
programs that manipulate running scripts.
>Also, without specific examples it was hard for me to understand how
>your proposed changes would work or whether they would have unwanted
>ramifications.
>
I'm not sure what you need in the way of examples, but I'll see what I can
come up with.
[snip]
>It's been a long time since lists evolved into their current form
>and my memory is sketchy. As I recall, when I first designed lists I
>didn't notice the similarities between lists and commands and thus
>didn't include all of the quoting features needed to use lists for
>manipulating commands. Once I realized this, I added the automatic
>quoting for "$", ";", etc. so that lists could be used to manipulate
>commands.
>
Of course these quoting features prohibit the use of lists for some types
of commands, and hence it would be nice if they were seperate.
>> 2) Confusion between "parsing" and "substitution"
>> I appreciate that the implementation may currently perform parsing and
>> substitution in parallel, but they are logically different phases.
>>
>> The parser splits things into words, and is concerned only with those
>> characters that seperate words (whitespace), and those characters that
>> group characters into a word (ignoring seperators).
>>
>> The substitution engine is then responsible for replacing some of
>> those words with other values. There is significant overlap between
>> the sets of characters that are special between these to phases, but
>> they are different phases. {} [] "" and \ are all used to group
>> characters in the first phase, {} [] \ and $ are all special to the
>> second phase.
>
>I have occasionally tried to make this distinction, but in fact the
>two phases are so closely tied together that it is very difficult to
>separate them logically. By the time you have described one phase you
>have mostly described the other phase as well.
>
Agreed, and that partly comes out in the description above. But note that
the distinction is valid, lists are parsed but not substituted commands.
Currently there is no way to perform substitution on them, this is what I
wish to change.
Dan Smart
John Ousterhout
>
> ....
> >isn't obvious to me why each of the issues you mentioned is in
> >fact a problem. Could you supply a simple (very concrete) example
> >that motivates each of your concerns, something like "I wanted to
> >write a script that does XYZ, but the closest I could get was ABC
> >because of QRS"? Said another way, are the issues you've raised
> >theoretical problems or practical problems?
>
>
> Thus the two main desires:
> (1)
> Making [subst] object safe. This is just a win. It would even bve
> acceptable if I had to pass a flag to force object safe substitution.
> Having an object safe subst that worked on lists would also help.
> (2)
> Seperating <quote> from [list]. This permits me to quote sections of lists,
> and pass the whole to eval. Thus allowing object safe evaluation with
> substitution.
Sorry for being dense, but I still can't figure out what you mean from
the wording above. Could you give a code example that illustrates the
effect you want to achieve with each of these? (You gave an example
for the second one, but you didn't explain the effect you were trying
to achieve so I didn't "get it".)
In any case, my recollection is that you were proposing to make changes
to Tcl that would create substantial compatibility problems. This would
be very difficult for us to do right now, so I would urge you to look
for ways of getting the effect you want by adding new commands rather
than changing existing features.
________________________________________________________________________
John Ousterhout 650-210-0102 tel
Chairman and Chief Technology Officer 650-230-4070 fax
Ajuba Solutions ous...@ajubasolutions.com
http://www.ajubasolutions.com
Christopher Oliver
Here is an example that I believe illustrates the problem, using JavaBlend and
IncrTcl (with protclsh83):
% package require Itcl
3.2
% package require java
1.2.6
% namespace import itcl::*
% class Foo {
> public variable a
> public variable b
> constructor args {
> eval configure $args
> }
> }
% Foo foo -a [java::new String foobar] -b [java::new java.util.Date]
foo
% set b [foo cget -b]
java0x2
% $b getYear
invalid command name "java0x2"
This example fails because "eval" doesn't preserve the objects passed to it and
the "java" command object "java0x2" gets garbage collected because there are no
longer references to it (well, actually it would fail anyway because IncrTcl
doesn't preserve Tcl objects passed to it either; But IncrTcl never sees the
original "java" objects as the arguments to eval were converted to a string
before IncrTcl was ever called).
I believe Dan is trying to invent a way to efficiently pass Tcl_Obj's intact
through eval while preserving backward compatibility in the case where this
behavior isn't required.
Chris
John Ousterhout wrote:
> Dan Smart wrote:
> >
> > ....
> >
> > ous...@scriptics.com (John Ousterhout) wrote in
> > <39D8DCC8...@scriptics.com>:
> >
> > >The description in your message is at a pretty abstract level; it
> > >isn't obvious to me why each of the issues you mentioned is in
> > >fact a problem. Could you supply a simple (very concrete) example
> > >that motivates each of your concerns, something like "I wanted to
> > >write a script that does XYZ, but the closest I could get was ABC
> > >because of QRS"? Said another way, are the issues you've raised
> > >theoretical problems or practical problems?
> >
> > ....
> >
> > Thus the two main desires:
> > (1)
> > Making [subst] object safe. This is just a win. It would even bve
> > acceptable if I had to pass a flag to force object safe substitution.
> > Having an object safe subst that worked on lists would also help.
> > (2)
> > Seperating <quote> from [list]. This permits me to quote sections of lists,
> > and pass the whole to eval. Thus allowing object safe evaluation with
> > substitution.
>
Mo
Well, the problem you describe boils down to the way that the internal
data type inside the Tcl_Obj gets released when the object gets converted
to another type. Another solution to this problem might be to look into
providing a way for an object to indicate to the system that it does not
want its internal rep to be released. The internal rep might be locked
down to a single type or there might be a third pointer in the object
that would keep track of the initial internal rep.
I have talked with Paul Duffin about some of these idea as they
related to the "feather" set of improvements he was working on. At any
rate, this avenue might stand a better chance of success than changing
the way eval works.
Mo DeJong
Red Hat Inc
Jeffrey Hobbs
Could you see if the following slight code change preserves
the object through eval?
Christopher Oliver wrote:
...
> Here is an example that I believe illustrates the problem, using JavaBlend and
> IncrTcl (with protclsh83):
> % package require Itcl
> 3.2
> % package require java
> 1.2.6
> % namespace import itcl::*
> % class Foo {
> > public variable a
> > public variable b
> > constructor args {
> > eval configure $args
instead of above, use:
eval [list configure] $args
> > }
> > }
> % Foo foo -a [java::new String foobar] -b [java::new java.util.Date]
> foo
> % set b [foo cget -b]
> java0x2
> % $b getYear
> invalid command name "java0x2"
>
> This example fails because "eval" doesn't preserve the objects passed to it and
> the "java" command object "java0x2" gets garbage collected because there are no
I'm not sure how incrTcl might affect this (how it passes things
around), but a change was made to 'eval' in 8.3 to specifically
preserve objects passed to eval, if they are all pure lists. This
means the following case should preserve the java object:
proc justdoit args {
eval $args [list getYear]
}
catch {java::new java.util.Date} jdate
justdoit $jdate
This is not flexible because of the getYear call, but I hope
that the idea is clear. I used 'catch' in line1 because just
in case you type it in interactively, it will prevent the
conversion of the object to a string.
Dan Kuchler
>
> Challenge:
> given the following c routines:
> ::rv::makeInteger which produces a Tcl_IntObj
>
> ::rv::send which publishes a message on Rendezvous.
> The types of the arguments to this routine are critical, as it
> needs to be able to distinguish between (for instance) a string that
> just happens to look like an integer, and an integer.
>
> Now given the following "list":
> (::rv::send dummysubject seq \$curSeq intVal [::rv::makeInteger 3])
> Passed perhaps as an argument to another routine which then does:
> set curSeq [::rv::makeInteger 3]
> ???? $thing-above
>
> What type should the list be, and how do I execute it?
Well, this is a little off topic, but I notice that you
are using a Tibco Rendezvous extension, and I was curious
whether this was one that you wrote, or whether it is one
that you purchased..
Just curious, since I have spent some time playing with
the Rendezvous APIs myself :)
--Dan
Dan Smart
>Chris,
>
>Could you see if the following slight code change preserves
>the object through eval?
>
>Christopher Oliver wrote:
>I'm not sure how incrTcl might affect this (how it passes things
>around), but a change was made to 'eval' in 8.3 to specifically
>preserve objects passed to eval, if they are all pure lists. This
>means the following case should preserve the java object:
>
If of course I want to perform substitution on the string passed to eval,
I'm stuffed.
Challenge:
given the following c routines:
::rv::makeInteger which produces a Tcl_IntObj
::rv::send which publishes a message on Rendezvous.
The types of the arguments to this routine are critical, as it
needs to be able to distinguish between (for instance) a string that
just happens to look like an integer, and an integer.
Now given the following "list":
(::rv::send dummysubject seq \$curSeq intVal [::rv::makeInteger 3])
Passed perhaps as an argument to another routine which then does:
set curSeq [::rv::makeInteger 3]
???? $thing-above
What type should the list be, and how do I execute it?
If I use eval:
If it's a real list, then curSeq will not be substituted.
And if it's a string, then the object rep of the last arg will be
destroyed.
So Answers on a postcard, or to the news group please...
Dan (who has been looking for a short example for a while) Smart
Darren New
> ::rv::send which publishes a message on Rendezvous.
> The types of the arguments to this routine are critical, as it
> needs to be able to distinguish between (for instance) a string that
> just happens to look like an integer, and an integer.
Sounds like this needs a more complex API, so you can specify such stuff. It
just sounds "wrong" to me to say "This Tcl interface needs to distinguish
between the integer 3, the float 3, and the string 3." You might as well
start using Ada. ;-)
--
Darren New / Senior MTS & Free Radical / Invisible Worlds Inc.
San Diego, CA, USA (PST). Cryptokeys on demand.
The tragedy of the commons applies to monitizing eyeballs, too.
John Ousterhout
>
> Challenge:
> given the following c routines:
> ::rv::makeInteger which produces a Tcl_IntObj
>
> ::rv::send which publishes a message on Rendezvous.
> The types of the arguments to this routine are critical, as it
> needs to be able to distinguish between (for instance) a string that
> just happens to look like an integer, and an integer.
This is a big problem. Tcl is organized around the principle that
types and representation have no impact on semantics. Even with the
addition of Tcl_Obj's in Tcl 8.0, the semantics are defined in terms
of string values. A string that looks like an integer *is* an
integer as far as Tcl is concerned; Tcl feels free to convert a
string that looks like an integer into an actual integer object, and
vice versa.
Unfortunately, you can write Tcl extensions that depend on preserving
a particular internal representation for an object, so that the object
doesn't behave the same as other objects with the same string value.
Jacl is one of these. It's very hard to make these things work cleanly,
because they violate one of the basic assumptions that has been in
Tcl from the beginning.
Although I haven't actually tried the experiment, I suspect that it
would be very difficult to evolve Tcl into a system that preserved
types the way you want. There are just too many assumptions in Tcl
that "everything is a string" and that values can be converted back
and forth from strings at will; to get around this, I suspect you
would have to define a fundamentally different language.
>
> Now given the following "list":
> (::rv::send dummysubject seq \$curSeq intVal [::rv::makeInteger 3])
> Passed perhaps as an argument to another routine which then does:
> set curSeq [::rv::makeInteger 3]
> ???? $thing-above
>
> What type should the list be, and how do I execute it?
>
> If I use eval:
> If it's a real list, then curSeq will not be substituted.
>
> And if it's a string, then the object rep of the last arg will be
> destroyed.
>
> So Answers on a postcard, or to the news group please...
You could always use list operators such as lsearch, lrange, and
concat to find the $curSeq element in the list and replace it with
the value of the $curSeq variable. Once you've done this I believe
you might be able to eval the list and get all the objects passed
through intact. However, this is any ugly solution because it depends
on a lot of information about how Tcl works internally; it would be
pretty non-obvious from the code you wrote what is going on and why
this particular code is needed.
Christopher Oliver
Yes, your changes do make this particular example work. (I tested it with a version
of Itcl/Itk which I hacked to be object-safe; if anyone has an interest in that, just
email me and I'll provide the code). This solution seems a bit ugly though, don't
you think?
Chris
Jeffrey Hobbs wrote:
> Chris,
>
> Could you see if the following slight code change preserves
> the object through eval?
>
> Christopher Oliver wrote:
> ...
> > Here is an example that I believe illustrates the problem, using JavaBlend and
> > IncrTcl (with protclsh83):
> > % package require Itcl
> > 3.2
> > % package require java
> > 1.2.6
> > % namespace import itcl::*
> > % class Foo {
> > > public variable a
> > > public variable b
> > > constructor args {
> > > eval configure $args
>
> instead of above, use:
> eval [list configure] $args
>
> > > }
> > > }
> > % Foo foo -a [java::new String foobar] -b [java::new java.util.Date]
> > foo
> > % set b [foo cget -b]
> > java0x2
> > % $b getYear
> > invalid command name "java0x2"
> >
> > This example fails because "eval" doesn't preserve the objects passed to it and
> > the "java" command object "java0x2" gets garbage collected because there are no
>
> I'm not sure how incrTcl might affect this (how it passes things
> around), but a change was made to 'eval' in 8.3 to specifically
> preserve objects passed to eval, if they are all pure lists. This
> means the following case should preserve the java object:
>
Dan Smart
>Dan Smart wrote:
>> ::rv::send which publishes a message on Rendezvous.
>> The types of the arguments to this routine are critical, as it
>> needs to be able to distinguish between (for instance) a string
>> that just happens to look like an integer, and an integer.
>
>Sounds like this needs a more complex API, so you can specify such
>stuff. It just sounds "wrong" to me to say "This Tcl interface needs to
>distinguish between the integer 3, the float 3, and the string 3." You
>might as well start using Ada. ;-)
>
Which is in fact how I solved the problem, there currently being no other
way to address it. This is not a very good example of the problem as it can
be addressed reasonably simply, good examples are however far more complex.
Still, your argument doesn't address the problem, weakly typed does not
mean untyped, tcl was untyped, now it is weakly typed. Integer 3, float 3.0
and "3" are all different things, most TCL interfaces preserve that
distinction, none of the substitution interfaces do.
Dan
Dan Smart
ous...@scriptics.com (John Ousterhout) wrote in
>Dan Smart wrote:
>>
>> Challenge:
>> given the following c routines:
>> ::rv::makeInteger which produces a Tcl_IntObj
>>
>> ::rv::send which publishes a message on Rendezvous.
>> The types of the arguments to this routine are critical, as it
>> needs to be able to distinguish between (for instance) a string
>> that just happens to look like an integer, and an integer.
>
>This is a big problem. Tcl is organized around the principle that
>types and representation have no impact on semantics. Even with the
>addition of Tcl_Obj's in Tcl 8.0, the semantics are defined in terms
>of string values.
With the addition of Tcl_Objs, TCL went from being an untyped language, to
a weakly typed language. Since 8.0 it has been possible to distinguish
between an integer, a float, and the string representation of same. Most
TCL interfaces preserve the difference, nothing involving substitution
does.
> A string that looks like an integer *is* an
>integer as far as Tcl is concerned; Tcl feels free to convert a
>string that looks like an integer into an actual integer object, and
>vice versa.
>
It only performs the conversion if you do something that demands such a
conversion. This is one of the things I really like about the way that
Tcl_Objs were added to TCL. If you avoid demanding the conversion, it
doesn't happen. Further, it is possible to distinguish in TCL between for
instance a real that is a real, and a real that is a string.
Unfortunately, today, any form of substitution demands a conversion to
string, most lose the object rep as well.
>Unfortunately, you can write Tcl extensions that depend on preserving
>a particular internal representation for an object, so that the object
>doesn't behave the same as other objects with the same string value.
>Jacl is one of these. It's very hard to make these things work cleanly,
>because they violate one of the basic assumptions that has been in
>Tcl from the beginning.
>
This is a problem we need to fix, and it is soluble, I believe that the
parts of my original proposal that didn't deal with '#' are fairly easy to
fit into TCL now, and will not affect backwards compatability.
Unfortunately, it is unlikely that I will be able to work on such a change
before christmas, otherwise I would, as proof of concept.
>Although I haven't actually tried the experiment, I suspect that it
>would be very difficult to evolve Tcl into a system that preserved
>types the way you want. There are just too many assumptions in Tcl
>that "everything is a string" and that values can be converted back
>and forth from strings at will; to get around this, I suspect you
>would have to define a fundamentally different language.
>
I think you are being unecessarily pessamistic. I agree that changing TCL
so that it is impossible to destory type information would indeed be
infeasible. Fixing TCL so it is possible to preserve type information is to
my mind eminently doable.
>>
>> Now given the following "list":
>> (::rv::send dummysubject seq \$curSeq intVal
>> [::rv::makeInteger 3])
>> Passed perhaps as an argument to another routine which then does:
>> set curSeq [::rv::makeInteger 3]
>> ???? $thing-above
>>
>> What type should the list be, and how do I execute it?
>>
>> If I use eval:
>> If it's a real list, then curSeq will not be substituted.
>>
>> And if it's a string, then the object rep of the last arg will
>> be destroyed.
>>
>> So Answers on a postcard, or to the news group please...
>
>You could always use list operators such as lsearch, lrange, and
>concat to find the $curSeq element in the list and replace it with
>the value of the $curSeq variable. Once you've done this I believe
>you might be able to eval the list and get all the objects passed
>through intact. However, this is any ugly solution because it depends
>on a lot of information about how Tcl works internally; it would be
>pretty non-obvious from the code you wrote what is going on and why
>this particular code is needed.
Unfortunately, with no way to no if a list element is intentionaly
preserved or not, there is no way of knowing whether elements begining with
$ need substituting or are a real value, further having got something that
does need substituting, you then need to frak around with a proc like:
proc mySubst {todo} {
uplevel 1 [string range $todo 1 end] foo
return $foo
}
If the element in question is []ed then I suspect you are SOOL.
An object safe subst solves part of the problem, a trivial implementation
of which would have the following semantics:
subst -osafe thing
looks at the first char of thing:
if it is a '$' then the remainder of the string is treated as a variable
ref and returned. If there are extraneous chars after the end of the var
name an error is raised.
If it is a '[' then if the last character is a '] the rest of the string is
executed and the result is returned, otherwise an error is raised.
Otherwise thing is returned.
Next extension:
If thing is a pure list, the above algorithm is applied to each element
that already has a string rep.
Next extension:
Provide a mechanism to flag a list element (or ideally any Tcl_Obj) as
being not substitutable. Even if the element/object has a string rep subst
should ignore it.(*)
At which point we are mostly there, Not as far as I'd like to go, as at
this point I would have to do:
eval [subst -osafe [list]]
The final step gels very nicely with the move to make [l*] subcommands,
which is to give the new command a flag which says whether to by default
set the 'substitutable' flag or not. At which point we make eval do an
automatic 'subst -osafe' on the list, and I'm happy as larry.
Dan "Assuming Larry is Really Happy" Smart
(*) As discussed else where this flag is tri-state deliberately protected,
deliberately unprotected, and neither.
Andreas Leitgeb
> John Ousterhout wrote:
>> A string that looks like an integer *is* an
>>integer as far as Tcl is concerned; Tcl feels free to convert a
>>string that looks like an integer into an actual integer object, and
>>vice versa.
Dan's example wasn't too bad. It just isn't necessary to come up with code
that breaks because of lost object-references.
The problem is there, because it is not possible in current Tcl to have
both substitution AND object-references together as shown in Dan's example.
Whether the programm fails, or maybe just performs horrible (due to
forth & back-conversion) is just a little detail about it.
Strings can't do it, because strings can be saved to file and reread later,
so they cannot possibly contain object-references.
it might boil down to:
- a mechanism to tag certain arguments to procedures, without having
to untag the other arguments, and without restricting the
arguments themselves in any way. ({tag}...-syntax creeps into mind)
- a bunch of commands to do selective (and object-safe !) substitution
inside normal lists (controlled by abovementioned tags), and a means
to pass such lists with substitution- AND object-information as
callback-scripts to all kinds of code where the actual "eval" is not
within reach.
Donal K. Fellows
Christopher Oliver <col...@mminternet.com> wrote:
> Yes, your changes do make this particular example work. (I tested it
> with a version of Itcl/Itk which I hacked to be object-safe; if
> anyone has an interest in that, just email me and I'll provide the
> code). This solution seems a bit ugly though, don't you think?
Well, the other simple alternatives are unfortunately inclined to give
semantic errors. For example:
set s {puts [ gets stdin ]}
eval $s
has the behaviour of echoing one line from stdin to stdout, but if you
modify Tcl so that objects are preserved across eval perfectly if they
are interpretable as list elements, then:
set s {puts [ gets stdin ]}
llength $s
eval $s
is a syntax error (since puts gets called with four arguments.) This
is a problem!
A year or two back, this was discussed in detail here on comp.lang.tcl
and the consensus reached was that you could only get away with the
perfect-preservation-across-eval trick if the lists being evaluated
are guaranteed to have come from something that is list generating
(instead of coming from some random string, which is the source of the
trouble with the above example.) Tcl currently implements an cautious
version of this, where the test is implemented in terms of the concept
of a "pure list" - a Tcl_Obj with a list rep but not a string rep - as
this was cheap to test for and cheap to implement, but there is a more
robust version possible. Implementing it would require quite a big
change to the core[*] though, so in the interest of keeping the bug
count down, it hasn't been done (yet.)
Apart from this _performance_optimization_, Tcl does not use the
concept of pure lists anywhere, and it is undocumented in the
distributed documentation for a very good reason. Extensions that
rely on this sort of thing are highly fragile[+], and are very much
not closely in line with the general semantic pattern of Tcl; the
correct way to pass objects through Tcl is to allocate some kind of
handle and explicitly release the handle once you've finished with it.
This might mean that you need to be clever about how you handle your
code, but that's the way it is.
Mind you, code which correctly uses the "pure list" optimization is
also generally robust code, and it will only cease to function as
expected if a very loudly-announced incompatible change happens. :^)
Donal.
[* The patch to add the current behaviour added one if-statement to
Tcl_EvalObjEx() together with matching support in Tcl_ConcatObj()
and a bugfix in Tcl_Main(). The more sophisticated version would
have modified all those places, as well as adding non-trivial
changes to virtually all the list code and a number of other sites.
And not that much code would actually see the difference, since the
most useful case is actually unoptimisable without a major overhaul
of the compiler itself. ]
[+ We may well change the behaviour in this area in the future without
warning! ]
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ fell...@cs.man.ac.uk
-- The guy who sells me my audio hardware explained that a computer will never
produce the same level of sound quality that a stereo will b/c stereo have
transistors and sound cards don't. --Matthew Garson <mga...@world.std.com>
Christopher Oliver
"Donal K. Fellows" wrote:
> Apart from this _performance_optimization_, Tcl does not use the
> concept of pure lists anywhere, and it is undocumented in the
> distributed documentation for a very good reason. Extensions that
> rely on this sort of thing are highly fragile[+], and are very much
> not closely in line with the general semantic pattern of Tcl;
I am aware of this point.
> the
> correct way to pass objects through Tcl is to allocate some kind of
> handle and explicitly release the handle once you've finished with it.
> This might mean that you need to be clever about how you handle your
> code, but that's the way it is.
I'm not sure what type of cleverness you're talking about. I think extensions like
TclBlend and TclMico and Feather add important, useful functionality to Tcl.
However, without reference-counting of their "object-like" commands they would be
essentially unusable. In Tk, because of the fact that widget commands are
automatically destroyed when their parent window is destroyed, the user can
reasonably manage explicit destruction of widgets. But if the user had to
explicitly destroy every Java or CORBA or Feather command the resulting programs
would be ugly and command "leaks" would become a big problem. (I can imagine that
this is already a problem for those who use IncrTcl for non-gui programs).
Chris
Donal K. Fellows
Christopher Oliver <col...@mminternet.com> wrote:
> I'm not sure what type of cleverness you're talking about. I think
> extensions like TclBlend and TclMico and Feather add important,
> useful functionality to Tcl. However, without reference-counting of
> their "object-like" commands they would be essentially unusable. In
> Tk, because of the fact that widget commands are automatically
> destroyed when their parent window is destroyed, the user can
> reasonably manage explicit destruction of widgets. But if the user
> had to explicitly destroy every Java or CORBA or Feather command the
> resulting programs would be ugly and command "leaks" would become a
> big problem. (I can imagine that this is already a problem for
> those who use IncrTcl for non-gui programs).
One technique that I've heard described was to put the references in
variables and to delete the handles when the variables are deleted (or
assigned to, I suppose, though the vars might be read-only instead.)
This would be reasonably easy to work with for most situations where
the referents have a syntactically limited usage. I can't remember
who described this way of working first, and what extension it was
described in relation to. It might have been Itcl, but I'm not sure.
Anyway, doing more than this requires a redesign of the *semantics* of
Tcl, and yet I don't see that as being easily done. And the design
really has to be done and explained well before the implementation
starts.[*]
Anyone who believes they have a clear idea of how they believe Tcl
could change to support this is urged to prepare a TIP on the matter.
TIPs (Tcl Improvement Proposals) are the only real way to make sure
that the matter is clearly set out. For some examples, see the
current TIP archive at:
http://www.cs.man.ac.uk/fellowsd-bin/TIP/
Included is a description of the format of TIPs and some of the
rationale behind them. More background can be had by reading the
archive of the tclcore mailing list (there is a way into that buried
in the TIP archive too.)
Donal.
[* Random thought: is it possible to substitute and still keep around
a collection of things that were substituted in? What would be the
performance and memory-consuption consequences of doing this? ]
--
Donal K. Fellows http://www.cs.man.ac.uk/~fellowsd/ fell...@cs.man.ac.uk
-- OK, there is the MFC, but it only makes the chaos object orientated.
-- Thomas Nellessen <nell...@gmx.de>
Peter da Silva
Andreas Leitgeb <a...@pc7499.gud.siemens.at> wrote:
> Strings can't do it, because strings can be saved to file and reread later,
> so they cannot possibly contain object-references.
The same problem exists for variable and procedures (a string can contain
$foo or [foo]), doesn't it? That is, this isn't a new problem.
--
`-_-' In hoc signo hack, Peter da Silva.
'U` "Milloin halasit viimeksi suttasi?"
Disclaimer: WWFD?
Andreas Leitgeb
>In article <slrn8v8et...@pc7499.gud.siemens.at>,
>Andreas Leitgeb <a...@pc7499.gud.siemens.at> wrote:
>> Strings can't do it, because strings can be saved to file and reread later,
>> so they cannot possibly contain object-references.
>
>The same problem exists for variable and procedures (a string can contain
>$foo or [foo]), doesn't it? That is, this isn't a new problem.
but $foo and [foo] are strings, consisting of the characters
"$", "f", "o", "o" and "[", "f", ... respectively, not
object-references in the sense they are being discussed in this thread.
If you place an object-reference into a string (with some syntax),
the string will not know, that it carries an object-ref, so if you
delete the string, the object will not notice, that it may be no
longer referenced.
In contrast, the variable or command foo does not care how often, if
at all it is "referenced".
Donal K. Fellows
<a...@pc7499.gud.siemens.at> writes
>If you place an object-reference into a string (with some syntax),
>the string will not know, that it carries an object-ref, so if you
>delete the string, the object will not notice, that it may be no
>longer referenced.
The easiest way of working around this is to extend the Tcl_Obj
structure with another field so that when a substitution is performed
into the string, the objects that were substituted are retained. That
would do the trick.
The biggest single down-side to this is that the Tcl_Obj structure is
very common and we're increasing its size by (at least[*]) four bytes.
The other potential problem, that old code will be allocating loads of
wrongly sized structures, is not really going to bite here since
everyone should be using Tcl_NewObj (or something built on it) to do the
allocation anyway; their code doesn't use the size of the structure.
You could also do all the management of the extra information quite
neatly behind the scenes of existing functions.
On balance, fixing this is possible, but it is not a change that it
would be a good idea to foist on people in a minor version change. IMHO
at least...
Were we to do this, a whole host of new features (e.g. lambda closures,
lexically scoped variables) would become practical. It is only the
complexity of lifetime management that's really stopping them now.
>In contrast, the variable or command foo does not care how often, if
>at all it is "referenced".
Strictly, the notion of lifetime is linked to that of the interpreter
(an explicitly managed resource) and the call-stack.
Donal.
[* It depends on the alignment, and the dependency is not straight
forward. ]
Andreas Leitgeb
> [about lifetime-management for objects embedded into strings:]
>The easiest way of working around this is to extend the Tcl_Obj
>structure with another field so that when a substitution is performed
>into the string, the objects that were substituted are retained. That
>would do the trick.
not really:
assume, variable a contains an objectref: "foo<objref>bar" where "<objref>"
is just meta-syntax for the objects representation in the string.
then what happens with: (will b be a carrier of the object-reference ?)
set b "$a"
regsub "oo.*ba" $a {&} b
set b [join [split $a]]
set b [string range $a 0 end]
... and many other string-manipulating code-snippets ...
A string is not really fit to maintain references to anonymous objects.
(not even with little extensions&patches)
Donal K. Fellows
In article <slrn90lk5...@pc7499.gud.siemens.at>, Andreas Leitgeb
<a...@pc7499.gud.siemens.at> writes
>not really:
> assume, variable a contains an objectref: "foo<objref>bar" where "<objref>"
> is just meta-syntax for the objects representation in the string.
>
> then what happens with: (will b be a carrier of the object-reference ?)
> set b "$a"
Yes.
> regsub "oo.*ba" $a {&} b
Possibly (assuming you meant [regexp] there, of course.) I could argue
this one either way.
> set b [join [split $a]]
No.
> set b [string range $a 0 end]
Same sort of answer as for [regexp].
> ... and many other string-manipulating code-snippets ...
>
>A string is not really fit to maintain references to anonymous objects.
>(not even with little extensions&patches)
Well, any other way of doing it requires major changes to the way Tcl
works, both as a library and as a language. My modest proposal at least
has the advantage of being compatible with the *current* semantics and
behaviour...
Donal.
Andreas Leitgeb
>> assume, variable a contains an objectref: "foo<objref>bar" where "<objref>"
>> is just meta-syntax for the objects representation in the string.
>> then what happens with: (will b be a carrier of the object-reference ?)
>> regexp "oo.*ba" $a {&} dmy b ;# my regsub-regexp-mixup corrected
>Possibly ...
>> set b [join [split $a]]
>No.
Why the former and not the latter ?
Why wouldn't the object in the latter example continue living in one element
of the temporary list and then continue living in the joined string ?
Which lowlevel actions on the string would cause the embedded object's
refcount to decrease ?
>>A string is not really fit to maintain references to anonymous objects.
>>(not even with little extensions&patches)
>Well, any other way of doing it requires major changes to the way Tcl
>works, both as a library and as a language. My modest proposal at least
>has the advantage of being compatible with the *current* semantics and
>behaviour...
I agree, though I conclude from it that it's not possible or at least not
practible to have object-lifetime managed by "substring"-lifetime,
which is the point I was trying to make.
Either
- We wait for a clean, bulletproof, sound solution to come.
(Godot might arrive earlier.) -- I'd be happy being proven wrong here.
- Or we have to consider some "semantical incompatibility",
- Or, Tcl will be stuck with only half-hearted object safety, as it now is.