Enhance [proc] for anonymous functions and closures
EL
I have filed the feature request below to the tracking system at
http://tcl.sourceforge.net/ and would like to have some feedback from
other users here as well...
How realistic is this, and how desireable? Is it possible at all, in a
certain time frame and what would this time frame be, approximately?
Asking as somebody who has no clue about interpreter implementation and
very few clue about compiler implementation...
Thanks,
Eckhard
----- Feature Request 1932618 -----
I would like to request that [proc] becomes a data type in addition to a
command, so that procs can be
- created on the fly, capturing the current environment and procedure stack
- created anonymously, e.g. [proc {} {...}] would return a new proc to
be used elsewhere.
- returned by other proc's
- given as arguments to other procs (including their arguments and the
environment in which they were created)
When this s possible, I think it would also be possible to have closures
in Tcl and better capabilities for functional programming - which would
enhance the language *a lot*
Is it possible to have [proc] as a Tcl_Obj* type?
Gerald W. Lester
--
+--------------------------------+---------------------------------------+
| Gerald W. Lester |
|"The man who fights for his ideals is the man who is alive." - Cervantes|
+------------------------------------------------------------------------+
EL
> I think this needs to be TIPed. See: wiki.tcl.tk/TIP
I thought for TIP's it would be necessary to have a reference
implementation, or at least a more concrete specification how it could
be achieved?
That I don't have, unfortunately... the text in the request is the only
thing I can actually specify and I guess this is too vague for a TIP, or no?
Eckhard
Bruce Hartweg
have you looked at the new apply command in 8.5?
http://www.tcl.tk/man/tcl8.5/TclCmd/apply.htm
also check out the TIP that introduced it
http://www.tcl.tk/cgi-bin/tct/tip/194
it has some discussion of uses and how to use it for
other FP constructs.
Bruce
Neil Madden
...
> ----- Feature Request 1932618 -----
> I would like to request that [proc] becomes a data type in addition to a
> command, so that procs can be
>
>
> - created on the fly, capturing the current environment and procedure stack
> - created anonymously, e.g. [proc {} {...}] would return a new proc to
> be used elsewhere.
> - returned by other proc's
> - given as arguments to other procs (including their arguments and the
> environment in which they were created)
>
> When this s possible, I think it would also be possible to have closures
> in Tcl and better capabilities for functional programming - which would
> enhance the language *a lot*
As Bruce Hartweg has pointed out, the [apply] command in 8.5 provides
first-class procs. Enhancing [proc] to allow a 2-arg case for a
constructor of such lambdas is a possibility, but I think it might cause
confusion[*]. I currently use:
proc lambda {params body} { list ::apply [list $params $body] }
(or a variation of this).
Closures are harder. See http://wiki.tcl.tk/17686 and
http://wiki.tcl.tk/20662 for a simple approach to creating immutable
closures (i.e., they capture the current values of variables rather than
the variables themselves). The problem with mutable variables is that
they have no natural string rep (e.g., if they are proc-local), so hard
to represent as a Tcl_Obj. The best I've come up with so far is to use
fully-qualified global/namespace variables as sort of references, but
you then need to deal with lifetime management.
[*] A generalisation of this idea would be to allow 0 or more name
arguments to proc, e.g. no-args = lambda, 1 arg = as now, many args =
namespace ensemble. See http://wiki.tcl.tk/20657
-- Neil
Neil Madden
...
> Closures are harder. See http://wiki.tcl.tk/17686 and
> http://wiki.tcl.tk/20662 for a simple approach to creating immutable
> closures (i.e., they capture the current values of variables rather than
> the variables themselves). The problem with mutable variables is that
> they have no natural string rep (e.g., if they are proc-local), so hard
> to represent as a Tcl_Obj. The best I've come up with so far is to use
> fully-qualified global/namespace variables as sort of references, but
> you then need to deal with lifetime management.
Sorry for the double post, but I should have mentioned that another
approach to mutable closures, used in the dictutils package, is to pass
around a lambda and a dictionary representing the environment
separately. When the lambda is executed it does something like a [dict
with], which allows the proc to update the dictionary and the new
version can then be passed on. It requires cooperation from the caller
of the lambda, but otherwise is a decent solution.
-- Neil
Gerald W. Lester
No you do not need it -- but it sure helps.
Joe English
> I would like to request that [proc] becomes a data type in addition to a
> command, so that procs can be
> - created on the fly, capturing the current environment and procedure stack
> - created anonymously, [...]
> - returned by other proc's
> - given as arguments to other procs [...]
> Is it possible to have [proc] as a Tcl_Obj* type?
> [... earlier ...]
> How realistic is this, and how desireable? Is it possible at all, in a
> certain time frame and what would this time frame be, approximately?
In Tcl: quite desirable (you're not the first to ask
for this set of features), but not very realistic
(which is why it hasn't been implemented yet).
See TIP#187 ("Procedures as values") and TIP#196 ("Tcl
commands as values"), for some earlier attempts at this,
and TIP#194 ("Procedures as values via 'apply'"),
which is what finally worked.
TIP#194 does *most* of what you ask for -- anonymous
procedures that can be passed to and returned from
other procedures -- but it can't capture the current
environment or procedure stack.
Several strategies have been proposed to implement
various aspects of lexical scoping and capturing
continuations. None have been workable. Some
were fundamentally incompatible with Tcl's EIAS semantics;
others were unworkable with Tcl's current implementation;
and the rest have been feasible but deemed not worth
the effort. Tcl just ain't Scheme.
However, Tcl does have pretty good support for the
"point-free" or "combinator-based" style of higher-order
functional programming; informally this is "Procedures
as Values via Command Prefixes". No TIP# for this one,
since Tcl has supported it pretty much all along
(although TIP#174 ("Math operators as commands"), TIP#112
("Ensembles are Namespaces are Commands"), and TIP#293
("Splat operator") have a nice synergy with this technique
that makes it work even better).
With this style you don't need lexical scope or call/cc,
so the fact that they're impossible in Tcl doesn't matter
all that much. And if you don't need full-blown
Scheme-style "real" lambda expressions, TIP#194's
"fake" lambdas are often suitable.
--Joe English
Donal K. Fellows
> I thought for TIP's it would be necessary to have a reference
> implementation, or at least a more concrete specification how it could
> be achieved?
No, but we won't (usually) vote on one until there is an implementation.
Donal.
Eckhard Lehmann
Thanks a lot for the TIP hints. I think TIP#187 comes pretty close to
my feature request. (Salvatore Sanfilippo also implemented the sugar
macro system, which I think is also a great enhancement of the Tcl
language).
> Several strategies have been proposed to implement
> various aspects of lexical scoping and capturing
> continuations. None have been workable. Some
> were fundamentally incompatible with Tcl's EIAS semantics;
> others were unworkable with Tcl's current implementation;
> and the rest have been feasible but deemed not worth
> the effort. Tcl just ain't Scheme.
Continuing the ideas of TIP#187... I guess that the "lambda" command
could have clientData associated in a Tcl_Obj*. Wouldn't it be
possible to capture variables and their current values at execution
time in the environment where the lambda is created - and associate
appropriate variables in the lambda's body before this is executed?
This could maybe be a way to implement lexical scoping?
Just an idea..
Eckhard
Larry W. Virden
> Is it possible at all, in a
> certain time frame and what would this time frame be, approximately?
As far as I am aware (and I hope the tcl maintainers can correct this
impression if I am wrong), there isn't a team "out there" somewhere
that is working on the items in the feature request section of the
project. That area is, rather, a "sticky pad" for ideas. If someone in
the community sees a request and decides that it is just what they
needed, and they have time and ability, then they might, some day,
write up a TIP < http://wiki.tcl.tk/TIP > to detail the change, go
through the process of having it approved, write up the code, and
integrate the code, tests, and documentation into the core.
On the other hand, there might not be anyone who likes the idea enough
to actually write any code.
Think of the Feature requests as a "wouldn't it be nice" type of
thing. Think of the TIP as a "hey, I'm working (or about to work) on
some new code - here's what I am planning to do; is this something
that the core team will permit into the core?"
So, you've taken a fine first step - put the idea out there to
discuss. The next step would be to write a prototype implementation -
or perhaps brainstorm here or in some other forum to your liking - on
the way to implement and then implement a first take on the code. And
of course, if you feel your C skills are a bit lacking, then
recruiting help in implementing might be your first step.
Alexandre Ferrieux
>
> [...] go through the process of having it approved,
> write up the code, [...]
>
If I read Donal correctly, this is the converse: first implement, then
discuss. I must admit this is a bit surprising, because it means
wasted efforts if your patch ends up being rejected. But in reality,
there are possibilities to get feedback before the final vote: TCT
members can't resist forever a question like "would you prefer A or
B" ;-)
-Alex
schlenk
Hmm, the refined way would be:
- Propose what you want to have
- Write a prototype patch so people can see that it in fact is
possible
- Let TCT vote
- Provide a full patch that fills in the missing pieces from the
prototype
Each of these stages allows some amount of discussion but unless you
can show at least a prototype its still discussing vapourware and gets
not as much attention.
Michael
Eckhard Lehmann
> Hmm, the refined way would be:
> - Propose what you want to have
> - Write a prototype patch so people can see that it in fact is
> possible
> - Let TCT vote
> - Provide a full patch that fills in the missing pieces from the
> prototype
When I have a prototype that works, it is probably the most complete
implementation - because otherwise it wouldn't make sense to create a
proof of concept that fails to show the possibility.
So what Alexandre says is basically right, you put a lot of work and
effort into it and in the end they will likely reject it or put it on
hold forever, and you don't see it in the language anyway. For me this
happened once already (TIP#290). Therefore I am biased towards not
starting any work unless it can be half way confirmed that it is
successful and of interrest.
In the end I think that TIP's and implementations that are created by
TCT members have a much higher chance to make it into the language
than TIP's from non TCT members. I think this makes sense, but there
must be a way for others to file and discuss feature requests. The
sourceforge tracker and this group are good places for this.
Eckhard
Alexandre Ferrieux
> In the end I think that TIP's and implementations that are created by
> TCT members have a much higher chance to make it into the language
> than TIP's from non TCT members. I think this makes sense, but there
> must be a way for others to file and discuss feature requests. The
> sourceforge tracker and this group are good places for this.
Yes, I too think it makes sense. Even if I invent something fabulous
(which I have yet to do ;-), I'll be happier to transfer my enthusisam
to a TCT member so that he can reimplement in agreement with 20 years
of craft, rather than to see my dumbo code accidentally pass the test
suite today and make it into the next Enceladus rover ;-)
Another good place to realize this symbiosis is the tcl-core list.
For example, a few weeks ago, after a wild brainstorming session here,
I went to tcl-core proposing a Mutability patch. I had very
interesting discussions on this with several TCT members; then I wrote
roughly 30% of what I set out to do (List, not Dicts), even
distributed the early patch, but then a fundamental objection was
found by Miguel. This is a virtuous process, and I don't regret a
thing. Maybe you can do the same.
-Alex
Donald G Porter
> If I read Donal correctly, this is the converse: first implement, then
> discuss.
From my perspective, the benefit of having a reference implementation
is that it fills in the gaps when the proposal itself is incomplete
or ambiguous. This is helpful because the population of folks who
contribute such things tend to be better coders of C/Tcl than writers
of precise specs in English.
So long as a proposal is complete and clear enough, I don't mind
approving it before implementation, though that leaves open the
possibility that an approved idea might languish unimplemented for
some time. (This has happened too).
--
| Don Porter Mathematical and Computational Sciences Division |
| donald...@nist.gov Information Technology Laboratory |
| http://math.nist.gov/~DPorter/ NIST |
|______________________________________________________________________|
Donal K. Fellows
> If I read Donal correctly, this is the converse: first implement, then
> discuss. I must admit this is a bit surprising, because it means
> wasted efforts if your patch ends up being rejected. But in reality,
> there are possibilities to get feedback before the final vote: TCT
> members can't resist forever a question like "would you prefer A or
> B" ;-)
We do it that way round because we've had to deal with the other way in
the past, and ended up with TIPs approved that *nobody* was willing to
implement. Forcing an implementation at least avoids that stupid situation.
Donal.
Eckhard Lehmann
<donal.k.fell...@manchester.ac.uk> wrote:
> We do it that way round because we've had to deal with the other way in
> the past, and ended up with TIPs approved that *nobody* was willing to
> implement. Forcing an implementation at least avoids that stupid situation.
I still think that there mus be a layer in between. You guys know the
core implementation of Tcl way better than I do as a simple user, and
it costs me *a lot* to step through the code, understand what is
happening when and how to change it according to the functionality I
would like to have. For you, who do that every day (?) it's kind of
nobrainer.
Everybody would save valuable time if non TCT members could just
submit and discuss feature requests and TCT members decide whether
they go into the core, when and how they go into the core - and then
do the implementation by themselves.
BTW, are there people working full time on Tcl implementation, maybe
at ActiveState?
Eckhard
miguel
> On 3 Apr., 16:57, "Donal K. Fellows"
> <donal.k.fell...@manchester.ac.uk> wrote:
>
>> We do it that way round because we've had to deal with the other way in
>> the past, and ended up with TIPs approved that *nobody* was willing to
>> implement. Forcing an implementation at least avoids that stupid situation.
We also largely avoid voting for TIPs that turn out to be unimplementable - ie,
cases of incapacity rather than unwillingness. Or those that have some obscure
nasty implication that only becomes obvious during implementation.
> I still think that there mus be a layer in between. You guys know the
> core implementation of Tcl way better than I do as a simple user, and
> it costs me *a lot* to step through the code, understand what is
> happening when and how to change it according to the functionality I
> would like to have. For you, who do that every day (?) it's kind of
> nobrainer.
> Everybody would save valuable time if non TCT members could just
> submit and discuss feature requests and TCT members decide whether
> they go into the core, when and how they go into the core - and then
> do the implementation by themselves.
Many TCT members are maintainers, but the two roles are different and should not
be confused. The TCT does no implementation as such. One could go as far as to
argue that the TCT does not own the core, it is only responsible for the APIs:
tcl.h, the stubs tables and the man pages.
Note that there are many maintainers outside the TCT too, and also plenty of
people who know the code's ins and outs who are not maintainers.
For the record, the set of TIPs where the provider of the ref implementation is
different from the initial proposer is not empty, I can point to at least one:
TIP 187. Note that the ref implementation was done by myself before I was
elected to the TCT. Note also that I would vote against 187 in the 8.x context
(not sure about 9.x); that vote was never called, but we approved 194 which is
very much a NO to 187.
So you do not need the TCT to write a TIP, and do not need it to provide a ref
implementation: just need to make it promising enough that someone with the
required knowledge implements it if you can't or won't do it yourself. You will
need a TCT member to call a vote on it though.
Miguel
EL
> So you do not need the TCT to write a TIP, and do not need it to provide
> a ref implementation: just need to make it promising enough that someone
> with the required knowledge implements it if you can't or won't do it
> yourself. You will need a TCT member to call a vote on it though.
Well, if there is still the possibility that it is neglected, why
provide a reference implementation?
I can do C coding as well, and I am not afraid to touch the Tcl code. My
point is more or less that it is a waste of time to do that, if the new
feature still has no or sparingly chance to go into the language. You
are 16 in the TCT, so there is always a good chance to get a NO vote for
a TIP. Not very constructive if you have already spent some nights on a
reference implementation..
Eckhard
miguel
Please also note that a TIP will not be voted on until there is a ref
implementation; but it will be discussed if there is enough detail to support a
discussion.
Some features have also been discussed to death prior to them being TIPed,
others have had competing TIPs. Main historical examples that comes to mind:
{expand} aka {*}, procedures as values.
schlenk
> miguel schrieb:
>
> > So you do not need the TCT to write a TIP, and do not need it to provide
> > a ref implementation: just need to make it promising enough that someone
> > with the required knowledge implements it if you can't or won't do it
> > yourself. You will need a TCT member to call a vote on it though.
>
> Well, if there is still the possibility that it is neglected, why
> provide a reference implementation?
If your TIP is crystal clear in what should be done and how it should
be done and it is evident that it is really implementable you don't
need a reference implementation. But to arrive at such a clear and
precisely worded TIP you nearly always have a working implementation
done anyway, because sometimes its the only way to know if something
can be done.
So consider it a discussion aid. Reference Implementation should show
that it is doable, so stuff like complete error handling, a full
testsuite etc. are not strictly required.
Yes, you might waste some time on writing a ref implementation,
especially if the idea itself is a bad one. But often you learn why
its a bad idea from doing the ref implementation and it isn't
discovered later that stuff is simply not implementable in the way it
was thought out.
Michael
EL
> Yes, you might waste some time on writing a ref implementation,
> especially if the idea itself is a bad one. But often you learn why
> its a bad idea from doing the ref implementation and it isn't
> discovered later that stuff is simply not implementable in the way it
> was thought out.
If other people, who know the Tcl code better, see that there is an
issue, then I can avoid banging my head against walls while trying to
implement it. This can be evaluated by a discussion with the experts -
much easier than with a try/error approach by myself. As said, I have
done this before and it is just a waste of time. There are really better
ways to learn...
Let me propose another way, one that is more valuable from my point of view:
I as a user think of a cool new feature, e.g. the proc==value thing, an
integrated debugger... you name it. Before I start to put any work into
that, I post my idea through a channel (the feature request system or
the TCT mailing list) and let TCT members, who *are* experts, decide
whether they find it cool as well and want to have it in the language.
If so, and they agree that it will go into the next major release, then
I can sit down and implement a patch. If they have objections against
the idea, then they can tell me that and make clear that this is not
going to be in any Tcl version (mayb for now or whatever).
Everybody has a benefit from this: I don't have to waste my time in a
useless reference implementation and can sleep some nights more. The TCT
in turn is not spammed with heaps of useless TIPs that sit in their
repository and need to be evaluated and maintained... because they have
ruled out many of them already before they are created.
Wouldn't this speed up the language evolution and be a much more
friendly environment for everybody who likes to contribute to core Tcl?
I for my part can say that I will never create a TIP + reference
implementation again, unless it is confirmed that the feature is of
interest and will really make it into the language, after it is finished.
Eckhard
Joe English
> If your TIP is crystal clear in what should be done and how it should
> be done and it is evident that it is really implementable you don't
> need a reference implementation. But to arrive at such a clear and
> precisely worded TIP you nearly always have a working implementation
> done anyway, because sometimes its the only way to know if something
> can be done.
+1.
> Yes, you might waste some time on writing a ref implementation,
> especially if the idea itself is a bad one. But often you learn why
> its a bad idea from doing the ref implementation and it isn't
> discovered later that stuff is simply not implementable in the way it
> was thought out.
+1.
-- JE
(I have nothing else to add).
Donal K. Fellows
> I as a user think of a cool new feature, e.g. the proc==value thing, an
> integrated debugger... you name it.
If you're having problems thinking of a new feature, look at our feature
request trackers. :-) On the other hand, the FRQs in those aren't sorted
at all.
> Before I start to put any work into
> that, I post my idea through a channel (the feature request system or
> the TCT mailing list) and let TCT members, who *are* experts, decide
> whether they find it cool as well and want to have it in the language.
You could also ask on comp.lang.tcl. The relevant questions for you are
"is it a good idea" and "is it possible"?
> If so, and they agree that it will go into the next major release, then
> I can sit down and implement a patch. If they have objections against
> the idea, then they can tell me that and make clear that this is not
> going to be in any Tcl version (mayb for now or whatever).
Ah, but we've had people promise stuff in the past and fail to deliver,
and so we'll not make any hard commitment until a patch is present.
(Whether or not we use the patch, well, that's a separate matter.)
However, provided you've got a good idea and an implementation, the
chances of getting the code into the next release are very good. They're
even better when you stop worrying about meta-questions and start
writing code. ;-)
Now, if we wrench things back to anonymous procedures, the issue there
relates to whether a value should have more meaning to it other than
what is present in its string representation. The fundamental semantic
model of Tcl 8 values says that it shouldn't, but perhaps that should be
relaxed for Tcl 9.0. (I'd be distinctly hesitant about doing that for
8.6 though; lots of work and the rabbit hole is very deep. We want 8.6
to get to beta *this* year!)
Donal.
Alexandre Ferrieux
<donal.k.fell...@manchester.ac.uk> wrote:
> [...] the issue there
> relates to whether a value should have more meaning to it other than
> what is present in its string representation. The fundamental semantic
> model of Tcl 8 values says that it shouldn't, but perhaps that should be
Donal, do you have a pointer to a discussion on this (which I
interpret as "opaque handles garbage-collected in EIAS world"),
fueling the idea that it is doable at all (Unless 9.0 is the bag of
all impossible things :-) ?
-Alex
Donal K. Fellows
> Donal, do you have a pointer to a discussion on this (which I
> interpret as "opaque handles garbage-collected in EIAS world"),
> fueling the idea that it is doable at all (Unless 9.0 is the bag of
> all impossible things :-) ?
Not offhand. It's really summing up discussions I've had with people
over many years, some of which were either by private email or even in
the bar at a Tcl conference. :-)
But as policies go, this one is key to how the values work in Tcl 8.*.
Some extensions (Tcl/Java, tcom) play fast-and-loose - I've done even
worse for some apps - but this in turn means that they can sometimes
break in unexpected ways.
Donal.