Work in Progress: Tcl Compiler
Adam Sah
In the week following the "announcement" of my ongoing work on inproving
performance of Tcl scripts, I was deluged with email requesting
information. As promised, I am posting the FAQ I put together on this
work in progress.
Feel free to mail additional questions, corrections, etc. to me at:
as...@cs.Berkeley.EDU.
---cut here---
Tcl Compiler (TC) Frequently Asked Questions
----------------------------------------------------------------------
There has been a lot of excitement regarding the Tcl Compiler project
currently underway here at UC Berkeley, as announced at the Tcl'93
Workshop. This FAQ (Frequently Asked Questions) list tries to answer
some of the more common inquiries as to this software.
This list is being maintained by Adam Sah (as...@cs.Berkeley.EDU). If you
have any questions/changes/additions, please send email.
People to contact:
TC (The Tcl Compiler) - contact: Adam Sah- as...@cs.Berkeley.EDU
Tcl in general - either: Adam Sah: as...@cs.Berkeley.EDU or
John Ousterhout: ous...@cs.Berkeley.EDU.
There's also the USENET newsgroup for discussion- comp.lang.tcl
Tcl information and sources are available from:
sprite.berkeley.edu and harbor.ecn.purdue.edu
Questions
----------------------------------------------------------------------
1. How much faster is it, anyway?
2. Do I have to change anything to use it? (if I don't change anything,
does this affect performance?)
3. I heard that you're adding types into this typeless language. Is this
true? Does this mean I have to declare types? What are you doing?!
4. What is the project status? When can I get my hands on it?
5. When can I expect a reasonably stable release to exist?
6. How can I compile up commands I embed into the interpreter using C?
("C command procedures")
7. Where can I get a copy of the paper presented at the Tcl'93 Conference?
8. Will TC be as portable as Tcl?
9. Are there any #pragma-like things where I can "talk" to the compiler
directly?
10. Isn't it possible to skirt this compilation stuff and just embed it
directly into the interpreter (so you don't need .tc files)?
11. From the look of your design, object-oriented Tcl won't run nearly as
fast, and neither will Tk code. Is this true?
12. What about profiling and code coverage?
Answers
----------------------------------------------------------------------
1. How much faster is it, anyway?
First of all, the numbers in the paper (see #7) are from a CISC machine-
5-7x performance. Subsequent numbers (as presented in the actual talk
given at the Tcl'93 Conference) on RISC machines show an 11x-20x
improvement. Jon Blow (my partner) and I believe that this is due to a
decrease in non-word-aligned addressing in compiled Tcl, as a result of
less string usage. For reference on the expense of non-word-aligned
addressing, see any one of the Alpha Architecture Handbooks availabel
from DEC.
These numbers also scale. We ran tests on several Tcl primitives,
including simple ones (like 'incr'), aggregates (like 'proc' and 'for')
and implemented boolean and math expressions. Our forthcoming list
implementation should easily maintain these figures. In short, we're
pretty confident that larger mostly-Tcl scripts will exhibit these kinds
of speedups.
In the worst-case scenarios we have tried, TC always exhibited _some_
performance gain on the normal Tcl interpreter. In the paper, we show
one degenerate case where the speedup is only 14%. We consider it
important that the performance never degrade relative to Tcl and will be
working hard to maintain this.
See the paper (#7) for more details. Also see question #11 for info on a
design change, which should help out the degenerate cases.
2. Do I have to change anything to use it? (if I don't change anything,
does this affect performance?)
Short answer: nothing, and if you don't you'll probably still win.
The way the compiler works is that we break commands into arguments.
Then, if the command is one that the compiler "knows about", ti is
compiled specially, whereby the arguments themselves are preparsed. This
provides substantial savings for aggregate commands like 'for', where the
body of the loop could be many commands. A large part of the project
involves correctly implementing the existing Tcl core command set (and
then the core Tk command set).
Hopefully, by the time it's released, most of the core will be
implemented, and so you won't have to do anything, unless you want to
compile the arguments for your own C command procedures. Note that
commands created using 'proc' are automatically compiled. See #6 for
details.
It's too early to gauge whether people are going to need to compile up
their commands to achieve real performance gains.
In the meantime, your old C command procedures will still work, but
won't be "compiled", which is to say that their arguments will be passed
as strings, which require parsing, etc.
3. I heard that you're adding types into this typeless language. Is this
true? Does this mean I have to declare types? What are you doing?!
Don't sweat it! It's not that bad. What's going on is that we're
associating a data type for the arguments to known commands. For
example, we call the condition argument in 'while', 'for', etc. a
"boolexpr" so that we can reuse the same parsing and execution routines.
Unless you are really doing something fancy, you'll never run into one of
these types.
4. What is the project status? When can I get my hands on it?
At the time of the conference (6/93), the compiler was at the
proof-of-concept stage, and badly needs a re-engineering job. This
shouldn't take long, since much of the code can be saved. Nevertheless,
this is going to stall new work for a little while.
I (Adam Sah) am a student of John Ousterhout's (creator of Tcl) and I'm
using the compiler to fulfill my project requirement for my Masters
degree here at Berkeley (on the way to a PhD). My self-imposed deadline
is approximately Feb, 1994 for its completion. With luck, I'll start to
ask for beta testers at the end of the summer, 1993.
The deal with Jon Blow is that he and I concocted this project as part of
a class on backends and optimization for a grad class here at Berkeley.
Unless something changes, the project is my responsibility after the
conference, so don't bother him about it :)
Expect a general beta test only after it's fairly stable- I don't have
time to deal with hordes of complaints, etc., and the compiler is too
fundamental a tool to leave junky bugs for users to find in use.
5. When can I expect a reasonably stable release to exist?
Figure that if I can keep to my schedule, then early 1994. If you're
planning on starting a big project that involves the compiler, please let
me know so that we can arrange things beforehand (as opposed to
uncovering bugs later on...).
6. How can I compile up commands I embed into the interpreter using C?
("C command procedures")
The compiler uses a configuration script (a Tcl script, of course) to
describe which commands to compile up and what argument types they
expect. See #2 and #3 for more info.
7. Where can I get a copy of the paper presented at the Tcl'93 Conference?
ftp:
sprite.berkeley.edu:pub/tcl/tc-tcl93.tex.gz
sprite.berkeley.edu:pub/tcl/tc-tcl93.ps.gz
If you cannot use FTP at all, there is a service called "ftpmail"
available from wrl.dec.com: you can send e-mail to this machine
and it will use FTP to retrieve files for you and send you the files
back again via e-mail. To find out more about the ftpmail service,
send a message to "ftp...@wrl.dec.com" whose body consists of the
single line "help".
8. Will TC be as portable as Tcl?
Yes. The command procedures you write using the existing argc/argv
interface will work as they always did (and won't require the TC
extensions). New style command procedures will require the extensions to
be used. When the compiler is released, I'm going to devote a lot of
manual to the "upgrade path" from Tcl to TC.
TC output files (.tc files) are architecture, int-size, and endian
independent. In other words, they're _very_ portable. Any machine with
32+ bit integers/pointers will run TC happily, barring weird OS
incompatibilities. I don't know what machines it will initially support
but I will try to leave room for an MSDOS port. DEC Alphas will almost
certainly be supported from the outset.
9. Are there any #pragma-like things where I can "talk" to the compiler
directly?
Not currently. This is largely because I don't want to futz with the
existing commands, intrude on the Tcl namespace, or try to play games in
comments. Really, #pragma isn't necessary.
10. Isn't it possible to skirt this compilation stuff and just embed it
directly into the interpreter (so you don't need .tc files)?
Well, yes, sort of. The decision to separate these was an early one,
based on it being a two-person project, the easy debugging offered by an
output file, and just not realizing that anything else was possible.
The compiler isn't far enough along in development to test the real
feasibility of this. There are lots of issues involved in this:
- how can this be done without _slowing_ down the system? (you're going
back to runtime parsing, which was the big expense we claimed to save
in the first place)
- only some commands are worth doing this for, and it's not clear how to do
this.
If you're really into this issue, let me know and I'll put it higher on
my list of things to do. In the meantime, this probably won't be part of
the initial versions, since I'm pretty sure that there's enough savings
gained in a full preparse to warrant having a separate compiler. If
subsequent tests show this to be otherwise, this may change.
11. From the look of your design, object-oriented Tcl won't run nearly as
fast, and neither will Tk code. Is this true?
Well, sort of. I'm going to change the design to fix this idiocy. The
short synopsis:
Remember how the compiler uses this config script to specify the types of
arguments for given commands? Well, what do you about variable numbers
of arguments? (answer: nothing, since how can you specify types for
something like this, without getting _really_ creative) So far, all of
the OOP proposals seem to play the same games as Tk:
proc doSomethingInteresting {widget} {
$widget configure -someOption foo -otherOption bar
}
Without playing some incredible flow analysis games, the compiler is not
going to be able to track down the object type of $widget. Think of this
as pointers-to-functions in C. As a compiler-writer, you're pretty hosed.
The solution is to have the C command procedures request the arguments,
rather than shove them down their throats. Currently, the runtime
arranges that the arguments that the callbacks receive are already
preparsed. The callbacks are not supposed to do any error checking or
parsing. A better way to do this is to have the callbacks request the
arguments in specified types, and if they don't match the requested
types, automatically convert them. This would allow for OOp Tcl and
other systems which can't be compiled easily.
Interestingly, you would still want everything else to stay the same, to
allow for preparsing of arguments for more normal-behaving commands, like
'incr', rather than runtime parsing, which is what this system entails.
This idea was proposed by John Ousterhout, and will almost certainly be
included in the first release.
12. What about profiling and code coverage?
This looks like a good thing to add TC, and should be fairly
straightforward. I'm looking into adding this. By the way, an
interesting project would be to integrate this support with an editor
(highlight expensive sections of code, etc.). Don't get me wrong, I'm
not suggesting yet another GUI editor- I'm saying that we should take an
existing one (mx? TkEmacs?) and add support for reading the profiler
output, whose format is not defined yet.
---cut here---
--
Thanks again,
-A.Sah'93 ...Adam Sah...asah@cs.Berkeley.EDU...
It's not the voltage, it's the amperage.
peter da silva
> 10. Isn't it possible to skirt this compilation stuff and just embed it
> directly into the interpreter (so you don't need .tc files)?
> - how can this be done without _slowing_ down the system? (you're going
> back to runtime parsing, which was the big expense we claimed to save
> in the first place)
You only need to parse the code once, instead of multiple times. It'd probably
be a hit for one-shots. I thought of doing something along these lines, but
only applying it to selected procedures. You'd use it like this:
proc foo {bar baz} {
...
}
bind foo # Using the SmallTalk terminology.
I think bind has been taken by "tk", right? Perhaps "compile" could be used
instead. Anyway, you'd make it so the command "compile" was a no-op in the
base language and in the pure compiler version, so at worst it wouldn't help
to put it in. It would never hurt.
--
Peter da Silva `-_-'
Network Management Technology Incorporated 'U`
12808 West Airport Blvd. Sugar Land, TX 77478 USA
+1 713 274 5180 "Na sema Jambo mbwa kali yake leo?"
Steve Nuchia
>instead. Anyway, you'd make it so the command "compile" was a no-op in the
>base language and in the pure compiler version, so at worst it wouldn't help
>to put it in. It would never hurt.
You'd get a bigger win if you used compile as an alternative keyword
with the same syntax as proc but slightly different semantics.
Specifically, it would execute faster if you snapped the command
name links as early as possible rather than as late as possible.
Unfortunately this changes the semantics, so you can't make it
transparent.
--
Steve Nuchia South Coast Computing Services, Inc. (713) 661-3301
Consultancy in technical computing. Communications services. Hardware.
4-port serial card for AT bus. Public shell/internet accounts in Houston.
Larry W. Virden
> name links as early as possible rather than as late as possible.
> Unfortunately this changes the semantics, so you can't make it
> transparent.
Could you not do a rename on proc, and substitute the compile keyword?
--
:s
:s Larry W. Virden INET: lvi...@cas.org
:s Personal: 674 Falls Place, Reynoldsburg, OH 43068-1614
Steve Nuchia
>Could you not do a rename on proc, and substitute the compile keyword?
Yes, but you don't want to unless the semantics are identical.