Symbol macros don't capture their lexical environments?
Ron Garret
(defmacro m0 () :foo)
(macrolet ((m1 () (m0)))
(macrolet ((m0 () :baz))
(macrolet ((m2 () (m0)))
(list (m1) (m2)))))
This returns (:foo :baz). However, if you do the analogous thing with
symbol macros:
(symbol-macrolet ((s1 (m0)))
(macrolet ((m0 () :baz))
(symbol-macrolet ((s2 (m0)))
(list s1 s2))))
you get (:baz :baz) on CCL, SBCL and CLisp.
Is this a bug in all three of these implementations, or is this actually
the correct behavior?
rg
Christophe Rhodes
The expansion of a regular macro call is the result of calling the macro
function on the macro form and the environment, which itself is the
value of the last form in the macro expansion's body. The expansion of
a symbol macro is the result of calling the associated macro function on
the symbol and the environment, which itself is the form specified as
the symbol macro's expansion (and here might be the source of your
confusion: not the value returned when calling that form).
Christophe
Jason Cornez
> (macrolet ((m1 () (m0)))
> (macrolet ((m0 () :baz))
> (macrolet ((m2 () (m0)))
> (list (m1) (m2)))))
I think you probably mean
(macrolet ((m1 () '(m0)))
(macrolet ((m0 () :baz))
(macrolet ((m2 () '(m0)))
(list (m1) (m2)))))
Because macros typically expand into forms. Symbol macros are defined
differently.
Same is true for global macros vs symbol macros.
Compare
(defmacro rg () '(list 'hello 'ron))
with
(define-symbol-macro rg (list 'hello 'ron))
-Jason
Thomas F. Burdick
This is the correct behavior. The right hand side of the binding is
not a form that evaluates to the expansion, it *is* the expansion.
IMHO, the correct use of symbol macros is as a thin layer of syntactic
sugar to convert what looks like a variable reference into what looks
like a function call. The meaningful transformations belong in the
macrolet side of things.
Thomas F. Burdick
mentioned the relationship between dynamic and lexical scopes. I guess
the chance of a civil discussion is higher here, so I'll try to reply
in this thread.
You're right that there is a tight connection, and you're also right
that others have noticed it before. From your surprise at it, I guess
you haven't ever written a compiler[*] (using Lisp as the language the
compiler is written in)? The way I like to phrase it is, the dynamic
contours of a compiler recapitulate the lexical contours of the
program being compiled. That's very different than the situation with
interpreters. There, the dynamic contours of the interpreter are the
dynamic contours of the program being interpreted.
And the result of this is that languages with dynamic variables make
simple compilers so easy to implement, it's fun. It's analagous to
call/cc: if you're disciplined, you can pass around and invoke
continuation functions in any old lexical lisp, so it's always
possible, but call/cc makes it easy to capture continuations. (This is
not an endorsement of call/cc, which I happen to think is a disaster
of a language construct).
[*] That's not a diss, just a guess, and if I'm wrong, no offense
intended. I happen to think that compilers are a very useful technique
that should be in all working programmers' box of tricks, but I'm so
far in the minority on that point that fighting the fight seems a bit
like my opponent is all the windmills of the world. So while I'd like
to think that most competant programmers have implemented little
compilers, and that they may have gotten certain insights from the
experience or maybe not, the reality is that if you did not have such
an insight, it's far more likely that it's because you haven't written
a compiler before. But I'll omitt that rant.
Ron Garret
<e6701c17-660b-46a0...@w19g2000yqk.googlegroups.com>,
"Thomas F. Burdick" <tbur...@gmail.com> wrote:
> You're right that there is a tight connection, and you're also right
> that others have noticed it before. From your surprise at it, I guess
> you haven't ever written a compiler[*]
Not a "real" one. I've written a lot of "toy" compilers, I've read a
fair number of books and papers about them, and once a long time ago
helped to port a "real" compiler (the T compiler) to a new operating
system. But I've never written a real compiler from scratch.
> The way I like to phrase it is, the dynamic
> contours of a compiler recapitulate the lexical contours of the
> program being compiled. That's very different than the situation with
> interpreters. There, the dynamic contours of the interpreter are the
> dynamic contours of the program being interpreted.
>
> And the result of this is that languages with dynamic variables make
> simple compilers so easy to implement, it's fun. It's analagous to
> call/cc: if you're disciplined, you can pass around and invoke
> continuation functions in any old lexical lisp, so it's always
> possible, but call/cc makes it easy to capture continuations. (This is
> not an endorsement of call/cc, which I happen to think is a disaster
> of a language construct).
That makes sense. Thanks!
> [*] That's not a diss,
I didn't take it as one.
> just a guess, and if I'm wrong, no offense
> intended. I happen to think that compilers are a very useful technique
> that should be in all working programmers' box of tricks, but I'm so
> far in the minority on that point that fighting the fight seems a bit
> like my opponent is all the windmills of the world.
I know that feeling!
> So while I'd like
> to think that most competant programmers have implemented little
> compilers, and that they may have gotten certain insights from the
> experience or maybe not, the reality is that if you did not have such
> an insight, it's far more likely that it's because you haven't written
> a compiler before. But I'll omitt that rant.
What is the best reference for getting the full scoop on the
lexical/dynamic binding connection?
rg
Ron Garret
<f45c99f3-0bf1-44af...@37g2000yqm.googlegroups.com>,
"Thomas F. Burdick" <tbur...@gmail.com> wrote:
> On Nov 19, 11:37 am, Ron Garret <rNOSPA...@flownet.com> wrote:
> > Consider the following code:
> >
> > (defmacro m0 () :foo)
> >
> > (macrolet ((m1 () (m0)))
> > (macrolet ((m0 () :baz))
> > (macrolet ((m2 () (m0)))
> > (list (m1) (m2)))))
> >
> > This returns (:foo :baz). However, if you do the analogous thing with
> > symbol macros:
> >
> > (symbol-macrolet ((s1 (m0)))
> > (macrolet ((m0 () :baz))
> > (symbol-macrolet ((s2 (m0)))
> > (list s1 s2))))
> >
> > you get (:baz :baz) on CCL, SBCL and CLisp.
> >
> > Is this a bug in all three of these implementations, or is this actually
> > the correct behavior?
>
> This is the correct behavior. The right hand side of the binding is
> not a form that evaluates to the expansion, it *is* the expansion.
I understand that. But that expansion is subject to further expansion.
For regular macros, that expansion occurs in the lexical scope of the
macro definition. For symbol macros that expansion seems to occur in
the lexical scope of the invocation, which seems wrong to me. But I
have not dug into to the spec to figure out if it actually *is* wrong.
rg
Pascal Costanza
(define-symbol-macro foo (bar)) is roughly equivalent to
(defmacro foo () '(bar))
In other words, whatever is the definition of a symbol macro is
implicitly quoted. You cannot express a computation as part of a symbol
macro definition, so whatever is the definition of a symbol macro is not
subject to macroexpansion itself.
If you need the result of a symbol macro be the result of a computation,
the trick is to define it alongside a 'real' macro, like this:
(defmacro foo () ... compute a result ...)
(define-symbol-macro foo (foo))
Likewise for local macros:
(macrolet ((foo () ... compute a result ...))
(symbol-macrolet ((foo (foo)))
...))
If you want to avoid accidental capture here, use the usual trick and
use a generated symbol for the name of the 'real' macro.
Pascal
--
My website: http://p-cos.net
Common Lisp Document Repository: http://cdr.eurolisp.org
Closer to MOP & ContextL: http://common-lisp.net/project/closer/
Kaz Kylheku
> Consider the following code:
Of course symbol macros don't capture their lexical environment.
You just have to redefine what ``lexical environment means''.
First, take a big black marker and strike out the glossary entry under L.
Then define lexical environment as ``something that contains only variable
bindings''.
Q.E.D. symbol macro bindings are not variable bindings; consequently, they
are not part of the lexical environment.
What is thought? Thought is simply word manipulation.
Tobias C. Rittweiler
> Consider the following code:
>
> (defmacro m0 () :foo)
>
> (macrolet ((m1 () (m0)))
> (macrolet ((m0 () :baz))
> (macrolet ((m2 () (m0)))
> (list (m1) (m2)))))
>
> This returns (:foo :baz).
Actually, that's not completely portable code. DEFMACRO may only store a
macro's definition in the /compilation environment/ which may be
distinct from the /evaluation environment/ where the actual macro
expander is evaluated. Cf. CLHS 3.2.3.1.1 and 3.2.1.
-T.
Ron Garret
Pascal Costanza <p...@p-cos.net> wrote:
I don't think that's true. If it were, the following:
(macrolet ((foo () :expanded))
(symbol-macrolet ((s (foo)))
s))
would return (foo) but in fact it returns :expanded.
In fact, you can use this to define a version of symbol-macrolet that
does the RIght Thing (simplified to allow only one symbol to be
macro-bound at a time in order to illustrate the principle):
(defmacro symbol-macrolet-that-does-the-right-thing
(symbol expansion &body body)
(let ((macro-name (gensym (symbol-name symbol))))
`(macrolet ((,macro-name () ,expansion))
(symbol-macrolet ((,symbol (,macro-name)))
,@body))))
Demo:
? (macrolet ((m1 () :level-1))
(symbol-macrolet-that-does-the-right-thing s (m1)
(macrolet ((m1 () :level-2))
s)))
:LEVEL-1
rg
Ron Garret
Huh? I don't see where that code relies on storing a macro's code in
anything but the compilation environment.
rg
Pascal Costanza
Yes, of course. 's expands into '(foo), which in turn gets expanded into
':expanded. However, the expansion of '(foo) into ':expanded happens
after expansion of 's is completed, not as part of expanding 's itself.
Consider the following case:
(macrolet ((foo () :expanded))
(macrolet ((bar () (foo)))
(bar)))
Here, '(bar) invokes the macro definition for 'bar whose body was
already expanded before, so the macro definition already contains just
':expanded. That's what I meant with "express computation as part of a
macro definition": The definition of a 'normal' macro is itself a
computation, whereas for a symbol macro, it's not.
Rob Warnock
+---------------
| "Thomas F. Burdick" <tbur...@gmail.com> wrote:
| > The way I like to phrase it is, the dynamic
| > contours of a compiler recapitulate the lexical contours of the
| > program being compiled. That's very different than the situation with
| > interpreters. There, the dynamic contours of the interpreter are the
| > dynamic contours of the program being interpreted.
That's clever, but not *quite* accurate. For an interpreter which
is interpreting a language with lexical scope [such as Scheme or the
lexically-scoped parts of CL], the dynamic contours of the interpreter
are *also* the lexical contours of the program being being interprested,
only the interpreter traces said contours over & over & over again, ;-}
not just once as with a compiler.
+---------------
| > And the result of this is that languages with dynamic variables make
| > simple compilers so easy to implement, it's fun.
+---------------
That's true enough. Just make your compiler's or interpreter's lexical
environment object be a global special, then LET-bind it to an extended
environment every time you enter a new lexical contour in the target
program. The automatic restoration of the global special on each LET
block exit will automagically prune the environment object.
+---------------
| What is the best reference for getting the full scoop on the
| lexical/dynamic binding connection?
+---------------
My favorite reference for such things is still Queinnec's "L.i.S.P.".
Though, as Thomas suggests, starting to sketch out the code for a small
compiler [or even an interpreter for a lexically-scoped language] should
quickly show you the parallel he's trying to make.
Even simpler, consider a code-walker that just wants to generate a
sorted cross-reference listing and show which symbols are used for
what & where. It's going to need to do a recursive traversal of the
code during which it will be building up and tearing down (or simply
abandoning to be GC'd) the lexical environments of the code that it's
walking. Dynamic variables are very convenient for holding the current
state of symbol->type/use mappings during said walk, especially if
the mappping is an a-list that's simply pushed onto as needed [and
auto-reverts when blocks are exited]. A common pattern is:
(let ((*state* *state*))
...do stuff...
(when (found-something-interesting)
(push interesting-thing *state*))
...do more stuff based on now-current state...
) ; done: *state* auto-reverts
Or maybe:
(let ((*state* (cons (make-new-stuff ...) *state*)))
...
) ; done: *state* auto-reverts
or:
(let ((*state* (extend-state *state* stuff...)))
...
) ; done: *state* auto-reverts
The point is that a trace of the dynamic state of *state* will be
more-or-less isomorphic to the static lexical contours of the code
being walked.
-Rob
-----
Rob Warnock <rp...@rpw3.org>
627 26th Avenue <URL:http://rpw3.org/>
San Mateo, CA 94403 (650)572-2607
Tobias C. Rittweiler
You call M0 during macro-expansion of M1 and M2. Evaluation of macro
expanders is performed within the /evaluation environment/ which, as
said, may be distinct from the /compilation environment/. For that to
work portably, you have to wrap the (DEFMACRO M0) in an EVAL-WHEN.
It's shown with example in 3.2.3.1.1 in combination with 3.2.1 to get
the subtle points and consequences right.
-T.
Ron Garret
"Tobias C. Rittweiler" <t...@freebits.de.invalid> wrote:
> Ron Garret <rNOS...@flownet.com> writes:
>
> > In article <87skcaq...@freebits.de>,
> > "Tobias C. Rittweiler" <t...@freebits.de.invalid> wrote:
> >
> > > Ron Garret <rNOS...@flownet.com> writes:
> > >
> > > > Consider the following code:
> > > >
> > > > (defmacro m0 () :foo)
> > > >
> > > > (macrolet ((m1 () (m0)))
> > > > (macrolet ((m0 () :baz))
> > > > (macrolet ((m2 () (m0)))
> > > > (list (m1) (m2)))))
> > > >
> > > > This returns (:foo :baz).
> > >
> > > Actually, that's not completely portable code. DEFMACRO may only store a
> > > macro's definition in the /compilation environment/ which may be
> > > distinct from the /evaluation environment/ where the actual macro
> > > expander is evaluated. Cf. CLHS 3.2.3.1.1 and 3.2.1.
> >
> > Huh? I don't see where that code relies on storing a macro's code in
> > anything but the compilation environment.
>
> You call M0 during macro-expansion of M1 and M2. Evaluation of macro
> expanders is performed within the /evaluation environment/ which, as
> said, may be distinct from the /compilation environment/. For that to
> work portably, you have to wrap the (DEFMACRO M0) in an EVAL-WHEN.
Only if you're compiling the code in a file. If you enter the code in
an interactive REPL it will work just fine.
Better yet, just change the example to this:
(macrolet ((m0 () :foo))
(macrolet ((m1 () (m0)))
(macrolet ((m0 () :baz))
(macrolet ((m2 () (m0)))
(list (m1) (m2))))))
All this seems like a bit of a nit w.r.t. the point I'm trying to make.
rg
Ron Garret
Pascal Costanza <p...@p-cos.net> wrote:
Before what? You could mean "before further expansion" (which is
correct) or you could mean "before BAR is ever invoked" (which is not).
It appears that you mean the latter when you go no to say:
> so the macro definition already contains just ':expanded.
No, that's not possible. Unless you have a compiler that can solve the
halting problem, there's no way for it to know that (BAR) will always
yield the same result. The example just happens to be one particular
case where you can tell, but that's easy to fix:
(defun foo-frob () ...) ; Happens to return :expanded
(macrolet ((foo () (foo-frob)))
(macrolet ((bar () (foo)))
(bar)))
FOO-FROB might just happen to return :expanded every time it is called,
but the compiler can't know that. So in general it has no choice but to
expand BAR when it is invoked, not when it is defined.
> That's what I meant with "express computation as part of a
> macro definition": The definition of a 'normal' macro is itself a
> computation, whereas for a symbol macro, it's not.
Yes, I understand all that, but you're still missing the point, which is
that the *further* macro-expansion of regular macros takes place in the
lexical environment of the macro's *definition*, whereas the further
expansion of symbol macros takes place in the lexical environment of the
(symbol) macro's *invocation*. This has nothing to do with how the
initial expansion is produced, and though I have not yet had a chance to
look, it appears more and more doubtful to me that this difference in
behavior is justified by the standard.
rg
Tobias C. Rittweiler
> All this seems like a bit of a nit w.r.t. the point I'm trying to make.
I'm sorry I somehow missed you were trying to make a point.
-T.
Pascal Costanza
I wasn't aware of these distinctions, and I find them weird. What is the
motivation for this? Under which circumstances does it make sense that
the compilation and the evaluation environment is not the same?
Ron Garret
"Tobias C. Rittweiler" <t...@freebits.de.invalid> wrote:
Did you not read the subject line of the thread?
rg
Pascal Costanza
Of course, it is. The '(foo) in the definition of bar is an invocation
of a macro, and it gets expanded when the definition of bar is compiled.
This happens before bar is ever invoked (and even when it's not invoked).
(It may not happen in interpreted code, though.)
Ron Garret
Pascal Costanza <p...@p-cos.net> wrote:
Oh, I think I get it now. It's not that "further macroexpansion"
happens in the lexical environment of the macro definition, it's that
the expansion function is defined in the lexical environment of the
macro definition (duh!). So this:
(macrolet ((m0 () :foo))
(macrolet ((m1 () (m0)))
(macrolet ((m0 () :baz))
(m1))))
returns :FOO but this:
(macrolet ((m0 () :foo))
(macrolet ((m1 () '(m0)))
(macrolet ((m0 () :baz))
(m1))))
return :BAZ.
Sorry for being obtuse.
rg
Pascal Costanza
Yep. And symbol-macrolet implicitly quotes its definition such that
(symbol-macrolet ((m1 (m0))) ...) is similar to the second example.
> Sorry for being obtuse.
No problem.
Tobias C. Rittweiler
Ron Garret
"Tobias C. Rittweiler" <t...@freebits.de.invalid> wrote:
> Ron Garret <rNOS...@flownet.com> writes:
>
> > In article <87ws1lg...@freebits.de>,
> > "Tobias C. Rittweiler" <t...@freebits.de.invalid> wrote:
> >
> > > Ron Garret <rNOS...@flownet.com> writes:
> > >
> > > > All this seems like a bit of a nit w.r.t. the point I'm trying to make.
> > >
> > > I'm sorry I somehow missed you were trying to make a point.
> >
> > Did you not read the subject line of the thread?
>
> Displaying confusion is not the same as making a point.
That's true, but it can be a side-effect of *trying* to make a point.
And your point would be...?
rg
Tobias C. Rittweiler
> In article <87pr7dg...@freebits.de>,
> "Tobias C. Rittweiler" <t...@freebits.de.invalid> wrote:
>
> > Ron Garret <rNOS...@flownet.com> writes:
> >
> > > In article <87ws1lg...@freebits.de>,
> > > "Tobias C. Rittweiler" <t...@freebits.de.invalid> wrote:
> > >
> > > > Ron Garret <rNOS...@flownet.com> writes:
> > > >
> > > > > All this seems like a bit of a nit w.r.t. the point I'm trying to make.
> > > >
> > > > I'm sorry I somehow missed you were trying to make a point.
> > >
> > > Did you not read the subject line of the thread?
> >
> > Displaying confusion is not the same as making a point.
>
> That's true, but it can be a side-effect of *trying* to make a point.
>
> And your point would be...?
I tried to shed some light on another confusion that I didn't see other
people to address.
-T.
Tobias C. Rittweiler
> Tobias C. Rittweiler wrote:
>
> > Ron Garret <rNOS...@flownet.com> writes:
> >
> > > Consider the following code:
> > >
> > > (defmacro m0 () :foo)
> > >
> > > (macrolet ((m1 () (m0)))
> > > (macrolet ((m0 () :baz))
> > > (macrolet ((m2 () (m0)))
> > > (list (m1) (m2)))))
> > >
> > > This returns (:foo :baz).
> >
> > Actually, that's not completely portable code. DEFMACRO may only store a
> > macro's definition in the /compilation environment/ which may be
> > distinct from the /evaluation environment/ where the actual macro
> > expander is evaluated. Cf. CLHS 3.2.3.1.1 and 3.2.1.
>
> I wasn't aware of these distinctions, and I find them weird. What is
> the motivation for this? Under which circumstances does it make sense
> that the compilation and the evaluation environment is not the same?
The issue
COMPILE-FILE-HANDLING-OF-TOP-LEVEL-FORMS
names VaxLisp in Current Practise as an example where that was the case,
and it also mentions controversy about the issue.
What I find most puzzling is that it additonally says that the
compilation environment inherits from the evaluation environment.
I guess this bit is necessary to portably allow DECLAIM to be seen as a
macro expanding to
(EVAL-WHEN (:COMPILE-TOPLEVEL :LOAD-TOPLEVEL :EXECUTE)
(PROCLAIM '(..declarations..)))
I wonder what VaxLisp had done with that. Anyone who knows? :-)
-T.
Duane Rettig
>
> >> Consider the following code:
>
> >> (defmacro m0 () :foo)
>
> >> (macrolet ((m1 () (m0)))
> >> (macrolet ((m0 () :baz))
> >> (macrolet ((m2 () (m0)))
> >> (list (m1) (m2)))))
>
> >> This returns (:foo :baz).
>
> > Actually, that's not completely portable code. DEFMACRO may only store a
> > macro's definition in the /compilation environment/ which may be
> > distinct from the /evaluation environment/ where the actual macro
> > expander is evaluated. Cf. CLHS 3.2.3.1.1 and 3.2.1.
>
> I wasn't aware of these distinctions, and I find them weird. What is the
> motivation for this? Under which circumstances does it make sense that
> the compilation and the evaluation environment is not the same?
This was discussed at length a few years back. A group search turns
up this thread:
which seems to have come from this conversation:
which may have come from this:
I'm not sure I want to get my head back into it at this time; it took
a lot of mindshare at the time to hold the nuances of the concepts,
though at that time it was easy because I was working on the
Environments Access module at the time.
Duane
Pascal Costanza
Are you really sure that these sections are relevant here, even if this
is file-compiled?
The HyperSpec entry for macrolet says this: "The macro-expansion
functions defined by macrolet are defined in the lexical environment in
which the macrolet form appears. Declarations and macrolet and
symbol-macrolet definitions affect the local macro definitions in a
macrolet, [...]"
This means that all invocations of m0, m1 and m2 should be expanded
correctly, independent of where the definitions are stored.
Right?
Ron Garret
Pascal Costanza <p...@p-cos.net> wrote:
> Tobias C. Rittweiler wrote:
> > Ron Garret <rNOS...@flownet.com> writes:
> >
> >> Consider the following code:
> >>
> >> (defmacro m0 () :foo)
> >>
> >> (macrolet ((m1 () (m0)))
> >> (macrolet ((m0 () :baz))
> >> (macrolet ((m2 () (m0)))
> >> (list (m1) (m2)))))
> >>
> >> This returns (:foo :baz).
> >
> > Actually, that's not completely portable code. DEFMACRO may only store a
> > macro's definition in the /compilation environment/ which may be
> > distinct from the /evaluation environment/ where the actual macro
> > expander is evaluated. Cf. CLHS 3.2.3.1.1 and 3.2.1.
>
> Are you really sure that these sections are relevant here, even if this
> is file-compiled?
>
> The HyperSpec entry for macrolet says this: "The macro-expansion
> functions defined by macrolet are defined in the lexical environment in
> which the macrolet form appears. Declarations and macrolet and
> symbol-macrolet definitions affect the local macro definitions in a
> macrolet, [...]"
>
> This means that all invocations of m0, m1 and m2 should be expanded
> correctly, independent of where the definitions are stored.
>
> Right?
No.
This turns out to be a very deep rabbit hole (as I am in the process of
discovering even as I write this). Duane just posted three links to
previous discussions about this. I strongly recommend going back and
re-reading those, otherwise we could be here a while.
To summarize the issue as succinctly as I can: the problem is that when
you write:
(macrolet ((m1 () (m0)) ...
instead of the more traditional
(macrolet ((m1 () `(m0)) ...
the expansion of m1 necessarily (according to the spec) takes place in
the evaluation environment, not the compilation environment. But when
file-compiling, DEFMACRO is only required to make the macro definition
available in the compilation environment.
This may not be completely correct. The issue apparently is very
complicated and contains many subtleties.
rg
Ron Garret
<0a26d596-f548-4fe8...@o9g2000prg.googlegroups.com>,
Duane Rettig <du...@franz.com> wrote:
I hope I can convince you to get your head back into it just far enough
to answer one question. The following example figured prominently in
your exposition of the issues back then:
(compile
(defun foo (x)
(macrolet ((bar ()
(setq *spy* x)))
(bar))))
but all implementations I tried this in (CCL, CLisp and SBCL) give an
error along the lines of that it's illegal to reference a lexically
bound variable in a macro expansion.
And yet you posted a transcript of running that code in CLisp without
generating an error, and instead "leaking" internal compiler information.
My question is: why was that not simply a bug in the version of CLisp
that you were using?
rg
Kaz Kylheku
> (compile
> (defun foo (x)
> (macrolet ((bar ()
> (setq *spy* x)))
> (bar))))
>
> but all implementations I tried this in (CCL, CLisp and SBCL) give an
> error along the lines of that it's illegal to reference a lexically
> bound variable in a macro expansion.
So, Common Lisp has a lexically-aware macro system that takes
care of a significant hygiene issue.
The problem with the reference is that the macro may be expanded anywhere
within the scope where it is visible, meaning that its body, and the
references it contains, are subject to code motion. The places where the same
macro bar is visible do not necessarily coincide with the places where the same
x is visible.
A hygienic macro system would just let you use x, an that x would refer
to the lexically visible x at the point of definition, no matter where
the body is pulled by expansions.
A non-hygienic macro system would just let you expand (bar) anywhere bar is in
scope, allowing x to refer to whatever x refers to at the expansion site.
Clearly, whereas DEFMACRO might be of the latter type, CL's lexical macros are
of neither kind.
People can now officially STFU about no macro hygiene in Lisp. :)
Ron Garret
Kaz Kylheku <kkyl...@gmail.com> wrote:
> On 2009-11-21, Ron Garret <rNOS...@flownet.com> wrote:
> > (compile
> > (defun foo (x)
> > (macrolet ((bar ()
> > (setq *spy* x)))
> > (bar))))
> >
> > but all implementations I tried this in (CCL, CLisp and SBCL) give an
> > error along the lines of that it's illegal to reference a lexically
> > bound variable in a macro expansion.
>
> So, Common Lisp has a lexically-aware macro system that takes
> care of a significant hygiene issue.
Actually, I'm pretty sure the spec simply says that the behavior is
undefined.
> The problem with the reference is that the macro may be expanded anywhere
> within the scope where it is visible, meaning that its body, and the
> references it contains, are subject to code motion. The places where the same
> macro bar is visible do not necessarily coincide with the places where the
> same x is visible.
No, visibility is not the problem. The problem is that X might not be
bound when BAR is expanded. Or, more to the point of the original
example (as far as I understand it), X might be bound by the file
compiler to some weird internal thingy.
> A hygienic macro system would just let you use x, an that x would refer
> to the lexically visible x at the point of definition, no matter where
> the body is pulled by expansions.
Yes, you can actually do that in CL with a little symbol-macro magic :-)
> People can now officially STFU about no macro hygiene in Lisp. :)
What, and give up all the fun? Never! :)
rg
Kenneth Tilton
Don't you mean "the correct"? You fucked up, admit it*, and move on.
kt
* What am I thinking?? You admit anything?!!! As for move on..oh, my.
Ron Garret
Kenneth Tilton <kent...@gmail.com> wrote:
No. Which one is "correct" depends on what you're trying to do.
> You fucked up, admit it*, and move on.
>
> kt
>
> * What am I thinking?? You admit anything?!!! As for move on..oh, my.
Why is it so important to you to get me to admit I made a mistake? And
why pounce on this stupid little bit of throwaway example code to make
your stand? What exactly is your problem, Kenneth Tilton? Don't you
have anything better to do than to try to show me up?
rg
Duane Rettig
> In article
> <0a26d596-f548-4fe8-9db7-61d26d324...@o9g2000prg.googlegroups.com>,
> Duane Rettig <du...@franz.com> wrote:
>
> > On Nov 20, 9:34 am, Pascal Costanza <p...@p-cos.net> wrote:
> > > I wasn't aware of these distinctions, and I find them weird. What is the
> > > motivation for this? Under which circumstances does it make sense that
> > > the compilation and the evaluation environment is not the same?
>
> > This was discussed at length a few years back. A group search turns
> > up this thread:
>
> >http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/d7...
> > 11665/79b63eed465b058c#79b63eed465b058c
>
> > which seems to have come from this conversation:
>
> >http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/17...
> > 0f113/ba7077ce7ecb5498?lnk=gst&q=compilation+environment#ba7077ce7ecb5498
>
> > which may have come from this:
>
> >http://groups.google.com/group/comp.lang.lisp/browse_thread/thread/60...
> > d7dd0/26a35ae73a5a69f0?lnk=gst&q=compilation+environment#26a35ae73a5a69f0
>
> > I'm not sure I want to get my head back into it at this time; it took
> > a lot of mindshare at the time to hold the nuances of the concepts,
> > though at that time it was easy because I was working on the
> > Environments Access module at the time.
>
> > Duane
>
> I hope I can convince you to get your head back into it just far enough
> to answer one question.
Oh, alright; you've hooked me, but probably only for a short time.
I've reread the three threads I found and reminded myself of what the
general issues were. Note that the three threads are indeed in
reverse chronological order, but the third (earliest) one isn't as
directly related to the first (most recent) two threads as I had
thought when I first skimmed through them.
The following example figured prominently in
> your exposition of the issues back then:
Actually, only the first (most recent) thread. I didn't see any
references to that particular code in the middle thread, though I did
see a reference to Peter Seibel's original code (which first appeared
at the beginning of the middle thread) in the most recent thread, as I
was explaining the reason for my answers. But let's look at what
you've written here:
> (compile
> (defun foo (x)
> (macrolet ((bar ()
> (setq *spy* x)))
> (bar))))
>
> but all implementations I tried this in (CCL, CLisp and SBCL) give an
> error along the lines of that it's illegal to reference a lexically
> bound variable in a macro expansion.
>
> And yet you posted a transcript of running that code in CLisp without
> generating an error, and instead "leaking" internal compiler information.
>
> My question is: why was that not simply a bug in the version of CLisp
> that you were using?
This is a great question, and the very reason why I had brought up
this code. In the most recent thread I had linked to, most of the
participants were taking a complex concept and confusing some aspects
of those concepts by conflating the two basic ANS concepts of "The
consequences are undefined/unspecified" and "An error is/should-be
signaled" (paraphrases of several similar items in 1.4.2 of the
spec). The difference between the two groups of "Error Terminology"
is that the former set places restrictions on the (conforming)
implementation, and the latter places restrictions on the programmer
(of a conforming program). The confusion comes when one writes
nonconforming code and then calls the behavior of a single
implementation a bug, especially when all other implementations
perform a particular way that might be popular. The thread
established that the above code was nonconforming; specifically there
is a "consequences are undefined" phrase applicable in that
situation. The parallel question you might ask here is "why would an
implementation behave that way, when all other implementations are
doing it another way?" and the answer is: "because it can; the code is
nonconforming and the consequences are undefined".
Note also that I didn't stop at trying to wrench people away from the
fallacious thinking about who is responsible for what conformance; I
was also trying to get people to think very slightly like
implementors, so that they could understand why code was resulting
(usually) in only one of two different behaviors - my argument was and
is that although we love to invoke the hyperbole when faced with "the
consequences are undefined" to include the cpu turning off your
building's alarm codes or executing the hcf instruction, but in
reality the more likely behaviors are taken from a very small set,
usually only one or two behaviors.
It is clear that the Clisp developers decided to change the behavior
of Clisp. Why might they have done this? Because (wait for it ...)
they can! - the consequences are undefined in this situation!
Duane
Ron Garret
<d3d917c1-e992-4a86...@h40g2000prf.googlegroups.com>,
Duane Rettig <du...@franz.com> wrote:
Thanks for taking the time to respond, I really appreciate it. But I
think that answer is a little facile, particularly in light of the HCF
hyperbole that you yourself brought up. Let me put my question more
precisely: why should leaking compiler information in response to a
lexical reference in a macro definition be considered reasonable
behavior? Why is that *not* like calling HCF or crashing or setting off
the nukes?
Let me give an example of the kind of answer I'm looking for. One might
ask why generating an error in this case is reasonable behavior, and the
answer is: because the macroexpansion function may have to be called at
macroexpansion time, but at that time the lexical variable which the
macroexpansion function closes over may not be bound, particularly if
you are file-compiling.
The kind of answer I'm looking for would go (I'm guessing) something
like, "Because when implementing a compiler you have to deal with X and
Y and Z, and one way to deal with those things happens to produce this
behavior." What I'm really trying to get at here is not the
idiosyncrasies of an old version of CLisp, but finding out what X and Y
and Z are.
rg
Duane Rettig
> Thanks for taking the time to respond, I really appreciate it.
You're welcome.
> But I
> think that answer is a little facile, particularly in light of the HCF
> hyperbole that you yourself brought up.
Not at all; it takes a lot of serious thought to fight the urge toward
hyperbole. It also takes a lot of serious thought to ask an
unanswerable question, in order to get people to think about what
they're assuming.
Let me put my question more
> precisely: why should leaking compiler information in response to a
> lexical reference in a macro definition be considered reasonable
> behavior? Why is that *not* like calling HCF or crashing or setting off
> the nukes?
You may get squeamish if someone starts bleeding, and your first
response might be "get a band-aid on that". But someone else might
say "what color is the blood? if it's white we need antibacterial
medication first." or "is it bleeding profusely? - a band-aid might
not help - we may need a tourniquet". i.e. it's not just a case of
being a catastrophe if you see someone bleeding; the blood itself
might carry information. In the same way, a program that bleeds
internals might seem nasty to someone who sees it, and the response is
"Fix that!" - but to someone else it might be something that says "oh,
yeah, we've got information crossing the boundary between compilation
and evaluation environments, and this is what we can do with this
info".
> Let me give an example of the kind of answer I'm looking for. One might
> ask why generating an error in this case is reasonable behavior, and the
> answer is: because the macroexpansion function may have to be called at
> macroexpansion time, but at that time the lexical variable which the
> macroexpansion function closes over may not be bound, particularly if
> you are file-compiling.
But the variable _is_ bound, according to CL's definition. It just
isn't bound to a _value_ and the binding isn't a lexical binding, but
a compiler binding. So perhaps a better way of stating a case for
generating the error is "because not generating the error is messy."
And as you have seen, this particular example is apparently answered
by all known CL implementations in the same way; they do indeed
generate the error. My whole point is that no implementation _must_
generate an error. Instead there might be useful information that can
be dealt with by not generating the error, but by letting such
information leak from one kind of environment to another (e.g.
transferring bindings from compilation to evaluation environments,
either with some sort of translation or even directly).
But note again that for this particular example all of the
implementations seem to agree that the information is not so useful.
> The kind of answer I'm looking for would go (I'm guessing) something
> like, "Because when implementing a compiler you have to deal with X and
> Y and Z, and one way to deal with those things happens to produce this
> behavior." What I'm really trying to get at here is not the
> idiosyncrasies of an old version of CLisp, but finding out what X and Y
> and Z are.
I don't have that kind of answer for this example; obviously compilers
didn't have to deal with this particular example, since they all
generate an error. That's not why I brought it up; my intention for
bringing it up was not to answer it, but to jolt others into realizing
that it is a hard question (and to jolt people out of their
assumptions about the other example in the previous thread, as well).
Duane
Ron Garret
<496f4d65-e0ab-4079...@j9g2000prh.googlegroups.com>,
Duane Rettig <du...@franz.com> wrote:
Hm... this is getting awfully Zen.
It seems to me that this all hinges on how broadly one interprets the
definition of "binding". That definition is "an association between a
name and that which the name denotes." So if one were to interpret this
definition broadly, one could write this:
(setf *bindings* '((x . 1) (y . 2)))
and say that X and Y are bound. Would you agree with that?
The problem is that you can push this to an extreme. For example:
(setf *vars* '(x y))
(setf *vals* '(1 2))
Are X and Y bound now? What about this:
(setf l1 '(x y))
(setf l2 '(1 2))
In general whether a symbol is bound depends on some convention. If I
interpret *bindings* as an alist, or *vars* and *vals* (or l1 and l2) as
somehow associated with each other, then X and Y are bound, otherwise
they are not.
On the other hand, if I write (defvar x 1) then X is unambiguously bound
because it has a binding according to a convention that is specifically
described in the standard: it has a dynamic binding in the variable
namespace (and accordingly that binding has certain properties on which
one can rely).
So, O Zen master Duane, if you will allow this humble student one more
question: these "compiler bindings" of which you speak, are they
bindings of the first or the second sort? i.e. is a "compiler binding"
a binding according to a convention that is specified in the standard
(and if so, where?) or is it a binding according to some unspecified
convention that is specific to a particular compiler implementation?
And if the latter is the case, how can you say so confidently that X
_is_ bound? Would it not be more correct to say that it *might* be
bound (to some compiler-specific thingy)?
rg
Duane Rettig
> > > The kind of answer I'm looking for would go (I'm guessing) something
> > > like, "Because when implementing a compiler you have to deal with X and
> > > Y and Z, and one way to deal with those things happens to produce this
> > > behavior." What I'm really trying to get at here is not the
> > > idiosyncrasies of an old version of CLisp, but finding out what X and Y
> > > and Z are.
>
> > I don't have that kind of answer for this example; obviously compilers
> > didn't have to deal with this particular example, since they all
> > generate an error. That's not why I brought it up; my intention for
> > bringing it up was not to answer it, but to jolt others into realizing
> > that it is a hard question (and to jolt people out of their
> > assumptions about the other example in the previous thread, as well).
>
> Hm... this is getting awfully Zen.
:-)
> It seems to me that this all hinges on how broadly one interprets the
> definition of "binding". That definition is "an association between a
> name and that which the name denotes." So if one were to interpret this
> definition broadly, one could write this:
>
> (setf *bindings* '((x . 1) (y . 2)))
>
> and say that X and Y are bound. Would you agree with that?
Absolutely. Note that that same glossary term says "... when the term
binding is qualified by the name of a namespace ..." and although we
think of our 5 traditional namespaces, if you follow the link to
namespace you find a much more general definition, especially
definition #2: "Any mapping whose domain is a set of names." So in
this case you are binding x and y in the '*bindings* namespace.
Who says CL is _just_ a Lisp2?
Note also that you could implement an environment as a set of
namespaces, so perhaps you could say that x and y are bound in the
*bindings* environment.
> The problem is that you can push this to an extreme. For example:
>
> (setf *vars* '(x y))
> (setf *vals* '(1 2))
>
> Are X and Y bound now? What about this:
>
> (setf l1 '(x y))
> (setf l2 '(1 2))
Yes, your intuition is following the definitional logic that I try to
follow; you _could_ give the *vars* and *vals* (or l1 and l2) grouping
a name, and thus call it a namespace, but that is indeed a stretch.
Note that some CL implementations use similar groupings (though they
are usually more like the alist style of your previous example, with
perhaps *venv*, *fenv*, etc naming the various namespaces. The
problem with even this approach is that although CL doesn't specify
environments completely, one restriction that is obvious is that an
environment mist be represented by exactly one object (either nil,
denoting the global environment, or some kind of lisp object for some
kind of lexical environment). So implementations have to put these
disjoint namespaces into one environment in order to conform to the
requirement that it be passable to macros and some other operators as
a single argument. It is unfortunate that X3J13 had to drop the
Environments chapter of CLtL2 (though it was the right thing to do at
the time), because if they had been able to standardize first-class
environments, we probably wouldn't be having so many conversations
about environmental things.
> In general whether a symbol is bound depends on some convention. If I
> interpret *bindings* as an alist, or *vars* and *vals* (or l1 and l2) as
> somehow associated with each other, then X and Y are bound, otherwise
> they are not.
Correct.
> On the other hand, if I write (defvar x 1) then X is unambiguously bound
> because it has a binding according to a convention that is specifically
> described in the standard: it has a dynamic binding in the variable
> namespace (and accordingly that binding has certain properties on which
> one can rely).
Correct. But imagine a more complete (or perhaps a next-generation)
spec with first-class environments defined. You would then not only
state that x is bound in the variable namespace, but depending on what
you are doing with that defvar form, you would say that x is bound in
the variable namespace of the global environment, or of the compiler
environment, or ...
Unfortunately, though the concepts of various environments are well
defined, the concept of bindings in particular environments is not so
well defined. For one thing, the CLtL2 definition of environments got
it wrong when it defined names in these objects to either be present
or absent; they do not have bindings in CLtL2, per se. All CL
compilers must implement some kind of binding style in whatever
compiler "environment" they implement, just as values are given to
variables bound in the regular interpreter environment.
> So, O Zen master Duane, if you will allow this humble student one more
> question:
Ah, but I see more than one question here :-)
these "compiler bindings" of which you speak, are they
> bindings of the first or the second sort? i.e. is a "compiler binding"
> a binding according to a convention that is specified in the standard
> (and if so, where?) or is it a binding according to some unspecified
> convention that is specific to a particular compiler implementation?
I'm hoping that in previous paragraphs I've already given you the
tools to answer the above questions yourself.
> And if the latter is the case, how can you say so confidently that X
> _is_ bound? Would it not be more correct to say that it *might* be
> bound (to some compiler-specific thingy)?
Well, I did also answer this two-part question above, but not
"confidently". But let me say this; if there is a CL compiler that
doesn't actually associate names to values, I'd be very interested in
seeing it. The reason I'm fairly confident in this is because when I
implemented the Environments Access module (http://www.lispwire.com/
entry-proganal-envaccess-des) I talked with a number of the CL vendors
and made sure that the "level 1" implementation worked in as many
versions as possible, And when I went to find out how they stored
their namespace information, the names were indeed bound in the
current contour to some kind of internal compiler structure.
Duane
Ron Garret
<fa25b254-77e5-41aa...@l2g2000yqd.googlegroups.com>,
Duane Rettig <du...@franz.com> wrote:
> > So, O Zen master Duane, if you will allow this humble student one more
> > question:
>
> Ah, but I see more than one question here :-)
See? That is why you are the master and I but the humble student. But
I think I feel some enlightenment coming on. :-)
Thanks!
rg
Thomas F. Burdick
> Ron Garret <rNOSPA...@flownet.com> wrote:
> +---------------
> | "Thomas F. Burdick" <tburd...@gmail.com> wrote:
> | > The way I like to phrase it is, the dynamic
> | > contours of a compiler recapitulate the lexical contours of the
> | > program being compiled. That's very different than the situation with
> | > interpreters. There, the dynamic contours of the interpreter are the
> | > dynamic contours of the program being interpreted.
> +---------------
>
> That's clever, but not *quite* accurate. For an interpreter which
> is interpreting a language with lexical scope [such as Scheme or the
> lexically-scoped parts of CL], the dynamic contours of the interpreter
> are *also* the lexical contours of the program being being interprested,
> only the interpreter traces said contours over & over & over again, ;-}
> not just once as with a compiler.
No, that's not true. Take this example:
(defun foo (x)
(+ x x))
(defun bar (x)
(if (numberp x)
(foo x)
(print x)))
The dynamic contours of a compiler recapitulate the lexical contours
of the above code, and whether the compiled language has dynamic or
lexical scope is irrelevant. In the most naive case, a compiler will
have one frame for (defun ...), a sub-frame for (x), a sub-frame for
(+ ...), a sub-sub-frame for x, and another for the second x. Then it
goes all the way back up to its "top" level, and starts down the
(defun bar ...) form, recapitulating the lexical structure of the form
being compiled with the code paths it takes.
Now take the case of an interpreter. Let's take the naive case again,
because it's easier. It sees (defun foo ...), registers the source to
be interpreted as a function associated with the symbol FOO, and then
continues to the next form. (defun bar ...) is treated similarly; it
does not trace through the IF, the NUMBERP, the (foo ...) and the
PRINT forms. Now when we interpret the call (bar 10) the interpreter
traces through the if, the numberp, the (foo ...), then in a dyamic
sub-context of interpreting the (foo ...) function call actually
starts tracing into the body of foo.
This is why it's so easy to leverage dynamic variables to implement a
dynamic-scoped interpreter OR a lexical-scoped compiler. You can of
course implement either scoping discipline in either an interpreter or
a compiler; and many interpreters have compiler-like elements to them,
where they do a light-weight pre-compilation step to produce something
that can be more efficiently interpreted. But one of the fundemental
differences between the two is that the code paths taken by a compiler
naturally mimic the textual shape of the code being compiled. And the
code paths taken by an interpreter mimic the code paths taken by the
program being interpreted.
> +---------------
> | > And the result of this is that languages with dynamic variables make
> | > simple compilers so easy to implement, it's fun.
> +---------------
>
> That's true enough. Just make your compiler's or interpreter's lexical
> environment object be a global special, then LET-bind it to an extended
> environment every time you enter a new lexical contour in the target
> program. The automatic restoration of the global special on each LET
> block exit will automagically prune the environment object.
You misunderstood what I meant by lexical contour. I didn't mean the
entry and exit points of variable scope, I mean literally the lexical
contours -- the textual shape -- of the program.
> +---------------
> | What is the best reference for getting the full scoop on the
> | lexical/dynamic binding connection?
> +---------------
>
> My favorite reference for such things is still Queinnec's "L.i.S.P.".
> Though, as Thomas suggests, starting to sketch out the code for a small
> compiler [or even an interpreter for a lexically-scoped language] should
> quickly show you the parallel he's trying to make.
No no no! The interpreter won't show you the point I was trying to
make...
As for LiSP, I agree that it's generally very useful for getting
insights into compilers and interpreters. But IIRC (and I don't have
my copy in front of me), doesn't he use only lexically-scoped lisps as
the implementation language? The point is still there to be seen in
what information is always passed around as "extra" arguments to all
functions, but he doesn't come out and make the point that he's
emulating dynamic variables (partly because he's not exactly doing
that, but something close).
I wonder if Richard Fateman would have better suggestions. I wonder if
he notices when his name is mentioned on c.l.l. He certainly has
experience teaching classes (and making reading lists) in writing
compilers using CL as the language to implement the compilers.
[snip]
> The point is that a trace of the dynamic state of *state* will be
> more-or-less isomorphic to the static lexical contours of the code
> being walked.
Isn't that what I was saying? :-)