Keywords and macros
Drew McDermott
some points raised in connection with the packages thread that I felt
were worth following up on.
Point 1: I was under the impression that the 'loop' macro's many
"local syntax markers" were exported from the :common-lisp package.
Someone reminded us that this is not so. To quote from Allegro's ANSI
CL hyperspec (no doubt Kent's usual lucid spec):
Loop keywords are not true keywords1; they are special symbols,
recognized by name rather than object identity, that are meaningful
only to the loop facility. A loop keyword is a symbol but is
recognized by its name (not its identity), regardless of the packages
in which it is accessible.
Point 2: Kent referred on the newsgroup to his past (futile) attempts
to get keywords to be "less than symbols," so that the print name of a
true symbol could *be* a keyword, i.e., a canonical string.
It seems to me that Point 1 is a damaging admission that there is
something wrong with the package approach. Indeed, someone coming
across this for the first time might say, "My oh my, Lisp is not
really a 'symbol-processing language' after all, because we are really
using symbols as glorified strings here." Granted, in the context of
macro expansion it really doesn't matter whether we test symbols using
'eq' or (lambda (s1 s2) (string= (symbol-name s1) (symbol-name s2))),
but if symbols can't deliver on their promise of being canonicalized
strings then there is something wrong.
The reason this ties in with Kent's point (point 2) is that it would
split symbol canonicalization off from symbol identity, thus easing
our conscience on both sides. Suppose we redesigned things thus:
Symbols behave just the way they do now, except their names are
keywords. 'symbol-name' can still return a string, but it is
understood as behaving as though it were
(lambda (s) (keyword-name (symbol-keyword s)))
Keywords print and read just the way they do now. They don't have
property lists, values, functions, etc., but they pretty much don't
have any of those things now. (Oops, they do have property lists.
They are hereby abolished!) They are self-evaluating, of course,
and implementations may choose to implement that with a value cell,
but it can't be set.
There is a predicate (canonically-eq s1 s2) that means exactly
(eq (symbol-keyword s1) (symbol-keyword s2)). Obviously, two
symbols have the same name if and only if they are canonically
equal, but the test is faster.
Macros like 'loop' can now unabashedly replace the language above
with a general prescription such as
Macros often require "local syntax markers" that mean something
only within the context of the macro. There are two ways to
implement such markers: as keywords or as symbols. The choice is
purely a matter of aesthetics. However, the macro implementer
should make sure that if symbols are used the macro implementation
tests potential occurrences within a macro call using
'canonically-eq' rather than 'eq' to avoid problems caused by the
macro call having been read in a different package from that used
by the macro implementation. We call a symbol used in this way a
"per-macro keyword."
And replace the language under loop with the simple statement:
Loop keywords are *per-macro keywords* (here insert hyperlink to
above), not true keywords.
Disclaimer: I don't know if this is what Kent had in mind.
Terminological issues:
Okay, "per-macro keyword" is not a very euphonious term. I'm open
to suggestion.
Perhaps "keyword" is bad terminology at this point, but then so is
"special"; some things just go on for historical reasons.
Pros and cons:
The big advantage of doing things this way is that it becomes
reasonably clear when a symbol should belong to a package and when it
shouldn't, and it provides a clear mechanism for a symbol "living" in
a package without really "belonging" to it. There are no
disadvantages that I can see.
Implementation issues:
Nothing prevents a Lisp implementation from doing what Kent and I
are suggesting right now (except for the property lists on
keywords). Although keywords don't have to be symbols, they aren't
required to be nonsymbols either. There is already a predicate
called 'keywordp'. (To my surprise.) All that is required is to
tweak 'intern' to handle the special case of interning to the
:keyword package, which should be a very fast check, unless someone
writes (intern "FOO" (concatenate 'string x1 x2)), which isn't going
to be fast no matter what.
The only problem is that if some implementations do 'canonically-eq'
fast and some do it slow, then if anyone depends on this predicate
in an inner loop they may get nervous.
Does anyone know of an implementation that does what I'm talking
about "under the hood"?
-- Drew McDermott
Barry Margolin
LOOP should not be considered representative of the way Common Lisp is
intended to be used. It was inherited from Maclisp, which didn't have
packages. In Common Lisp we use :keywords for the kinds of things that
LOOP keywords are used for, but for both readability and compatibility
purposes we decided not to use them in LOOP. To make them work using
ordinary symbol EQ checking, we would have had to clutter up the
COMMON-LISP package with exported symbols like FROM, TO, etc., and a
programmer working in a package that doesn't inherit from COMMON-LISP would
have to write things like:
(cl:loop cl:from 1 cl:to 10
cl:do ...)
which is really ugly, compared to:
(cl:loop from 1 to 10
do ...)
>The reason this ties in with Kent's point (point 2) is that it would
>split symbol canonicalization off from symbol identity, thus easing
>our conscience on both sides. Suppose we redesigned things thus:
>
> Symbols behave just the way they do now, except their names are
> keywords. 'symbol-name' can still return a string, but it is
> understood as behaving as though it were
> (lambda (s) (keyword-name (symbol-keyword s)))
>
> Keywords print and read just the way they do now. They don't have
> property lists, values, functions, etc., but they pretty much don't
> have any of those things now. (Oops, they do have property lists.
> They are hereby abolished!) They are self-evaluating, of course,
> and implementations may choose to implement that with a value cell,
> but it can't be set.
>
> There is a predicate (canonically-eq s1 s2) that means exactly
> (eq (symbol-keyword s1) (symbol-keyword s2)). Obviously, two
> symbols have the same name if and only if they are canonically
> equal, but the test is faster.
If an implementor things that it's necessary to optimize (string=
(symbol-name x) (symbol-name y)), they can do something similar to your
suggestion. I don't see why this needs to be exposed in the language
semantics. While your scheme makes this rare comparison faster, it slows
down the more common operation of printing a symbol, since this now
requires a double indirection (from the symbol to the keyword, then from
the keyword to its name). Either way, the performance differences are
miniscule, so I don't think performance should be an issue for it.
This seems like a big change just to clean up the way LOOP is described.
And LOOP will still be a wart, because it will be the only language feature
that uses "per-macro keywords". Providing this interface and giving a name
to them would encourage other macro designers to do something similar, and
we don't want more things like LOOP.
--
Barry Margolin, bar...@genuity.net
Genuity, Woburn, MA
*** DON'T SEND TECHNICAL QUESTIONS DIRECTLY TO ME, post them to newsgroups.
Please DON'T copy followups to me -- I'll assume it wasn't posted to the group.
Erik Naggum
| Point 1: I was under the impression that the 'loop' macro's many "local
| syntax markers" were exported from the :common-lisp package. Someone
| reminded us that this is not so. To quote from Allegro's ANSI CL
| hyperspec (no doubt Kent's usual lucid spec):
|
| Loop keywords are not true keywords1; they are special symbols,
| recognized by name rather than object identity, that are meaningful
| only to the loop facility. A loop keyword is a symbol but is
| recognized by its name (not its identity), regardless of the packages
| in which it is accessible.
Why is this such a big deal to you?
///
--
In a fight against something, the fight has value, victory has none.
In a fight for something, the fight is a loss, victory merely relief.
Post with compassion: http://home.chello.no/~xyzzy/kitten.jpg
Kent M Pitman
> I have been out of circulation for the last few weeks, but there were
> some points raised in connection with the packages thread that I felt
> were worth following up on.
>
> Point 1: I was under the impression that the 'loop' macro's many
> "local syntax markers" were exported from the :common-lisp package.
> Someone reminded us that this is not so. To quote from Allegro's ANSI
> CL hyperspec (no doubt Kent's usual lucid spec):
>
> Loop keywords are not true keywords1; they are special symbols,
> recognized by name rather than object identity, that are meaningful
> only to the loop facility. A loop keyword is a symbol but is
> recognized by its name (not its identity), regardless of the packages
>
> in which it is accessible.
>
> Point 2: Kent referred on the newsgroup to his past (futile) attempts
> to get keywords to be "less than symbols," so that the print name of a
> true symbol could *be* a keyword, i.e., a canonical string.
>
> It seems to me that Point 1 is a damaging admission that there is
> something wrong with the package approach.
Hmm. I can see how you might _think_ that. "admission", by the way,
is usually a verb of intent. I doubt I'd intend to damage my case. ;)
> Indeed, someone coming
> across this for the first time might say, "My oh my, Lisp is not
> really a 'symbol-processing language' after all, because we are really
> using symbols as glorified strings here." Granted, in the context of
> macro expansion it really doesn't matter whether we test symbols using
> 'eq' or (lambda (s1 s2) (string= (symbol-name s1) (symbol-name s2))),
You mean just STRING=. You probably don't realize STRING= takes symbol
arguments. Or aren't wanting to admit it here. See below for exposition.
> but if symbols can't deliver on their promise of being canonicalized
> strings then there is something wrong.
Here's another take on the syntax:
Are you familiar with the terminology of designators, btw? One real
terminological victory I feel like I scored with ANSI CL did not change
any semantics but added tremendous expressive power in seeing what had
previously been seen as "special cases" as more consistent, on average,
than people often see these things.
There are a number of functions that take strings as args but also take
other things that can pass for strings. For example, (string< 'foo 'bar)
is a valid call because the symbol FOO designates the string "FOO" and
the symbol BAR designates the string "BAR".
I would make the case that LOOP really takes not symbols but symbolic string
designators as arguments. It would probably make the spec look even more
regular to simply say that string designators themselves are allowed here
and therefore allow (LOOP "FOR" X "FROM" 1 "TO" 10 "DO" (PRINT X)). This
would work fine since right now there are no non-compound forms allowed.
> The reason this ties in with Kent's point (point 2) is that it would
> split symbol canonicalization off from symbol identity, thus easing
> our conscience on both sides.
I like my simplification better because it's simpler to explain.
I think everyone would agree that these aren't non-symbols.
They are symbols being used not for their identity but for their name.
Just as the symbol FOO in (symbol-name 'FOO) is being used for its name
and not its identity. I'll get the same answer from (symbol-name '#:FOO),
for example. But that doesn't mean I should create special terminology
to talk about symbols when I'm focusing on each different attribute of them,
any more than I want to call FOO in (get 'foo 'bar) a "property symbol"
instead of a "symbol". It's just a symbol. Symbols have numerous aspects.
That's what makes them useful.
> Suppose we redesigned things thus:
>
> Symbols behave just the way they do now, except their names are
> keywords. 'symbol-name' can still return a string, but it is
> understood as behaving as though it were
> (lambda (s) (keyword-name (symbol-keyword s)))
>
> Keywords print and read just the way they do now. They don't have
> property lists, values, functions, etc., but they pretty much don't
> have any of those things now. (Oops, they do have property lists.
> They are hereby abolished!)
I think it's considered legit to put packaged symbols on a keyword's
plist. I wouldn't mind giving up that power, but it might break
conforming programs.
> They are self-evaluating, of course,
> and implementations may choose to implement that with a value cell,
> but it can't be set.
>
> There is a predicate (canonically-eq s1 s2) that means exactly
> (eq (symbol-keyword s1) (symbol-keyword s2)).
If a system already did this canonicalization of names hack, it could
optimize (string= (the symbol s1) (the symbol s2)) to
(eq (symbol-name s1) (symbol-name s2))
without needing a new operator.
> Obviously, two
> symbols have the same name if and only if they are canonically
> equal, but the test is faster.
IMO, this should just be an optimization issue.
> Macros like 'loop' can now unabashedly replace the language above
> with a general prescription such as
>
> Macros often require "local syntax markers" that mean something
> only within the context of the macro. There are two ways to
> implement such markers: as keywords or as symbols. The choice is
> purely a matter of aesthetics. However, the macro implementer
> should make sure that if symbols are used the macro implementation
> tests potential occurrences within a macro call using
> 'canonically-eq' rather than 'eq' to avoid problems caused by the
> macro call having been read in a different package from that used
> by the macro implementation. We call a symbol used in this way a
> "per-macro keyword."
>
> And replace the language under loop with the simple statement:
>
> Loop keywords are *per-macro keywords* (here insert hyperlink to
> above), not true keywords.
I'd be happy to just make a term "guide word" and say
Guide words are string designators used by some macros to denote
syntax information that must be constant at compile time.
For example, if a hypothetical WHENEVER macro takes guide words
of THEN and ELSE, then it follows that (a) forms like
(WHENEVER x "THEN" y "ELSE" z) and
(WHENEVER x :then y :else z) and (WHENEVER x THEN y ELSE z) are
equivalent [in any package], since the names of guide words are used
only for the string they designate, and (b) neither
symbols nor strings are allowed as forms at toplevel of the
WHENEVER macro.
> Disclaimer: I don't know if this is what Kent had in mind.
It's not. However, that in and of itself, fortunately for all of us,
even me, doesn't render your idea dead. I don't own the language nor
am I always right. Even here, I have an opposing opinion, but I'm curious
what others think...
> Terminological issues:
>
> Okay, "per-macro keyword" is not a very euphonious term. I'm open
> to suggestion.
Mostly I don't like it because it suggests a "type" issue, yet it
really seems to connote a "usage" issue. When viewing the program as
data instead of program, for example, it would not be a keyword.
It's a horrible thing (and not my choice) that LOOP used this term
"keyword" at all in this context. I tried to grandfather in the old
usage for compatibility reasons, but I don't want to promote it.
I use the word guide word myself, because it's more general.
Incidentally, for those who think this is a Lisp-only issue, I think
prepositions are "guide words" in English.
> Perhaps "keyword" is bad terminology at this point, but then so is
> "special"; some things just go on for historical reasons.
Well, we can try not to make things worse.
> Pros and cons:
>
> The big advantage of doing things this way is that it becomes
> reasonably clear when a symbol should belong to a package and when it
> shouldn't, and it provides a clear mechanism for a symbol "living" in
> a package without really "belonging" to it. There are no
> disadvantages that I can see.
I don't see how this can possibly be known to the reader without
confronting packages, and so I don't see how it can avoid being about
packages. Maybe I'm misunderstanding.
Drew McDermott
LOOP should not be considered representative of the way Common Lisp is
intended to be used. It was inherited from Maclisp, which didn't have
packages. In Common Lisp we use :keywords for the kinds of things that
LOOP keywords are used for,
If an implementor things that it's necessary to optimize (string=
(symbol-name x) (symbol-name y)), they can do something similar to your
suggestion. I don't see why this needs to be exposed in the language
semantics.
And LOOP will still be a wart, because it will be the only language feature
that uses "per-macro keywords". Providing this interface and giving a name
to them would encourage other macro designers to do something similar, and
we don't want more things like LOOP.
However, I find that my own (much cleaner!) macros often look a bit cluttered with all those colons, especially when the "guide words" (in Kent's phrase) are short. For instance, I have a macro 'datafun' that is used thus:
(datafun flag sym
(defun (x y z) ...))
Which tells some S-expression processor (associated with the flag, a symbol itself), that if it encounters an S-expression beginning (sym ...) it should call the associated anonymous function. Note that that there is no name following the 'defun'; it is automatically generated by the 'datafun' macro. Now, some editors, and no doubt some human readers, are confused by the 'defun' followed by no name; the editor tends to be fooled into thinking (x y z) is the name of the function, and various annoying indentation consequences follow. So, I think, I will allow an optional guide word '^' to follow the defun:
(datafun flag sym
(defun ^ (x y z) ...))
reminding the human that the name is assigned by 'datafun', and fooling the editor into thinking '^' is the name of the function. Alas, this runs afoul of package boundaries; if 'datafun' is exported but '^' is not, the macro will malfunction. So, is this an improvement?
(datafun flag sym
(defun :^ (x y z) ...))
It's what I opted to do. But I could see a reasonable person opting for the 'loop' approach instead. Obviously, this has nothing to do with efficiency issues, and hopefully nothing to do with whether my macro or 'loop' is worthy; the issue arises over and over.
-- Drew McDermott
P.S. Someone will no doubt point out that my macro could be shortened to
(datafun flag sym (x y z) ...)
Of course it could. But I have independent reasons for wanting
this device to work with nonstandard function definers. So the full
truth is that 'defun' is not the only thing that can appear here.
In fact, any definer at all should be allowed, provided the name of the
thing defined normally appears right after the definer, in which case it
may be replaced by ':^'.
Erann Gat
<drew.mc...@yale.edu> wrote:
> For instance, I have a macro 'datafun' that is used thus:
>
> (datafun flag sym
> (defun (x y z) ...))
>
> Which tells some S-expression processor (associated with the flag, a symbol
> itself), that if it encounters an S-expression beginning (sym ...) it
should call
> the associated anonymous function. Note that that there is no name
following the
> 'defun'; it is automatically generated by the 'datafun' macro. Now,
some editors,
> and no doubt some human readers, are confused by the 'defun' followed by
no name;
Indeed. Why didn't you just use lambda instead?
E.