keywords in Common Lisp and (MIT) Scheme
Erik Naggum
as MIT Scheme. the keywords are distinguished from other symbols by a
trailing colon. this notation has been picked up by Draft International
Standard 10175 Document Style Semantics and Specification Language (DSSSL).
I have written some Common Lisp code to process SGML documents, and thought
I'd try my hand at some DSSSL implementation, since the two standards are
intimately connected. however, this has clashed with a most annoying and
exceedingly trivial little problem.
a trailing package marker (:) in a symbol, as in foo:, causes `read' to
signal an error, as CLtL2 (p 521) indicates it should. however, ANSI CL
2.3.5 (5) says undefined patterns including the package marker are reserved
for implementation-dependent use. well, that's not entirely bad if I can
find a way to do it.
is there a reasonable way I can convince the Lisp reader that a trailing
colon is not really a major disaster, but largely equivalent to a keyword?
I have been messing with the reader before, with moderate success, but this
one seems to be out of my league. I can disable the colon as the package
marker, and instead look for a trailing colon in all my symbols, but if I
do this, I have to implement substantial parts of the reader myself, as I
can't get into its internals sufficiently to just fix this minor problem.
do others do this, or do they switch to Scheme just because of a colon
problem?
#<Erik>
--
miracle of miracles. look what the Net dragged in.
Len Charest
Erik Naggum <er...@naggum.no> wrote:
>a trailing package marker (:) in a symbol, as in foo:, causes `read' to
>signal an error, as CLtL2 (p 521) indicates it should. however, ANSI CL
>2.3.5 (5) says undefined patterns including the package marker are reserved
>for implementation-dependent use. well, that's not entirely bad if I can
>find a way to do it.
CLtL2, page 253 agress with the ANSI doc regarding trailing package
markers.
>is there a reasonable way I can convince the Lisp reader that a trailing
>colon is not really a major disaster, but largely equivalent to a keyword?
Unfortunately, the 'character attributes' of the colon are hard-wired
into the Common Lisp reader. See CLtL2, page 541, for the description
of SET-SYNTAX-FROM-CHAR. In my experience, the *only* time this
limitation gets in the way is when you what to reprogram the behavior
of colon.
You could beg your vendor for a (non-portable) solution. Otherwise you
could disable the colon with:
(set-macro-character #\: #'(lambda (stream char) (values)))
in which case the colon is effectively treated as whitespace (i.e., it
behaves as a token separator). You may be able to isolate this effect
by using a special readtable when you expect to parse keywords.
--
Len Charest, Jr. =Speed=of=Lightning==>
JPL Artificial Intelligence Group
cha...@underdog.jpl.nasa.gov <<<Power of Thunder>>>
Steve Haflich
From: cha...@underdog.jpl.nasa.gov (Len Charest)
In article <19950125T0...@naggum.no>,
Erik Naggum <er...@naggum.no> wrote:
>a trailing package marker (:) in a symbol, as in foo:, causes `read' to
>signal an error, as CLtL2 (p 521) indicates it should. however, ANSI CL
>2.3.5 (5) says undefined patterns including the package marker are reserved
>for implementation-dependent use. well, that's not entirely bad if I can
>find a way to do it.
CLtL2, page 253 agress with the ANSI doc regarding trailing package
markers.
>is there a reasonable way I can convince the Lisp reader that a trailing
>colon is not really a major disaster, but largely equivalent to a keyword?
Unfortunately, the 'character attributes' of the colon are hard-wired
into the Common Lisp reader.
Yes, this doesn't prevent portable code from assigning an
interpretation to any reserved reader syntax, although it's a litle
more work than one would like.
First, make sure you understand the details how the CL reader works,
and particularly that it processes extended tokens in two distinct
phases. First a token is collect (CLtL Section 22.1.1) and then it is
parsed into a number of symbol (CLtL Section 22.1.2).
To change the way symbols are parsed, you indeed need to make
readtable changes. Specifically, change the dispatch of every escape
or constituent character to a new reader macro that you write. This
macro must duplicate the behavior of the token-collecting phase,
i.e. rules 4 through 10 in section 22.1.1. Whether you cheat on
escape conventions etc. is up to you, but the algorithm is simple to
write just by following the given rules carefully.
Once an entire token has been collected, examine the collected token
string and see if its syntax conforms to the extension you are
implementing. If so, do it. If not, lambda-bind *READTABLE* back to
the original readtable and call READ-FROM-STRING on the token. This
trick saves you from having to reimplement the really difficult parts
of the reader, especially all the number constructing code.
IMO it is unfortunate that the token-collection phase and the
token-parsing phases were not separated and made visible by being tied
to exported, named functions. That would have eliminated the useless
duplication of the token-colleciton algorithm.
This strategy is awkward (aside from the implementation effort) only
in that by using multiple readtables, it prevents subsequent
modification of `the' readtable by other unrelated applications in the
same lisp image. But this is true any time an application makes
readtable changes for any purpose. The CL readtable mechanism is
elegant in the ability instantaneously to lambda bind the whole
readtable, but deficient in facilities for independent incremental
customizations.
William G. Dubuque
|Date: Sat, 28 Jan 1995 18:01:49 GMT
In article <19950125T0...@naggum.no>,
Erik Naggum <er...@naggum.no> wrote:
>is there a reasonable way I can convince the Lisp reader that a trailing
>colon is not really a major disaster, but largely equivalent to a keyword?
|Steve described how to implement a wrapper around the CL tokenizer
|by remapping all constituent and escape chars to macro chars.
While implementing a wrapper around the tokenizer is
conceptually the correct way to proceed, practically it will
probably turn out to be very inefficient compared to the
built-in tokenizer, especially in highly optimized lisp
implementations.
For the problem at hand--redefining the meaning of ':' in
tokens--there is a much more efficient solution (and
arguably simpler), see my earlier reply below.
If one needed to perform more complex parsing of tokens--
such as redfining number syntax--then one might need to
resort to a tokenizer wrapper as sketched by Steve.
As a long time symbolic math hacker (Macsyma) one cannot
help but consider some of these parsing issues due to the
highly sophisticated syntax encountered in various
mathematical languages.
Its unfortunate that one needs to resort to such
convolutions due to CL deficiencies that could have been
easily remedied.
-Bill
--------
From: w...@zurich.ai.mit.edu (William G. Dubuque)
Newsgroups: comp.lang.lisp
Subject: redefining meaning of colon in symbols [was: syntax of Re: keywords in Common Lisp and (MIT) Scheme]
Date: 26 Jan 95 10:34:14
Organization: M.I.T. Artificial Intelligence Lab.
NNTP-Posting-Host: martigny.ai.mit.edu
In-reply-to: Erik Naggum's message of 25 Jan 1995 07:05:19 UT
References: <19950125T0...@naggum.no>
Distribution:
From: Erik Naggum <er...@naggum.no>
Date: 25 Jan 1995 07:05:19 UT
is there a reasonable way I can convince the Lisp reader that a trailing
colon is not really a major disaster, but largely equivalent to a keyword?
Yes, this can be accomplished while still preserving the
efficiency of the built-in CL tokenizer and reader, but it is not
simple (in spite of Steele's comment on p. 510 of CLtL2 that "The
reader is also parameterized in such a way that it can be used as
a lexical analyzer for a more general user-written parser.").
First, define ':' as a terminating macro char that returns a
single value that is a unique object of your choice that will be
used to denote the occurrence of colons in objects that have been
read.
Next, you must redefine every macro character in your language
that may read objects containing your new style symbols.
For example, for the '(' macro char, you must wrap its definition
so that you recursively walk the list that it reads and scan for
the unique colon markers, and do whatever you desire to them and
any preceding symbol. You can optimize this so that the walk
need only be done if a colon was actually encountered.
For quote, backquote and comma macro chars: if they read a
symbol, you need to peek ahead for a possible ':'.
And similarly for any other macro chars in your language.
The only performance hit you take is the cost of the above macro
char wrappers, which should be negligible since in all cases you
can call the built-in macro char and then perform a quick
analysis on its result.
Its unfortunate that there is no clean way to do this in CL.
With a little more parameterization, the reader could have been
much more general.
-Bill