Pining for a Paul Graham-style "aha!" moment with Lisp/macros
Ralph Richard Cook
then C, C++, now Java. I look at other languages for the heck of it
(Forth, Smalltalk, Python, Ruby) and to learn other concepts that
don't often crop up in my current stable of languages I work in.
There have been some Lisp-related articles on slashdot lately, one of
which led me to Paul Graham's article "Beating the Averages" which I'm
sure most of you have read:
http://www.paulgraham.com/avg.html
The article quotes Eric Raymond's "How to Become a Hacker",
Lisp is worth learning for the profound enlightenment experience
you will have when you finally get it; that experience will make
you a better programmer for the rest of your days, even if you
never actually use Lisp itself a lot.
When working in Pascal I didn't really *get* pointers until I learned
C, which made me a better Pascal programmer. When working in C++ I
didn't really *get* interfaces until I learned Java, well, you get the
idea.
What I'm looking for is that same kind of "aha" moment with macros,
and symbolic programming in general. Looking on the web I see macros
defined but nowhere do I see a good example of how it was used to do
something that would be very hard to do or just can't be done in
anohter language. For that matter, the definitions of "symbolic
programming" I've seen haven't been that enlightening either.
Anything on the Web that can help me out? I'm not quite ready to
commit to slog through 600 pages of an $80 book on what may be a wild
goose chase for me.
Duane Rettig
> What I'm looking for is that same kind of "aha" moment with macros,
> and symbolic programming in general. Looking on the web I see macros
> defined but nowhere do I see a good example of how it was used to do
> something that would be very hard to do or just can't be done in
> anohter language. For that matter, the definitions of "symbolic
> programming" I've seen haven't been that enlightening either.
Since macros operate on Lisp language elements and generate new
constructs of Lisp language elements, it is a closed system, and in
a sense nothing can be done with macros that can't be done in other
languages. However, what CL macros do for you is to show you source
code for otherwise opaque implementations, and _that_ can't be done
in most other languages.
> Anything on the Web that can help me out? I'm not quite ready to
> commit to slog through 600 pages of an $80 book on what may be a wild
> goose chase for me.
Take whatever version of CL you're experimenting with, and remember
this mantra:
(pprint (macroexpand '(<form>)))
You can start with something basic:
(pprint (macroexpand '(defun foo (x y) (+ x y))))
and you can then move on to other defining forms or other macros
defined either by CL or by your implementation. I always find
Allegro CL's def-foreign-call form to be very interesting:
(pprint (macroexpand '(ff:def-foreign-call sbrk ())))
The "aha" experience you achieve might be the realization that the
language is not only doing a lot under the hood for you, but that
you can lift the hood and see how the engine is built. And then you
can copy the concepts you see and build your own engine. It's one
of the ways CL is the one of the most extensible languages I know.
--
Duane Rettig Franz Inc. http://www.franz.com/ (www)
1995 University Ave Suite 275 Berkeley, CA 94704
Phone: (510) 548-3600; FAX: (510) 548-8253 du...@Franz.COM (internet)
Patrick Wray
"Ralph Richard Cook" <ral...@bellsouth.net> wrote in message
news:3c5c02eb...@news.atl.bellsouth.net...
[...]
> What I'm looking for is that same kind of "aha" moment with macros,
> and symbolic programming in general. Looking on the web I see macros
> defined but nowhere do I see a good example of how it was used to do
> something that would be very hard to do or just can't be done in
> anohter language.
If you're receptive to the ideas in Paul Graham's articles, check out his
book "On Lisp". He wrote it mainly to address the above issue.
As luck would have it, he made "On Lisp" available online yesterday:
http://www.paulgraham.com/onlisptext.html
Kaz Kylheku
>When working in Pascal I didn't really *get* pointers until I learned
>C, which made me a better Pascal programmer. When working in C++ I
>didn't really *get* interfaces until I learned Java, well, you get the
>idea.
I've had a different experience. I was exposed to C upon entering
university. In the C course, we had to write an interpreter for a
tiny subset of PostScript. I was introduced to interfaces by the idea
that the professor set before the class. For bonus points, one could
use a struct containing function pointers to define the graphics output
driver of the interpreter, have the driver fill it with pointers to its
own implementations of the functions, and provide a way to select the
driver by name on the command line. I understood I/O hooks from hacking in
machine language on 8 bit microcomputers, and had used callback functions,
but this was still an ``Aha'' moment. I managed to ``subcontract'' the
development (properly credited, of course) of an X window display
driver to a senior student, whom I gave only the interface spec.
Another ``aha'' moment.
>What I'm looking for is that same kind of "aha" moment with macros,
>and symbolic programming in general.
One way to ``get it'' instantly, is to first spend years scratching
away in a stone age programming language like C, and struggle with its
macros to badly emulate language features that you would like to have,
running against limits in what you can do, and how cleanly you can do it.
So upon diving into Lisp, I had the ``aha'' moment nearly right away.
It seems that can't write a 200 line program in Lisp without at least
one macro in it. ;)
To understand symbolic processing, write a compiler in a Stone Age
programming language like C++ or Java. Compiler construction *screams*
for dynamic typing and symbolic processing. Do you ever use the Yacc
parser generator? One of the first declarations in a Yacc source file is
a %union { } which combines objects of different types into one. This
allows for a generic parser stack that can hold integers, identifiers,
strings, non-terminal symbols, whatever. In the C translation, you get
an ugly union which just overlays types in a completely unsafe way. What
you are doing right there is starting to badly emulate a feature that
is clean and safe in Lisp.
In Lisp, macros *are* little compilers. Compiler writing is so easy that
you don't even notice you are doing it. The lexical analysis and parsing
is taken care for you, as is the flexible representation of symbolic
data, so you just have to focus on the aspects that are unique to your
task at hand.
>Looking on the web I see macros
>defined but nowhere do I see a good example of how it was used to do
>something that would be very hard to do or just can't be done in
>anohter language.
Such macros are in the language itself. Take the setf macro. Prior
to setf, one had to use different functions to destructively store
a value to different kinds of places. For example, to store
into the CDR field of a cons cell, it would be (rplacd cell value).
To store into the CAR, it is (rplaca cell value). Storing to
a variable bound to a symbol (setq symbol value) and so on.
The setf macro unites assignment so that you simply use the
same form which expresses *access* to the location in order
to perform a *store*. E.g. (setf (car cell) value)
(setf symbol value) and so on. In other words, setf implements
what some other language slike C call an ``lvalue'': an expression
that designates a place, and can be the target of an assignment.
Moreover, setf is extensible. The Lisp programmer can create
new kinds of places for setf to understand, and methods for them
to store values there.
Imagine if C didn't have lvalues. What code could you write in C to
invent them? You couldn't. They have to be hacked into the compiler,
which is closed to you. Aha!
The setf macro is a little lvalue compiler which knows how to grok
the place, and then generate code to assign to it. So (setf (car cell)
value) becomes, quite possibly, (rplaca cell value) or maybe something
implementation-specific.
In general, if you want to ``get it'', you have to take on a tough
project. Something you have been wanting to hack on, but kept putting
it off because C, C++ or Java kept getting in the way.
I once wrote a macro preprocessor in C just for fun, a project called
MPP. It's close to 15000 lines of C and hardly does anything. It
never acquired all the features I wanted, and I realized that they
would be quite hard to put in. That hurt. A lot of those 15000 lines
are reinvented wheels. A red-black tree associative container, a hash
table module, a module to emulate exception handling in C, a module for
dynamic reference-counted strings; linked lists, etc. In other words,
a good part of what is already standard in Common Lisp before you write
single line of code!
I'm nearly certain I could write a MPP workalike in Lisp in 1500
lines. So *statically*, judging by lines of code, I'd have 10:1 increase
in productivity. But *Dynamically*, it would be even better, because the
C program would probably require something like a 100 lines of change
to achieve a feature for every 1 line of change to do the same feature
in the Lisp version. And those 100 lines of change would be scattered,
not all in one place. If the C version required 1500 lines of change to
get somewhere, that's already equal in size to the entire Lisp equivalent!
Paul Tarvydas
@news.atl.bellsouth.net:
> (Forth, Smalltalk, Python, Ruby) and to learn other concepts that
> don't often crop up in my current stable of languages I work in.
Do let me encourage you to continue looking at lisp. I've been at it, on
and off since 1977 (interspersed with many years grinding my brain down
with things like C, C++, Eiffel, Perl, Smalltalk, etc etc) and I still
keep finding those woderful aha's in lisp.
Things that I understand better because of my exposure to lisp include:
- dynamic typing (variables aren't typed, values are)
- garbage collection (Java has garbage collection, but I would argue that
the combination of other features in Java hide the wonderfullness of GC,
Smalltalk might be another place to really learn about GC)
- lists - an incredibly versatile, simple, efficient data structure that
seems to get bolluxed up in just about every other language (as soon as
someone tries to express List as a class, they tend to murder the
concept)
- simplicity - that programming in a single paradigm (say, OOP) and
"simplicity" are not equivalent
- procedural programming, recursion, OOP, functional programming,
declarative programming, functions as first-class data types, symbolic
programming, data-driven programming
- programming environments
- and all sorts of stuff that is now so second-nature to me that I can't
notice it or describe it to others.
It might help you to understand why people rave about lisp macros, if you
consider this over-simplification: you can imagine that the lisp
compiler contains an interpreter for the full language inside of the
compiler. As it compiles lisp source code, the code can direct the
compiler to define new compile-time functions (extensions to the
compiler) using *anything* in the lisp language. Using a few compile-
time operators, like 'back-quote', 'comma' and 'at' (`,@ resp.), the
programmer can direct the compiler to call these compile-time functions
and to splice their results into the source code before further
compilation.
Furthermore, lisp "source code" is stored as lists (not text) so all of
the powerful list-operators that lisp provides are available for editing
macros and creating lisp code at compile time. If you look at macro
code, you'll see that many macros do things like "looping" over the
incoming source code (the arguments of the macros) and manipulating the
code. For example, if you wanted to re-implement the "if" statement
using keywords "then" and "else", you would write a function that loops
through all of the elements of a list (which happens to be the source
code that your function (macro) is going to transform). You test to see
if the current item is a symbol, if it is, you test to see if it is the
symbol "then". You collect up all of the items into a list until you hit
the symbol "else". Then you create a new list, and collect items into
it, until you hit the end of the if-statement. You splice the lists
together to form a legal lisp expression and return it (to the compiler,
which then splices this result into the program it's compiling). You do
all this in Lisp.
Furthermore, lisp comes with built-in lisp source-code parsing operators.
The only other language, that I know of, that comes with self-parsing
operators is Tcl. [Tcl treats all programs as strings, lisp treats all
programs as lists].
It's as if Java came with a built-in YACC and a Java grammar that you
could manipulate at run-time using your program. The combination of
features in lisp (simple syntax, code as lists, powerful list operators)
makes it unnecessary to use YACC at runtime in lisp to manipulate the
lisp language.
I saw someone have this aha, using Tcl, recently: the fact that code is
stored as a fundamental data structure, means that you can read code into
a database (a hash) and "fire" the code at will. This means that
(trusted) programs can send code to each other or that you can store a
fact base as a file of executable code. I wrote an article for Dr.
Dobb's in '78 about this (it was a simple lisp editor, implemented as a
read-loop - it would read a command in from the keyboard, in lisp syntax,
lisp would convert the text into a list containing symbols, then the
command would be executed by simply eval'ing the new list).
Over-simplification of symbolic programming: using words (symbols) as
data. In lisp, code is read from source files into the lisp environment.
Almost the first thing that happens is that the reader hashes every
incoming word (symbol) into a global hash table (this global hash table
*is* the environment). From that point on, all symbols are simply
carried around as pointers into the hash table - i.e. symbols in lisp are
really inexpensive. So, in lisp, it costs you nothing to say THIS-IS-A-
SYMBOL instead of something like 57 (i.e. words (symbols) are as cheap to
use as numbers, in lisp). Likewise, there is only one copy of THIS-IS-A-
SYMBOL in the environment (hash table), no matter how many times it
appears in the source.
Immediately, a whole class of programming noise disappears, such as
#define's in C or final consts (the closest C analogy to symbols is
enums, you can't do anything with enums, except move them around). When
you are mentally free to use words (symbols) instead of numbers, your
programming style changes dramatically (you've had an aha). You tend to
express ideas in human-readable terms, the debugger shows you things in
human-readable terms, etc, etc. The fact that you can stop thinking
about converting data into numbers or other data structures, drops your
cognitive load and allows you to solve bigger problems before your brain
explodes.
If you know something about the guts of compilers, you can consider
symbols to be "tokens". The lisp environment can be over-simplified to
something like an interactive version of the insides of a compiler. The
lisp reader reads text files, tokenizes them and parses them, then
deposits the tokenized data into the "environment". You, the programmer,
sit inside of the environment with a bag of tools and a bag of tokenized
code. You invoke tools that operate on the tokenized stuff floating
around in this soup called an environment. One of the tools is a
compiler. Another tool is an evaluator - evaluate (compile/run,
interpret, whatever) this lump of tokens and show me the result.
This is the kind of programming that the eXtreme programming people are
talking about - try (test) bits of code until you get it right, then move
on. Build in small steps, always be prepared to show working code (build
often).
For comparison, Smalltalk takes this environment-as-soup concept to an
extreme - Smalltalk is text-file hostile, which makes things like Envy
necessary. Lisp works with "normal" text files that look like normal
source code. The good stuff happens when those text files are read into
the environment.
> Anything on the Web that can help me out? I'm not quite ready to
> commit to slog through 600 pages of an $80 book on what may be a wild
> goose chase for me.
All of Paul Graham's "On Lisp" code is at
http://www.paulgraham.com/lispcode.html
'On Lisp' is the book to read to grok macros, but the book is very hard
to get these days.
pt
Paolo Amoroso
> In Lisp, macros *are* little compilers. Compiler writing is so easy that
By the way, compiler macros are also worth a look. The original poster may
want to check the entry for the DEFINE-COMPILER-MACRO form in the Common
Lisp HyperSpec.
Paolo
--
EncyCMUCLopedia * Extensive collection of CMU Common Lisp documentation
http://web.mclink.it/amoroso/ency/README
[http://cvs2.cons.org:8000/cmucl/doc/EncyCMUCLopedia/]
Vladimir Zolotykh
> Allegro CL's def-foreign-call form to be very interesting:
>
> (pprint (macroexpand '(ff:def-foreign-call sbrk ())))
I've tried to see it. Honestly I understood almost nothing. You
called it 'very interesting'. Would you be so kind to add some words
of explanation ? I like interesting things but not always easy to
understand it especially w/o vast knowledge of ACL 'implementation'
functions.
--
Vladimir Zolotykh gsm...@eurocom.od.ua
Tim Bradshaw
> On Sat, 02 Feb 2002 20:58:03 GMT, k...@accton.shaw.ca (Kaz Kylheku) wrote:
>
> > In Lisp, macros *are* little compilers. Compiler writing is so easy that
>
> By the way, compiler macros are also worth a look. The original poster may
> want to check the entry for the DEFINE-COMPILER-MACRO form in the Common
> Lisp HyperSpec.
>
And symbol macros. You don't hear much about them (less even than
compiler macros maybe) but you can do really nice tricks. My current
system has a
`variable' called something like *gui-style* - if you assign to it
then magic things happen including closing lots of windows and so on.
Of course using a name implying a special variable is pretty dumb, I
should invent some other naming convention for it.
--tim
Kaz Kylheku
Bradshaw wrote:
>Paolo Amoroso <amo...@mclink.it> wrote in message news:<O15dPNoyRJAwv6CC=h6ekJJF9=U...@4ax.com>...
>> On Sat, 02 Feb 2002 20:58:03 GMT, k...@accton.shaw.ca (Kaz Kylheku) wrote:
>>
>> > In Lisp, macros *are* little compilers. Compiler writing is so easy that
>>
>> By the way, compiler macros are also worth a look. The original poster may
>> want to check the entry for the DEFINE-COMPILER-MACRO form in the Common
>> Lisp HyperSpec.
>>
>
>And symbol macros. You don't hear much about them (less even than
>compiler macros maybe) but you can do really nice tricks.
Indeed, nice tricks like implement with-slots.
David Hanley
I had fooled with lisp, but hadn't really 'gotten it.'
I was coding in C and i started to miss certian lisp functions (
mapcar, remove-if-not, reduce ) i was glad they were in the lisp
compiler.
Then i realized that they were very easy to program in lisp, even if
they weren't there. This was NOT true for C or C++!. I could make
such features myself.
Since then, i've been a much more flexible and generic coder. I add
little features i need instead of bemoaning their not being there.
dave
Duane Rettig
Well, if you've understood almost nothing, then at least it's not
nothing. Tell me what you have understood of it so far, and I'll
fill in at least some of the rest.
Vladimir Zolotykh
>
> Well, if you've understood almost nothing, then at least it's not
> nothing. Tell me what you have understood of it so far, and I'll
> fill in at least some of the rest.
I've just tried to describe what I understand and what do not. When I
write 'I understand' I don't mean 'I know all details, how it works,
which cases etc.' but only 'I have a point from which I could go
further when needed.' So let me to proceed.
(progn
;; What purpose the following form stands for ? Which cases it works
;; in ? E.g., what I should write to see it in action ? Deciding on
;; EVAL-WHEN it works during compilation only, but that is all I
;; could say about it. It seems all defined functions go in
;; excl::.functions-defined.
(eval-when (:compile-toplevel)
(excl::check-lock-definitions-compile-time
'sbrk 'function 'foreign-functions:def-foreign-call (fboundp 'sbrk))
(push 'sbrk excl::.functions-defined.))
;; The next form I don't understand also. Why EVAL-WHEN needed ?
;; What is the value of system::direct-ff-call initially (probably
;; it initialized in ff package). What would happen if SBRK wasn't
;; removed from SYSTEM::DIRECT-FF-CALL ?
(eval-when (compile load eval)
(remprop 'sbrk 'system::direct-ff-call))
;; The following three forms I understood (more or less), except
;; excl::get-ff-apply-closure. Probably it returns the closure that
;; being funcalled gives the call to sbrk(2).
(setf (fdefinition 'sbrk)
(excl::get-ff-apply-closure
(excl::determine-foreign-address '("sbrk" :language :c) 514 nil)
'(:int (integer * *))))
(record-source-file 'sbrk)
'sbrk)
--
Vladimir Zolotykh gsm...@eurocom.od.ua
Kenny Tilton
> nothing. Tell me what you have understood of it so far, and I'll
> fill in at least some of the rest.
I'll bite:
> CG-USER(7): (pprint (macroexpand '(ff:def-foreign-call sbrk ())))
> Warning: String arguments passed to `SBRK' will be converted because :STRINGS-CONVERT is assumed
> true.The with-native-string macro can be used for explicit string conversions around the
> foreign calls. This warning is suppressed when :STRINGS-CONVERT is specified in the
> def-foreign-call.
>
> (PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL)
> (EXCL::CHECK-LOCK-DEFINITIONS-COMPILE-TIME 'SBRK 'FUNCTION
> 'FOREIGN-FUNCTIONS:DEF-FOREIGN-CALL (FBOUNDP 'SBRK))
> (PUSH 'SBRK EXCL::.FUNCTIONS-DEFINED.))
> (EVAL-WHEN (COMPILE LOAD EVAL) (REMPROP 'SBRK 'SYSTEM::DIRECT-FF-CALL))
> (SETF (FDEFINITION 'SBRK)
> (EXCL::GET-FF-APPLY-CLOSURE
> (EXCL::DETERMINE-FOREIGN-ADDRESS '("sbrk" :LANGUAGE :C) 514 NIL)
> '(:INT (INTEGER * *))))
> (RECORD-SOURCE-FILE 'SBRK) 'SBRK)
What has this got to do with macros and/or symbolic programming per se?
That was the question. And it was a newbie question, so no way they
should be expected to realize:
> eval-when normally appears as a top level form, but it is meaningful
> for it to appear as a non-top-level form.
> However, the compile-time side effects described in Section 3.2 Compilation
> only take place when eval-when appears as a top level form.
We had a chat here recently questioning if macros were ever even
necessary, so it's no good to show a lot of cool stuff going on, the
newbie question was "why macros and symbolic programming". ie, what they
can they do other approaches cannot.
--
kenny tilton
clinisys, inc
---------------------------------------------------------------
"I don't think the heavy stuff is gonna come down for a while."
- Carl, Caddy Shack
Duane Rettig
> > Well, if you've understood almost nothing, then at least it's not
> > nothing. Tell me what you have understood of it so far, and I'll
> > fill in at least some of the rest.
>
> I'll bite:
Ouch :-)
> > CG-USER(7): (pprint (macroexpand '(ff:def-foreign-call sbrk ())))
> > Warning: String arguments passed to `SBRK' will be converted because :STRINGS-CONVERT is assumed
> > true.The with-native-string macro can be used for explicit string conversions around the
> > foreign calls. This warning is suppressed when :STRINGS-CONVERT is specified in the
> > def-foreign-call.
> >
> > (PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL)
> > (EXCL::CHECK-LOCK-DEFINITIONS-COMPILE-TIME 'SBRK 'FUNCTION
> > 'FOREIGN-FUNCTIONS:DEF-FOREIGN-CALL (FBOUNDP 'SBRK))
> > (PUSH 'SBRK EXCL::.FUNCTIONS-DEFINED.))
> > (EVAL-WHEN (COMPILE LOAD EVAL) (REMPROP 'SBRK 'SYSTEM::DIRECT-FF-CALL))
> > (SETF (FDEFINITION 'SBRK)
> > (EXCL::GET-FF-APPLY-CLOSURE
> > (EXCL::DETERMINE-FOREIGN-ADDRESS '("sbrk" :LANGUAGE :C) 514 NIL)
> > '(:INT (INTEGER * *))))
> > (RECORD-SOURCE-FILE 'SBRK) 'SBRK)
>
> What has this got to do with macros and/or symbolic programming per se?
It's what the macro produced, under the hood.
> That was the question. And it was a newbie question, so no way they
> should be expected to realize:
>
> > eval-when normally appears as a top level form, but it is meaningful
> > for it to appear as a non-top-level form.
> > However, the compile-time side effects described in Section 3.2 Compilation
> > only take place when eval-when appears as a top level form.
But it is not necessary to understand this paragraph for this macroexpansion,
because the eval-when clauses are presumably at top-level for this
discussion.
> We had a chat here recently questioning if macros were ever even
> necessary, so it's no good to show a lot of cool stuff going on, the
> newbie question was "why macros and symbolic programming". ie, what they
> can they do other approaches cannot.
The significance of macros is not what they are, but what they produce.
You can take a defining macro, and expand it to all sorts of code which
does huge amounts of work that would otherwise be tedious and error-prone
to code by hand. Or, you can expand a complex defstruct or foreign type
accessor, and have it produce what eventually compiles down to one
instruction. That seems to me to be pretty powerful, and I wouldn't
want to give something like that up in my language of choice.
Duane Rettig
> Duane Rettig wrote:
> >
> > Well, if you've understood almost nothing, then at least it's not
> > nothing. Tell me what you have understood of it so far, and I'll
> > fill in at least some of the rest.
>
> I've just tried to describe what I understand and what do not. When I
> write 'I understand' I don't mean 'I know all details, how it works,
> which cases etc.' but only 'I have a point from which I could go
> further when needed.' So let me to proceed.
Yes, that's the right way to go about doing it. It is not necessary
to understand all of what you see. Perhaps that is one of the first
great points about lisp; you can expose parts of it a little at a time,
without having to understand all of it at once. Of course, other
languages that have macro capabilities can do similar things, but
usually it's an all-or-nothing thing (have you ever tried to read
the .i file output from "cc -P xxx.c", where xxx.c includes lots of .h
files?). CL has macro expansion tools which let you work on all or
parts of the expansion process.
For others' benefit, the following form is what was expanded by
Vladimir from the form '(ff:def-foreign-call sbrk ()); I will be
adding to the good effort he started and explain some of the mechanism
that he is seeing. Note that the specifics may or may not have to do
with macros per-se (yes, Kenny, I saw your posting just recently) but
it does demonstrate the fact that macros allow one to see down into the
implementation as far as the implementor wishes to expose it. In other
languages, the implementation is opaque, or at most one or two levels
of macros (which ironically tend to be designed for information _hiding_).
One more note before I start; a lot of the implementation of macros
which define things has to do with eval-when. A subtlety of the
code-as-data feature of Lisp which we have talked about at length
on a thread several months ago is the "when" of Lisp; most languages
only have one "time" available to its programs, and that is "execute"
time. Compile time is not available to a program; it is done by a
compiler in a separate process. Macroexpand time is done by a
preprocessor or by the same compiler. Read-time? Also done by the
compiler. In fact, since parsing is a useful run-time feature, many
parsers are built in other programming languages, to fulfill a need
that is not a need in Lisp; with the lisp reader you already have
powerful parsing capability for lisp syntax at all levels (macro, code,
data).
Because Lisp offers distinctions for "times" as e.g. read-time,
compile-time, load-time execute-time, etc., it is both foreign to
non-lisp programmers and it is exciting when a new lisp programmer
"gets" it (via that "aha" experience). Ironically, I "got" the
times issue when I learned Forth, with its <builds does> construct
(which distinguishes between definition time and execution time).
OK, now let's assume that the expanded form was a top-level form
in a file. A progn doesn't change top-level-ness, so all of the
top-level forms within the forms are also at top-level.
> (progn
>
> ;; What purpose the following form stands for ? Which cases it works
> ;; in ? E.g., what I should write to see it in action ? Deciding on
> ;; EVAL-WHEN it works during compilation only, but that is all I
> ;; could say about it. It seems all defined functions go in
> ;; excl::.functions-defined.
>
> (eval-when (:compile-toplevel)
> (excl::check-lock-definitions-compile-time
> 'sbrk 'function 'foreign-functions:def-foreign-call (fboundp 'sbrk))
> (push 'sbrk excl::.functions-defined.))
When at top-level, the :compile-toplevel simply tells the compiler to
evaluate the form at compile-time. Note that there are actually
two forms here; The second one, as you noted, is doing some
bookkeeping for the current with-compilation-unit (implied or explicitly
specified), and will keep any calls to sbrk within the file being
compiled from warning that there is no such definition. Similar things
are done with defun, and if you say (compile (defun foo () (bar)))
then you will get a warning, unless you have already defined bar
or unless it is also defined in the same compilation unit (a file
compilation serves as a simple compilation-unit).
Also though, the first form is checking the symbol and its package
against package-locking, and warns or errors if there is some kind
of violation. This occurs at compile-time, so that the decision as
to whether to warn or to error at compile-time can be made separately
from the decision to error and refuse to store the new value by the
(setf fdefinition) form, below. This separation between compile-time
and load-time is critical to the ability of Lisp to be friendly to the
user while being harsh to mistakes; a programmer creating a defining
macro might want to warn only during compile-time so that the user
can gather all of the warnings and respond to all at one time, and
the programmer can also arrange for the load-time action to be
either a continuable error or an error, depending on the level of
recovery desired.
> ;; The next form I don't understand also. Why EVAL-WHEN needed ?
> ;; What is the value of system::direct-ff-call initially (probably
> ;; it initialized in ff package). What would happen if SBRK wasn't
> ;; removed from SYSTEM::DIRECT-FF-CALL ?
>
> (eval-when (compile load eval)
> (remprop 'sbrk 'system::direct-ff-call))
The three qualities compile, load, and eval are actually synonyms for
:compile-toplevel, :load-toplevel, and :execute. It essentially means
all times except for read-time. So whenever the compiler encounters
this form, it will excecute it, and whenever the file is loaded into
the lisp (either compiled or in source form) it will be evaluated.
Now, as to the form itself, it simply means that whenever the defining
form is encountered, the 'sbrk symbol will no longer have the
direct-ff-call property regardless of whether it did earlier. This
keeps us from keeping such an out-of-date property from a previously
defined function called sbrk. Please note the second macroexpansion
below for an example of this property being stored.
> ;; The following three forms I understood (more or less), except
> ;; excl::get-ff-apply-closure. Probably it returns the closure that
> ;; being funcalled gives the call to sbrk(2).
>
> (setf (fdefinition 'sbrk)
> (excl::get-ff-apply-closure
> (excl::determine-foreign-address '("sbrk" :language :c) 514 nil)
> '(:int (integer * *))))
>
> (record-source-file 'sbrk)
> 'sbrk)
Yes, that's precisely right. We use a technique quite often to save a lot
of space, in creating closure objects which only contain the closed-over
variables and a template function. In this case, the code is fairly
trivial, so I'll show it:
(defun get-ff-apply-closure (address returning)
(named-function foreign-function
(lambda (&rest args)
(declare (dynamic-extent args))
(sys::ff-apply address returning nil args))))
Note first the presence of an Allegro CL extension called named-function.
It is described, as is this similar closure-building technique, in
http://www.franz.com/support/documentation/6.1/doc/pages/operators/excl/named-function.htm
The two arguments (address and returning) are the closed-over variables,
and the function template implements the code. In fact, since
get-ff-apply-closure is compiled, so too is the named-function that is
returned. This allows the functionality which is stored into sbrk's
function cell to be pre-compiled, even if the lisp image you are working
on has had the compiler removed.
Finally, compare the above macroexpansion of
(ff:def-foreign-call sbrk ())
which specifies no arguments and thus does no argument checking, and
which returns an integer value by default, against the following
definition for which one (integer, by default) argument is specified,
and for which I have given the :call-direct option:
CL-USER(6): (pprint (macroexpand '(ff:def-foreign-call sbrk (n) :call-direct t)))
(PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL)
(EXCL::CHECK-LOCK-DEFINITIONS-COMPILE-TIME 'SBRK 'FUNCTION
'FOREIGN-FUNCTIONS:DEF-FOREIGN-CALL (FBOUNDP 'SBRK))
(PUSH 'SBRK EXCL::.FUNCTIONS-DEFINED.))
(EVAL-WHEN (COMPILE LOAD EVAL)
(SETF (GET 'SBRK 'SYSTEM::DIRECT-FF-CALL)
(LIST '("sbrk" :LANGUAGE :C) T :C '(:INT (INTEGER * *))
'((:INT (INTEGER * *))) T 2)))
(SETF (FDEFINITION 'SBRK)
(LET ((EXCL::F
(NAMED-FUNCTION SBRK
(LAMBDA (N)
(EXCL::CHECK-ARGS '((:INT (INTEGER * *))) 'SBRK
N)
(SYSTEM::FF-FUNCALL (LOAD-TIME-VALUE
(EXCL::DETERMINE-FOREIGN-ADDRESS
'("sbrk" :LANGUAGE :C)
2
NIL))
'(:INT (INTEGER * *)) N
'(:INT (INTEGER * *)))))))
(EXCL::SET-FUNC_NAME EXCL::F 'SBRK)
EXCL::F))
(RECORD-SOURCE-FILE 'SBRK) 'SBRK)
CL-USER(7):
I'll let you do the same analysis on this one if you like; obviously
there is more to this definition, and there are things which will be
opaque, but it should help you to understand how to perform some
experimentation with macroexpansion to see into the implementation
somewhat.
Note that the same macroexpansions can be done on defun forms, and even on
defmacro forms, and for any macro form (even macros which do not define).
Try (pprint (macroexpand '(defmacro foo (x) `(list ,x)))), for example.
Well, this turned out to be longer than I expected, but hopefully I've
communicated some of what I see in macros that makes CL especially
powerful.
Kenny Tilton
Duane Rettig wrote:
> Note that the specifics may or may not have to do
> with macros per-se (yes, Kenny, I saw your posting just recently)...
damn, the one time I really needed Usenet to lose an article. sorry
about my sh*t-eating tone in that.
--
kenny tilton
clinisys, inc
---------------------------------------------------------------
"We have a pond and a pool. The pond would be good for you."
- Ty to Carl, Caddy Shack
Thomas F. Burdick
> The significance of macros is not what they are, but what they produce.
> You can take a defining macro, and expand it to all sorts of code which
> does huge amounts of work that would otherwise be tedious and error-prone
> to code by hand. Or, you can expand a complex defstruct or foreign type
> accessor, and have it produce what eventually compiles down to one
> instruction. That seems to me to be pretty powerful, and I wouldn't
> want to give something like that up in my language of choice.
Hopefully the OP is still around to read this. That's a good
explanation although the timing is a bit late (unless he's still
around of course).
--
/|_ .-----------------------.
,' .\ / | No to Imperialist war |
,--' _,' | Wage class war! |
/ / `-----------------------'
( -. |
| ) |
(`-. '--.)
`. )----'
Vladimir Zolotykh
more familiar with such 'cool stuff' and expansions like that produced
by
(pprint (macroexpand '(ff:def-foreign-call sbrk (n) :call-direct t)))
don't fright me any more.
BTW, I find the following
(macroexpand '(do ((x 1 (1+ x)) (y 2 (1+ y))) ((> x 10) x) (print x)))
(macroexpand '(and a b c))
also illustrative.
--
Vladimir Zolotykh gsm...@eurocom.od.ua
Ralph Richard Cook
had some mini-insights as I've read the replies.
I'd like to thank Paul Graham for making On Lisp available (surely he
reads the posts on comp.lang.lisp, at least the ones with his name in
the subject line :-) ).
This is from the perspective of a Java programmer that's dipping his
toe in the Lisp waters and have read these posts and the first three
chapters of "On Lisp".
Things that don't impress me as much as you'd think:
Dynamic typing - I was exposed to that in Python & other scripting
languages
Garbage collection - Java has it
Anonymous functions - They're cool, and they may be simpler to do in
Lisp, but you can do much of the same thing in Java with anonymous
inner classes if you'd like
The setf macro - Kaz Kylheku pointed out that this replaces what
C-like languages have as 'lvalues', a compiler feature. However, in
Lisp there really are no 'lvalues', everything is a function call. The
setf macro doing different things based on object type, and the
ability to add new setf's (storing a value as XML in a database table
or something else equally wacky) could be done with OOP polymorphism.
Things that do impress me:
Lists/S-expressions as a 'built-in' data type - As Paul Tarvydas
points out, when other languages try to put lists in classes it can
really mess things up. One thing I think of as an advantage in Java is
the OOP nature of it but it still has primitives. Sometimes a function
just needs to return the number 3, and that's certainly more efficient
than an number object that has the value of 3, like it would be in
Smalltalk. Lisp does this by making lists an efficient primitive. Once
I 'got' lists I see why (cdr aList) is no big deal, you just move your
'pointer' to the next element in the list, so all those recursive
functions that pass the cdr of a list to themselves is an efficient
way of working on a list.
What I'm going to do now is finish 'On Lisp' and try my hand at a home
project using Lisp. That should help me form an opinion on macros
since everyone here says you can't write a significant Lisp program
without winding up making a macro or two. It will probably be a while
but I'll report back here with a Google link to this thread.
Thanks again,
Richard
Kenny Tilton
Ralph Richard Cook wrote:
>
> Things that don't impress me as much as you'd think:
>
> Dynamic typing - I was exposed to that in Python & other scripting
> languages
>
> Garbage collection - Java has it
Well I'll be impressed when I find them both in one language, along with
maturity, an ANSI standard, performance close to C, generic methods,
macros, special variables, restarts from a backtrace after fixing the
code, and macros like the one I did this evening which expands this tidy
little form:
(DEF-C-SLOT names (person)
:echo ((self new-names) (format t "~&You can call me ~a" new-names))
:unchanged-if (() (equal new-Value old-Value)))
...into all the plumbing required by my constraints engine for it to
work...
(progn (defmethod names ((self person)) (c-slot-value self 'names))
(defmethod (setf names) (new-value (self person))
(setf (c-slot-value self 'names) new-value))
(eval-when (compile load) (setf (get 'names :ephemeral) nil))
(def-c-echo (names) ((self person) new-names old-value)
(format t "~&you can call me ~a" new-names))
(defmethod c-unchanged-p ((self person) (slotname (eql 'names))
new-value old-value)
(declare (ignorable new-value old-value)) (equal new-value
old-value)))
The macro looks a little bit like this:
(DEFMACRO def-c-slot (slotname (&optional (classname t)) &key ephemeral
echo unchanged-if)
`(progn
(DEFMETHOD ,slotname ((self ,classname))
(c-slot-value self ',slotname))
(DEFMETHOD (setf ,slotname) (new-value (self ,classname))
(setf (c-slot-value self ',slotname) new-value))
(eval-when (compile load)
(setf (get ',slotname :ephemeral) ,ephemeral))
,(when echo
(destructuring-bind ((&optional (self-var 'self)
(new-value-var 'new-value)
(old-value-var 'old-value))
&body echo-body) echo
`(DEF-C-ECHO (,slotname) ((,self-var ,classname)
,new-value-var ,old-value-var)
,@echo-body)))
,(when unchanged-if
(destructuring-bind ((&optional (self-var 'self)
(new-value-var 'new-value)
(old-value-var 'old-value))
&body unchanged-body) unchanged-if
`(DEFMETHOD c-unchanged-p ((,self-var ,classname)
(slotname (eql ',slotname))
,new-value-var ,old-value-var)
(declare (ignorable ,new-value-var ,old-value-var))
,@unchanged-body)))))
Aha?
Friedrich Domincus
> I want to thank everyone for their responses to my original post. I've
> had some mini-insights as I've read the replies.
>
> I'd like to thank Paul Graham for making On Lisp available (surely he
> reads the posts on comp.lang.lisp, at least the ones with his name in
> the subject line :-) ).
>
> This is from the perspective of a Java programmer that's dipping his
> toe in the Lisp waters and have read these posts and the first three
> chapters of "On Lisp".
>
> Things that don't impress me as much as you'd think:
It's not about impressing you. Fact is that
>
> Dynamic typing - I was exposed to that in Python & other scripting
> languages
is much older and have had it forever.
>
> Garbage collection - Java has it
Well from what do they take their idea? Fact is again tha Lisp has GC
for Ages.
>
> Anonymous functions - They're cool, and they may be simpler to do in
> Lisp, but you can do much of the same thing in Java with anonymous
> inner classes if you'd like
Great of course it is much less work to declare inner classes. How
much do you have to write for it? Well you might have to implement a
"top-clas" than derive different classes and call their action during
a polymorphig dispatch. I can do the same Lisp but who would do that?
>
> What I'm going to do now is finish 'On Lisp' and try my hand at a home
> project using Lisp. That should help me form an opinion on macros
> since everyone here says you can't write a significant Lisp program
> without winding up making a macro or two.
Well why should you not beeing able to get along without Macros?
Friedrich
Ola Rinta-Koski
> without winding up making a macro or two.
You can, strictly speaking, but macros tend to make the program
shorter by allowing more reuse of code. Shorter means also clearer,
since you can hide all the messy details inside the macro instead of
the program logic - they won't go away, but they'll be concentrated
in one place.
--
Ola Rinta-Koski o...@cyberell.com
Cyberell Oy +358 41 467 2502
Rauhankatu 8 C, FIN-00170 Helsinki, FINLAND www.cyberell.com
Larry Kramer
>
> ral...@bellsouth.net (Ralph Richard Cook) writes:
> > since everyone here says you can't write a significant Lisp program
> > without winding up making a macro or two.
>
> You can, strictly speaking, but macros tend to make the program
> shorter by allowing more reuse of code. Shorter means also clearer,
> since you can hide all the messy details inside the macro instead of
> the program logic - they won't go away, but they'll be concentrated
> in one place.
Well, let me vent a little bit...
First of all, I disagree with the statement that "you can't write
a significant Lisp program without winding up making a macro or two."
I've worked on many signfician Lisp programs that didn't use macros.
In addition, I don't think code reuse is a good reason for writing
macros. If you have a block of code with a bunch of messy details
-- especially if you find yourself repeating it -- make it into
a function or a method.
As a matter of fact, I subscribe to the rule that I will only
write a macro when it is absolutely necessary -- for instance,
when a function would suffice, but it needs to handle its
arguments in a special way.
I've worked on too many Lisp projects where people that preceded
me used macros where they did not need to, thus making the code
more impenetrable. It's much easier to read function calls than
having to macroexpand a bunch of code to figure out what's going
on.
Larry
Duane Rettig
Hi, Larry!
Larry Kramer <lkra...@cs.cmu.edu> writes:
> Ola Rinta-Koski wrote:
> >
> > ral...@bellsouth.net (Ralph Richard Cook) writes:
> > > since everyone here says you can't write a significant Lisp program
> > > without winding up making a macro or two.
> >
> > You can, strictly speaking, but macros tend to make the program
> > shorter by allowing more reuse of code. Shorter means also clearer,
> > since you can hide all the messy details inside the macro instead of
> > the program logic - they won't go away, but they'll be concentrated
> > in one place.
>
> Well, let me vent a little bit...
>
> First of all, I disagree with the statement that "you can't write
> a significant Lisp program without winding up making a macro or two."
I agree with you, and also disagree with the above statement. It is
indeed possible to create significant CL programs without using any
macros. CL would be turing complete without macros (except, of course,
that some CL functionality is _defined_ as macros).
> I've worked on many signfician Lisp programs that didn't use macros.
I hope you meant "create" rather than "use". Just by using CL you are
using macros.
> In addition, I don't think code reuse is a good reason for writing
> macros. If you have a block of code with a bunch of messy details
> -- especially if you find yourself repeating it -- make it into
> a function or a method.
> As a matter of fact, I subscribe to the rule that I will only
> write a macro when it is absolutely necessary -- for instance,
> when a function would suffice, but it needs to handle its
> arguments in a special way.
That's one way to use macros. However (and especially regarding
code reuse) a macro which serves as a function generator can in fact
make the generation of many similar-but-slightly-different functions
or methods easier and more error-free. The trick in making such
macros readable is in realizing that a macro is itself a little
transformation program, and in that respect must be built with the
same careful design as any function. It's OK to call functions
at macroexpand time, and it's ok to break up a macro into several
pieces. I've seen many macros that were obviously designed by
programmers who were in a hurry to get to the "real" functionality
(the generated code) and who thus threw the macro together with no
consideration for maintainability or reuse. When you see something
like this, don't blame it on macros in general, blame it on bad
programming, just as you would blame a badly written function on
bad programming.
> I've worked on too many Lisp projects where people that preceded
> me used macros where they did not need to, thus making the code
> more impenetrable. It's much easier to read function calls than
> having to macroexpand a bunch of code to figure out what's going
> on.
Yes, programmers sometimes tend to get carried away when they first
discover macros. This is true in any language. I remember when I
was in a different job, in a C shop, where an ex-Pascal programmer
had defined several macros like:
#define begin {
#define end }
which of course made the C code completely unreadable (not because
it looked like Pascal, which is in an of itself fairly readable, but
because it looked like C trying to look like Pascal in a C shop).
Macro usage and design is an art, just like program design. Overuse or
underuse will still lead to working programs, but just the right amount
of use will lead to beautiful working programms.
Bijan Parsia
[snip]
> Lists/S-expressions as a 'built-in' data type - As Paul Tarvydas
> points out, when other languages try to put lists in classes it can
> really mess things up. One thing I think of as an advantage in Java is
> the OOP nature of it but it still has primitives. Sometimes a function
> just needs to return the number 3, and that's certainly more efficient
> than an number object that has the value of 3, like it would be in
> Smalltalk.
Bzzt. 3, being a SmallInteger, is an immediate object. That is, its value
is encoded in its reference.
Even if not, what *kind* of efficiency are you talking about? Space
"efficiency" is mostly recovered by immediate encoding. Speed of
operations in a tight loop is a different problem, but not because of the
object representation *per se*, but because of the need to do dynamic
dispatch. This can be optimized in a lot of cases.
I'm not sure how often a function *really needs* to return a 3 value
rather than a 3 object. Numerically intensive code is hardly the standard,
and there are *way* better choices than most Java systems if you are doing
numerics (Common Lisp, for example, which standardly represents FIXNUMs as
Smalltalk does, but has optional type declarations, etc. which ease a
variety of operations). Even in a naive Smalltalk system, you can punch
our of the dispatch loop in a variety of ways (e.g., Squeak has several
"IntegerArray" classes which define primative arithmetic operations over
their members which never see the dispatch loop; this isn't as elegant as
fully optimization, but it's rather easy to implement and deal with in a
cross platform way).
Cheers,
Bijan Parsia.
Ola Rinta-Koski
> Larry Kramer <lkra...@cs.cmu.edu> writes:
> > Ola Rinta-Koski wrote:
> > > macros tend to make the program shorter by allowing more reuse
> > > of code.
> > I don't think code reuse is a good reason for writing macros. If
> > you have a block of code with a bunch of messy details --
> > especially if you find yourself repeating it -- make it into a
> > function or a method.
Of course, if at all possible. But often it's not, or would make the
code much more convoluted.
> However (and especially regarding code reuse) a macro which serves
> as a function generator can in fact make the generation of many
> similar-but-slightly-different functions or methods easier and more
> error-free.
This is part of what I was getting at with my comment regarding code
reuse. However, a lot of my macros are similar to existing WITH-*
macros, which establish bindings and so on, but aren't function
generators. So it's syntactic sugar, but very useful as such.
Here's an example of HTML generation with macros. I think the
easiest way of outputting matching HTML start and end tags is with
macros, and it couldn't be done with functions in a similar way.
TABLE, TR and TD are HTML-generating macros, and there's also
DESTRUCTURING-DOLIST which I noticed I was writing a lot and so
decided to turn it into a macro for shorter code and IMO nicer
syntax - the definition is also below.
(defmacro destructuring-dolist ((list &rest vars) &body body)
(let ((item (gensym)))
`(dolist (,item ,list)
(destructuring-bind ,vars ,item ,@body))))
;; example of use of HTML generator macros
(when items
(h2 stream "Upcoming events:")
(table (stream)
(destructuring-dolist (items id starts expires priority
status category title description)
(declare (ignore id priority status category))
(tr (stream)
(td (stream)
(format stream "~%~A~@[ to ~A~]~%"
starts
(unless (same-day starts expires)
expires)))
(td (stream)
(format stream "~A (~A)<BR>~&"
title description))))))
bluefox420
of "aha" moments. Check out Chap. 6 where he shows some real funky
stuff with closures.
Cheers.