[OT] Power of Lisp macros?

5 views
Skip to first unread message

Adriano Nagelschmidt Rodrigues

unread,
Oct 23, 2002, 5:43:08 PM10/23/02
to perl6-l...@perl.org
Hi,

Perl is my favorite language, and I'm eagerly following Perl 6 development. So
I would like to ask this question here. Sorry if I'm being inconvenient...

Do you think that Lisp macros make the language more powerful than others (eg
Perl)? I mean, do they really give a competitive advantage, or are they being
overrated (see below)?

I've been reading Paul Graham's articles (www.paulgraham.com), they are very
interesting. Here's a paragraph from "Beating the Averages":


But I think I can give a kind of argument that might be convincing.
The source code of the Viaweb editor was probably about 20-25%
macros. Macros are harder to write than ordinary Lisp functions,
and it's considered to be bad style to use them when they're not
necessary. So every macro in that code is there because it has to
be. What that means is that at least 20-25% of the code in this
program is doing things that you can't easily do in any other
language. However skeptical the Blub programmer might be about my
claims for the mysterious powers of Lisp, this ought to make him
curious. We weren't writing this code for our own amusement. We
were a tiny startup, programming as hard as we could in order to
put technical barriers between us and our competitors.


What do you think? I have very little Lisp experience...

Thanks,

--
Adriano

Luke Palmer

unread,
Oct 23, 2002, 5:58:54 PM10/23/02
to te...@safira.com, perl6-l...@perl.org
> Date: Wed, 23 Oct 2002 18:43:08 -0300
> From: Adriano Nagelschmidt Rodrigues <te...@safira.com>

>
> Hi,
>
> Perl is my favorite language, and I'm eagerly following Perl 6
> development. So I would like to ask this question here. Sorry if I'm
> being inconvenient...
>
> Do you think that Lisp macros make the language more powerful than
> others (eg Perl)? I mean, do they really give a competitive
> advantage, or are they being overrated (see below)?

If you define "powerful" as "can do more things," then of course not.
Lisp is implemented in C, and C's macros are certainly not essential
to its functionality. But think of what macros in general provide:

* Multi-platform compatability
* Easier maintenance

Perl has no problem with the former. It's multi-platform by nature.
But is has as much of a problem with the latter as any other language,
except Lisp. That is one of the continuing strong points of Lisp: it
can change very, very quickly.

However, they are intending to make it possible to write things like
C<if> with subs, which will imply most of the power of
macros... though I imagine it won't be possible to have the level of
introspection lisp macros have (infinite). But, that design team is
very clever, so you never know.

This kind of thing should reveal itself in the next Apocalypse. Then
we can see Larry's "vision," and make appropriate adjustments in terms
of that. Right now, it's very fuzzy.

Oh, and you aren't being inconvenient. These kinds of questions are
what the list is for: ask away!

Luke

Adriano Nagelschmidt Rodrigues

unread,
Oct 24, 2002, 11:26:41 AM10/24/02
to perl6-l...@perl.org
Luke Palmer writes:
> > Do you think that Lisp macros make the language more powerful than
> > others (eg Perl)? I mean, do they really give a competitive
> > advantage, or are they being overrated (see below)?
>
> If you define "powerful" as "can do more things," then of course not.

No, of course. I guess any language is a Turing Machine, after all...

I mean power in the sense of "more high level", that could be measured in
(fewer) lines of code. Would I exaggerate if I said that the C/Perl
compression rate could approach 10 in certain cases?

Then I could point you to some benefits Perl has over C, for some classes of
problems: faster development, easier maintenance, capability to better develop
bigger systems.

And Perl excels in solving a wide range of problems.

> Lisp is implemented in C, and C's macros are certainly not essential
> to its functionality. But think of what macros in general provide:
>
> * Multi-platform compatability
> * Easier maintenance
>
> Perl has no problem with the former. It's multi-platform by nature.
> But is has as much of a problem with the latter as any other language,
> except Lisp. That is one of the continuing strong points of Lisp: it
> can change very, very quickly.

Yes. And what would this kind of "meta programming" allow? Perhaps thoughts
like this:

"Now I need code for these n cases. I will just write a macro."

Maybe it makes complex problems suddenly appear more "tractable", allows for
more code reuse/factorization?

> However, they are intending to make it possible to write things like
> C<if> with subs, which will imply most of the power of
> macros... though I imagine it won't be possible to have the level of
> introspection lisp macros have (infinite). But, that design team is
> very clever, so you never know.

Well, I have to confess that I'm asking about macros, but I don't even
remember when was the last time I used a closure ;-)

> This kind of thing should reveal itself in the next Apocalypse. Then
> we can see Larry's "vision," and make appropriate adjustments in terms
> of that. Right now, it's very fuzzy.

Nice, I'm looking forward to reading it.

> Oh, and you aren't being inconvenient. These kinds of questions are
> what the list is for: ask away!

Thanks!

Best regards,

--
Adriano

Adam Turoff

unread,
Oct 24, 2002, 6:38:33 PM10/24/02
to perl6-l...@perl.org
On Thu, Oct 24, 2002 at 12:26:41PM -0300, Adriano Nagelschmidt Rodrigues wrote:
> Luke Palmer writes:
> > Lisp is implemented in C, and C's macros are certainly not essential
> > to its functionality. But think of what macros in general provide:
> >
> > * Multi-platform compatability
> > * Easier maintenance
> >
> > Perl has no problem with the former. It's multi-platform by nature.
> > But is has as much of a problem with the latter as any other language,
> > except Lisp. That is one of the continuing strong points of Lisp: it
> > can change very, very quickly.
>
> Yes. And what would this kind of "meta programming" allow? Perhaps thoughts
> like this:
>
> "Now I need code for these n cases. I will just write a macro."
>
> Maybe it makes complex problems suddenly appear more "tractable", allows for
> more code reuse/factorization?

Damian's Switch.pm is like a Lisp macro. It extends Perl syntax for a
certain kind of problem, and makes it easier to write a common code
pattern. The thought process might go something like this:

"I want to check a variable against a variety of conditions, and
I'm tired of writing the same long-winded and error prone
if/elsif/elsif/elsif/else cascade. This is a common 'switch'
statement, except that I want to match a variety of conditions
intelligently (integers, string equality, regexes, etc.)."

When Paul Graham writes that 25% of his Viaweb code was macros,
he's really saying that Common Lisp wasn't well suited to his
problem domain (writing an ecommerce app). However, Common Lisp
*allowed* itself to be extended in that directon, with macros.
The result is that the 75% of his code that comprised the guts of
Viaweb could have been written without macros (or in a language
other than Common Lisp), but would have taken significantly more
effort and code (and led to more bugs).

Perl source filters are similar to Lisp macros to a small degree.
Lisp macros are Lisp functions that are invoked at compile time to
transform a *tokenized* Lisp program (using Lisp data structures)
into a different set of tokens (which are then compiled). Source
filters tend to act on raw text, not tokenized source code, and
need to deal with the problematic aspects of Perl syntax (comments,
Pod, HEREDOCs, etc.) every time they are written/invoked.

Because Lisp macros are real Lisp code, they are far more powerful than
the textual substitutions that pass for "C macros".

Perl6 is moving to include something like Lisp macros. Perl5 source
filters are a rough approximation, and a preview of things to come.

Z.

Guillaume Germain

unread,
Oct 24, 2002, 6:31:54 PM10/24/02
to Luke Palmer, te...@safira.com, perl6-l...@perl.org
On Wednesday 23 October 2002 17:58, Luke Palmer wrote:
> > From: Adriano Nagelschmidt Rodrigues <te...@safira.com>
>> [...]

> > Do you think that Lisp macros make the language more powerful than
> > others (eg Perl)? I mean, do they really give a competitive
> > advantage, or are they being overrated (see below)?
>
> If you define "powerful" as "can do more things," then of course not.

We're probably not talking about Turing-completeness here, so I'm not sure
your point is valid. You sure can 'do more thing' in a language with
Lisp-style macros (read _On Lisp_ and blow your mind). As for the competitive
advantage etc., this does have a lot to see with the programmer. As for OOP
or patterns or whatever, there is no silver bullet.

> Lisp is implemented in C, and C's macros are certainly not essential
> to its functionality.

Humph. What does the fact that Lisp *can be* implemented in C have to see
with C macros? BTW, it would be an insult to Lisp to compare C's
macros to Lisp's macros. C macros are a stupid text replacement facility,
they don't have anything to see with Lisp-style macros.

> But think of what macros in general provide:
>
> * Multi-platform compatability
> * Easier maintenance

That's for C macros. It doesn't make much sense if you are talking about
Lisp macros.

> However, they are intending to make it possible to write things like
> C<if> with subs, which will imply most of the power of
> macros...

Well, if it means controling evaluation and making arbitrary code
manipulation, then yes, probably...

> though I imagine it won't be possible to have the level of
> introspection lisp macros have (infinite).

?

Manipulating arbitrary AST in Perl would be very complicated, those 'pesky'
parenthesis in Lisp account for a lot of the magic that can go on while
keeping things at a reasonable amount of complexity.

Guillaume

Marco Baringer

unread,
Oct 25, 2002, 6:04:12 AM10/25/02
to perl6-l...@perl.org
Luke Palmer <fibo...@babylonia.flatirons.org> writes:

> If you define "powerful" as "can do more things," then of course not.
> Lisp is implemented in C, and C's macros are certainly not essential

[aside: most "major" common lisp implementations (cmucl, sbcl,
openmcl, mcl, allegro and lispworks) are all native compilers
implemented in lisp (and some assembler for boot straping). CLISP is
the only "major" implementation whose core is in fact implemented in a
C dialect (for speed reasons (the virtual machine is really slow))]

> But think of what macros in general provide:
>
> * Multi-platform compatability
> * Easier maintenance

* Creating/Embedding custom languages. aka - adapting the
langauge to your problem domain.

common lisp macros allow you to locally extend the vocabulary of
common lisp in the same way engineers have locally (ie within the
engineering domain) extended english with new syntax/semantics to deal
with engineering problems.

macros are functions which are run when the source code is read
(parsed). the argument to a macro is source code (expressed as a data
structure and not simple text) and the return value is source code
(not text). this is a fundamental difference between C's
text-processing macros, without this macros lose most of their power
and become too hard to write to be used.

common lisp's LOOP is a great example of what you can do with macros
(and the best iteration construct around). see
http://www.lispworks.com/reference/HyperSpec/Body/06_a.htm for the
spec and http://www.ai.sri.com/~pkarp/loop.html for a tutorial.

random points and counter-points:

- what macros are really good at is embedding mini-languages and
creating new idioms, this often goes with, but is not nessecarily
related to, reducing lines of code. example: CLOS/MOP (common lisp
object system/meta object protocol) are implemented as macros on top
of non-OO lisp (this statement maybe be a lie if you go deep enough
into the specifics of some implementations).

- the power of lisp's macros is that they allow you to perform
arbitrary code transformations by working with code as if it was
data. at one point there was discussion about having perl subs with
"auto-args" (i forget where i read about this) where by the
arguments to the sub where determined by parsing the body of the sub
itself and looking at what variables where used, this is a trivial
macro in lisp. adding this to perl5 required a source filter which
took forever to write and was never used because is was never
reliable enough (this may say more about my capabilities as a
programmer than about perl5 source filters).

- everything you can do with macros you can do without, since macros
always get expaned (translated) into "regular" common lisp
code. however, sometimes (like with CPS) hand writing the output is
prohibitly difficult.

- some people consider macros to actually reduce maintainability since
they perform arbitrary code manipulations, so you have _no_ idea of
what is going on if you don't know what the macro does. macros which
introduce new symbols are especially vulnerable to this.

- any sufficently powerful tool can be used to shot yourself in the
foot (or blow off your head). i doubt this java-esque argument
(let's "protect" the programmers from themselves) has any weight
with perl programmers, but it's something i've heard more than once.

- writing realiable/robust source filters is hard (do able, but hard,
even with The Damien's Filter::Simple). writing grammars is better,
but still hard, and more importantly, both require a rdically
different mind set from "regular" programming. the ease of writing
lisp macros is largely due to the fact that lisp has no syntax
(almost), and that lisp's syntax is programmable. perl6 will have
the second and can't do much about the first (sort of goes against
"different things should look different").

just another lurker's rant...
--
-Marco
Ring the bells that still can ring.
Forget your perfect offering.
There is a crack in everything.
That's how the light gets in.
-Leonard Cohen

Piers Cawley

unread,
Oct 25, 2002, 7:24:38 AM10/25/02
to af...@corp.vlex.com, Marco Baringer, perl6-l...@perl.org
Angel Faus <af...@corp.vlex.com> writes:

> Speaking about macros, I renember reading somewhere something about
> Scheme hygenic macros, but i didn't really understood it.
>
> Do they solve the maintenance problems of Lisp macros? Would they be
> applicable to perl?

Scheme hygenic macros do a lot of the bookkeeping for you, so you
don't have to muck about with gensym and are generally safe. The
problem is (judging by what the Common Lisp types say, I don't have
experience in this area myself), sometimes you need to do things that
would be considered 'dangerous' and, unless the scheme implementation
you're working with has some none standard extensions giving a scheme
like C<defmacro>, you're up the well known creek without a paddle.

--
Piers

"It is a truth universally acknowledged that a language in
possession of a rich syntax must be in need of a rewrite."
-- Jane Austen?

Angel Faus

unread,
Oct 25, 2002, 11:02:37 AM10/25/02
to Marco Baringer, perl6-l...@perl.org

Speaking about macros, I renember reading somewhere something about
Scheme hygenic macros, but i didn't really understood it.

Do they solve the maintenance problems of Lisp macros? Would they be
applicable to perl?

Thanks for any tips,

-angel

Larry Wall

unread,
Oct 25, 2002, 12:54:20 PM10/25/02
to Marco Baringer, perl6-l...@perl.org
On 25 Oct 2002, Marco Baringer wrote:
: Luke Palmer <fibo...@babylonia.flatirons.org> writes:
: > But think of what macros in general provide:

: >
: > * Multi-platform compatability
: > * Easier maintenance
: * Creating/Embedding custom languages. aka - adapting the
: langauge to your problem domain.
:
: common lisp macros allow you to locally extend the vocabulary of
: common lisp in the same way engineers have locally (ie within the
: engineering domain) extended english with new syntax/semantics to deal
: with engineering problems.

Only up to a point. Engineers sometimes muck with the language at
the parse level, before the macro processor even has a shot at it.
Lisp gets away with this only because its syntax unambiguously
distinguishes verbs from nouns. But engineers are always messing
around with their word categories. How about using a verb as a
predicate adjective:

All systems are go for launch.

That's probably derived from something more like:

All systems are "go" for launch.

So a macro system that takes preparsed text is still not powerful
enough. It could be argued that you just pass in a string of
data tokens without parentheses to get any arbitrary language,
but you still can't parse a sentence like:

All systems are ( for launch.

: macros are functions which are run when the source code is read


: (parsed). the argument to a macro is source code (expressed as a data
: structure and not simple text) and the return value is source code
: (not text). this is a fundamental difference between C's
: text-processing macros, without this macros lose most of their power
: and become too hard to write to be used.

Yes, source filters have the same problem.

: - what macros are really good at is embedding mini-languages and


: creating new idioms, this often goes with, but is not nessecarily
: related to, reducing lines of code. example: CLOS/MOP (common lisp
: object system/meta object protocol) are implemented as macros on top
: of non-OO lisp (this statement maybe be a lie if you go deep enough
: into the specifics of some implementations).

Support for mini-languages is a major design goal for Perl 6.

: - the power of lisp's macros is that they allow you to perform


: arbitrary code transformations by working with code as if it was
: data. at one point there was discussion about having perl subs with
: "auto-args" (i forget where i read about this) where by the
: arguments to the sub where determined by parsing the body of the sub
: itself and looking at what variables where used, this is a trivial
: macro in lisp. adding this to perl5 required a source filter which
: took forever to write and was never used because is was never
: reliable enough (this may say more about my capabilities as a
: programmer than about perl5 source filters).

But we want auto-args by marking the args themselves, not by
mentioning a special macro name in front. So support has to be
built-in.

: - everything you can do with macros you can do without, since macros


: always get expaned (translated) into "regular" common lisp
: code. however, sometimes (like with CPS) hand writing the output is
: prohibitly difficult.

Sure.

: - some people consider macros to actually reduce maintainability since


: they perform arbitrary code manipulations, so you have _no_ idea of
: what is going on if you don't know what the macro does. macros which
: introduce new symbols are especially vulnerable to this.

Well, same is true of any built-in. But macros get really nasty if
they cause your program to throw error messages that are impossible
to understand.

: - any sufficently powerful tool can be used to shot yourself in the


: foot (or blow off your head). i doubt this java-esque argument
: (let's "protect" the programmers from themselves) has any weight
: with perl programmers, but it's something i've heard more than once.

Actually, it has a lot of weight, but not in the sense of preventing
Perl programmers from using the powerful features. What we really
try to do is to avoid requiring the novice programmer to know abou the
powerful features before they need to know them. If a Perl programmer
has to do grammar munging in order to write a CGI script, something
is terribly wrong. They might use a module that does grammar munging
on their behalf, but that's different, because presumably someone
else with more expertise wrote that module. So grammar munging is
there to make life easier for today's source filter writers, not to
make life harder for the novice.

: - writing realiable/robust source filters is hard (do able, but hard,


: even with The Damien's Filter::Simple). writing grammars is better,
: but still hard, and more importantly, both require a rdically
: different mind set from "regular" programming. the ease of writing
: lisp macros is largely due to the fact that lisp has no syntax
: (almost), and that lisp's syntax is programmable. perl6 will have
: the second and can't do much about the first (sort of goes against
: "different things should look different").

Interestingly (you will appreciate this as an Italian, or at least
a resident of Italy), I was reading Umberto Eco's _The Search for a
Perfect Language_, and he makes the point that, over the centuries,
many of the designers of "perfect" languages have fallen into the trap
of trying to make similar things look similar. He goes on to argue
that similar things should look different, because when you don't,
you end up with too little redundancy for effective communication.

Suppose you have a system in which all farm animals are classified
into the same category, and distinguished by one letter in their
name. All farm animals begin with, say, "snarfu". So we get:

snarfum "cow"
snarfun "horse"
snarfux "chicken"
snarfuk "pig"
snarfuh "goose"
...

As you can see, it would be really easy in such a system for you to
tell your spouse to kill the goose, and have them think you wanted them
to kill the horse instead, especially if the next word happened to
start with a nasal sound. You might send someone to milk the horse
instead of the cow. Or you might think you're going to have chicken
pot pie for dinner, and get pig pot pie instead.

Of course, if you mishear a different letter, you might get a category
error, and end up baking a spouse pot pie. But you'd probably notice
that miscommunication...

This problem doesn't arise as much when similar words look and sound
quite different, as they do in natural languages. But it does make
those languages harder to learn.

Written languages (such as computer languages) are perhaps a bit more
forgiving, since you don't have to parse them on the fly with only
one chance to hear--you can stare at them until you see that you have
a -n instead of a -m, or a qx// instead of a qw//. Or a !! instead
of a ||, for that matter...

But it does mean that it would probably a mistake to force all
user-defined quote constructs to be of the form qX//. Up till now
I've carefully chosen letters for X that don't have the same overall
shape, so that qr, qx, qq, etc. all look quite different. Extending
that after the manner of "perfect" languages will reduce readability.

Larry

Rich Morin

unread,
Oct 26, 2002, 2:18:47 PM10/26/02
to perl6-l...@perl.org
At 9:54 AM -0700 10/25/02, Larry Wall wrote:
>Suppose you have a system in which all farm animals are classified
>into the same category, and distinguished by one letter in their
>name. All farm animals begin with, say, "snarfu". So we get: ...

A similar problem exists with street names. Some housing developments
use very similar names for all of their streets, causing quite a bit
of confusion for newcomers. On a slightly larger scale, a short stretch
of Interstate 280 (in the SF Bay Area) has the following offramps:

Farm Hill
Page Mill
Sand Hill

that is, /.a.. .ill/.

-r
--
email: r...@cfcl.com; phone: +1 650-873-7841
http://www.cfcl.com/rdm - my home page, resume, etc.
http://www.cfcl.com/Meta - The FreeBSD Browser, Meta Project, etc.
http://www.ptf.com/dossier - Prime Time Freeware's DOSSIER series
http://www.ptf.com/tdc - Prime Time Freeware's Darwin Collection

Reply all
Reply to author
Forward
0 new messages