First own macro: How evil is eval?

[Axel Reichert]

Hello,

finally I felt the need to write my first macro. I have read Practical
Common Lisp's chapters 7 and 8 and made my way up to half of chapter 7
of "On Lisp", sometimes think I got it, but when trying a write it down
I struggle a lot, so Perlism 116 applies:

http://www.cs.yale.edu/homes/perlis-alan/quotes.html

(-:

Here is what I am trying (all simplified): I have an expensive function,
say

(defun expensive-defun (list)
(sleep 3)
(mapcar #'1+ list))

that is called over and over again by

(defun frequently-run-defun (number)
(case number
(1 (expensive-defun (list 0 1 2)))
(2 (expensive-defun (list 1 2 3)))
(3 (expensive-defun (list 2 3 4)))))

In fact, it would be possible and of course faster at runtime to
calculate the results in advance, such as

(defun frequently-run-defun-fast (number)
(case number
(1 '(1 2 3))
(2 '(2 3 4))
(3 '(3 4 5))))

But the original representation as (list 0 1 2), ... would feel much
more natural (in the sense of domain-specific thinking) in the source
code and so I had the idea to write a macro that transforms

(cheap-macro (list 0 1 2))

or

(cheap-macro '(0 1 2))

into

'(1 2 3)

and then have

(defun frequently-run-defun-macro (number)
(case number
(1 (cheap-macro (list 0 1 2)))
(2 (cheap-macro (list 1 2 3)))
(3 (cheap-macro (list 2 3 4)))))

which gives me more intuitively readable AND faster running code,
so the best of both worlds.

I tried (more poking around in the dark) various combinations of quotes,
backquotes and commas, but failed miserably. What did work was

(defmacro cheap-macro (list)
`(quote ,(eval `(mapcar #'1+ ,list))))

but I cannot belief that I will have to fall back to eval for such a
trivial macro. It is much more likely that I am still lacking
understanding.

What would be the "best-practice" way to implement this macro?

Best regards

Axel

[Jeff Barnett]

I suggest you look up "memo functions" and see if that technology would
help you.
--
Jeff Barnett

[Axel Reichert]

Jeff Barnett <j...@notatt.com> writes:

> I suggest you look up "memo functions" and see if that technology
> would help you.

Are you referring to "memoization"? I am aware of the concept, but
dismissed it so far, because I am using the lists resulting from the
"case" form for some lookup with a more or less random number, so not
repeating often. Roughly like

(position-if #'(lambda (elt) (> elt (random 1.0)))
resulting-list-from-case-form)

However, I might put this technique to good use for the "case" form
itself. Thanks for the idea, which is somewhat orthogonal to my original
question.

Axel

[Jeff Barnett]

Yes I was talking about memo functions and I was thinking of using them
to save some of the work done by expensive-defun.
--
Jeff Barnett

[Spiros Bousbouras]

How much generality do you want ? Do you want something like
(cheap-macro (append (list 1 2) (list 3 4)))

to also work ? If you want this much generality , I don't see how you can
avoid EVAL .If on the other hand you only want cheap-macro to work with
a list argument and all the elements of the list will be constant numbers
then writing (cheap-macro (list 0 1 2)) or (cheap-macro '(0 1 2))
is redundant , all you really need is (cheap-macro (0 1 2)) .In that case

(defmacro cheap-macro (&rest list)
(append (list 'quote) (mapcar (function expensive-defun) list)))

and

(defun frequently-run-defun-macro (number)
(case number

(1 (cheap-macro (0 1 2)))
(2 (cheap-macro (1 2 3)))
(3 (cheap-macro (2 3 4)))))

do the job.

> and then have
>
> (defun frequently-run-defun-macro (number)
> (case number
> (1 (cheap-macro (list 0 1 2)))
> (2 (cheap-macro (list 1 2 3)))
> (3 (cheap-macro (list 2 3 4)))))
>
> which gives me more intuitively readable AND faster running code,
> so the best of both worlds.
>
> I tried (more poking around in the dark) various combinations of quotes,
> backquotes and commas, but failed miserably. What did work was
>
> (defmacro cheap-macro (list)
> `(quote ,(eval `(mapcar #'1+ ,list))))
>
> but I cannot belief that I will have to fall back to eval for such a
> trivial macro. It is much more likely that I am still lacking
> understanding.
>
> What would be the "best-practice" way to implement this macro?

By the way , expensive-defun suggests to me that executing the DEFUN form
will be expensive whereas what you really mean is that the function will
be expensive. So expensive-function is a more suggestive name IMO.

--
vlaho.ninja/prog

[Kaz Kylheku]

On 2021-11-28, Axel Reichert <ma...@axel-reichert.de> wrote:
> What would be the "best-practice" way to implement this macro?

One way would be:

> (defun frequently-run-defun (number)
> (case number

> (1 (load-time-value (expensive-defun (list 0 1 2))))
> (2 (load-time-value (expensive-defun (list 1 2 3))))
> (3 (load-time-value (expensive-defun (list 2 3 4))))))

When this code is compiled, the compiler arranges for the expensive
calculations to be run one time when the compiled code is loaded.

The load-time values then become de facto literal objects that
are just retrieved whenever the expression's value is needed.

There is some eval in there somewhere under the hood; we just hid
it.

If you want to write a macro which takes a piece of syntax as an
argument (the usual habit of a macro) and converts the value of
that syntax to a literal, you cannot avoid eval.

That evaluation is not taking place in the usual time that macros
normally arrange: the evaluation time of the generated code.

When you let things evaluate in their usual time, then eval is
invisible. When you need unusual evaluation, then unless there
is an existing gadget for doing that (e.g. load-time-value if
you want load-time evaluation), then you will end up coding an eval
somewhere.

In the TXR Lisp language, I provided a built-in operator called
macro-time. If you write (macro-time E), E gets evaluated in the
top-level environment, to obtain a value V, and then (quote V) is the
result. Thus, you don't have to use eval. For some reason, I designated
this as a special operator. It is actually implemented right inside the
macro expander as a special case, though it could have been a macro.

In any case, the implementation of macro-time cheerfully calls eval.

Macros themselves are based on eval. Think about it; the Lisp
interpreter or compiler is scanning your code to expand macros. It sees
(mac 1 2 3). What does it do? It invokes the mac macro expander
function. But that is evaluation! The mac function contains expressions:
expressions that give the default values of optional and keyword
parameters, and expressions that make up its body. These (at least in
an abstract sense) are evaluated by eval. The (mac 1 2 3) call is
effectively evaled, and replaced with the result.

You just don't see explicit uses of the eval functions if you stick
to invoking evaluation in all the situations that the language
designers designed for you.

"evil eval" really just refers to going off the beaten path,
according to someone's opionion of what that is.

E.g. I designed macro-time for you in a certain non-CL dialect; so if
you are in that dialect and use macro-time, you are not perpetrating any
use of eval.

If you're in a dialet that doesn't have macro time, you have to write
your own, which involves eval, and so then you are raising eyebrows.

However, until this point in the article, I have been lying about
something, and it's time to come clean.

There is a way of writing macro-time in Common Lisp without using eval.
You can disguise eval by stashing an expression into the body of a
macrolet (local macro), and the macrolet can be generated by a macro:

(defmacro macro-time (expr)
(let ((secret-mac (gensym)))
`(macrolet ((,secret-mac () `(quote ,,expr)))
(,secret-mac))))

This works because invocation of macros is evaluation (IOEMIE?)

What is this doing? When we make a call such as

(macro-time (read-contents-of-file-as-string "/etc/shadow"))

first the macro-time macro is invoked. It calls (gensym) to allocate a
unique symbol G and binds that to secret-mac. Then it generates this
code:

(macrolet ((G () `(quote ,(read-contents-of-file-as-string "/etc/shadow"))))
(G))

This code replaces the (macro-time ...) invocation. So now the expander
is processing a macrolet. It establishes the definition of the local
macro G and processes the (G) form which tells it to call that macro.

The macro is called and so the `(quote ...) backquote expression is
evaluated: THERE IS OUR HIDDEN EVAL.

Summary: we got an expression to evaluate at macro-expansion time by
inserting it into the body of a locally defined macro, and throwing the
invocation of that macro into the path of the macro-expander.

P.S. there is no funtion read-contents-of-file-as-string, but you
can easily write one.

--
TXR Programming Language: http://nongnu.org/txr
Cygnal: Cygwin Native Application Library: http://kylheku.com/cygnal

[Axel Reichert]

Spiros Bousbouras <spi...@gmail.com> writes:

> If on the other hand you only want cheap-macro to work with a list
> argument and all the elements of the list will be constant numbers

Yes.

> then writing (cheap-macro (list 0 1 2)) or (cheap-macro '(0 1 2)) is
> redundant , all you really need is (cheap-macro (0 1 2)) .

That was the point I was crucially missing, you made my day, many
thanks!

Because of this misconception one of my tries, the rather "canonical"

(defmacro cheap-macro (list)
`(mapcar #'1+ ,list))

was dismissed by me to early: Since, say,

(cheap-macro (list 1 2 3))

evaluated to

(2 3 4)

I thought this to be not enough and tried hard to somehow put a quote in
front of this. Had I tried this macro "downstream" I would have seen
that it does the job.

Hopefully I will become more confident with the quoting business further
down the macro road ...

> By the way , expensive-defun suggests to me that executing the DEFUN
> form will be expensive whereas what you really mean is that the
> function will be expensive. So expensive-function is a more suggestive
> name IMO.

Yes, you are right.

Thanks again!

Axel

[Axel Reichert]

Kaz Kylheku <480-99...@kylheku.com> writes:

> One way would be:
>
>> (defun frequently-run-defun (number)
>> (case number
>> (1 (load-time-value (expensive-defun (list 0 1 2))))
>> (2 (load-time-value (expensive-defun (list 1 2 3))))
>> (3 (load-time-value (expensive-defun (list 2 3 4))))))
>
> When this code is compiled, the compiler arranges for the expensive
> calculations to be run one time when the compiled code is loaded.
>
> The load-time values then become de facto literal objects that
> are just retrieved whenever the expression's value is needed.

All this reads similar to compiler macros (I found them in Edmund
Weitz's "Common Lisp Recipes", tried them, but also screwed up).

> When you need unusual evaluation, then unless there is an existing
> gadget for doing that (e.g. load-time-value if you want load-time
> evaluation), then you will end up coding an eval somewhere.

[...]

> You just don't see explicit uses of the eval functions if you stick
> to invoking evaluation in all the situations that the language
> designers designed for you.

Great explanations, thank you.

> "evil eval" really just refers to going off the beaten path

Well, my instinct was that almost certainly I do not have to leave the
beaten path for "my first macro" (apart from mimicking textbook
examples), something that seemed like a trivial exercise.

> there is no funtion read-contents-of-file-as-string, but you can
> easily write one.

Or use one provided by various utility packages.

Best regards

Axel

[Axel Reichert]

r...@zedat.fu-berlin.de (Stefan Ram) writes:

> In TeX, one can use \expandafter to add another level of makro
> expansion to a function definition.

While I have written several (La)TeX packages, at that time I did not
have any Lisp knowledge. This is quite an interesting tangent, since
back then much of the (plain) TeX macro business sounded like Greek to
me. Probably it will be rewarding to resurrect some layers of fossil
knowledge in my brain, even though it will not solve my problem at
hand. Anyway, much appreciated!

Axel

[Axel Reichert]

Axel Reichert <ma...@axel-reichert.de> writes:

> Spiros Bousbouras <spi...@gmail.com> writes:
>
>> If on the other hand you only want cheap-macro to work with a list
>> argument and all the elements of the list will be constant numbers
>
> Yes.
>
>> then writing (cheap-macro (list 0 1 2)) or (cheap-macro '(0 1 2)) is
>> redundant , all you really need is (cheap-macro (0 1 2)) .
>
> That was the point I was crucially missing, you made my day, many
> thanks!
>
> Because of this misconception one of my tries, the rather "canonical"
>
> (defmacro cheap-macro (list)
> `(mapcar #'1+ ,list))
>
> was dismissed by me to early: Since, say,
>
> (cheap-macro (list 1 2 3))
>
> evaluated to
>
> (2 3 4)
>
> I thought this to be not enough and tried hard to somehow put a quote in
> front of this. Had I tried this macro "downstream" I would have seen
> that it does the job.

Of course, my defmacro above does not the job, but rather mimics a
defun. While the result of the evaluation is fine, I was interested in
getting the same result after macro expansion. I still have to learn a
lot.

But your hint with append, quote etc. was helpful. By now I have a
working solution with

(defmacro cheap-macro (&rest args)
(append (list 'quote)
(list (expensive-function args))))

that is called by

(cheap-macro 1 2 3)

, so I got rid of the list argument along the way.

Axel

[Kaz Kylheku]

On 2021-11-29, Stefan Ram <r...@zedat.fu-berlin.de> wrote:

> Axel Reichert <ma...@axel-reichert.de> writes:
>>(1 (expensive-defun (list 0 1 2)))
>

> In TeX, one can use \expandafter to add another
> level of makro expansion to a function definition.
>

> \def\a{abc}
> \def\x{\a}
> \expandafter\def\expandafter\y\expandafter{\a}
> \show\x
> \show\y
> \end

This just looks like macro bodies are implicitly
quasiquoted, and expandafter is a kind of unquote operator
which can be invoked in the middle of such a body.

(define-symbol-macro a `"abc")

(define-symbol-macro x `a)

(define-symbol-macro y `,a)

x -> a
y -> "abc"

Or, well, not exactly.

It's like \expandafter is a control sequence that has a state
machine hidden behind it? It is called three times, to collect three
arguments, and when it has all three, it then acts on them to evoke the
semantics.

\expandafter \def \expandafter \ident \expandafter {body}

The quote protection is removed from body allowing its
expansion to be combined with the collected \def\ident.

(defmacro expandafter (defop ident expr)
`(,defop ,ident ,(eval expr)))

(define-symbol-macro a `"abc")

(define-symbol-macro x 'a)

(expandafter define-symbol-macro y 'a)