Unbound identifier error with syntax transformer that uses syntax-generating helper procedure

[Greg Rosenblatt]

Hi, I'm having trouble writing a syntax transformer that uses a syntax-generating procedure defined elsewhere.

When the procedure is defined locally, everything is fine.

When the procedure is defined outside the transformer, I have to do a dance to make the procedure visible at the right phase, which seems to work.  However, upon use I get:

> racket unbound-identifier.rkt

unbound-identifier.rkt:9:7: lambda: unbound identifier;

 also, no #%app syntax transformer is bound

  context...:

   #(1973 module unbound-identifier 0) #(2181 module) #(2811 macro) #(2822 local)

   #(2823 intdef) #(2824 module (unbound-identifier utilities) -1)

  other binding...:

   #<module-path-index:(racket)>

   #(1972 module) #(1973 module unbound-identifier 0)

  at: lambda

  in: (lambda (i) (displayln (quasiquote (input: (unquote input)))))

  context...:

   standard-module-name-resolver

 

I wrote this self-contained example using a submodule, but the error also occurs when requiring the module from another file.  What am I doing wrong?  I imagine it's something silly.

#lang racket

(provide this-works this-does-not-work)

(module utilities racket/base

  (provide compile-test)

  (define (compile-test)

    #`(lambda (i) (displayln `(input: ,input)))))

(require (for-syntax 'utilities))

(define-syntax (this-works stx)

  (syntax-case stx ()

    ((_ input)

     (let ()

       (define (compile-test)

         #`(lambda (i) (displayln `(input: ,input))))

       #`(#,(compile-test) input)))))

(define-syntax (this-does-not-work stx)

  (syntax-case stx ()

    ((_ input to-do ...)

     (let ()

       #`(#,(compile-test) input)))))

(this-works 3)

(this-does-not-work 3)

[Alexis King]

The short answer is that you need a (require (for-template racket/base))
in your utilities submodule:

(module utilities racket/base
(provide compile-test)

(require (for-template racket/base))

(define (compile-test)
#`(lambda (i) (displayln `(input: ,i)))))

But this answer probably isn’t especially helpful towards debugging
similar problems in the future, so let me give a longer explanation.

Racket’s macro system has phases. Phase 0 corresponds to runtime, phase
1 corresponds to phase 0’s compile-time, phase 2 corresponds to phase
1’s compile-time, etc. This space of phases is unbounded. Each phase
contains a completely distinct set of bindings, so when you write, say,
`let` at phase 0, it isn’t necessarily the same `let` as the one you use
at phase 1.

The code you write in the body of a define-syntax definition is in phase
1, since it is evaluated at compile-time. In your top-level module, you
use #lang racket, which happens to provide the bindings from racket/base
at both phase 0 and phase 1. This is why you can use `let` from
racket/base inside your this-works macro — it was provided at that phase
by #lang racket. The code in your template, in this case #`(lambda (i)
....), ends up getting evaluated at runtime, so it uses the phase 0
bindings.

You might have already known all that, since your question is about the
utilities submodule, but I wanted to include that explanation for
context. This module is interesting, since you require it for-syntax in
your enclosing module. This has the effect of *shifting* the phases of
your utilities submodule, so its phase 0 ends up aligning with phase 1
of the enclosing module. Now, the language of this submodule is
racket/base, which provides bindings for `provide` and `define`, but
what about the code in the template?

Well, from the utilities module’s perspective, that code is actually
going to be evaluated one phase level below phase 0: phase -1! This
phase-shifting that happens when you import things for-syntax is why
negative phases are meaningful — even though phase -1 doesn’t really
make any sense in isolation, after the phase-shifting that the
for-syntax import causes, phase -1 becomes phase 0.

Racket manages all this complicated bookkeeping behind the scenes, so
you never need to worry about which *absolute* phase your code will be
used at. What you do need to worry about is which *relative* phase
pieces of code will end up at. In your utilities submodule, the `lambda`
identifier in the template will be evaluated at relative phase level -1,
so you need to ensure racket/base’s bindings are in scope at phase level
-1. This is what for-template does: it is like for-syntax, but it shifts
imports a phase level down instead of a phase level up.

(Note that (for-template (for-syntax ....)) is a no-op, since the shifts
cancel each other out. It may be educational to think about the
implications of this for your program.)

Alexis

[Greg Rosenblatt]

Thanks, that explanation helped.  I had gaps in my knowledge.

I also ended up daisy-chaining yet another submodule nested within the first, to require it both normally and for-template due to braid-shaped phase dependencies.  Nothing seems to have gone wrong.

[Matthias Felleisen]

Here is a variant of the program that asserts a phase level and breaks if it is not the expected one. This is just a programmatic confirmation of Alexis’s explanation of course. 

#lang racket

(module syntax-assertions racket 

  (provide assert-level)

  (define (assert-level i)

    (define observed

      (- (variable-reference->phase (#%variable-reference))

         (variable-reference->module-base-phase (#%variable-reference))))

    (unless (= i observed)

      (raise-syntax-error #f (format "expected phase level: ~a, actual phase level ~a" i observed)))))

(require (for-syntax (submod "." syntax-assertions)))

;; ---------------------------------------------------------------------------------------------------

(module utilities racket/base

  (require (submod ".." syntax-assertions))

  

  (provide compile-test)

  (define (compile-test)

    (assert-level 1) ;; <— you want to run at this level 

    #`(lambda (i) (displayln `(input: ,input)))))

(require (for-syntax 'utilities))

(define-syntax (this-works stx)

  (syntax-case stx ()

    ((_ input)

     (let ()

       (define (compile-test)

         (assert-level 1) ;; <— you want to run at this level