Hygiene for a curried macro

[rocketnia]

Hi all,

I've been experimenting with a custom system of managed local variables, and I came up with a hygiene test case that was failing. So then I tried the same test case with plain Racket variables, and it failed that way too. Here's a minimalistic example.

Basically, this is a curried macro: The user supplies a variable and gets a macro out, and then the user calls that macro. Can the second macro bind the variable the user supplied to the first one? I thought it would be able to, but this doesn't currently seem to be case on Racket v7.8 [cs].

Could anyone explain what's going on with this? Is there a workaround if I want to write this kind of macro? Should I file a bug in Racket? This looks pretty close to R5RS Scheme, so I wonder what the situation in the broader Scheme world is like, too.

#lang racket

(require rackunit)

(define-syntax-rule

  (let-second-and-create-let-third var let-third body-of-let-second)

  (let-syntax ([let-third

                 (syntax-rules ()

                   [(let-third body-of-let-third)

                    (let ([var "third"])

                      body-of-let-third)])])

    ; This binding shows that the first macro *does* manage to bind

    ; the given variable, even though the second macro doesn't.

    (let ([var "second"])

      body-of-let-second)))

(check-equal?

  (let ([x "first"])

    (let-second-and-create-let-third x let-third

      (let-third

        x)))

  "third"

  "Test that a macro generated by a macro can bind a variable the user supplied to the generator macro")

; FAILURE

; actual: "second"

; expected: "third"

You can also find this code in Gist form here: https://gist.github.com/rocketnia/cb83da2cfcddbf614dfe1dfc5e08792c

Thanks in advance for any insight you have about what's going on here.

- Nia

[Philip McGrath]

Hi Nia,

Here's a variant that passes your test:

#lang racket

(require rackunit
         syntax/parse/define)

(define-syntax let-second-and-create-let-third
  (syntax-parser
    [(_ var let-third body-of-let-second)
     #'(let ([var "second"])
         (let-syntax ([let-third
                       (syntax-parser
                         [(_ body-of-let-third)
                          #:with var* (syntax-local-introduce #'var)
                          #'(let ([var* "third"])
                              body-of-let-third)])])
           body-of-let-second))]))

(check-equal?
  (let ([x "first"])
    (let-second-and-create-let-third x let-third
      (let-third
        x)))
  "third"
  "Test that a macro generated by a macro can bind a variable the user supplied to the generator macro")

You need `syntax-local-introduce` (or another mechanism, like `datum->syntax`) for `let-third` to be able to capture `x` in its body: otherwise, the expander will see that it is a macro-introduced binding. You also need to address the fact that each `let` in the expansion creates a new scope: if the binding to `"second"` isn't in scope for the right-hand-side of `let-third`, you can get an ambiguous binding (see: https://www.cs.utah.edu/plt/scope-sets/pattern-macros.html#(part._pattern-ambiguous)). In this case I just moved the `let-syntax` inside the `let`, but you could also use something like `letrec-syntaxes+values`.

-Philip

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/71daa8da-25cf-426b-b709-ee9ed25b53f0n%40googlegroups.com.

[Philip McGrath]

I'll also put in a plug for DrRacket's Macro Stepper, which can show your scopes in pretty colors! (And precise numbers.) In particular, in the "Stepper > Foreground colors" menu, you can toggle between "By macro scopes" and "By all scopes".

-Philip

On Wed, Sep 30, 2020 at 3:40 AM Philip McGrath <phi...@philipmcgrath.com> wrote:

Hi Nia,

Here's a variant that passes your test:

#lang racket

(require rackunit
         syntax/parse/define)

(define-syntax let-second-and-create-let-third
  (syntax-parser
    [(_ var let-third body-of-let-second)
     #'(let ([var "second"])
         (let-syntax ([let-third
                       (syntax-parser
                         [(_ body-of-let-third)
                          #:with var* (syntax-local-introduce #'var)

                          #'(let ([var* "third"])
                              body-of-let-third)])])
           body-of-let-second))]))

(check-equal?
  (let ([x "first"])
    (let-second-and-create-let-third x let-third
      (let-third
        x)))
  "third"
  "Test that a macro generated by a macro can bind a variable the user supplied to the generator macro")

You need `syntax-local-introduce` (or another mechanism, like `datum->syntax`) for `let-third` to be able to capture `x` in its body: otherwise, the expander will see that it is a macro-introduced binding. You also need to address the fact that each `let` in the expansion creates a new scope: if the binding to `"second"` isn't in scope for the right-hand-side of `let-third`, you can get an ambiguous binding (see: https://www.cs.utah.edu/plt/scope-sets/pattern-macros.html#(part._pattern-ambiguous)). In this case I just moved the `let-syntax` inside the `let`, but you could also use something like `letrec-syntaxes+values`.

-Philip

On Wed, Sep 30, 2020 at 2:46 AM rocketnia <rok...@gmail.com> wrote:

--

[Ryan Culpepper]

Yes, the behavior you're seeing is a consequence of hygiene, and you should see the same behavior in other Scheme implementations.

When the expander gets to the `let-third` call, there is a `var` identifier in the macro's template that is used as a binder, and there is a `var` identifier in the macro's argument that is used as a reference. Hygiene says that a macro-introduced binder does not capture a macro-argument reference.

The fact that they both originate from the *same* identifier given to `let-second-and-create-let-third` is irrelevant. The hygiene condition for `let-third` requires that they be treated differently. After all, in "ordinary" hygienic macros, the identifiers are the same also, but the fact that one is in the macro template and one is passed to the macro distinguishes them.

If you want to do higher-order macro programming, the lesson is that hygienic macros are not *pure* abstractions for syntax in the same way that Racket's closures are pure abstractions for computation. That becomes a problem when you want to generate macro definitions that contain identifiers that they use as binders.

Philip provided one solution (and thanks for the macro stepper plug!). The other solution is to pass `var` as an argument to `let-third`.

Ryan

--

[rocketnia]

Thanks Philip and Ryan,

I do need to get more familiar with the macro stepper. :) I tried it out for maybe the first time just now, but I think Philip's screenshot already helped me understand what's going on.

I recall that when Racket invokes a macro, it puts a scope on the inputs first (the macro-introduction scope), and then it flips that scope afterward so that it occurs only on the rest of the macro's output. This keeps variable occurrences that are introduced in the output from binding variables that were already in the input, and vice versa.

What's going on here seems to be that `let-third` uses `x`, but doesn't take `x` as its own input, so this scope-flipping treats it like it's an identifier the `let-third` macro introduces. If I think about the example in terms of currying, then `x` *is* an input -- it just isn't an input to this particular stage of the macro. But this isn't something Racket's hygiene is prepared to treat as an input.

And you know what? I guess I'm convinced. When I write out this far simpler example...

(let ([x "first"])

  (let-syntax ([let-second
                 (syntax-parser
                   [(_ body)
                    #'(let ([x "second"])
                        body)])])
    (let-second
      x)))

...I take one look at it, and I don't expect the `(let ([x "second"]) ...)` to have anything to do with the other two occurrences of `x`. My original macro expands into a macro like this, so it doesn't work either.

I guess my intent clouded my judgment. I must have gone in assuming I could write a macro that created a `let` that shadowed the variable I wanted it to shadow, but that doesn't even work when the variable is hardcoded like this.

Philip's technique with the scope introduction with `syntax-local-introduce` looks good. I even tried `syntax-local-introduce` almost exactly like that myself before posting here, and I think I ran into the exact issue that Philip fixed by moving the `(let ([var "second"]) ...)` around. So there is a workaround to this, which is nice.

I didn't actually have a macro to write; I just wanted a test case to ensure my custom variables obeyed the kind of hygiene Racket's existing variables did. Now that I understand this better, I know that what I'm seeing isn't a broken behavior.

Thanks again,

Nia