Re: [racket-dev] try the new macro expander

[Alexis King]

I’ve just taken the time to reimplement how my curly-fn package handles the transformation in an attempt to make it a little less unwieldy. Specifically, the transformation itself is now done via macro expansion rather than being performed at read-time. This means that the curly brace syntax itself is now expanded extremely simply.

This:

#{+ 1 %}

just becomes this:

(curly-fn {+ 1 %})

and `curly-fn` is a syntax transformer that performs all the relevant introspection required to expand into a `lambda`.

Of course, that expansion doesn’t quite work. The `curly-fn` macro is defined in another module, so the curly brace syntax actually expands into this:

#'(let ()
(local-require curly-fn/private/curly-fn-transformer)
(curly-fn {+ 1 %}))

Overall, this seems to work okay.

I then decided to implement the strategy of using make-syntax-introducer to scope everything properly. This actually worked great—the following now works absolutely fine:

(let ([lambda 0])
(#{+ 1 %} 3))
; => 4

So hygiene is now preserved, but unfortunately, another one of my packages that depends on curly-fn still throws an error. Specifically, the error complains about the `let` containing the `local-require` has no binding. What’s particularly odd, though, is that this error only occurs in my documentation that uses `include-extracted` from `scribble/extract`.

After some investigation, it looks like I can get it to fail when using syntax-local-lift-require. Here’s a minimal example, again using two Racket source files:

; source.rkt
#lang curly-fn racket/base
#{values}

; lift.rkt
#lang racket
(define-syntax (lift stx)
(syntax-case stx ()
[(_ module-path)
(syntax-local-lift-require #'module-path #'id)]))
(lift "source.rkt")

That fails with the following error:

let: unbound identifier;
also, no #%app syntax transformer is bound
context.:
#(252056 module) #(252057 module) #(252265 module reader -1)
#(252266 module reader -1) #(255222 module) #(255223 module source 0)
other binding.:
#<module-path-index:(racket/base)>
#(255032 macro) #(255222 module) #(255223 module source 0) in: let

Is this a bug with syntax-local-lift-require in the new macro expander, or is it my fault (more likely)? For the version of curly-fn that causes this error, see the “stx-transformer” branch:

https://github.com/lexi-lambda/racket-curly-fn/tree/stx-transformer

Thanks, and sorry for the long message,
Alexis

[Alexis King]

I have been testing mostly using a combination of DrRacket and `raco test`. However, with that begin-for-syntax bug fixed, I’ve determined what seems to be the problem (and is reminiscent of a similar problem in the previous version).

This works:

#lang curly-fn racket
(begin-for-syntax
(#{+} 1 2 3))

This doesn’t:

#lang curly-fn racket
(require (for-meta 2 racket/base))
(begin-for-syntax
(begin-for-syntax
(#{+} 1 2 3)))

This can be “fixed” by adding (require (for-meta 2 racket/base)) to curly-fn/lang/reader, but of course this is a hack more than anything else. Is there any good solution to this problem, or is this still mostly unsolvable under the current system?

Alexis

> On Jul 19, 2015, at 11:10 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
>
> I will be able to look later today.
>
> Are you running in DrRacket or using plain `racket`? There's a problem
> with DrRacket at the moment where it will make some programs work when
> they should fail. I expect to look into that soon; meanwhile, plain
> `racket` doesn't have the problem.

[Alexis King]

One more thing: introducing a new scope with make-syntax-introducer seems to break DrRacket’s Check Syntax arrows for the whole module. I don’t understand exactly how DrRacket’s arrows work, but my preliminary debugging in the macro stepper has left me a little confused.

Whether I’m using curly-fn or plain racket/base, I’ve found that the “marks” the macro stepper displays on some identifiers for a simple program to be confusing. I wrote a simple test program:

(define foo #f)
foo

And I inspected the marks on each `foo` identifier throughout the expansion process.

I found that in every single step, both `foo` identifiers had the same two marks, as I expected. However, in the very last step (the step that supposedly doesn’t do anything, when it just says “Expansion finished”), the second `foo` identifier mysteriously changed and gained two extra marks. I have no idea why.

Sorry for all my questions, but I’m quite curious about how this process works, but the information my debugging has provided has seemed confusing and unintuitive thus far.

[Matthew Flatt]

At Sun, 19 Jul 2015 12:21:02 -0700, Alexis King wrote:
> This works:
>
> #lang curly-fn racket
> (begin-for-syntax
> (#{+} 1 2 3))
>
> This doesn’t:
>
> #lang curly-fn racket
> (require (for-meta 2 racket/base))
> (begin-for-syntax
> (begin-for-syntax
> (#{+} 1 2 3)))
>
> This can be “fixed” by adding (require (for-meta 2 racket/base)) to
> curly-fn/lang/reader, but of course this is a hack more than anything
> else. Is there any good solution to this problem, or is this still
> mostly unsolvable under the current system?

Unfortunately, this is still an unsolved problem.

[Matthew Flatt]

At Sun, 19 Jul 2015 15:46:07 -0700, Alexis King wrote:
> One more thing: introducing a new scope with make-syntax-introducer seems to
> break DrRacket’s Check Syntax arrows for the whole module.

That makes sense in retrospect. Adding an extra scope makes
`syntax-original?` produce #f for everything in whole module, and that
makes DrRacket ignore the identifiers.

I think `make-syntax-introducer` should probably accept an optional
argument to specify that the new scope should *not* indicate
non-original syntax.

> I don’t understand
> exactly how DrRacket’s arrows work, but my preliminary debugging in the macro
> stepper has left me a little confused.
>
> Whether I’m using curly-fn or plain racket/base, I’ve found that the “marks”
> the macro stepper displays on some identifiers for a simple program to be
> confusing. I wrote a simple test program:
>
> (define foo #f)
> foo
>
> And I inspected the marks on each `foo` identifier throughout the expansion
> process.
>
> I found that in every single step, both `foo` identifiers had the
> same two marks, as I expected. However, in the very last step (the
> step that supposedly doesn’t do anything, when it just says
> “Expansion finished”), the second `foo` identifier mysteriously
> changed and gained two extra marks. I have no idea why.

I don't know yet, either, but I'll investigate.

[Robby Findler]

On Mon, Jul 20, 2015 at 8:16 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
> At Sun, 19 Jul 2015 15:46:07 -0700, Alexis King wrote:
>> One more thing: introducing a new scope with make-syntax-introducer seems to
>> break DrRacket’s Check Syntax arrows for the whole module.
>
> That makes sense in retrospect. Adding an extra scope makes
> `syntax-original?` produce #f for everything in whole module, and that
> makes DrRacket ignore the identifiers.

Not sure if it is relevant, but if you put the property
'original-for-check-syntax on an identifier, check syntax will treat
it as if it was original.

Robby

[Matthew Flatt]

In this case, I think that would mark too many identifiers as original,
such as identifiers introduced by macros that are defined in the same
module.

[Gustavo Massaccesi]

Slightly off-topic: I didn't notice 'original-for-check-syntax. A few
weeks ago I had problems with a missing arrows in DrRacket, and I
searched for 'disappeared-use and I got
http://docs.racket-lang.org/reference/stxprops.html?q=disappeared-use#%28idx._%28gentag._188._%28lib._scribblings%2Freference%2Freference..scrbl%29%29%29

I think there should be a margin note from there to
http://docs.racket-lang.org/tools/Check_Syntax.html?q=disappeared-use#%28idx._%28gentag._28._%28lib._scribblings%2Ftools%2Ftools..scrbl%29%29%29

Gustavo

> --
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.
> To post to this group, send email to racke...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/CAL3TdOMQeBLO--bjJp12Ch8%2BSpm7VXPyRCEEu6bOwzSrKr-m8Q%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

[Alexander D. Knauth]

Wouldn't it only do that if the identifiers were original to begin with?
I mean, if they were introduced by macros, then they wouldn't be original to begin with, and so a function produced by (make-syntax-introducer) wouldn't make it original unless it already was?

[Robby Findler]

On Mon, Jul 20, 2015 at 8:49 AM, Gustavo Massaccesi <gus...@oma.org.ar> wrote:
> Slightly off-topic: I didn't notice 'original-for-check-syntax. A few
> weeks ago I had problems with a missing arrows in DrRacket, and I
> searched for 'disappeared-use and I got
> http://docs.racket-lang.org/reference/stxprops.html?q=disappeared-use#%28idx._%28gentag._188._%28lib._scribblings%2Freference%2Freference..scrbl%29%29%29
>
> I think there should be a margin note from there to
> http://docs.racket-lang.org/tools/Check_Syntax.html?q=disappeared-use#%28idx._%28gentag._28._%28lib._scribblings%2Ftools%2Ftools..scrbl%29%29%29

Ah, good idea. I've put a link.

Robby

[Robby Findler]

I'm not sure if you're asking about Check Syntax or not, but what it
does is ignore identifiers, unless they are either syntax-original? or
they have that property.

Robby

[Alexis King]

> That makes sense in retrospect. Adding an extra scope makes
> `syntax-original?` produce #f for everything in whole module, and that
> makes DrRacket ignore the identifiers.
>
> I think `make-syntax-introducer` should probably accept an optional
> argument to specify that the new scope should *not* indicate
> non-original syntax.

I’m returning to this problem now, so may I ask if you’ve given any more thought to this issue? If that’s the right approach, how hard would it be to implement that change for `make-syntax-introducer`? Is the usage of `'original-for-check-syntax` relevant here (I don’t think there was a clear consensus reached)?

Alexis

[Matthew Flatt]

I think the change to `make-syntax-introducer` is straightforward, and
it still seems like the right idea, but I haven't gotten there, yet. (I
hope to catch up on many things next week.)

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-dev+...@googlegroups.com.
> To post to this group, send email to racke...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/racket-dev/3F280188-3684-4B50-BE24-0AC1F6BFFD
> 8C%40gmail.com.

[Alexander D. Knauth]

Is there any reason not to have `make-syntax-introducer` functions preserve `syntax-original?`-ness whenever it's given something `syntax-original?` ?

[Matthew Flatt]

I think it would not matter in some cases. The original intent, though,
was to create a scope that's like a macro-introduction scope --- where
the result should not be considered part of the original program, and
the added scope may be the only indication of that. A grep through some
sources shows that `make-syntax-introducer` is used that way in several
places, so a backward-incompatible change seems like a bad idea.

[Alexander D. Knauth]

Is it relying on the resulting syntax not being `syntax-original?` though?
And even if it is, would the input syntax not be original most of the time anyway?

> --
> You received this message because you are subscribed to the Google Groups "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-dev+...@googlegroups.com.
> To post to this group, send email to racke...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-dev/20150725155005.BA7EF6501CC%40mail-svr1.cs.utah.edu.

[Matthew Flatt]

Yes, those uses rely on the resulting syntax not being `syntax-original?`.

I've pushed the change to `make-syntax-introducer`. Pass a true
argument to `make-syntax-introducer` to create a scope that acts like a
use-site scope (no effect on `syntax-original?`) instead of a
macro-introduction scope.

> https://groups.google.com/d/msgid/racket-dev/3FBB766D-5E35-4038-8D9C-81A6709294
> 02%40knauth.org.

[Alexis King]

Oh wow, thanks for the prompt update. I was in the process of trying to pick my way through the C code to figure out how I might be able to implement it myself, but I must admit I wasn’t getting anywhere fast. ;)