try the new macro expander

[Matthew Flatt]

The new macro expander is ready for you to try:

http://www.cs.utah.edu/~mflatt/tmp/scope-snapshot/

What you should expect:

* If your library has no macros, or if it uses only pattern-matching
macros that don't expand to submodules or units, then it should work
with the new expander without changes.

* If your library includes a macro that expands to a submodule, then
there's a good chance that the macro will need to change. The old
expander allowed even pattern-based macros to generate submodules
that bind parts of the macro's input. More generally, the new
expander doesn't throw away as much pre-expanded context on a
submodule form as the old expander.

* If your library includes a macro that expands to a unit, and if a
signature or definition are not both introduced by the macro or from
the macro use site, then you might run into an unbound-identifier or
missing-definition error. It's difficult to pin down what the old
macro did, and in the few cases where I've seen a difference between
the old and new version, I've been inclined to say that the old
behavior was wrong. This change could be enough of a problem that we
have to do something different, through, so please let me know if
you run into it.

* A macro that uses `syntax-local-get-shadower` like will have to be
revised, and it should be revised to avoid that function. A macro
that uses `syntax-local-make-delta-introducer` definitely will have
to be revised, because that function is no longer supported. Most
macros don't use such obscure functions.


Most packages build ok using the above snapshot:

http://next-pkg-build.racket-lang.org.s3-website-us-west-2.amazonaws.com/

Several failures are "ambiguous binding" errors. That error usually
indicates an unavoidable incompatibility between the old and new macro
systems. I'm interested to hear whether the package authors can
understand the errors and fix them and whether the incompatibilities
seem manageable.

(Thanks to Greg for fixing a problem in the "rackjure" package before
the build. That problem was due to a failure of the old macro expander
to report an import--definition conflict.)


If you're prefer to work from the "scope" branch of

https://github.com/mflatt/racket

then it's simplest to add the catalog

http://www.cs.utah.edu/~mflatt/tmp/scope-snapshot/catalog/

before "http://pkgs.racket-lang.org/" so that you get compatible
versions of the handful of packages that I modified.


I've also updated the write-up at

http://www.cs.utah.edu/~mflatt/scope-sets-4/

The changes are relatively small and mostly in section 9.

[Matthew Flatt]

I see that the `declare` macro is using `syntax-local-get-shadower` as
a way of storing information in the compile-time environment. The
`syntax-local-get-shadower` function is still the only primitive to
provide that functionality. It's an awkward primitive, and I considered
replacing it with a primitive that is more streamlined to that task,
but it seems more likely that we'll eventually provide a nicer API as a
library built in terms of `syntax-local-get-shadower`.

So, for this macro, it makes sense to continue with
`syntax-local-get-shadower` for now.

The macro currently fails in the new expander for two reasons:

* There's a bug in the new expander that makes `syntax-local-context`
produce an inconsistent value. That's the source of of the
duplicate-definition error.

I've pushed a repair to the "scope" branch, and I'll update the
snapshot later today.

* The strategy of having a fresh mark on `old-vars` (by not using
`syntax-local-introduce`) doesn't work anymore. An extra scope on
`old-vars` doesn't prevent it from being captured by the new `vars`
binding.

I think the solution is to use `local-expand` before generating the
`vars` definition, so that `local-expand` sees the old `vars`
binding. Or, if it's important to delay the lookup that
`local-expand` performs until `vars` is called, then introduce an
indirection through a fresh name/scope for each context and use
`vars` to record that name/scope.

At Mon, 13 Apr 2015 16:09:43 +0200, Jens Axel Søgaard wrote:

> 2015-04-13 14:40 GMT+02:00 Matthew Flatt <mfl...@cs.utah.edu>:
> > The new macro expander is ready for you to try:

> ...

>
> > * A macro that uses `syntax-local-get-shadower` like will have to be
> > revised, and it should be revised to avoid that function. A macro
> > that uses `syntax-local-make-delta-introducer` definitely will have
> > to be revised, because that function is no longer supported. Most
> > macros don't use such obscure functions.
>

> The following example shows two macros declare and vars that work
> together.
> The declare macro is used to declare identifiers (have some property) and
> the macro vars shows how a macro can access the declared identifiers that
> are in scope. Here vars simply expands to a list of the declared
> identifiers.
>
> Example:
>
> (declare a) ; module level declaration
> (display (vars))
> (let ()
> (display (vars)) ; (1)
> (declare x) ; here (1) and (2) are to display the same
> variables (same scope)
> (display (vars)) ; (2)
> (let ()
> (display (vars))
> (declare s)
> (declare t)
> (display (vars)))
> (declare y)
> (display (vars))) ; check that s and t are no longer in scope
> (declare b)
>
> ; displays: (b a)(y x b a)(y x b a)(t s y x b a)(t s y x b a)(y x b a)
>
> My solution in the old macro system used syntax-local-get-shadower
> together with syntax-local-introduce to "refresh" the vars identifier
> in order to redefine its meaning in each new scope.
>
> In what direction should I look to fix this example?
>
> [The error message with the new macro expander is:
> internal definition: duplicate binding name in: vars]
>
>
> #lang racket
> (require (for-syntax syntax/parse racket/format))
>
> (begin-for-syntax
> (define module-level-variables '())
> ; each internal definition contexts is given an index
> (define intdefs (make-hasheq))
> ; each index is associated to a list of declared names
> (define local-variables (make-hash))
>
> (define (new-intdef? intdef) (hash-has-key? intdefs intdef))
> (define (index-of-intdef intdef) (hash-ref! intdefs intdef (hash-count
> intdefs)))
> (define (add-local-var! index var)
> (hash-update! local-variables index (λ (old-vars) (cons var old-vars))
> '())))
>
> (define-syntax (vars stx)
> (with-syntax ([vs module-level-variables])
> #''vs))
>
> (begin-for-syntax
> (define refresh-identifier (compose syntax-local-introduce
> syntax-local-get-shadower)))
>
> (define-syntax (declare stx)
> (syntax-parse stx
> [(_ id)
> (define var (syntax->datum #'id))
> (define ctx (syntax-local-context))
> (cond
> [(eq? ctx 'module)
> (set! module-level-variables (cons var module-level-variables))
> #'(begin)]
> [(list? ctx) ; internal definition context
> (define old-scope? (new-intdef? ctx))
> (define index (index-of-intdef ctx))
> (add-local-var! index var)
> (cond [old-scope? #'(begin)]
> [else (with-syntax ([vars (refresh-identifier
> #'vars)]
> [old-vars
> (syntax-local-get-shadower #'vars)]
> [index index]
> [.... #'(... ...)])
> #'(define-syntax (vars st)
> (define locals (hash-ref local-variables
> index))
> (define others (local-expand #'(old-vars)
> (syntax-local-context) #f))
> ; others = (quote (var ...)) so we skip the
> quote here:
> (with-syntax ([(local ....) locals]
> [(_ (other ....)) others])
> #`'(local .... other ....))))])]
> [else (error 'declare
> (~a "declarations are only allowed at the module level "
> "and in internal definition contexts"))])]))
>
> (declare a)
> (display (vars))
> (let ()
> (display (vars))
> (declare x)
> (display (vars))
> (let ()
> (display (vars))
> (declare s)
> (declare t)
> (display (vars)))
> (declare y)
> (display (vars)))
> (declare b)
>
> ; displays: (b a)(y x b a)(y x b a)(t s y x b a)(t s y x b a)(y x b a)

[Jens Axel Søgaard]

2015-04-13 14:40 GMT+02:00 Matthew Flatt <mfl...@cs.utah.edu>:

> The new macro expander is ready for you to try:

...

>  * A macro that uses `syntax-local-get-shadower` like will have to be
>    revised, and it should be revised to avoid that function. A macro
>    that uses `syntax-local-make-delta-introducer` definitely will have
>    to be revised, because that function is no longer supported. Most
>    macros don't use such obscure functions.

[Anthony Carrico]

On 04/13/2015 08:40 AM, Matthew Flatt wrote:
> I've also updated the write-up at
>
> http://www.cs.utah.edu/~mflatt/scope-sets-4/

From scope-sets-4:
> In a macro-extensible language, expanding a use of a macro creates a
> new scope in the same way that a binding form creates a new scope.
> Starting with

> (let ([x 1]) (let-syntax ([m (lambda (stx) #'x)]) (lambda (x) (m))))

> the right-hand side of the m binding is has the scope set {a}, while
> the final m has scope set {a, b, c} corresponding to the let,
> let-syntax, and let forms.

Typos:

1. "is has"

2. "let, let-syntax, and let" -> "let, let-syntax, and lambda"

--
Anthony Carrico

[Alexander D. Knauth]

On Apr 13, 2015, at 8:40 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:

> Most packages build ok using the above snapshot:
>
> http://next-pkg-build.racket-lang.org.s3-website-us-west-2.amazonaws.com/
>
> Several failures are "ambiguous binding" errors. That error usually
> indicates an unavoidable incompatibility between the old and new macro
> systems. I'm interested to hear whether the package authors can
> understand the errors and fix them and whether the incompatibilities
> seem manageable.

For a “hygienic" reader-extension, that outputs syntax with lexical context telling it where an identifier should be from so it isn’t captured, what should I do in the new expander?

For example the afl and rackjure pkgs both provide a reader extension that looks like for example:
#λ(+ % 1)
Which is read as: (sort of)
(lambda (%) (+ % 1))
Where the `lambda` identifier has lexical context that says that it comes from racket/base.

But with the new macro expander, I’m seeing errors like this:
lambda: identifier's binding is ambiguous
context...:
#(42661 module) #(42683 module reader 0) #(42873 module)
#(42874 module test-afl-at-exp-racket 0) #(43994 local) #(43995 intdef)
matching binding...:
#<module-path-index:(racket/base)>
#(42661 module) #(42683 module reader 0)
matching binding...:
#<module-path-index:(racket/base)>
#(42661 module) #(42683 module reader 0)
matching binding...:
#<module-path-index:(racket/base)>
#(42873 module) #(42874 module test-afl-at-exp-racket 0)
in: lambda
compilation context...:
/home/racket/build-pkgs/user/.racket/snapshot/pkgs/afl/afl/tests/test-afl-at-exp-racket.rkt

So, is there a way to do this in the new macro expander?

[Matthew Flatt]

At Wed, 15 Apr 2015 17:43:42 -0400, "Alexander D. Knauth" wrote:
>
> On Apr 13, 2015, at 8:40 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
>
> > Most packages build ok using the above snapshot:
> >
> > http://next-pkg-build.racket-lang.org.s3-website-us-west-2.amazonaws.com/
> >
> > Several failures are "ambiguous binding" errors. That error usually
> > indicates an unavoidable incompatibility between the old and new macro
> > systems. I'm interested to hear whether the package authors can
> > understand the errors and fix them and whether the incompatibilities
> > seem manageable.
>
> For a “hygienic" reader-extension, that outputs syntax with lexical context
> telling it where an identifier should be from so it isn’t captured, what should
> I do in the new expander?
>
> For example the afl and rackjure pkgs both provide a reader extension that
> looks like for example:
> #λ(+ % 1)
> Which is read as: (sort of)
> (lambda (%) (+ % 1))
> Where the `lambda` identifier has lexical context that says that it comes from
> racket/base.
>

> But with the new macro expander, I’m seeing errors like this: [...]

>
> So, is there a way to do this in the new macro expander?

I don't think a "hygienic" reader extension works in the old expander,
either. The `lambda` produced by the `afl` or `rackjure` reader doesn't
reliably refer to the `lambda` from `racket/base`, because a local
binding for `lambda` captures the reader-produced `lambda`:

#lang rackjure
(let ([lambda 5])
#λ(+ % 1)) ; => unbound `%1`


There's another problem, but it's more subtle:

#lang afl racket/kernel
#λ(+ % 1)

This one goes wrong because there's no run-time dependency on
`racket/base`. (In the case of `#lang rackjure`, that particular problem
doesn't happen, because the reader is bundled with a regular module
dependency that pulls along `racket/base`.)

Some of us refer to these problems as the "graphical syntax problem",
because we've run into them when attempting to implement non-textual
syntax. We don't yet have a solution, but I hope we find one
eventually.

Meanwhile, I think readers have to produce syntax objects that have no
lexical information (i.e., no scopes).

[Alexander D. Knauth]

On Apr 16, 2015, at 8:17 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:

> I don't think a "hygienic" reader extension works in the old expander,
> either. The `lambda` produced by the `afl` or `rackjure` reader doesn't
> reliably refer to the `lambda` from `racket/base`, because a local
> binding for `lambda` captures the reader-produced `lambda`:
>
> #lang rackjure
> (let ([lambda 5])
> #λ(+ % 1)) ; => unbound `%1`

I know, but it still works in the old expander if there is a module-level binding conflicting with it, so that’s a little bit of success, and it also works if the language doesn’t provide a lambda at all, so that’s some more success, but I was hoping that a better expansion model would be able to solve this problem for local bindings too.

> There's another problem, but it's more subtle:
>
> #lang afl racket/kernel
> #λ(+ % 1)
>
> This one goes wrong because there's no run-time dependency on
> `racket/base`.

I hadn’t thought of this one, but it makes sense, and I also think it’s reasonable to expect it to only work for a language that at least depends on racket/base, but if I remember correctly, it did work for a language that depends on racket/base but didn’t provide it. I’ll add that to the docs for afl, for now.

> Meanwhile, I think readers have to produce syntax objects that have no
> lexical information (i.e., no scopes).

Okay, but I was hoping otherwise.

Alex Knauth

[Gustavo Massaccesi]

Just for fun. Don't try this at home:

#lang racket
(define-syntax-rule (quasiquote x) (reverse 'x))
`(1 2 3 4)

Gustavo

[Alexander D. Knauth]

The problem is that the backquote character is an unhygienic reader macro.
But it was intended to work that way as an abbreviation, so that’s why it's not a problem.

On Apr 16, 2015, at 4:09 PM, Jos Koot <jos....@gmail.com> wrote:

> I tried it and it works as I expected.
> So what is the problem?
> Jos Koot

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Developers" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-dev+...@googlegroups.com.
> To post to this group, send email to racke...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/racket-dev/CAPaha9MC%2BruE_BpfLmBFW6nsktOj
> 0uRB6VUEHqRBpfXpaOPYdQ%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Robby Findler]

On Thu, Apr 16, 2015 at 2:17 PM, Alexander D. Knauth
<alex...@knauth.org> wrote:
> Okay, but I was hoping otherwise.

If all the problems were solved then we wouldn't have anything to do, right?!

Maybe this one is for you. :)

Robby

[Alexander D. Knauth]

On Apr 16, 2015, at 5:01 PM, Jos Koot <jos....@gmail.com> wrote:

> But:
>
> (define quasiquote list)
> (car '`x) ; -> quasiquote as well, which is not obvious.
>
> Racket has become very far in hygienic macros.
> Is is time to think about hygienic read-tables too?
> I have no idea how to acccomplish that. Sorry.

I think that even if the backquote where a “hygienic” reader macro, this would still produce ‘quasiquote.

That’s because `x would read as (quasiquote x), where quasiquote is bound to, well, who cares what it’s bound to, it’s quoted as a symbol.

But if it’s not quoted then:
For an unhygienic backquote reader macro:
(define quasiquote list)
(define x 5)
`x ; -> (quasiquote x), where quasiquote is bound to the definition above -> (list x) -> (list 5) -> ‘(5)
For a hygienic backquote reader macro:
(define quasiquote list)
(define x 5)
`x ; -> (quasiquote x), where quasiquote is bound to quasiquote from racket -> ‘x

This already works in some cases with the current macro expander, if I extend the readtable with a mapping of the backquote character to a procedure that returns a syntax object that has lexical context telling it where the quasiquote identifier came from. But with the current macro expander, there are many cases where hygienic reader macros won’t work properly, which Matthew Flatt pointed out.

[Alexander D. Knauth]

On Apr 16, 2015, at 8:17 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:

> I don't think a "hygienic" reader extension works in the old expander,
> either. The `lambda` produced by the `afl` or `rackjure` reader doesn't
> reliably refer to the `lambda` from `racket/base`, because a local
> binding for `lambda` captures the reader-produced `lambda`:
>
> #lang rackjure
> (let ([lambda 5])
> #λ(+ % 1)) ; => unbound `%1`

I just solved this problem for afl! (with the current macro expander)
https://github.com/AlexKnauth/afl/commit/130216c2b85077aef893f7d0828d69394bceece7

[Jos Koot]

I see what you mean, I think:

For example:

#lang racket
(car '`x); -> quasiquote, which is quite obvious.

But:

(define quasiquote list)
(car '`x) ; -> quasiquote as well, which is not obvious.

Racket has become very far in hygienic macros.
Is is time to think about hygienic read-tables too?
I have no idea how to acccomplish that. Sorry.

Thanks, Jos Koot.

[Jos Koot]

I tried it and it works as I expected.
So what is the problem?
Jos Koot

-----Original Message-----
From: gus....@gmail.com [mailto:gus....@gmail.com] On Behalf Of Gustavo
Massaccesi
Sent: jueves, 16 de abril de 2015 21:59
To: Alexander D. Knauth
Cc: Matthew Flatt; Racket Devs
Subject: Re: [racket-dev] try the new macro expander

[Matthew Flatt]

Yes, with the current expander, adding a fresh mark on the
reader-introduced identifiers makes them impossible to bind (except by
picking apart the form produced by `#λ...` to extract the special
identifiers).

Although that strategy doesn't work with the set-of-scopes expander,
you can get the same result for both expanders by making the symbolic
half of the identifier fresh, instead of the lexical-context half. That
is, with

(define-syntax (define-unbindable-id stx)
(syntax-case stx ()
[(_ name id)
(with-syntax ([gen-id (string->uninterned-symbol
(symbol->string
(syntax-e #'id)))])
#`(begin
(require (only-in racket/base [id gen-id]))
(define name (quote-syntax gen-id))))]))

(define-unbindable-id afl-lambda-id lambda)

use `afl-lambda-id` in place if `#'lambda`.


I see that you've set up `afl` so that `#λ...` will work in phases -10
to 10. I also see that `rackjure` doesn't do that, supporting only
phase 0. It may be worth noting that the change applied to `rackjure`
means that `#λ...` can no longer work in phases other than 0 (since
client code cannot introduce a suitable binding). And we'd probably
agree that `afl`'s support for specific phases is a hack. The
uninterned-symbol approach has the same limitations.


Those limitations make me feel that these approaches are the wrong
direction for a general solution. Both approaches introduce a dimension
(i.e., scopes) into reader results that isn't supposed to be at that
level, similar to the way that syntax objects are not supposed to be
"3-D". Another way to put it is that `read` is meant to work for
reading code (although with a loss of source-location information), not
just `read-syntax`.

Meanwhile, I think that reader extensions should be able to define new
kinds of literals, and those literals should work with `quote`. While
`#λ...` isn't that kind of extension, it should have a more predictable
result under `quote. But `quote` is a bigger problem for other
extensions that we've tried, such as images or quaternions.

Maybe the answer is that reader extensions produce a certain kind of
prefab struct, and something in the expander level should allow those
prefabs to be handled differently, even under `quote`. Something like
that might enable extensions that are abstract and work across phases
while also giving languages control over what kinds of values are
allowed.

[Alexander D. Knauth]

On May 9, 2015, at 8:59 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:

> Yes, with the current expander, adding a fresh mark on the
> reader-introduced identifiers makes them impossible to bind (except by
> picking apart the form produced by `#λ...` to extract the special
> identifiers).
>
> Although that strategy doesn't work with the set-of-scopes expander,
> you can get the same result for both expanders by making the symbolic
> half of the identifier fresh, instead of the lexical-context half. That
> is, with
>
> (define-syntax (define-unbindable-id stx)
> (syntax-case stx ()
> [(_ name id)
> (with-syntax ([gen-id (string->uninterned-symbol
> (symbol->string
> (syntax-e #'id)))])
> #`(begin
> (require (only-in racket/base [id gen-id]))
> (define name (quote-syntax gen-id))))]))
>
> (define-unbindable-id afl-lambda-id lambda)
>
> use `afl-lambda-id` in place if `#'lambda`.

Do you mean that the afl reader would add a (define-unbindable-id ….) to the module body?
But then what if I use something like #lang afl syntax/module-reader or something else that’s dependent on stuff like that?
Plus there’s also the possibility of one of begin, require, only-in, define, or quote-syntax being shadowed, right?
Or do you mean that this would work if I put define-unbindable-id in once in afl/reader.rkt, or something else?

> I see that you've set up `afl` so that `#λ...` will work in phases -10
> to 10. I also see that `rackjure` doesn't do that, supporting only
> phase 0. It may be worth noting that the change applied to `rackjure`
> means that `#λ...` can no longer work in phases other than 0 (since
> client code cannot introduce a suitable binding). And we'd probably
> agree that `afl`'s support for specific phases is a hack.

Yes; that’s why before I had been asking about whether there was a way to create an identifier that told the macro expander that it’s supposed to be bound to lambda from racket/base no matter what phase level it’s in, but you already answered that question.

> The uninterned-symbol approach has the same limitations.

I think I’m misunderstanding something here, so this depends on you answer to my first few questions, but how?

> Those limitations make me feel that these approaches are the wrong
> direction for a general solution. Both approaches introduce a dimension
> (i.e., scopes) into reader results that isn't supposed to be at that
> level, similar to the way that syntax objects are not supposed to be
> "3-D". Another way to put it is that `read` is meant to work for
> reading code (although with a loss of source-location information), not
> just `read-syntax`.

Even if you pretend that it never worked at all, though, Why shouldn’t the reader be able to tell the macro expander about how identifiers should be bound?
I understand the problem with 3d-syntax based on the separation of phase-levels and the separate-compilation guarantee, but is there a similar argument for why this shouldn’t work?

> Meanwhile, I think that reader extensions should be able to define new
> kinds of literals, and those literals should work with `quote`. While
> `#λ...` isn't that kind of extension, it should have a more predictable
> result under `quote. But `quote` is a bigger problem for other
> extensions that we've tried, such as images or quaternions.

Do you mean that you would expect (quote #fn(+ % 1)) to produce the same procedure as it would without the quote?
But that’s starting to get into more 3d-syntax, right?
This is off topic and has nothing to do with #fn, but would the structs for the images or quaternions need to be defined in a cross-phase-persistent module?

> Maybe the answer is that reader extensions produce a certain kind of
> prefab struct, and something in the expander level should allow those
> prefabs to be handled differently, even under `quote`. Something like
> that might enable extensions that are abstract and work across phases
> while also giving languages control over what kinds of values are
> allowed.

Um, I think I’m lost again. How would this work?
Are you saying that this would be an alternative to 3d-syntax, or would this be used for things like #fn?

[Matthew Flatt]

At Sat, 9 May 2015 10:10:12 -0400, "Alexander D. Knauth" wrote:
>
> On May 9, 2015, at 8:59 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
>
> > Yes, with the current expander, adding a fresh mark on the
> > reader-introduced identifiers makes them impossible to bind (except by
> > picking apart the form produced by `#λ...` to extract the special
> > identifiers).
> >
> > Although that strategy doesn't work with the set-of-scopes expander,
> > you can get the same result for both expanders by making the symbolic
> > half of the identifier fresh, instead of the lexical-context half. That
> > is, with
> >
> > (define-syntax (define-unbindable-id stx)
> > (syntax-case stx ()
> > [(_ name id)
> > (with-syntax ([gen-id (string->uninterned-symbol
> > (symbol->string
> > (syntax-e #'id)))])
> > #`(begin
> > (require (only-in racket/base [id gen-id]))
> > (define name (quote-syntax gen-id))))]))
> >
> > (define-unbindable-id afl-lambda-id lambda)
> >
> > use `afl-lambda-id` in place if `#'lambda`.
>
> Do you mean that the afl reader would add a (define-unbindable-id ….) to the
> module body?

> [...]

> Or do you mean that this would work if I put define-unbindable-id in once in
> afl/reader.rkt, or something else?

"No" to the former, "yes" to the latter. The macro and use would be in
the reader's implementation. The reader would return the identifier
bound to `afl-lambda-id` where it currently returns a `lambda`
identifier.

> Plus there’s also the possibility of one of begin, require, only-in,
> define, or quote-syntax being shadowed, right?

Yes, you'd need `(define-unbindable-id afl-define-syntax define-syntax)`,
and so on, for all the identifiers produced by reading `#λ...`.



For the rest of your message below: I don't know. An answer to these
questions is research --- maybe a good topic for you! I'm happy to
discuss the issues further, but e-mail is not great for that kind of
discussion; if you're interested, maybe we could try a different mode.

Meanwhile, relevant to this thread, a question is whether the
set-of-scopes expander likely to lose anything --- relative to the
current expander --- that's needed for a good solution to the problem.
I think it isn't, mostly because we didn't have a good answer so far.

[Alexander D. Knauth]

Thanks!

[Matthew Flatt]

At Sat, 9 May 2015 06:59:53 -0600, Matthew Flatt wrote:
> At Thu, 16 Apr 2015 17:39:50 -0400, "Alexander D. Knauth" wrote:
> >
> > On Apr 16, 2015, at 8:17 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
> >
> > > I don't think a "hygienic" reader extension works in the old expander,
> > > either. The `lambda` produced by the `afl` or `rackjure` reader doesn't
> > > reliably refer to the `lambda` from `racket/base`, because a local
> > > binding for `lambda` captures the reader-produced `lambda`:
> > >
> > > #lang rackjure
> > > (let ([lambda 5])
> > > #λ(+ % 1)) ; => unbound `%1`
> >
> > I just solved this problem for afl! (with the current macro expander)
> >
> https://github.com/AlexKnauth/afl/commit/130216c2b85077aef893f7d0828d69394bceec
> e
> > 7
>
> Yes, with the current expander, adding a fresh mark on the
> reader-introduced identifiers makes them impossible to bind (except by
> picking apart the form produced by `#λ...` to extract the special
> identifiers).
>
> Although that strategy doesn't work with the set-of-scopes expander,

Another idea:

Adding a fresh mark isn't sufficient, because the reader-introduced
identifier will still have all the marks of the enclosing module, and
so it will see any bindings created by the module.

You want the reader-introduced identifier to be *missing* a scope that
the module has, so it won't see any of the module's bindings. It occurs
to me now that you can arrange it to be missing a scope, because `#lang
afl` controls the `read-syntax` call for the entire module. That
`read-syntax` can be wrapped to add a scope, and the scope can be
canceled for the reader-introduced identifiers in the readtable action
for `#λ`.


Specifically, starting back with the pre-set-of-scopes implementation
of afl's "reader.rkt", add

(define current-afl-introduce (make-parameter values))

and set the new parameter in `wrap-reader` to a scope-toggling
function:

(define (wrap-reader p)
(lambda args
(define orig-readtable (current-readtable))
(define introduce (make-syntax-introducer))
(parameterize ([current-readtable (make-afl-readtable orig-readtable)]
[current-afl-introduce introduce])
;; This is not quote right, because `p` could be `read`
;; instead of `read-syntax`. Maybe check that the result from
;; `(apply p args)` is syntax object, and use `introduce` only
;; then:
(introduce (apply p args)))))

and use the parameter's value in `parse` instead of creating an
introducer there:

(define (parse ....)
....
(define intro (current-afl-introduce))
....)

[Alexander D. Knauth]

Oh! That sounds like the perfect solution!

Thanks!

I'll try that.