how do you read, manipulate, debug scope sets?

[zeRusski]

I wrote a macro which introduced an implicit binding <~ so that it could be used

in expressions at the use-site. Initially did it with

#+begin_src racket

  ;; inside syntax-parse

  (datum->syntax this-syntax #'<~)

#+end_src

followed by macro introduced expr that binds it, then the use-site macro-input

that uses it. Think (let/ec <~ macro-input-body).

Worked just fine when tested at top-level or module begin or in expression

position, but then suddenly broke when I wrote another define-like macro whose

body expanded into the macro above. Turns out scopes of <~ at use-site and one I

introduced in a macro didn't match, at least that's what I surmount from the

message below. I was originally going to ask if someone could teach me to read

these messages, but then I found ~syntax-debug-info~ in docs :) and IIUC the

message below tells me there are two identifier bindings where the error occurs

whose scope-sets share some scopes namely "common scopes ...", but neither one's

scope-set is a subset of the other hence the error. Am I reading it right?

#+begin_src racket

; /Users/russki/Code/tilda/prelude/tilda.rkt:303:20: <~: unbound identifier

;   in: <~

;   context...:

;    #(2212719 use-site) #(2212754 intdef) #(2212808 local)

;    #(2212809 intdef) [common scopes]

;   other binding...:

;    local

;    #(2212718 macro) [common scopes]

;   common scopes...:

;    #(2198084 module) #(2198091 module tilda) #(2212726 local)

;    #(2212727 intdef) #(2212737 local) #(2212738 intdef) #(2212741 local)

;    #(2212742 intdef) #(2212745 local) #(2212746 intdef) #(2212749 local)

;    #(2212750 intdef) #(2212753 local)

#+end_src

I fixed the above with some guesswork that amounted to replacing datum->syntax

with

#+begin_src racket

  (syntax-local-introduce #'<~)

#+end_src

which IIUC simply flips the scopes so now <~ is use-site and may as well be part

of the macro input. Right?

Suddenly I find myself playing games with hygiene and not really knowing the

rules.

Are there any tutorials that show you how to use things documented in Syntax

Transformers chapter of the Reference?

How do you debug these scope games?

How do you introduce or capture identifier bindings (break hygiene)?

Can you temporarily unbind an identifier (for the extent of some expr), so

basically remove or trim some scopes from identifiers that occur in macro input? I

suppose there are several possible cases here: 

- trim or replace scopes of ids whose sets match those at use-site, guessing this

  won't unbind "shadowing" identifiers (let or define introduced in your macro

  input) i.e. those with extra scopes in addition to use-site,

- how do we deal with those, could we trim ids whose scope sets are supersets of

  use-site?

- assuming I know how to do the above, do I walk the syntax tree and trim those

  scopes every time I find matching id or is there a better way?

At this point I'd like to better understand how to manipulate sets of scopes and

verify the result. Could someone kindly teach me or point out good reads or

examples?

Thanks

[Matthias Felleisen]

> On May 6, 2019, at 10:53 AM, zeRusski <vladile...@gmail.com> wrote:
>
> Suddenly I find myself playing games with hygiene and not really knowing the rules.

Hygiene is a default not an absolute. The idea of hygiene is that, unless the macro writer goes out of his way, the expander assumes that identifiers coming from the macro’s input and those introduced by the macro (AND identifiers defined and used in the context of the macro definition vs identifiers defined in the macro’s use context .. which is not your current problem) are not related.

If you, as a macro writer, wish to break this default assumption, you need to work for it.

The macro system allows certain manipulations of scope. You have discovered syntax-local-introduce and its use for flipping scopes. And yes, there are more games you can play and you may wish to explore those but they are not for situations like yours. [In your specific example, I think it is often better to pass the identifier into the macro.] For examples, you may wish to look at Alexis’s Hackett implementation. Not all of what you asked for is doable, and I can’t even imagine you’d need all of these.

— Matthias

[Matthew Butterick]

On May 6, 2019, at 7:53 AM, zeRusski <vladile...@gmail.com> wrote:

Are there any tutorials that show you how to use things documented in Syntax

Transformers chapter of the Reference?

How do you debug these scope games?

I agree there is something of a gap in the Racket docs / tooling, because scopes are a relatively recent addition to the macro expander. You might  want to take a look at the `scope-operations` and `debug-scopes` packages.

[1] https://docs.racket-lang.org/scope-operations/index.html

[2] https://docs.racket-lang.org/debug-scopes/index.html

[zeRusski]

Thanks Matthew I'll have a look

All:

I published a link to the code I had in mind in a separate thread, so if interested check it out. I don't want to pollute this one if someone takes on the challenge of answering in more general terms.

[Michael Ballantyne]

It's not an answer to your broader question, but I suspect what you actually want here is

#+begin_src racket

  ;; inside syntax-parse

  (datum->syntax this-syntax '<~)

#+end_src

Notice that the second argument to datum->syntax should be a symbol here, rather than a syntax object. If you provide a syntax object as you did in your original code, datum->syntax simply returns that syntax unchanged.

I don't expect using syntax-local-introduce will work reliably; before the change to the scope sets model of hygiene in Racket 6.3 it was another way to accomplish this purpose, but no longer.  For example, it breaks when the macro introducing the binding and its use are in different modules. Below, (m introduced) works in a but fails in b.

#+begin_src racket

#lang racket

(module a racket

  (define-syntax (m stx)

    (syntax-case stx ()

      [(_ body)

       (with-syntax ([introduced (syntax-local-introduce #'introduced)])

         #'(let ([introduced 5])

             body))]))

  (m introduced)

  (provide m))

(module b racket

  (require (submod ".." a))

  (m introduced))

#+end_src

If you haven't read the papers describing Racket's hygiene algorithm and module system, you might find them helpful. The documentation isn't enough on its own, unfortunately.

For hygiene: http://www.cs.utah.edu/plt/scope-sets/

For phasing: https://www.cs.utah.edu/plt/publications/macromod.pdf 

[zeRusski]

#+begin_src racket

  ;; inside syntax-parse

  (datum->syntax this-syntax '<~)

#+end_src

Notice that the second argument to datum->syntax should be a symbol here, rather than a syntax object. If you provide a syntax object as you did in your original code, datum->syntax simply returns that syntax unchanged.

Damn, you totally caught it! Indeed, I was an idiot and used syntax where symbol was meant. Thank you

 

I don't expect using syntax-local-introduce will work reliably; before the change to the scope sets model of hygiene in Racket 6.3 it was another way to accomplish this purpose, but no longer.  For example, it breaks when the macro introducing the binding and its use are in different modules. Below, (m introduced) works in a but fails in b.

And this totally makes sense now that, thanks to your example, I've seen the error msg its produced. Indeed, the two sets of scopes differ, so you end up with an unbound identifier there. Also confirmed with my own implementation and switched it back to datum->syntax as you suggested above and now escape with <~ works as expected. At least the error matches my mental model even if this means I no longer understand the purpose of syntax-local-introduce.

I think I ought to re-read the syntax model chapter in docs and read the papers you mentioned. Maybe even make it a regular exercise :)

Thank you Michael