Identifier equality of identifiers stashed in preserved syntax properties

[Alexis King]

In the language I am working on, I store identifiers in preserved
syntax properties to attach type information to pieces of syntax.
More specifically, I store identifiers inside prefab structs, which
are then stored within preserved syntax properties. This seems to
work fine when my program spans a single module, but it breaks in
a confusing way when I use an identifier from another module.

Specifically, I have two identifiers from the “same” place, but the
macro that attaches the syntax property is used in two different
modules. I would expect these identifiers to be free-identifier=?,
but they are not. Looking at what identifier-binding returns, I get
the following results:

(#<module-path-index:()> List #<module-path-index:()> List 0 0 0)
(#<module-path-index:(rascal/prelude)> List #<module-path-index:(rascal/prelude)> List 0 0 0)

Obviously, the module path index values are different, but I don’t
know why or how to fix it. My attempt to create a minimal reproducible
test case led to me filing racket/racket#1495[1], but that issue
seems to be slightly different from my problem.

If I stick an identifier in a preserved syntax property, what are
the guarantees I have about it? Is using free-identifier=? on it
from another module dangerous in some way? The documentation doesn’t
seem to mention anything about this, but in the meantime, I’m
honestly a little unsure how to even begin to fix this (aside from
radically changing how my program works).

[1]: https://github.com/racket/racket/issues/1495

[Matthew Flatt]

Putting identifiers in syntax properties causes them to be hidden from
various scope and module-path-shifting operations. Probably, you're
seeing the effect of hiding an identifier from path shifting (when an
expanded module is compiled or when a compiled module is declared and
instantiated in at a given module path and phase).

I'd like to have a better story for identifiers in syntax properties.
Meanwhile, when I have run into this problem, I've usually been able to
avoid it by putting the identifier in a `quote-syntax` form. That is,
instead of putting a 'my-info property on some expression

e

I made a macro or other transformer generate

(begin
(quote-syntax (my-info <info>))
e)

and a traversal of an expression looks for <info> in expressions that
match that pattern.

If that strategy doesn't work for you (without major changes to your
program), then I'm afraid I don't have a better answer at the moment.

> --
> You received this message because you are subscribed to the Google Groups
> "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to racket-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Alexis King]

That makes sense; thank you for your quick reply. It might be
possible to do something like what you describe, but I do have a
little more context that makes this sort of tricky. I’m trying to
not just store identifiers but also store prefab structs containing
identifiers. The issue I’m running into is that embedding prefab
structs in syntax objects seems to “flatten” the contained syntax
objects. For example, if I do something like this:

(begin-for-syntax
(struct prefab-box (val) #:prefab)
(syntax-parse
(local-expand
#`(let-syntax ([x (λ (stx)
(syntax-property #'(void) 'prop
#,(prefab-box #'add1)))])
x)
'expression '())
#:literals [let-values]
[(let-values () (let-values () x))
(prefab-box-val (syntax-property #'x 'prop))]))

The result of this is not the syntax #'add1, but the symbol 'add1.
The reason this happens makes some intuitive sense (though my
understanding of what happens is not precise enough to really explain
it), but it’s not good enough for what I need.

A hacky solution I’ve tried has been to attach the prefab struct
to a piece of “proxy” syntax as a property, like this:

(define proxy-stx
(syntax-property #'proxy 'prop (prefab-box #'add1)))

Then I can replace the #,(prefab-box #'add1) with the following:

(syntax-property #'#,proxy-stx 'prop)

…and I get #'add1 instead of 'add1. This hack seems to cause problems
when it spans multiple modules, though, as I’m discovering. Is there
some other way I can embed a prefab struct in a piece of syntax
without the syntax objects being lost when it is evaluated as an
expression?

[Ryan Culpepper]

On 10/25/2016 08:04 PM, Alexis King wrote:
> That makes sense; thank you for your quick reply. It might be
> possible to do something like what you describe, but I do have a
> little more context that makes this sort of tricky. I’m trying to
> not just store identifiers but also store prefab structs containing
> identifiers. The issue I’m running into is that embedding prefab
> structs in syntax objects seems to “flatten” the contained syntax
> objects. For example, if I do something like this:
>
> (begin-for-syntax
> (struct prefab-box (val) #:prefab)
> (syntax-parse
> (local-expand
> #`(let-syntax ([x (λ (stx)
> (syntax-property #'(void) 'prop
> #,(prefab-box #'add1)))])
> x)
> 'expression '())
> #:literals [let-values]
> [(let-values () (let-values () x))
> (prefab-box-val (syntax-property #'x 'prop))]))
>
> The result of this is not the syntax #'add1, but the symbol 'add1.

Consider that (eval (list 'quote #'+)) also evaluates to a symbol, for
about the same reason.

Try local-expanding either of these terms instead:

#`(let-syntax ([x (λ (stx)
(syntax-property #'(void) 'prop

(prefab-box #'add1)))])
x)

or

#`(let-syntax ([x (λ (stx)

#'#,(syntax-property #'(void) 'prop
(prefab-box #'add1)))])
x)

and the box will have an identifier in it.

Ryan

[Alexis King]

> On Oct 25, 2016, at 17:27, Ryan Culpepper <ry...@ccs.neu.edu> wrote:
>
> Consider that (eval (list 'quote #'+)) also evaluates to a symbol, for about the same reason.

Yes, this does make sense, even if I couldn’t put it quite so nicely. :)

> Try local-expanding either of these terms instead:
>
> #`(let-syntax ([x (λ (stx)
> (syntax-property #'(void) 'prop
> (prefab-box #'add1)))])
> x)
>
> or
>
> #`(let-syntax ([x (λ (stx)
> #'#,(syntax-property #'(void) 'prop
> (prefab-box #'add1)))])
> x)
>
> and the box will have an identifier in it.

I can’t quite do either of these, unfortunately, since in my case, I
need to defer attaching the syntax property until `x` is actually used
(I apply an instantiate-type function to the prefab struct value at each
use site), and the prefab struct value itself is not known ahead of time
(it is produced by another function, and it can be a relatively complex
structure).

The approach I’ve taken so far is the cheat with the piece of “proxy”
syntax, which is closer to your second example, since it uses syntax
properties to circumvent evaluation. It causes the issue mentioned at
the beginning of this thread, though.

It seems like perhaps, in theory, I could come up with a function
prefab->expression which would take a prefab struct value and produce a
syntax object containing an expression that would evaluate to the same
thing, but this is not only an extremely ugly hack but one that won’t
work unless I know all the different kinds of prefab structs ahead of
time (and I would like users to be able to use arbitrary ones). So I
feel a bit stuck.

[Alexis King]

As a small followup to this, I managed to come up with the following
hacky workaround, which seems to be working alright so far.
Specifically, it’s a function that converts arbitrary preservable values
into expressions that evaluate to themselves:

(define preservable-property->expression
(match-lambda
[(and (app prefab-struct-key (? values prefab-key))
(app struct->vector (vector _ fields ...)))
#`(make-prefab-struct
(quote #,prefab-key)
#,@(map preservable-property->expression fields))]
[(? list? lst)
#`(list #,@(map preservable-property->expression lst))]
[(cons a b)
#`(cons #,(preservable-property->expression a)
#,(preservable-property->expression b))]
[(? syntax? stx)
#`(quote-syntax #,stx)]
[(and (or (? boolean?) (? symbol?) (? number?) (? char?)
(? string?) (? bytes?) (? regexp?))
datum)
#`(quote #,datum)]
[other
(error
'preservable-property->expression
"disallowed value within preserved syntax property\n value: ~e"
other)]))

(That doesn’t currently handle all cases, like boxes/vectors/hash maps,
but it would be easy enough to extend.)

This is obviously a total hack, but it seems to work alright for this
particular case, and I thought I’d share it in case anyone else runs
into the same problem and this is helpful for them, too.