format-id doesn't preserve preserved?-edness of syntax-property?

78 views
Skip to first unread message

William J. Bowman

unread,
Dec 11, 2016, 7:20:12 PM12/11/16
to Racket Users
Before I file a bug report, I want to make sure something *else* isn't going on.

In "format-id1.rkt" (https://gist.github.com/wilbowma/247d15e0e0bed6b239584854e79b5015), I define and export some identifiers with preserved syntax-properties. The identifier "id" is created using format-id with #:props to attach a syntax-property. The identifier "id2" is created using generate-temporary and a syntax-property is attached manually. I also use these the syntax-property in the same file. If I run or compile the file, no errors.

In "format-id2.rkt" (https://gist.github.com/wilbowma/ed071343c8c651f3a78d690a4ab7a392), I use the syntax-properties in exactly the same way. However, if I run or compile the file, I get errors that indicate "id" is missing it's syntax-property.

--
William J. Bowman
Northeastern University
College of Computer and Information Science
signature.asc

Alex Knauth

unread,
Dec 11, 2016, 7:54:32 PM12/11/16
to William J. Bowman, 'William J. Bowman' via Racket Users

> On Dec 11, 2016, at 7:20 PM, 'William J. Bowman' via Racket Users <racket...@googlegroups.com> wrote:
>
> Before I file a bug report, I want to make sure something *else* isn't going on.

It looks like "something else" is going on; according to syntax-property-preserved? it's still marked as preserved in the program below.

#lang racket
(require racket/syntax)

(define stx (syntax-property #'x 'my-prop 5 #t))
(syntax-property-preserved? stx 'my-prop)
;=> #true

;; transfer properties from stx
(define stx* (format-id #f "y" #:props stx))
(syntax-property-preserved? stx* 'my-prop)
;=> #true

Alex Knauth

William J. Bowman

unread,
Dec 12, 2016, 12:55:31 PM12/12/16
to Racket Users
Ah, thanks.

Anyone else have other ideas for problems using syntax-properties, or maybe make-rename-transformers (with not-free-identifier=?), across modules?

--
William J. Bowman
signature.asc

William J. Bowman

unread,
Dec 13, 2016, 2:23:26 PM12/13/16
to Racket Users
After a debugging session with Leif, the problem is actually a user-error caused by sublte issues with macros-generating-macors:

In the failing case, I was generating the following

(with-syntax ([x (format-id ...)])
#`(begin
(define x body)
(define-syntax id
(make-rename-transformer #'x))))

Notice that #'x is not the same identifier as x, and thus does not have the same syntax-properties.

Why does this issue only comes up across a file boundary? I suspect this has to do with the behavior of make-rename-transformer. Since #'x was not getting 'not-free-identifier=? correctly, make-rename-transformer was optimizing this away, although I thought I was preventing that.

--
William J. Bowman

PS: Thanks Leif!
> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.


signature.asc

Alexis King

unread,
Dec 13, 2016, 5:14:36 PM12/13/16
to William J. Bowman, Matthias Felleisen, Racket Users
> On Dec 13, 2016, at 11:23 AM, 'William J. Bowman' via Racket Users
> <racket...@googlegroups.com> wrote:
>
> Notice that #'x is not the same identifier as x, and thus does not
> have the same syntax-properties.

I’m not convinced this is correct.

If you insert println statements in the body of define^, inside the
with-syntax block, you get the following:

(println (syntax-property #'x 'definition))
(println (syntax-property-preserved? #'x 'definition))

; => #<syntax:format-id1.rkt:49:12 id^>
; => #t

The syntax properties are definitely maintained in templates used
by `syntax`.

> Why does this issue only comes up across a file boundary? I suspect
> this has to do with the behavior of make-rename-transformer. Since
> #'x was not getting 'not-free-identifier=? correctly,
> make-rename-transformer was optimizing this away, although I thought
> I was preventing that.

This is more interesting, and it was my first instinct, too. For
that reason, I replaced uses of make-rename-transformer with uses
of make-variable-like-transformer from syntax/transformer, but the
behavior was the same, so I don’t think this has to do with
make-rename-transformer specifically. However, something more
interesting happens.

I changed the macro that define^ defines to insert some debugging
calls on macro invocation:

(define-syntax (id stx)
(println (syntax-property #'x 'definition))
(println (syntax-property-preserved? #'x 'definition))
((set!-transformer-procedure
(make-variable-like-transformer #'x))
stx))

The output of this is much more interesting. When running the source
module, format-id1.rkt, the result is as one would expect:

; => #<syntax:format-id1.rkt:49:12 id^>
; => #t

However, when using the defined identifier from format-id2.rkt, the
result is different:

; => #f
; => #f

This is extremely interesting, since the first example demonstrates
that the syntax property is most definitely marked as preserved,
yet it gets lost by the time format-id2 tries to get at it, despite
the fact that it should be the same exact syntax object.

My guess as to what’s going on here is that the issue is not with
make-rename-transformer but with quote-syntax. The above program
expands into a macro definition that has (quote-syntax x) in it,
and the x has a preserved syntax property on it. However, that
property is not actually preserved, despite being marked as preserved.
This is different from other uses of preserved syntax properties
that attach their syntax properties to expressions in the resulting
program, rather than quoted pieces of syntax.

This feels very confusing, and it certainly feels like a bug, but
maybe Matthew has an explanation for this behavior that I don’t
currently understand?

Alexis

Matthew Flatt

unread,
Dec 14, 2016, 10:14:55 AM12/14/16
to William J. Bowman, Alexis King, Matthias Felleisen, Racket Users
I think there are two problems in this example.


The first problem is that preserved syntax properties were not actually
preserved in bytecode for a syntax object that has no source location.
(I'm assuming that Alexis's tests were run in DrRacket in it's default
mode of compiling bytecode for `require`d files.) I'll push a repair
for that bug.


The second problem is that syntax objects as property values don't work
the way that you want. That limitation is why, even with the
property-presevation bug fixed, the `free-identifier=?` test fails.
Unfortunately, that's a limitation of syntax objects and properties
that's difficult to resolve.

To explain a little more, consider the program

https://gist.github.com/mflatt/d569d8f3a89e6815cf18df26adc11053

For now, look at just the `property` submodule. The
`quote-syntax/add-example-property` macro is like `quote-syntax`,
except that it adds an 'example property that holds the current time at
compile time.

An expression like

(syntax-property (quote-syntax/add-example-property id) 'example)

accesses that compile-time value at run time. If you compile the
program to bytecode, the value shown by running the module later still
reports the time that the module was compiled.

Now consider the enclosing module, which uses the `property` submodule
and accesses the time for both a freshly quoted term and the value of
`id` as exported by the submodule:

(require 'property)
(syntax-property (quote-syntax/add-example-property new-id) 'example)
(syntax-property id 'example)

I think you won't be too surprised that the freshly quoted identifier
has a newer time than the one reported for `id`. Even though both
values were obtained from the same `compile-time` variable, each module
compilation (submodule and enclosing module) gets a fresh instance of
`compile-time`.


In "format-id1.rkt", instead of `compile-time` bound to a timestamp,
you have a quoted syntax object that you can image being implicitly
lifted as

(define-for-syntax compile-time #'id^)

In much the same way that `(current-inexact-milliseconds)` in

(define-for-syntax compile-time (current-inexact-milliseconds))

produces a different result for each subsequent compile-time use of the
modules, `#'id^` produces a different syntax object depending on how
its enclosing module is instantiated. Imagine taking the compiled
bytecode, saving it in two places, and loading it as a module from each
place; the `#'id` evaluations for the copies will produce identifiers
that claim to be from different modules relative to the filesystem.

So, here's the limitation: the implicit lifting that happens for a
quoted syntax object, which enables a kind of re-evaluation of that
syntax object in each module instance, doesn't happen for
syntax-property values. More generally, changes to the scope of a
syntax object are not propagated to property values that happen to be
themselves syntax objects. When you put a syntax object into a
property, then you get whatever that syntax object meant at the time it
was attached as a property.

That might still sound ok for your case, because you compile and use
"format-id1.rkt" in-place. Still, the information attached to `#'id^`
changes. While compiling the module, the attached binding information
says "the current module being compiled". When compiling
"format-id2.rkt", the attached binding information for that `#'id^`
says "format-id1.rkt".


To summarize, don't try to attach syntax objects as property values
like that. I know this advice sounds ironic, given that the original
use of properties was for syntax-valued keys like 'origin. Properties
like 'origin are meant to be extracted from just-expanded programs,
though, as opposed to used in a later compilation step. At the same
time, those uses of properties are why we can't just make
`syntax-property` reject property values that are syntax objects.

Sam Tobin-Hochstadt

unread,
Dec 14, 2016, 10:23:29 AM12/14/16
to Matthew Flatt, William J. Bowman, Alexis King, Matthias Felleisen, Racket Users
On Wed, Dec 14, 2016 at 10:14 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
> More generally, changes to the scope of a
> syntax object are not propagated to property values that happen to be
> themselves syntax objects. When you put a syntax object into a
> property, then you get whatever that syntax object meant at the time it
> was attached as a property.

This is a limitation of syntax properties that I've long had to work
around in Typed Racket as well. Are there fundamental reasons why we
couldn't have a kind of syntax property that scopes propaged to? Would
this be something that would be easier to implement in the new
expander?

Sam

Matthias Felleisen

unread,
Dec 14, 2016, 11:09:20 AM12/14/16
to Sam Tobin-Hochstadt, Matthew Flatt, William J. Bowman, Alexis King, Racket Users
I can’t believe I fell for this again. Ryan has explained this more than once to me . . . and yet. In this spirit, I support Sam’s question :-)

Matthew Flatt

unread,
Dec 14, 2016, 12:43:38 PM12/14/16
to Matthias Felleisen, Sam Tobin-Hochstadt, William J. Bowman, Alexis King, Racket Users
Yes, it's obviously easier to implement this new feature with the new
expander.

But right now, as Sam well knows, I'm open to patches only as long as
they make things no slower and no bigger. :) The current implementation
is reliably within a factor of 2 of the old one, but we're still
working to close the gap.

Matthew Flatt

unread,
Dec 14, 2016, 12:51:50 PM12/14/16
to Matthias Felleisen, Sam Tobin-Hochstadt, William J. Bowman, Alexis King, Racket Users
At Wed, 14 Dec 2016 10:43:34 -0700, Matthew Flatt wrote:
> But right now, as Sam well knows

I'd like to clarify that this is because Sam has been very helpful in
making things better --- not that he's been asking for new things.

Alexis King

unread,
Dec 14, 2016, 6:48:13 PM12/14/16
to Matthew Flatt, Stephen Chang, Alex Knauth, Matthias Felleisen, Racket Users
> On Dec 14, 2016, at 7:14 AM, Matthew Flatt <mfl...@cs.utah.edu> wrote:
>
> To summarize, don't try to attach syntax objects as property values
> like that. I know this advice sounds ironic, given that the original
> use of properties was for syntax-valued keys like 'origin. Properties
> like 'origin are meant to be extracted from just-expanded programs,
> though, as opposed to used in a later compilation step. At the same
> time, those uses of properties are why we can't just make
> `syntax-property` reject property values that are syntax objects.

What are the implications of this for the Types as Macros/Turnstile
stuff that Stephen and Alex have been working on, given that they
use heavily use syntax objects in syntax properties as part of
attaching type information to identifiers? Are they doing something
to work around this and/or using properties in a way that avoids
this issue? Or is it likely that they’ll run into this for programs
that span multiple modules?

(Obviously, I have a personal motive for asking this question, since
I’ve been playing with Turnstile and the techniques it uses myself.
So far I’ve been able to write programs that span multiple modules
without much issue, modulo an issue I asked about on this mailing
list a month or two back, but I’d like to know if I’m relying on
some sort of undefined behavior.)

William J. Bowman

unread,
Dec 14, 2016, 9:47:55 PM12/14/16
to Alexis King, Matthew Flatt, Stephen Chang, Alex Knauth, Matthias Felleisen, Racket Users
On Wed, Dec 14, 2016 at 03:50:28PM -0800, Alexis King wrote:
> What are the implications of this for the Types as Macros/Turnstile
...
>
> (Obviously, I have a personal motive for asking this question, since
Based on the names of functions in my example, you'd never guess how I ran into this problem.
(Hint: It was implementing dependent types in Racket using Types as Macros.)

--
William J. Bowman
signature.asc

Dupéron Georges

unread,
Dec 15, 2016, 6:13:32 AM12/15/16
to Racket Users
Le jeudi 15 décembre 2016 00:48:13 UTC+1, Alexis King a écrit :
> without much issue, modulo an issue I asked about on this mailing
> list a month or two back

Here's (what I suspect is) the thread Alexis mentions, for those who feel curious: https://groups.google.com/forum/#!topic/racket-users/TGax2h8dVxs (Identifier equality of identifiers stashed in preserved syntax properties).

Stephen Chang

unread,
Dec 15, 2016, 11:54:35 AM12/15/16
to Alexis King, Matthew Flatt, Alex Knauth, Matthias Felleisen, Racket Users
> What are the implications of this for the Types as Macros/Turnstile
stuff?

As William mentioned, he's also working on a language using the Types
as Macros approach. Alex and I have run into the same issue, but it's
always with compiled files (without compiling, requires and provides
appear to behave as expected). Unfortunately, we have not figured out
a good solution either. Could we borrow any ideas from Typed Racket
here?

> but I’d like to know if I’m relying on some sort of undefined behavior

In some sense, the entire Types as Macros approach is "undefined
behavior" :/ because we are using the expander for unintended
purposes. So thank you (and William and everyone else that has tried
it out) for diving in and helping us figure out what works and what
needs more work.

But now that we've shown that such an approach is both viable and
wanted by Racket programmers, for the rough parts where we cannot find
a good solution, the next step might be to address these issues with
contributions to the expander itself.

Just to throw them out there, other issues that I'm working on (in
addition to scopes on stx props) are:
- Better support (over stop lists) for "delimited" expansion. For
example, I often want to "fully expand an expression, but not past a
specified language".
- Better support for expanding expressions with free identifiers in a context.
- Better support for mapping stx objs back to the surface stx, in a
hygienic way.

William J. Bowman

unread,
Dec 15, 2016, 5:06:43 PM12/15/16
to Matthew Flatt, Alexis King, Matthias Felleisen, Racket Users
All,

Thanks for help debugging and explaining this.

FYI, with Matthew's bug fix merged, and help from Alexis, I've managed to work around my issue. It was pretty much the same issue described in thread linked to by Dupéron, https://groups.google.com/forum/#!topic/racket-users/TGax2h8dVxs, and also in the Racket GitHub issue https://github.com/racket/racket/issues/1495.

The workaround seems to be: if you want to stash an identifier in a syntax-property, expand into syntax that attaches the syntax-property. That is, don't do (define x (syntax-property #'id 'prop #'stx)) and then use x in a syntax object, but generate #`(... (syntax-property #'id 'prop #'stx ...))

In my example, the identifier that caused issues was generated by define^, which directly called syntax-property in a for-syntax helper function. The identifier that worked fine was generated by define^^, which generated syntax that included a syntax-property.

Maybe we shouldn't be stashing identifiers in syntax-properties at all, but in practice this workaround seems to work (in the implementations of Alexis' Rascal and my Cur).

--
William J. Bowman
> syntax-property values. More generally, changes to the scope of a
> syntax object are not propagated to property values that happen to be
> themselves syntax objects. When you put a syntax object into a
> property, then you get whatever that syntax object meant at the time it
> was attached as a property.
>
> That might still sound ok for your case, because you compile and use
> "format-id1.rkt" in-place. Still, the information attached to `#'id^`
> changes. While compiling the module, the attached binding information
> says "the current module being compiled". When compiling
> "format-id2.rkt", the attached binding information for that `#'id^`
> says "format-id1.rkt".
>
>
> To summarize, don't try to attach syntax objects as property values
> like that. I know this advice sounds ironic, given that the original
> use of properties was for syntax-valued keys like 'origin. Properties
> like 'origin are meant to be extracted from just-expanded programs,
> though, as opposed to used in a later compilation step. At the same
> time, those uses of properties are why we can't just make
> `syntax-property` reject property values that are syntax objects.
>
>
> At Tue, 13 Dec 2016 14:16:49 -0800, Alexis King wrote:
signature.asc
Reply all
Reply to author
Forward
0 new messages