questions about top-level-bind-scope in root-expand-context

[Yongming Shen]

Hi,

I noticed that top level (define-values ...) forms create bindings that include the top-level-bind-scope (of the namespace's root-expand-context), and have two questions related to this.

First, in the source file expander/expand/bind-top.rkt, there is a comment that says "When compiling `(define-values (x) ...)` at the top level, we'd like to bind `x` so that a reference in the "..." will point back to the definition, as opposed to being whatever `x` was before." Isn't this the exact opposite of what (define-values (x) ...) should do?

Second, ignoring the comment mentioned above, but still consider `(define-values (x) ...)`. It appears that the binding of `x` to `...` with the top-level-bind-scope is only used by `(#%top . x)` (either explicit or implicit). The only exception seems to be in contexts where neither `x` nor #%top are binded, in which case `x`, not wrapped by #%top, also uses that binding. The case of `(#%top . x)` is confusing because even without the top-level-bind-scope binding of `x`, `(#%top . x)` can still locate `(define-values (x) ...)`, otherwise mutually recursive functions won't work at the top level. As for the exception case, it seems rare enough that it can be ignored. So my question is, what's benefit does the top-level-bind-scope bring?

Thanks!

[Matthew Flatt]

At Sat, 21 Mar 2020 00:00:07 -0700 (PDT), Yongming Shen wrote:
> First, in the source file expander/expand/bind-top.rkt, there is a comment
> that says "When compiling `(define-values (x) ...)` at the top level,
> we'd like to bind `x` so that a reference in the "..." will point back to
> the definition, as opposed to being whatever `x` was before." Isn't this
> the exact opposite of what (define-values (x) ...) should do?

Here's an example scenario that the current rule is meant to cover:

> (require module-that-defines-fib)

> fib ; refers to binding from `module-that-defines-fib`

> (define (fib n)
(if (< n 2)
1
(+ (fib (- n 1)) (fib (- n 2))))) ; refers to this definition

> (fib 27) ; => 514229

If a programmer usually expects the `fib`s in the definition's `...`
region here to refer to the new definition, not to the imported
binding, then the implemented rule is the right one. If programmers
expect that `fib`s in the `...` to refer to the imported binding, then
I'd agree with you. But I think the former is more often the case.

Neither rule is obviously right, and if we make the example more
complicated with more macros involved, I'm pretty sure there's no way
to implement either rule consistently: the top level is hopeless. The
case of a single recursive function seems common enough that we've
picked a rule and implementation to make it work.

> Second, ignoring the comment mentioned above, but still consider
> `(define-values (x) ...)`. It appears that the binding of `x` to `...` with
> the top-level-bind-scope is only used by `(#%top . x)` (either explicit or
> implicit). The only exception seems to be in contexts where neither `x` nor
> #%top are binded, in which case `x`, not wrapped by #%top, also uses that
> binding. The case of `(#%top . x)` is confusing because even without the
> top-level-bind-scope binding of `x`, `(#%top . x)` can still locate
> `(define-values (x) ...)`, otherwise mutually recursive functions won't
> work at the top level. As for the exception case, it seems rare enough that
> it can be ignored. So my question is, what's benefit does the
> top-level-bind-scope bring?

Just to restate (to make sure I understand), you're pointing out that
an unbound identifier is treated the same as a binding using the
top-level scope (i.e., both refer to a top-level variable) --- so why
bother with the top-level scope?

Although the result is the same for variable references, it's not the
same for macro bindings or for imported bindings. And, then, there's no
way to predict in advance that an identifier will be used as a variable
and omit the top-level scope in that case.

Stepping back a little, it may not be the right design choice to treat
an unbound identifier as a reference to a top-level variable. But some
sort of default is necessary to support the traditional top level. And
treating unbound identifiers this was avoids a[nother] special
treatment of top-level scopes, where an unbound variable could be
treated as a variable reference only if it has a top-level scope.

Really, the only consistent approach that I know is to not have an
interactive top level, but that's not an attractive option.


Matthew

[Yongming Shen]

Hi Matthew,

Thank you for the quick reply!

I tried the example you gave for my first question and it resulted in an error.

I have the following as `module-that-defines-fib`:

  #lang racket

  (provide fib)

  (define fib "fib")

And this is the error that I got (using Racket 7.6):

  ; application: not a procedure;

  ;  expected a procedure that can be applied to arguments

  ;   given: "fib"

  ; [,bt for context]

I think this is because `(define-values (x) ...)` expands `...` without the top-level-bind-scope, even when expand-context-to-parsed? is #t (according to expander/expand/top.rkt). Is this a bug?

Related to your answer to my second question, `define-syntaxes` similarly does not add the top-level-bind-scope when expanding `...`. Does this mean that even for `define-syntaxes`, `...` won't use the top-level-bind-scope binding(s) after all?

A little bit off-topic, in the definition of define-values (in expander/expand/top.rkt), there is `(define-match m s ...)`, but for define-syntaxes it is `(define-match m disarmed-s ...)`. Is this difference significant? Or does define-match not care whether `s` or `disarmed-s` is used?

Thanks,

Yongming

[Matthew Flatt]

At Mon, 23 Mar 2020 01:45:40 -0700 (PDT), Yongming Shen wrote:
> I tried the example you gave for my first question and it resulted in an
> error.

Oops --- you're right. I lost track of what we try to make work at the
top level.

> I think this is because `(define-values (x) ...)` expands `...` without the
> top-level-bind-scope, even when expand-context-to-parsed? is #t (according
> to expander/expand/top.rkt). Is this a bug?

Since the behavior goes far back, I think this is the behavior that we
decided to settle for.

> Related to your answer to my second question, `define-syntaxes` similarly
> does not add the top-level-bind-scope when expanding `...`. Does this mean
> that even for `define-syntaxes`, `...` won't use the top-level-bind-scope
> binding(s) after all?

The way that evaluation, binding, and expansion are interleaved means
that a `define-syntaxes` macro can refer to itself in expansions. The
binding of an identifier in a macro template is resolved after the
macro is applied.

The difference with `define` is that the right-hand side is
expanded/compiled before `define` binds.

> A little bit off-topic, in the definition of define-values (in
> expander/expand/top.rkt), there is `(define-match m s ...)`, but for
> define-syntaxes it is `(define-match m disarmed-s ...)`. Is this difference
> significant? Or does define-match not care whether `s` or `disarmed-s` is
> used?

Using `disarmed-s` in the definition of `define-values` is probably
a better idea, and I'll look into that more.

It think it turns out not to matter, normally, because `define-values`
is transparent to syntax arming via `syntax-arm` with a #t second
argument (which is what the expander does). But it would be better to
not rely on that.

[George Neuner]

On Mon, 23 Mar 2020 01:45:40 -0700 (PDT), Yongming Shen
<sym...@gmail.com> wrote:

>Hi Matthew,
>
>Thank you for the quick reply!
>
>I tried the example you gave for my first question and it resulted in an
>error.
>I have the following as `module-that-defines-fib`:
>
> #lang racket
> (provide fib)
> (define fib "fib")
>
>And this is the error that I got (using Racket 7.6):
>
> ; application: not a procedure;
> ; expected a procedure that can be applied to arguments
> ; given: "fib"
> ; [,bt for context]

I've run into this problem before ... I don't recall the official
explanation, but my takeaway was that Racket does not permit you to
directly *export* a value - you have to export a function or macro
that produces the value.

E.g.,
#lang racket
(provide fib)
(define (fib) "fib")

>I think this is because `(define-values (x) ...)` expands `...` without the
>top-level-bind-scope, even when expand-context-to-parsed? is #t (according
>to expander/expand/top.rkt). Is this a bug?
>Related to your answer to my second question, `define-syntaxes` similarly
>does not add the top-level-bind-scope when expanding `...`. Does this mean
>that even for `define-syntaxes`, `...` won't use the top-level-bind-scope
>binding(s) after all?
>
>A little bit off-topic, in the definition of define-values (in
>expander/expand/top.rkt), there is `(define-match m s ...)`, but for
>define-syntaxes it is `(define-match m disarmed-s ...)`. Is this difference
>significant? Or does define-match not care whether `s` or `disarmed-s` is
>used?

I don't know the internals so I can't evaluate your theory.


>Thanks,
>Yongming

George

[Hendrik Boom]

On Mon, Mar 23, 2020 at 02:46:53PM -0400, George Neuner wrote:
> On Mon, 23 Mar 2020 01:45:40 -0700 (PDT), Yongming Shen
> <sym...@gmail.com> wrote:
>
> >Hi Matthew,
> >
> >Thank you for the quick reply!
> >
> >I tried the example you gave for my first question and it resulted in an
> >error.
> >I have the following as `module-that-defines-fib`:
> >
> > #lang racket
> > (provide fib)
> > (define fib "fib")
> >
> >And this is the error that I got (using Racket 7.6):
> >
> > ; application: not a procedure;
> > ; expected a procedure that can be applied to arguments
> > ; given: "fib"
> > ; [,bt for context]
>
>
> I've run into this problem before ... I don't recall the official
> explanation, but my takeaway was that Racket does not permit you to
> directly *export* a value - you have to export a function or macro
> that produces the value.

That seems like an extremely arbitrary restriction.

>
> E.g.,
> #lang racket
> (provide fib)
> (define (fib) "fib")
>
>
> >I think this is because `(define-values (x) ...)` expands `...` without the
> >top-level-bind-scope, even when expand-context-to-parsed? is #t (according
> >to expander/expand/top.rkt). Is this a bug?
> >Related to your answer to my second question, `define-syntaxes` similarly
> >does not add the top-level-bind-scope when expanding `...`. Does this mean
> >that even for `define-syntaxes`, `...` won't use the top-level-bind-scope
> >binding(s) after all?
> >
> >A little bit off-topic, in the definition of define-values (in
> >expander/expand/top.rkt), there is `(define-match m s ...)`, but for
> >define-syntaxes it is `(define-match m disarmed-s ...)`. Is this difference
> >significant? Or does define-match not care whether `s` or `disarmed-s` is
> >used?
>
> I don't know the internals so I can't evaluate your theory.
>
>
> >Thanks,
> >Yongming
>
> George
>

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/2l0i7flr0to9msl6sg165vptiec0qq26l7%404ax.com.

[George Neuner]

On Mon, 23 Mar 2020 16:57:26 -0400, Hendrik Boom
<hen...@topoi.pooq.com> wrote:

>On Mon, Mar 23, 2020 at 02:46:53PM -0400, George Neuner wrote:
>> On Mon, 23 Mar 2020 01:45:40 -0700 (PDT), Yongming Shen
>> <sym...@gmail.com> wrote:
>>
>> >I have the following as `module-that-defines-fib`:
>> >
>> > #lang racket
>> > (provide fib)
>> > (define fib "fib")
>> >
>> >And this is the error that I got (using Racket 7.6):
>> >
>> > ; application: not a procedure;
>> > ; expected a procedure that can be applied to arguments
>> > ; given: "fib"
>> > ; [,bt for context]
>>
>>
>> I've run into this problem before ... I don't recall the official
>> explanation, but my takeaway was that Racket does not permit you to
>> directly *export* a value - you have to export a function or macro
>> that produces the value.
>
>That seems like an extremely arbitrary restriction.

Well, Racket considers exported objects to immutable. You can export
a simple value (e.g., "foo" or 42) but it can only be used as a
constant. If what you need is a module variable then the only way to
expose it is via an access function.

AFAIHS, the OP has not shown us the code which uses the provided
object, so we really don't know what it was trying to do when the
error occurred.

George

[Alexis King]

> On Mar 23, 2020, at 13:46, George Neuner <gneu...@comcast.net> wrote:
>
> I've run into this problem before ... I don't recall the official
> explanation, but my takeaway was that Racket does not permit you to
> directly *export* a value - you have to export a function or macro
> that produces the value.
>
> E.g.,
> #lang racket
> (provide fib)
> (define (fib) "fib")

I’m not sure what issue you may have run into in the past, but Racket certainly has no issue with exporting a value. (After all, functions are values.) The issue described in this thread is (a) entirely syntactic (it’s an issue of binding/scoping, not runtime behavior), and (b) only occurs at the top-level (i.e. in the REPL, outside of a module). Matthew has already provided a detailed explanation of why the issue occurs.

It’s true that you cannot set! imported bindings, but that’s quite separate from the problem discussed in this thread (which appears to be resolved).

[Hendrik Boom]

On Mon, Mar 23, 2020 at 05:55:14PM -0400, George Neuner wrote:
> On Mon, 23 Mar 2020 16:57:26 -0400, Hendrik Boom
> <hen...@topoi.pooq.com> wrote:
>
> >On Mon, Mar 23, 2020 at 02:46:53PM -0400, George Neuner wrote:
> >> On Mon, 23 Mar 2020 01:45:40 -0700 (PDT), Yongming Shen
> >> <sym...@gmail.com> wrote:
> >>
> >> >I have the following as `module-that-defines-fib`:
> >> >
> >> > #lang racket
> >> > (provide fib)
> >> > (define fib "fib")
> >> >
> >> >And this is the error that I got (using Racket 7.6):
> >> >
> >> > ; application: not a procedure;
> >> > ; expected a procedure that can be applied to arguments
> >> > ; given: "fib"
> >> > ; [,bt for context]
> >>
> >>
> >> I've run into this problem before ... I don't recall the official
> >> explanation, but my takeaway was that Racket does not permit you to
> >> directly *export* a value - you have to export a function or macro
> >> that produces the value.
> >
> >That seems like an extremely arbitrary restriction.
>
> Well, Racket considers exported objects to immutable. You can export
> a simple value (e.g., "foo" or 42) but it can only be used as a
> constant. If what you need is a module variable then the only way to
> expose it is via an access function.

That makes sense.

Or I suppose you could build a one-element mtable array and export that.
The you still can't change the variable, but can change teh value inside it.

-- hendrik

>
> AFAIHS, the OP has not shown us the code which uses the provided
> object, so we really don't know what it was trying to do when the
> error occurred.
>
> George
>

> --
> You received this message because you are subscribed to the Google Groups "Racket Users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/p9bi7fpigiqm3tc3togm4vmontlv5v70ti%404ax.com.

[Yongming Shen]

Oops, I'm afraid things are getting off topic. Just to clarify, my question is really as the title states, about top-level-bind-scope, which is used by the expander to achieve certain effects when expanding define-values/define-syntaxes forms at the top level. I'm mainly trying to get a better understanding of those effects, and whether top-level-bind-scope is necessary to implement them.

You received this message because you are subscribed to a topic in the Google Groups "Racket Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/racket-users/7Y-a1r9oRPI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/20200324001128.u5rp6pw677lbhleg%40topoi.pooq.com.

[Yongming Shen]

Hi Matthew,

Thanks for the explanations. But I'm still not convinced that the top-level-bind-scope is needed. This is my current understanding. The purpose of the  top-level-bind-scope is to support recursion better at the top level. But for the case of `(define-values (x) ...)`, if `x` is not defined yet, then implicit #%top in `...` will let `...` refer to `x`. If `x` is defined, then the old definition will be used by `...`. Either way, the top-level-bind-scope is not needed. For the case of `(define-syntaxes (x) ...)`. As you explained, a macro can naturally recursively refer to itself, simply because of how macro expansion works, so the top-level-bind-scope is again not needed for recursion. Is my understanding correct?

[Yongming Shen]

I have done some further digging. Looks like it is non-trivial to remove the top-level-bind-scope and still preserve the current behaviors of top-level `define-values` and `define-syntaxes`. In particular, through `as-expand-time-top-level-bindings` in "expander/expand/bind-top.rkt", both top-level `define-values` and `define-syntaxes` use `select-defined-syms-and-bind!/ctx` from "expander/expand/def-id.rkt", which selects variable-symbols and binds them at the same time. So after a variable symbol is selected, a binding must be created for it. For top-level `define-values` and `define-syntaxes`, using the top-level-bind-scope for that mandatory binding is necessary to achieve the current top-level behaviors. On the other hand, uses of the top-level-bind-scope that are related to `#%top` (such as in "expander/expand/expr.rkt" and "expander/expand/main.rkt") appear to be non-essential, but may offer minor performance advantages (really not sure).