Using the top level to fix unbound identifier

[Jonathan Simpson]

I have a bug I'm trying to fix in #lang magic(https://github.com/jjsimpso/magic). I'm pretty sure I know what the cause is and I know what I want to do, but I don't know how to do it. The situation is this:

In the following magic code, a new function(called a named query in magic) tga-image is defined and later used inside the definition of the function test-scope. The issue is that tga-image is an unbound identifier in test-scope. If tga-image is used outside of a function there isn't a problem because the syntax object passed to use has the binding. What I would like to do is define tga-image at the top level. When test-scope is being expanded it doesn't find the binding for tga-image and tags it with #%top, so I think putting tga-image at the top level will solve my problem. I also think that this is not a bad solution for my use case.

#lang magic


# display tga bitmap image information
0 name  tga-image
>2 byte <34        Targa image data


0 name  test-scope
>2 byte <34        Test Scope
>0 use  tga-image

After parsing, this code becomes:

(module magic-mod magic/expander
  (magic
   #f
   #f
   (named-query
    (name-line (offset 0) (name-type "name") "tga-image")
    (level)
    (line (offset 2) (type (numeric "byte")) (test (numtest "<" 34)) (message "Targa image data")))
   (named-query
    (name-line (offset 0) (name-type "name") "test-scope")
    (level)
    (line (offset 2) (type (numeric "byte")) (test (numtest "<" 34)) (message "Test Scope"))
    (level)
    (line (offset 0) (type (use "use")) (test (use-name "tga-image"))))))

This is the most relevant code in my expander (I highlighted creation of name's binding in red):

(define-syntax (named-query stx)
  (syntax-case stx (name-line)
    [(_ (name-line (_ 0) (_ "name") magic-name))
     (with-syntax ([name (datum->syntax stx (string->symbol (syntax->datum #'magic-name)))])
       #'(define name
           (lambda (new-offset) (void))))]
    [(_ (name-line (_ 0) (_ "name") magic-name) . rst)
     (with-syntax ([name (datum->syntax stx (string->symbol (syntax->datum #'magic-name)))]
                   [modified-rst (cons (datum->syntax #'rst always-true-line) #'rst)])
       #'(define name
           (lambda (new-offset)
             (syntax-parameterize ([name-offset (make-rename-transformer #'new-offset)])
               ;(printf "name: offset = ~a~n" name-offset)
               (query . modified-rst)))))]))

(define-syntax (magic-module-begin stx)
  (define (query? expr)
    (if (and (pair? expr)
             (equal? (car expr) 'query))
        #t
        #f))
  (define (named-query? expr)
    (if (and (pair? expr)
             (equal? (car expr) 'named-query))
        #t
        #f))
  (define (wrap-with-delimiter-print expr)
    (list 'when* expr '(printf "*** ")))

  (let ([exprs (cdadr (syntax->datum stx))])
    ;(display queries)
    (let ([queries (filter query? exprs)]
          [named-queries (filter named-query? exprs)])
      #`(#%module-begin
         #,@named-queries
         (define (magic-query)
           (or #,@queries))
         (define (magic-query-run-all)
           ; any-true? creates a binding for last-level-offset which we probably don't want here. investigate.
           (any-true? #,@(map wrap-with-delimiter-print queries)))
         (provide magic-query magic-query-run-all)))))

The full code is at https://github.com/jjsimpso/magic/blob/master/expander.rkt.

If there isn't an easy way to pass the top level syntax object into named-query, then I may have to do something with syntax parameters. I think that would be more complicated since I'm potentially defining lots of functions.

Any help is appreciated! I thought this would be an easy one, but I've spent a while on it already. Hopefully there is a simple solution.

-- Jonathan

[Tom Gillespie]

Without have read this carefully, are you maybe looking for https://docs.racket-lang.org/reference/syntax-util.html#%28def._%28%28lib._racket%2Fsyntax..rkt%29._format-id%29%29? If you just pass string->symbol and then use datum->syntax you will get binding errors.

Tom

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/ee664882-1dec-4bb8-b66b-bd6ee80316dc%40googlegroups.com.

[Jonathan Simpson]

I changed the lines in named-query from

[name (datum->syntax stx (string->symbol (syntax->datum #'magic-name)))

to

(format-id stx "~a" (syntax-e #'magic-name))

and I still get the same error. I also switched to using format-id where I use the name, with no change either. Am I using this correctly?

The format-id change is easier to read, so it is worth keeping even if it doesn't solve this issue. 

-- Jonathan

To unsubscribe from this group and stop receiving emails from it, send an email to racket...@googlegroups.com.

[Jonathan Simpson]

On Tuesday, August 13, 2019 at 10:41:33 PM UTC-4, Jonathan Simpson wrote:

I changed the lines in named-query from

[name (datum->syntax stx (string->symbol (syntax->datum #'magic-name)))

to

(format-id stx "~a" (syntax-e #'magic-name))

Of course this should be

[Tom Gillespie]

It is the same error, but I think it is occurring at a different place. Running raco macro-stepper on name-test.rkt  the original error was happening in define-values (because format-id was not used). After making the format-id changes it occurs during the expansion of use-name when expanding the line macro. Therefore I don't think that putting it in #%top will fix your problem because it is now a hygiene error. I'm not entirely sure why it complains but it seems like it is related to 2.4 Internal definitions from Macros that work together (https://www.cs.utah.edu/plt/publications/jfp12-draft-fcdf.pdf), local-expand and syntax-local-make-definition-context seem like they might be what is needed? Best,

Tom

PS I've encountered similar issues a number of times and haven't taken the time to figure out the right way to deal with it, so your issue is a nice clear cut example.

To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/94501425-0fae-4cea-86b0-1aaa1ac574cc%40googlegroups.com.

[Jonathan Simpson]

The error during the expansion of use-name in the line macro is the same as what I was seeing before I made the format-id change. I'm running Racket 6.11 so perhaps that explains why I wasn't seeing the problem you were without format-id. I really have no idea, but at least it does look like we are seeing the same error now.

I'll take a look at the documentation you linked. I thought that I was out of the woods as far as hygiene went because it works in the case where the named-query tga-image is used by a regular query(as opposed to another named-query/function). But I''ll be the first to admit I could be wrong here.

Just for reference, here is the last bit of the macro expansion the step before the error:

                     (#%app:53

                      magic-test:54

                      (#%app:55 +:56 new-offset '2)

                      (#%app:57 type:54 (quote:54 (numeric "byte")) (quote:54 (numtest "<" 34)))

                      (#%app:58 compare:54 (quote:54 (numtest "<" 34)) (quote:54 (numeric "byte")))

                      '"Test Scope")

                     (#%app:59 (#%top . tga-image) 0)

                     #t))))

             #f)))))

   (define-values:61 (magic-query) (lambda:62 () (or)))

   (define-values:63 (magic-query-run-all) (lambda:64 () (any-true?)))

   (#%provide:65 (expand:65 (provide-trampoline:65 magic-query magic-query-run-all)))))

tga-image: unbound identifier in module

tga-image

-- Jonathan

To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/94501425-0fae-4cea-86b0-1aaa1ac574cc%40googlegroups.com.

[Matthew Butterick]

On 13 Aug 19, at 7:03 PM, Jonathan Simpson <jjsi...@gmail.com> wrote:

In the following magic code, a new function(called a named query in magic) tga-image is defined and later used inside the definition of the function test-scope. The issue is that tga-image is an unbound identifier in test-scope. If tga-image is used outside of a function there isn't a problem because the syntax object passed to use has the binding. What I would like to do is define tga-image at the top level.

You might consider the technique described here, including the `find-unique-var-ids` function:

https://beautifulracket.com/basic-2/variables-and-input.html

Since you're using `brag`, you can wrap your future identifiers in, say, an `id` rule that you splice away in the grammar. So your parse tree ends up the same, but now that `id` rule leaves a residue as a syntax property. 

Then, as part of your `#%module-begin` prep work, you can use this property to pull out the identifiers and write them into the top level of your new module as `define` expressions (with dummy values, that would get replaced later with `set!`). 

There may be a way of accomplishing the same thing more elegantly with syntax lifts and captures. Like the lost city of El Dorado, I searched for this beauty, but expired on the jungle floor before discovering it. Since then, however, many people smarter than me have seen that page, so by Cunningham's Law, if there were a better method, I probably would've been told by now. 

[Jonathan Simpson]

I will take a look at this as well. It may be a few days before I have time to devote to this again, but I appreciate everyone's help so far.

-- Jonathan

[Jonathan Simpson]

I still need to work through the Beautiful Racket link Matthew sent, so perhaps that will address my problem, but in the meantime I think I've increased my understanding of the issue.

I realized that if I parsed the to-be-defined function names as symbols instead of strings I wouldn't need to generate new syntax objects for them in the macros. So I did that and am now using the parsed symbols from the source code directly and no longer need to create the identifiers with a with-syntax and format-id combo. That simplifies my macros a little but I still have the same unbound identifier problem.

I also realized that I have the same unbound identifier issue when I try to define a recursive function(or named-query in my language) that calls itself. Interestingly, the macro stepper tool understands that the function identifier and the recursive call to it are the same identifier until part way through the macroexpansion. See attached screenshots.

Surely defining a recursive function with a macro is a fairly common occurrence. It is the expansion of the query macro within the function that appears to be the problem. The syntax object passed to query has its own lexical environment that doesn't include the containing function. This is the point of hygiene I guess. I'm not exactly sure how to use it, but would local-expand be a method to expand query in it's surrounding lexical environment?

-- Jon

[Sorawee Porncharoenwase]

Surely defining a recursive function with a macro is a fairly common occurrence. It is the expansion of the query macro within the function that appears to be the problem. The syntax object passed to query has its own lexical environment that doesn't include the containing function. This is the point of hygiene I guess. I'm not exactly sure how to use it, but would local-expand be a method to expand query in it's surrounding lexical environment?

It is, and normally it’s not a problem because people generally write macros hygienically. However, in your case, several macros are written in defmacro-style (syntax->datum the whole syntax object to S-expression, and then manipulate the S-expression instead). That is usually the cause of problems.

Here’s an example of how to write module-begin hygienically without using syntax->datum:

......

;; my own macro implementations to demonstrate that it works

(define-syntax (named-query stx)
  (syntax-parse stx
    ;; make definition
    [(_ ({~datum name} name) e) #'(define (name) e)]))

(define-syntax (query stx)
  (syntax-parse stx
    [(_ e) #'e]))

(define-syntax (line stx)
  (syntax-parse stx
    [(_ e) #'e]))

(define-syntax (use-name stx)
  (syntax-parse stx
    ;; add application
    [(_ e) #'(e)]))

;; end my own macro implementations

(define-syntax (wrap-with-delimiter-print stx)
  (syntax-parse stx
    [(_ expr) #'(when* expr (printf "*** "))]))

(define-syntax (magic-module-begin stx)
  (define (query? expr)
    (syntax-parse expr
      [({~literal query} . _) #t]
      [_ #f]))

  (define (named-query? expr)
    (syntax-parse expr
      [({~literal named-query} . _) #t]
      [_ #f]))

  (syntax-parse stx
    [(_ ({~datum magic} exprs ...))
     #:with (queries ...) (filter query? (attribute exprs))
     #:with (named-queries ...) (filter named-query? (attribute exprs))
     #'(#%module-begin
        named-queries ...
        (define (magic-query) (or queries ...))
        (define (magic-query-run-all)
          ; any-true? creates a binding for last-level-offset which we probably don't want here. investigate.
          (any-true? (wrap-with-delimiter-print queries) ...))
        (provide magic-query magic-query-run-all))]))

(provide
 (except-out (all-from-out racket/base) #%module-begin) 
 (rename-out [magic-module-begin #%module-begin])
 use-name query named-query line)

Then this code works correctly:

#lang racket

(module magic-mod magic/expander
  (magic
   (named-query (name tga-image) (line 1))
   (named-query (name test-scope) (line (use-name tga-image)))
   (query (use-name test-scope))))

For your code, the next thing I would suggest looking into is to avoid using syntax->datum in the query macro (and also the whole S-expression mangling in expander-utils) and use syntax-parse on the syntax object instead. As a bonus, syntax-parse would let you avoid things like cadddadr.

 

-- Jon

On Wednesday, August 14, 2019 at 10:49:35 PM UTC-4, Jonathan Simpson wrote:

I will take a look at this as well. It may be a few days before I have time to devote to this again, but I appreciate everyone's help so far.

-- Jonathan

On Wednesday, August 14, 2019 at 1:00:54 AM UTC-4, Matthew Butterick wrote:

On 13 Aug 19, at 7:03 PM, Jonathan Simpson <jjsi...@gmail.com> wrote:

In the following magic code, a new function(called a named query in magic) tga-image is defined and later used inside the definition of the function test-scope. The issue is that tga-image is an unbound identifier in test-scope. If tga-image is used outside of a function there isn't a problem because the syntax object passed to use has the binding. What I would like to do is define tga-image at the top level.

You might consider the technique described here, including the `find-unique-var-ids` function:

https://beautifulracket.com/basic-2/variables-and-input.html

Since you're using `brag`, you can wrap your future identifiers in, say, an `id` rule that you splice away in the grammar. So your parse tree ends up the same, but now that `id` rule leaves a residue as a syntax property. 

Then, as part of your `#%module-begin` prep work, you can use this property to pull out the identifiers and write them into the top level of your new module as `define` expressions (with dummy values, that would get replaced later with `set!`). 

There may be a way of accomplishing the same thing more elegantly with syntax lifts and captures. Like the lost city of El Dorado, I searched for this beauty, but expired on the jungle floor before discovering it. Since then, however, many people smarter than me have seen that page, so by Cunningham's Law, if there were a better method, I probably would've been told by now. 

--
You received this message because you are subscribed to the Google Groups "Racket Users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to racket-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/racket-users/cef06a2b-0d07-462d-806f-1d9804fca973%40googlegroups.com.

[Jonathan Simpson]

Thanks very much for this! I hadn't considered the effect that stripping the syntax information from the input to query would have. It has always been a goal of mine to go back and rewrite all the macros -- and especially the ugly stuff you mentioned in expander-utils.rkt -- with syntax-parse. Your example should help me with that.

The parse-levels and transform-levels functions in expander-utils.rkt might be difficult to write in syntax-parse, although I'd love to do it that way if possible. Would replacing my s-expr manipulation to operate on syntax objects(via syntax->list, stx-car, and so on) achieve the same result? Regardless, I will look into syntax-parse because I don't particularly like what I have now and adding more complication to it is not something I'd relish doing.

-- Jon

To unsubscribe from this group and stop receiving emails from it, send an email to racket...@googlegroups.com.

[Jonathan Simpson]

I just wanted to follow up to say that I finally got this working! I finally re-implemented all of my non-hygienic macros with syntax-parse, which resolved the unbound identifiers.

Once again, thanks to everyone who offered help and suggestions.

-- Jon