Thanks

I would recommend the main papers on syntax-case and its workings as a  
good supplement to the guides, and you can also check out TSPL4 [1] as an  
additional reference to the syntax-case system, variations (minor, ish) of  
which are in all R6RS capable Schemes (Racket included).

        Aaron W. Hsu

[1] http://www.scheme.com/tspl4/

        Aaron W. Hsu

1. Learn syntax-rules.
2. Learn the syntax-case library.
3. Learn Rackets macro system.

  For syntax-rules: I suggest you to search for "syntax-rule
tutorial" and  pick any of the short  introductions you find
this  way,  just  to  get  you  started;  then  read  "JRM's
Syntax-rules Primer  for the Merely  Eccentric"[1], which is
long and should satisfy your needs.

  For   syntax-case:  I   have   found  neither   satisfying
documentation, nor tutorials,  for learning it without pain;
this  includes  my own  introduction  to  the R6RS  relevant
chapter[2].   I think  that  both the  chapter  in the  R6RS
document  and   the  chapter  in   "The  Scheme  Programming
Language"[3] are  for people who "already get  it", the same
goes for all  the papers I have seen  so far: people writing
those  have  long  forgotten  what  does it  mean  to  be  a
beginner.   Also, to learn  syntax-case one  has to  build a
mental  model of  how the  macro expander  works, and  it is
totally opaque at  first.  The only way is  to have patience
with yourself and try to read everything you stumble upon.

  I  have not  learned Racket  macros, so  I  cannot suggest
anything; when  I tried to read  their documentation without
knowing syntax-case first,  I understood almost nothing; now
that I know a bit  of syntax-case I can understand something
more of  what I  remember; so I  guess that,  after learning
syntax-case,  you  will  be  able  to  use  their  reference
material with some success.

HTH

One thing I probably should have made more clear is that I'm at the
point where I can pretty easily write things with syntax-rules but
there are things I want to do which I think it is preventing me from
doing (as it should) so I think I need to understand syntax-case &
friends more thoroughly.

The "things I want to do" are essentially introduce bindings which the
user of the macro can see (which I think is the same thing as "not be
hygenic"), so I want something like this to "work":

(define-syntax bodging
  (syntax-rules ()
    ((bodging form ...)
     (let ((x 1)) form ...))))

(bodging x)

I do understand why syntax-rules is making it not "work", and I'm glad
it does so in general.

I'm aware that there are things out there which essentially look like
the CL macro system, but I don't want to use them as I already
understand that (though, if they can be implemented in terms of
syntax-case &c it might well be interesting to look at their
implementation to see how they do it).

In addition to already understanding them, they would have the advantage
of letting you write code more easily transposable to Common Lisp.

(define-syntax bodging
  (syntax-rules ()
    ((bodging var form ...)
     (let ((var 1)) form ...))))

(bodging x x)

by expanding to:

(let ((x 1)) x)
;;     ^     ^
;;     |     |
;;     |      -- reference position
;;      -- binding position

To be able to write:

(bodging x)

and  achieve the  same result,  we  want to  generate the  X
identifier in the  binding position as if it  was present in
the input form; here:

(define-syntax bodging
  (lambda (macro-use-syntax-object-holding-input-form)
    (syntax-case macro-use-syntax-object-holding-input-form ()
      ((bodging form ...)
       (let ((the-dirty-identifier (datum->syntax (syntax bodging) 'x)))
         (quasisyntax
          (let (((unsyntax the-dirty-identifier) 1))
            form ...)))))))

or more briefly but completely equivalent:

(define-syntax bodging
  (lambda (stx)
    (syntax-case stx ()
      ((bodging form ...)
       #`(let ((#,(datum->syntax #'bodging 'x) 1))
           form ...)))))

notice how  QUASISYNTAX, UNSYNTAX and SYNTAX  are similar to
QUASIQUOTE, UNQUOTE and QUOTE and the abbreviations:

#` #, #'

are similar to the abbreviations:

` , '

        Aaron W. Hsu

(define-syntax (bodging x)
   (syntax-case x ()
     [(k form ...)
      (with-implicit (k x)
        #'(let ([x 1]) form ...))]))

(define-syntax (bodging x)
   (syntax-case x ()
     [(k form ...)
      (with-syntax ([x (datum->syntax #'k 'x)])
        #'(let ([x 1]) form ...))]))

I am not sure how many of these forms and extensions Racket has, but they  
are trivial to provide and writing those might be an interesting exercise.  
The with-implicit form [1] makes it cleaner to do these sorts of things,  
and the with-syntax form allows a cleaner final syntax, so that you do not  
need to use quasisyntax and quasiunquote.

Just to give you some variety to practice your macros on.

If you really want to play around with how hygiene works, there is an  
interesting example in CSUG8 that shows how one might implement a module  
system with interfaces and implementation separately by writing macros  
around a combined module form `module'. [2] It's a good exercise, and you  
can write a naive `module' form for Racket as well to try these things  
out. There are some gotchas to watch out for as well. [3]

        Aaron W. Hsu

[1] http://www.scheme.com/csug8/syntax.html#./syntax:s11
[2] http://www.scheme.com/csug8/syntax.html#./syntax:s19
[3] http://my.opera.com/arcfide/blog/2010/08/25/syntax-brain-twister

[Because of this the following explanation is rough, but it should be
fine to understand how to use `syntax-case'.  No need to flame with
the 2000 mistakes I'm liable to make.]

The main idea with Racket's macro system (and probably with other
`syntax-case' systems) is that macros are syntax to syntax functions,
just like `defmacro', except that instead of raw s-expressions you're
dealing with syntax object.  This becomes very noticeable when
identifiers are handled: instead of dealing with symbols, you're
dealing with these syntax values (called "identifiers" in this case)
that are essentially a symbol and some opaque information that
represents the lexical scope for its source.  In several `syntax-case'
systems this is the only difference from `defmacro' macros, but in the
Racket case this applies to everything -- identifiers, numbers, other
immediate constants, function applications, etc -- they are all the
usual sexpr values that you're used to, except wrapped with additional
information.  Another thing that is unique to Racket is this extra
information: in addition to the opaque lexical context, there is also
source information and arbitrary properties (there are also
certificates, but that's ignorable for this level.).

With this in mind, explaining the rest is not too difficult:

* (syntax-source stx), (syntax-position stx), (syntax-line stx),
  (syntax-column stx) -- retrieve parts of the source information.

* (syntax-e stx) -- takes a syntax value and returns the value it
  "wraps".  For example, if `stx' is an identifier you'd get a symbol,
  and if it's a number you'd get the number etc.  If it's a
  parenthesized form, you'd get a list of syntax values for the
  subforms.  Note that the list can be improper, with the last
  element being a syntax object that contains a proper list.

* (syntax->list stx) -- since `syntax-e' might return an improper list
  for a list syntax, this utility makes things easier and will use
  `syntax-e' again in these cases and will return a proper list.  If
  the syntax is itself not a proper list, it returns #f.

* (syntax->datum stx) -- takes a syntax value and returns the plain
  s-expression that it holds.  This is done by recursive uses of
  `syntax-e'.  (It would be a simple definition.)

* (syntax-property stx prop) -- returns the given property value from
  stx, if any, and #f if none.  For example, try
    (syntax-property #'[foo] 'paren-shape)

* There is no accessor for the lexical scope (and you don't need one).

* To create a piece of syntax you use `datum->syntax', and you give it
  an s-expression which will be the "contents" of the resulting syntax
  object.  (The input can contain syntax values, which are left as
  is.)  But when you do that you need to give it the other bits --
  including the lexical context thing, which you have no access to.
  The way that's done is:

    (datum->syntax context-stx input-sexpr)

  This returns a syntax value that wraps the `input-sexpr' value,
  using the lexical scope from `context-stx'.  There is actually
  another optional argument that specifies the source (either using
  another syntax object, or as an explicit list), and another for
  copying the properties from, etc.  So a common thing to do when you
  want to create an unhygienic user-visible binding is:

    (datum->syntax something 'foo something something)

  where `something' is some syntax value that you get from the user
  input to the macro.  It makes a `foo' identifier that has the same
  lexical context information, same source, and same properties.

* There is also (quote-syntax blah) which creates a quoted syntax,
  with its lexical source from the place it appears.

* Finally, `define-syntax' does the magic of tying a name with a
  transformer function.

And that's almost everything that you need in order to write hygienic
(and non-hygienic) macros.  Very inconveniently.

For example, here's a simple `while' macro:

   (define-syntax (while stx)
     (define subs (syntax->list stx))
     (datum->syntax
      stx
      `(let loop ()
         (when ,(cadr subs)
           ,@(cddr subs)
           (loop)))
      stx))

which breaks like this:

  (define x 2)
  (let ([let 5])
    (while (< x 10) (printf "x = ~s\n" x) (set! x (add1 x))))

The problem is that all of those quoted names are getting the context
of the user input, which is not the right thing (it's close to a
defmacro).  To fix this, you need to `quote-syntax' all of these
identifiers, so they'll have the macro source instead of the input
source (to try this, you'll need (require (for-syntax racket/base)),
but that's a different story):

   (define-syntax (while stx)
     (define subs (syntax->list stx))
     (datum->syntax
      stx
      `(,(quote-syntax let) ,(quote-syntax loop) ()
         (,(quote-syntax when) ,(cadr subs)
           ,@(cddr subs)
           (,(quote-syntax loop))))
      stx))

But that's clearly insane...  More than being tedious, it's still
incorrect since all of those parens will have the user's lexical
context (which means that it will use the user's notion of function
application via the special `#%app' macro).  Instead of doing this, a
better approach would be to create the resulting syntax with the
lexical context of the macro source by changing just that argument:

   (define-syntax (while stx)
     (define subs (syntax->list stx))
     (datum->syntax
      (quote-syntax here)
      `(let loop ()
         (when ,(cadr subs)
           ,@(cddr subs)
           (loop)))
      stx))

And that's simple again, and works fine.

The problem is that it's tedious wrt to deconstructing the input
(which is trivial in this case), and wrt slapping together an output
value -- and that's where `syntax-case' comes in.  It addresses the
both by using pattern matching, where identifiers in patterns are
bound as "syntax patterns".  A new form is added -- `syntax' -- which
is similar to a `quote', except that (a) it actually quotes things
similarly to `quote-syntax', with the lexical context of the `syntax'
form; and (b) pattern variables are substituted with what they
matched.  The above macro becomes much easier:

   (define-syntax (while stx)
     (syntax-case stx ()
       [(_ test body ...)
        (syntax (let loop () (when test body ... (loop))))]))

The first line specifies that you want to match the `stx' input
syntax, and that you have no "keywords" (in the same sense as in
`syntax-rules').  The second line is the pattern that is matched
against this input -- with two pattern variables that match the second
subexpression and the sequence of expressions from the third and on.
(The first subexpression is matched against `_' which matches anything
and doesn't bind a pattern variable -- it's often not needed, since
it's just the macro name.)  The last line is the output, specified
using `syntax', which means that it's very similar to the previous
version where everything is given the lexical context of the macro and
the two pattern variables are replaced with the two matches (which
means a splicing subtitution for `body').

Now, say that you want an unhygienic user-visible piece of syntax.
For example, bind the always entertaining `it' thing to the test
result.  This:

   (define-syntax (while stx)
     (syntax-case stx ()
       [(_ test body ...)
        (syntax (let loop ()
                  (let ([it test])
                    (when it body ... (loop)))))]))

won't work because `it' has the macro source -- it's hygienic and
therefore not visible.  Instead, you need to use `datum->syntax' with
the user syntax:

   (define-syntax (while stx)
     (syntax-case stx ()
       [(_ test body ...)
        (let ([it (datum->syntax stx 'it)])
          (syntax (let loop ()
                    (let ([it test])
                      (when it body ... (loop))))))]))

But this doesn't really work since `it' needs to be bound as a pattern
variable rather than a plain binding.  `syntax-case' can be used here
again: (syntax-case <something> () [foo <body>]) will match `foo'
against the <something> syntax, and it will be bound as a pattern
variable in the <body>.  So:

   (define-syntax (while stx)
     (syntax-case stx ()
       [(_ test body ...)
        (syntax-case (datum->syntax stx 'it) ()
          [user-it
           (syntax (let loop ()
                     (let ([user-it test])
                       (when user-it body ... (loop)))))])]))

  http://docs.racket-lang.org/guide/macros.html

  http://blog.racket-lang.org/2008/02/dirty-looking-hygiene.html

Most of the problem, really, is that I know the CL macro system - which
is the FORTRAN IV of macro systems - so well that it's impeding my
understanding of how Scheme's work I think.

(collecting
  ...
  (collect x)
  ...
  (foo (lambda (x) (collect x)))
  ...)

where COLLECT is a local macro (in the current CL implementation: it
should actually be a local function but I didn't understand that when I
wrote a very long time ago, hence the annoying wrapping-it-in-lambda
thing above) which will collect its argument into a list, with the
result of the whole form being the list.

In CL, COLLECT obviously needs to exist as a symbol even outside of the
macro: in the current implementation it does not have a global
definition, but it actually should have one which simply signals an
error.  So that's pretty like your example.

Once again thanks to everyone who's answered here: I didn't expect to
get such a lot of good answers.

--tim

(define (call/collecting thunk)
  ;; mindless reverse-the-list version
  (let ((s '()))
    (thunk (lambda (e)
             (set! s (cons e s))))
    (reverse s)))

(define-syntax (collecting stx)
  (syntax-case stx ()
    ((collecting)
     #'(quote ()))
    ((collecting form ...)
     #`(call/collecting (lambda (#,(datum->syntax stx 'collect))
                          form ...)))))

And now, for instance, (collecting (collect 1) (collect 2)) -> (1 2).

  People  have difficulties  in  learning Scheme's  hygienic
macros, it is a fact for me.  Even if, at some point in your
life,   you   are  comfortable   in   using  an   "ordinary"
preprocessor (like  GNU m4), IMHO  there is very  little you
can   reuse  in   your  brain   from  past   experiences  in
non-esoteric  languages:  Scheme   hygienic  macros  are  on
another level.

See http://dx.doi.org/10.3217/jucs-016-02-0271

Pascal

That's what's so great with Common Lisp macros:
they're written in Common Lisp.