macros generating macros in modules

[Abe Schneider]

I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro, I can write:

macro meta_macro(x)
  quote
    macro foo(y)
      $x+y
    end
  end
end

and then call it and the generated macro:

@meta_macro(5)
println(@foo(3))
# 8

Okay, wasn't entirely sure that was going to work, but it appears to. However, if I try to move the exact same macro code into a module, it no longer works:

module meta_macro_mod

export @meta_macro, @foo

macro meta_macro(x)
  quote
    macro foo(y)
      $x+y
    end
  end
end

end

Even without trying to invoke `@foo`, when I execute `@meta_macro`, I get:

ERROR: syntax: invalid macro definition


It seems like how the macro gets evaluated changes when it's in the module. If I change it to a module to just doing an 'include', everything works again.

[Patrick O'Leary]

On Saturday, April 4, 2015 at 9:04:10 AM UTC-5, Abe Schneider wrote:

I should start off, not entirely sure this is an okay thing to do with Julia. Suppose I want to create a macro that generates another macro...

I'm not sure whether this should work or not, but either way I'm not sure how it's any more expressive than macros alone? Can you describe what it is you want to do at a high level in a little more detail? There might be another way.

[Jameson Nash]

using functions is often much simpler to write and use:

function meta_foo(x)
    function foo(y)
      x+y
    end
end

foo2 = meta_foo(5)
println(foo2(3))

# 8

[Abe Schneider]

The issue I'm dealing with is that I have a macro that I want to pass a list of functions that the macro can employ for parsing grammars. The macro itself exists already, and it currently has its own static list of functions. I'm trying to figure out a way to allow the user of the library to customize it.

For example:

@grammar foo begin
  start = list(integer, r", ")
end

where `list` is a user supplied function. The issue that I keep running into is that there doesn't seem to be any easy way of just passing in a function to a macro. Turning an expression into a function is problematic, because `eval`s are: (a) only work in the modules namespace, and (b) are frowned on.

I've thought about manually looking up the symbol name in the various namespaces, but while I've found methods to list the functions, I haven't found a way to do the reverse.

Ideally, it would be nice to generate an expression with `call` that took a symbol and a namespace, that way no lookups would have to be done.

Having hit a dead end in this direction, it occurred to me that it might be possible to do something like this:

@set_parsers(list, etc)
@grammar foo begin
...
end

where '@set_parsers` would generate the `@grammar` macro with the static list it already has plus whatever else the user adds in.

Which works, except for the whole module issue.


Thanks,
A

[Jameson Nash]

I think the underlying issue may be that you are trying to have the macro do too much, when you should instead be just doing source-code transforms and preprocessing. One such example of a great use of a macro is simply:

macro special_foo(syntax_tree)

    return quote

        special_foo( $(QuoteNode(syntax_tree) )

    end

end

[Abe Schneider]

I don't understand how what I'm trying to do falls outside the domain of transforming code. My main goal is to take a user-defined grammar and create relevant code.

More to the point, I think the general ability to generate expressions based on functions/namespaces is also solidly in the domain of transforms. I can imagine plenty of use-cases for language-bindings, communication, etc.

Yes, there are many simple macros one can create, which are useful, but there exists many useful things one can do with macros that can get more complex.

A

[Patrick O'Leary]

Let me ask the question slightly differently. Without constraining yourself to legal Julia syntax, what would you like to go in, and what do you want to come out? I get the feeling there's a design here that doesn't have this complexity--my intuition is that a generic function can be used to drive this just fine (called within a macro to facilitate the final splice.) But that's still hard to concretely suggest.

[Abe Schneider]

I think if I just copy bits of the current code in it might be the easiest. I have the following macro:

macro grammar(name, definitions)
  parsers = [:+, :*, :?, :|, :-, :^, :list, :integer, :float]
  mapped_parsers = map_symbol_to_function(parsers)
  return parseGrammar(name, definitions, ParserData(mapped_parsers))
end

where parseGrammar goes through the definitions and converts them into actual code. Its output is an Expr tree. The key part here, is that per line, another function `parseDefinition` is recursively called. The simplified version of it is:

function parseDefinition(name::String, ex::Expr, pdata::ParserData)
  # ...

  # look up the symbol in the expression tree to get the function
  parser = get(pdata.parsers, ex.args[1], nothing)
  if parser !== nothing
    # call function on the rest of the expression
    rule = parser(name, pdata, ex.args[2:end])
  end
end

Thus, all the parsers listed above will be called appropriately if their symbol is encountered. This used to be a monolithic function, but it seemed a much better design to keep each parser as its own thing. The parsers themselves are not complex. Here's the definition of the or-rule:

function|(name::String, pdata::ParserData, args::Array)
  left = parseDefinition("$(name)_1", args[1], pdata)
  right = parseDefinition("$(name)_2", args[2], pdata)
  return OrRule(name, value)
end

The `map_symbol_to_function` thus traverses the original `parsers` list created and maps the symbol to this function.


So, all of this currently works (and is viewable to anyone who is interested to see the full versions on PEGParser's github).

The issue is that currently `parsers` is a fixed list. I'm trying to figure out how to let the users of the library add their own. For example, suppose and `integer` parser wasn't yet made. Someone could create their own:

function integer(name::String, pdata::ParserData, args::Array)
  return IntegerRule(name)
end

and then pass it into the macro somehow in so when `parseDefinition` is called, it will be used.

Originally, the macro was written to allow custom parsers to be passed in with the grammar definition:

@grammar foo [integer] begin
  start = list(integer, r",[ \t\n]*")
end

Thus, the `grammar` macro would just append whatever you listed to the `parsers` list. However, the mapping of symbols to functions will fail because of the namespace issue.

Maybe, as others have argued, that's okay, because I'm trying to do too much in a macro. That's fine, so the next logical conclusion is that I should just generate code that does the mapping.

Also fine. BUT, don't forget that this mapping is there to support the function of macro. Specifically, the `parsers` list is used in the `parseDefinition` function for transforming a grammar definition into code. Thus, that means I would need a macro that generates code for another macro, bringing us to the original question posed.

One possible way around this might be that the macro doesn't actually have to generate another macro (which I think is probably the more elegant solution), but rather you could generate variables which the second macro can use. For example:

@set_parsers(integer)
# creates a parsers = [...] list
@grammar foo begin
  # checks if `parsers` list exists, if not creates its own
end

However, this approach seems dangerous, as it introduces variables that the user is unaware of.

[Toivo Henningsson]

If users should supply additional parsers (which parse sub-ASTs, right?) then it's not really possible to hide all the metaprogramming machinery from them anyway. So why not let the user create the macro, and just supply functions that make writing the macro implementation as easy as possible?

[Abe Schneider]

That's definitely an option, though it seems sub-optimal to me if it's possible to automate it from a user's perspective. It may be that it just isn't possible, but if so, is it because it's something the language doesn't currently support or does it go against the philosophy of the language?

Specifically, several possible methods could provide the necessary mechanism:

(1) Ability to get a function based on a signature and a namespace. Yes, you can get the names of functions in a namespace, but I don't see a way to convert that name to an actual function without an eval.

(2) Ability to create an call-expression with a namespace indicate to use the function from that namespace. For example: Expr(:call, :sym, :foobar_namespace, args). This means no additional lookup is required on the user's part.

(3) Allow macros to generate macros. This is already partially supported, though it's not clear if this is intentional or not. c++ has shown that template programming can be both useful and powerful (e.g. offloading computation to compile time rather than runtime).


All of these methods are general and not specific to the problem.