Simplifying macros...

[Mike Allen]

Hi All!

BTW, before getting started, I just want to say thanks to everyone involved in Scala for producing such an incredible programming language!  Scala has transformed my coding, and my enjoyment of coding.

I'm very interested in the new (experimental) macro features, which remind me very much of the approach taken by a specialist, proprietary programming language called SLX (Simulation Language with eXtensibility), in which the "extensibility" bit comes in the form of its ability to define macros and even new statements.  (In turn, SLX's extensibility features are based upon a language whose name I cannot recall that was developed at the University of Michigan.)

However, I personally find SLX macros/statements a lot simpler to use than Scala's equivalent (as I currently understand them, at least - please feel free to correct me if my understanding of Scala macros is wrong).  Since SLX is such an obscure language, I just thought I'd share this with you...

Incidentally, SLX is a C-like language for Windows that has recently added some object-oriented features (a variant named SLX 2).  It is developed by a company called Wolverine Software.  There is a student version available for free download (use subject to a license agreement), which contains a more thorough document describing this feature (typically in "C:\Program Files\Wolverine\SLX\Doc\Chapter5.pdf" post installation).

Precursor Packages

In Scala, it is currently necessary to compile macro code separately, before compiling any code that uses those macros.  This means that, in eclipse, sbt, maven, etc., you need to have two projects instead of one when macros are introduced.  In SLX, the compiler can compile and load macro/statement definitions on a single compiler run.

(Incidentally, there is clearly a security risk in doing this, since someone could execute malicious code just by getting you to compile their sources, but I think that's possibly true of how Scala works currently too, right?  Maybe macros could be signed, or the compiler could ask for confirmation before compiling, etc.)

Instead of packages, SLX has simple flat "modules", the contents of which are placed into global scope.  If a module is flagged as a "precursor module" using the precursor statement, then the SLX compiler knows that it contains statement/macro definitions that need to be compiled and loaded into itself in order to process the remainder of the source.  One restriction in SLX is that it cannot expand a macro/statement definition that it has not already encountered, so the order in which modules are defined and compiled matters - I guess that may be a sticking point for Scala too.

Code defined in a precursor module is also present in the resulting executable, so that code that is required for macro/statement definitions can be shared with the code that uses those macro/statements.

For example:

// Precursor module: compiled and loaded when encountered by compiler, also written out to executable.

precursor module CompileAndLoadStuff {

  // definition of macros/statements

}

// Regular module: just processed and written out to executable.

module RegularStuff {

  // code that uses macros/statements defined in CompileAndLoadStuff

}

Macro/Statement Definitions

Inside these precursor modules, SLX allows macros and statements to be defined.  I don't want to go into too much detail about the syntax of this, which is unimportant, but here's how a basic assert statement might be defined in SLX:

precursor module Assert {

  // Define statement of the form "assert (condition);".

  statement assert (#condition);

  definition {

    if (!debugDisabled) {

      expand (#condition) "if (!(#)) {\n";

      expand (#condition) "  SLXDebugger (\"Assertion failed: #\");";

      expand "}\n";

    }

  }

}

module DoStuff {

  // ...

  assert (PI == 3.142);

}

What's happening here is that the statement expression defines a statement and a pattern that describes its syntax.  (A macro is similar but does not have a trailing semi-colon - in effect, a Scala macro is equivalent to an SLX macro or statement depending upon how it's used.)  The definition that follows, which is part of the statement expression, is written in regular SLX code.  The definition code writes replacement code to be used when the statement is encountered, using the expand statement.  This is compiled and loaded by the compiler.

When the assert (PI == 3.142) statement is encountered, the compiler pattern matches the assertion with the definition of the assert statement, then executes the associated definition code, which then effectively re-writes the statement to be one of the following:

If debugDisabled is true:

module DoStuff {

  // ...

}

If debugDisabled is false:

module DoStuff {

  // ...

  if (!(PI == 3.142)) {

    SLXDebugger ("Assertion failed: PI == 3.142");

  }

}

The re-rewritten code is then compiled and written to the output.

Indeed, it's worth noting that precursor modules can themselves make use of statements and macros defined in other precursor modules (the whole of the precursor module must be compiled and loaded before SLX can expand any contained statements or macros).

Summary

Both of the techniques above seem to me to be a lot simpler than the approach Scala currently adopts (although, granted, Scala includes type information and is far more sophisticated overall) from the perspective of someone writing a macro definition.

Do you think either of the above could be adopted by Scala?  Would they be useful?  Feasible?  Just curious...

Best Regards,

Mike

[Rogach]

It seems that there is hope to simplify this situation: https://gist.github.com/xeno-by/5058410 (and relevant commit: https://github.com/scalamacros/kepler/commit/2a92d61610b9756323e93460385b2c31d9048b2e).

[Eugene Burmako]

Hi Mike,

Thanks a lot for a detailed formulation of your thoughts!

Before we begin, I would like to mention that it looks like the whole "macros are hard to write" thing gets quite a bit of traction recently (e.g. take a look here: http://stackoverflow.com/questions/15401982/pass-closure-to-scala-compiler-plugin). Therefore I want to explicitly state that macro UX can and will be improved - we already have a number of improvements in macro paradise and some more are on their way.

Now back to the topic. If you don't mind, I'll ask some questions directly to you instead of downloading the student version of SLX and digging into it (which of course would be better, but then I'd have to come back with an answer in a few days, not today; therefore I would very much appreciate if you could help me out here).

1) Precursor packages. Can they depend on non-precursor packages? Can they depend on each other? If they do, how does the compiler figure out in what order it has to compile them?

2) Indeed, macros are, to a degree, a security risk. Even though we haven't evaluated this in detail, this risk looks similar to the risk of linking to a library and then running that library when debugging, right?

3) From what I can see, macros in SLX work expand into strings, i.e. they are very much like C macros. How do you take care of escaping in such a system? Is there a way to make it hygienic (i.e. prevent name collisions between generated identifiers and user-written identifiers)?

4) If I understand correctly, you find the string-based approach of SLX simpler than what's currently in Scala (expansion into manually written ASTs), right? Would quasiquotes change your opinion: http://docs.scala-lang.org/overviews/macros/quasiquotes.html?

Cheers,
Eugene

--
You received this message because you are subscribed to the Google Groups "scala-language" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-languag...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Lex Spoon]

It's nice to see an explicit discussion about making macros nicer!
Macros are a killer feature in Scala these days; I expect they will
become a must-have feature in languages once more designers see how
well they work in Scala.

The claim about security is misleading. You have the exact same
problem if you use SBT, Maven, Ant, or Make: the build tool can build
a component that it then executes as part of the build.

The argument about string templating versus AST rewriting is probably
unresolvable. People who like string templating like that it's fully
general and that it is easy to use for simple tasks. However, it has
problems with escaping, that make it not so easy for a serious effort.
It has problems with static checking, which goes against the overall
typed-language style of Scala. And it has big problems with the IDE.
While I don't have a detailed defense of AST-based macros handy, I
confess they feel much better to me--assuming, to be sure, that they
can be made expressive enough. I can't speak to that personally, since
I haven't had an excuse to use them seriously.

On compilation staging, it's certainly nice if there is an easy way to
improve things. However, it seems to require either that programmers
insert staging directives such as "precursors" within the code, or
that the compiler do automatic binding time analysis. Both of those
solutions seem worse to me than the current solution of having the
programmer sequence things via the build tool. To make this easier,
perhaps we could come up with a convention for source code
directories? Instead of just "src/main", make it "src/main.1",
"src/main.2", etc.?

Lex

[Mike Allen]

Hi Eugene,

Thanks for your quick reply!  It's good to know that you're working on making this process more straightforward.

Here's the answers to your questions, to the best of my knowledge.

Answer 1:

No, a precursor package cannot depend upon a non-precursor package, but it can depend upon other precursor packages.  The restriction is necessary because non-precursor packages are not loaded into the compiler after compilation, and, therefore, are unavailable to the compiler and to the contents of a statement/macro definition.

As for the compilation order, SLX makes no decisions about the order of compilation - it's pretty much under the direct control of the programmer. A single file is specified to the compiler, and that file includes others (which may, in turn, include yet other files) using an "import" statement. Import works like a "#pragma (header)", or like a guarded #include, in C and simply replaces the import statement with the contents of the specified file (or with nothing, if the file has already been included).

A precursor module's contents are only loaded into the compiler once the whole module has been compiled.  As a result, two precursor modules cannot be dependent upon each other: if precursor module A tries to make use of a function, class, statement, macro, etc. defined in precursor module B, then B must be fully compiled and loaded into the compiler before A is compiled.  Consequently, B cannot depend upon elements defined in A.  (Things may be a little more sophisticated than this, but I doubt it: SLX is a very straightforward language.)

I would guess that it would be possible for the Scala compiler to automate module determination to a degree by figuring out the dependencies (of what would be "precursor packages", or some such) and compiling in the right order, but co-dependent precursor packages would obviously lead to compiler errors.

Answer 2:

I've been puzzling over this for a while.  Like you say, it's possibly no less risky that running a piece of code from a third party.  However, the difference is that you might not expect the compiler to be executing the code it's compiling - that's a little out of the ordinary and may take people by surprise: you wouldn't have to run the resulting program to expose yourself to any risks in that code.

Answer 3:

Aha!  You spotted the deficiencies in the simple example I provided!  ;-)

Yes, SLX macros are a little like C macros in that the expansion is first to a string that is then compiled like regular code, but also more sophisticated because of the fact that the string is generated by the macro, according to circumstances, inside the compiler.  That much is clear.

As for escaping, I typically wrote a function (in a precursor module) that performed the escaping and called it when appropriate in generating the expansion.  That was not a part of SLX.  However, SLX added a syntax that made it unnecessary to escape quote characters - describing that would likely add more noise to the discussion, so I'm omitting it.

Protecting the resulting expansion from collisions with other element names in scope was an issue too.  Typically, this could be addressed by wrapping the output in braces, ensuring a local scope for any elements defined within the expansion (that is so trivial that SLX ought to have done it automatically).

Answer 4:

I'm not necessarily saying that the SLX approach is easier - it just seems that way to me right now.  The SLX approach has relatively few surprises and the compiler generates meaningful messages when the code fails - but, clearly, if you write a bad macro, the SLX compiler can't do much about it until another piece of code attempts to use it.  That's a definite downside.

I've struggled with learning to apply Scala's current approach to writing macros - probably because there are no examples right now that do what I'm trying to do.  Once I get off the beaten path, I've found things pretty confusing!

I'll look at the quasi-quotes link you have provided and get back to you on that.

Incidentally, one other advantage of the approach that SLX takes is that the text expansion of a statement/macro is available to the debugger to step through as though it was regular code.  If I recall (it's a couple of years since I used SLX regularly), you could toggle the expansion of a statement/macro when debugging code.

Thanks again!

Best Regards,

Mike

[Mike Allen]

Hi Lex,

I agree with every point you make - particularly about macros being a killer feature - with the exception of the last one.  Having to split projects into two causes me a LOT of headaches.  As a user, I'd rather the compiler handled the complexities of macro generation rather than - an extra keyword seems like a small price to pay!  ;-)

Best Regards,

Mike

[Eugene Burmako]

I've yet to comment the point that've been made here, but one thing I'd like to mention right away is that templating and ASTs aren't mutually exclusive: http://docs.scala-lang.org/overviews/macros/quasiquotes.html.

--

[Rex Kerr]

On Tue, Mar 19, 2013 at 9:38 AM, Lex Spoon <l...@lexspoon.org> wrote:

It's nice to see an explicit discussion about making macros nicer!
Macros are a killer feature in Scala these days; I expect they will
become a must-have feature in languages once more designers see how
well they work in Scala.

Well, only to the extent that Scala can in major ways assist the tedious process of rewriting ASTs.  It's not like AST manipulation is a new thing in computer science.

So far, macros look to me like a killer feature only by virtue of how incredibly limited Java is, and how eager one is to not be so limited once one uses Scala for a bit.  Maybe quasiquotes and some decent documentation will be the magic recipe for genuinely easy and powerful macro writing; right now (in 2.10) they give you very desirable capabilities but only after a lot of hard work.

If creating something like the "fragment" method below becomes easy with macros, I will indeed call them a killer feature:

  sealed trait Foo { def bar: Int }
  final case class One(x: Int) extends Foo { def bar = x }
  final case class Two(x: Int) extends Foo { def bar = 2*x }
  final case class Three(x: Int) extends Foo { def bar = 3*x }

  val xs = List(One(5), Two(6), Three(4), Two(2), Three(0))

  val (ones, twos, threes) = fragment(xs){
    case One(n) => One(2*n)
    case Two(n) => Two(n)
    case Three(n) = Three(2*n/3)
  }

where you get warnings if you don't cover all cases, and this works like three distinct collect statements.

The fragment function is conceptually as simple as (imagined notation)

  unicorn def fragment[X,Y](xs: C[X] <: Seq[X]){f => X => Y]} = {
    val types = f.cases.map(_.returntype).map(z => C.newBuilder[z])
    xs.foreach{x => f.cases{c => types(c.returntype) += c(x) }}
    types.values.map(_.result).force
  }

if types and ASTs are first-class entities.

  --Rex

[Lex Spoon]

Totally agreed about quasi-quoting being a big help. As someone who
might or might not have spent some time in the Scheme Underground, I
am very happy to see the idea being proposed for Scala.

I should have mentioned earlier that GWT has had great success with
string templating in their "generator" system. Essentially, you call
GWT.create(Foo.class), where "Foo" is an interface. The compiler looks
up a generator for Foo's that the programmer supplies, and that
generator emits Java source code for a subclass of Foo. It's the bare
minimum of what you need for in-compiler code generation, and GWT
users find it to provide a huge increase in capability.

Nonetheless, ASTs are better. Structure is a good thing. You wouldn't
want lambdas that are based on string templating....

-Lex

[Eugene Burmako]

It would be nice to see a language, which lets one intermix compile-
time and runtime code in this manner with resonable efficiency. Even
in Lisp they gave up on fexprs.

[Rex Kerr]

Agreed--and we've enough decades extra experience with "reasonable efficiency" that I don't think it's out of reach any more.  But obtaining the resources necessary for such a project is not something I'd know how to do successfully or I'd probably already be doing it.

So in lieu of that, my hope is that macros come as close as can reasonably be expected.  (Well, I guess my hope is actually slightly unreasonable, but still within the realm of the possible.)

  --Rex

[Rintcius Blok]

I agree with every point you make - particularly about macros being a killer feature - with the exception of the last one.  Having to split projects into two causes me a LOT of headaches.  As a user, I'd rather the compiler handled the complexities of macro generation rather than - an extra keyword seems like a small price to pay!  ;-)

I am curious to learn what the headaches are that you encounter. To me the project split looks like the way to go. I guess for the same reason why I like the set up of SBT projects (by putting build code in a project dir; and how it allows arbitrary nesting).

Rintcius

[Mike Allen]

For me, the problems are many.

Take one of my projects, which is a library that is built using Maven and distributed via the Maven central repository.  To enable use of macro's (and I very much want to be able to use them), I have to do the following IN THE SIMPLEST CASE:

1. Do lots of non-value-added editing of parent & child POM files to make it appear to users of the library that nothing has changed, including merging all the sub-project jar files into a single jar that can be distributed.

2. Move all of my existing code into a sub-project.

3. Create another sub-project for my macro implementations.

But wait!  What if some of my macro implementations need to use elements defined in the main project, which is not unreasonable?  Now I have to identify all such code and move it - and the code upon which it depends - from the "uses macros project" to the "macro implementation project".  That's not exactly a trivial exercise.  I'll then rejoice at the knowledge that some code belonging to package "somepackage" is defined in the "uses macros project" and other code in the same package needs to go into the "macro implementation project".  Nice!

But there's more!  Now, what if a macro definition itself uses another macro?  Now, I've got to create ANOTHER project that contains the implementation of the macro upon which my consumer macro depends.  So instead of one package, I now have THREE!  And things can clearly escalate beyond that very easily.  Each time I need to split a project, I have an enormous amount of overhead to solve one trivially simple problem - and none of it makes me more productive or adds a huge amount of value.

In fact, the economic consequences of needing to split my code into a number of dependent projects is going to dissuade me from even considering the use of macros, and I'm sure that others will feel the same.

So, to me, this is a very big issue.  If Scala could follow the SLX approach, and allow macros to be defined and used on one compiler run, then this issue goes away completely.

Best Regards,

Mike

[Rintcius Blok]

Thanks for clarifying, Mike. That doesn't sound like the most fun thing to do indeed. Just curious, why do you need to make it appear to users that nothing has changed?

I can see you like to simplify things, but in my mind a project is the entity where one separates these kind of things (build definitions is another example).
I cannot see this working without separating projects, so the only option I see would be to *simplify the separation of projects*.

I am not sure whether you just use Maven repository or also for project definition. If you use Maven for project definition, then I think there could be a big win for you, if you switch to a build tool that makes defining a project more light-weight and flexible. Would that be an option? E.g. in SBT it is already pretty light-weight (then it could come down to defining an extra source root for every project).

[Mike Allen]

Hi Rintcius,

Thanks for your comments.

The reason I don't want users to notice that anything has changed is so that their existing code won't break.

I strongly disagree that creating new projects is the way to go.  I have one project and one code base (actually, I have a number of such projects) and I want to keep things at that level.  If I choose to have sub-projects, then I can do so, and I agree that it would be nice if that could be done more simply, but I don't see why I should be forced to do so.  Similarly, if I use Maven, I don't see why I should be forced to switch to a different build tool, not that I've got anything against sbt.

As I pointed out in my earlier post, the status quo requires me to do a lot of huffing and puffing just to be able to use macros.  Furthermore, I've pointed out that I've used a language (SLX) that doesn't require splitting projects to enable macros of a similar nature to Scala's.  SLX macros were a joy to use and I made extensive use of them; Scala's macros, well, not so much right now.  But, as Lex pointed out earlier in this thread, macros are a killer feature.

To me, the fact that SLX can make writing and using macros so simple means that there has to be a better way - a simpler way - of supporting macro development in Scala.  That's one of the reasons I started this thread.  (The other reason, simplifying how the macros themselves are written, has gone away a little.  I guess I've come to terms with using ASTs, although I didn't really want to have to dive that deep.  C'est la Vie!)

Incidentally, lest anyone think I'm ungrateful, I want to reiterate that words can't describe how much I love coding in Scala!  Thanks again to everyone who has played a part in its development!

However, are we happy with the need to split projects that define and use macros?  If not, then what might be done to address it?  (After all, would Scala have automated type inference if it didn't believe in making things easier for the programmer?)

I'm proposing that it ought to be possible and feasible for Scala to support single projects that both define and employ macros.  How might we do this?

Obviously, the approach that SLX takes (in which the programmer dictates the order in which files are compiled) is unappealing for Scala.  This implies that the Scala compiler needs to figure out how to compile sources containing macro dependencies, so that it completes compilation of macro implementations before completing the compiling of instances of those macros.  It's clearly an understatement to say that that is nontrivial.  But I would argue that it shouldn't simply be dismissed because it's not easy to do.

At the same time, we don't want to push the burden onto the programmer, because the harder we make it for them, the less likely they are to even use macros - thereby causing them to peddle inferior solutions to their users.  ;-)  (Forcing coders to split projects kills macros as a feature, IMHO.)

So, to get the ball rolling, here's one suggestion: let's say that we have a new keyword (I'll call it "X" to avoid religious wars over the name) that indicates that a block of code, a package, object, class, function, etc. is to be compiled and then loaded into the compiler once it has been compiled.  All macro implementations and code used by macro implementations would have to be decorated with X.  (Would it be an error to reference a function in a macro statement that wasn't decorated with X?  Not if a jar file with the definition was already available, but if the function is part of the same project, then you'd get an error if it wasn't.)  Whenever the compiler encounters such code, it compiles it and loads it upon completion.

If the compiler encounters an instance of a macro, but does not yet have the implementation available, then it suspends processing of the associated element - say by adding it to a queue of outstanding work - then continues with what it can do.  When the implementation is encountered and loaded, it could resume compilation of the associated instances.

Similarly, if during compilation of a macro implementation, it encounters a reference to a function, class, etc. that has yet to be compiled, then it would suspend compilation of the definition until the referenced code is available.  (I guess if the referenced code is defined in the same X block, then the compiler ought to be smart enough to know that it will be available by the time the block has finished being compiled.)  Again, once the referenced code is available, the compiler could resume compiling the referencing code.

If all sources have been processed and there is still outstanding work in its queue, then it's time to start dishing out error messages.  Maybe the compiler could warn, for instance, that a class required by a macro implementation was present in the code, but that it was not decorated with X.  Or that an X block Y depended upon code in an X block Z, and code in Z depended upon code in Y - causing a circular dependency.

So, not knowing much (or anything, if I'm honest) about the Scala compiler, would this be feasible?  Anyone have any better ideas?  Just trying to get some debate going...

Thanks for reading this far!

Best Regards,

Mike

[sreque]

I'm not a 100% expert in this subject, but I thought I'd provide some more background on the problem of compilable macros and importing macros in the same compilation unit. My belief that solving these problems isn't easy and involves a number of tradeoffs.

Scala macros are inspired at least in part by Nemerle macros, which also enforce separate compilation of macros. If you read section 9.4 of their paper (http://nemerle.org/metaprogramming.pdf) you'll see that they don't go into a lot of detail but basically say that Scheme has dealt with this problem already and run into significant complications on their end.

In the world of Scheme,  in order to support using macro definitions in the same compilation unit in a principled manner, your language has to define some notion of a reflective tower (see http://srfi.schemers.org/srfi-72/srfi-72.html#tower) in order to separate expand and runtime environments. There are also some tooling complications that occur when you when you mix macro compilation and execution into the same run. The paper "Compilable and Composable Macros" (see http://www.cs.utah.edu/plt/publications/macromod.pdf) provides some more background information on the subject.

Template Haskell has some similar limitations (http://www.haskell.org/ghc/docs/7.6.2/html/users_guide/template-haskell.html) but it looks like it does a better job overall. It looks like in template Haskell you can compile module A and module B together if A uses macros defined in B, but only if B makes no reference to A, even transitively. The Haskell compiler will therefore be able to compile module B before it begins even type-checking module A. Scala might be able to do something like this, but I don't know how much work would be involved or what kind of effect it would have on downstream tooling.

Hopefully that provides some useful information on why this likely a hard problem, and one that has been encountered multiple times by different languages implementing macros

Regards,

  Sean Reque

[Mike Allen]

Hi!

Just to be clear, I'm not talking about defining and using macros in the same compilation unit - just on the same compilation run.

Mike

[Scott Carey]

Separating peojects with macros is the right thing to do. Your users will appreciate the modularization. Projects that bundle everything into one jar are oftwn my worst enemies as a consumer of many libraries.

As a producer of libraries, additional modularization is some up front work, but it nearly always has long term benefits.

Lastly, even with maven you could do this in one project if the macros are internal (not part of your api). Treat macros like code generators and leverage the generate-sources phase.

If your users consume your code via maven, additional dependencies can be made invisible. Their pom does not change. Transitive dependencies are easy.

* As a user, if the macros are not for me to use in my runtime, I don't want them bundled in the jar. Therefore they should be a separate maven artifact anyway.
* As a user, if the macros are to be used by my compile but not my users runtime, I don't want them bundled in the same artifact either.

This is akin to test artifacts from the dependency and bundling perspective in several ways. Maven does not have a code gen step that is as sophisticated as the test step, unfortunately. It does have one however.

[Paul Phillips]

On Wed, Mar 27, 2013 at 8:01 AM, Scott Carey <scott...@gmail.com> wrote:

Separating peojects with macros is the right thing to do.  Your users will appreciate the modularization.  Projects that bundle everything into one jar are oftwn my worst enemies as a consumer of many libraries.

This is a non-sequitur. I could as easily say "separating projects with classes named Bippy is the right thing to do. Your users will appreciate the modularization. Projects that bundle Bippies into the same jar are my worst enemies."

Modularization is only an asset to the extent that it is consonant with useful code partitions. Macros are a cross-cutting concern. They're completely decoupled from meaningful module boundaries.

[Rodrigo Cano]

I agree, yet one of the arguments is right, if you modularize the macros in a separate library, you can declare them as a provided dependency, which causes them to not be packaged with your app, and that make sense, since the macros are only run during compile time, transitively this could apply to the the scala-reflect.jar and scala-compiler.jar that your macros might depend upon.

[Mike Allen]

Hi Scott,

The macros I'm planning to implement are intended for both internal and external use - it makes a lot of sense for them to be part of the same jar file, IMHO.

However, if you refer to my post above, you'll see that it's possible to have some macros that utilize other macros, which in turn could utilize yet other macros - so are you recommending a whole tree of projects/jar files?  As I pointed out, that's a lot of effort on my part for very little perceived reward.  Furthermore, since these macros can utilize regular elements within the libraries, there's potentially a huge code management issue - which project/jar do I put class X into (given that it might or might not be referenced from zero or more macros)?  Finally, some of the code and macros logically belong in the same package - but if they're spread across different jar files, then I can't lock them and prevent user code from adding itself to those packages and by-passing element access protections (I've worked hard to make the user interface as simple as possible and don't want to lose all of that).

However, I didn't think of using the generate-sources phase to pre-compile the macro code - that could conceivably allow me to keep all my sources in one project (albeit with some filtering).  That could be a useful workaround, but I'd still prefer a solution (ability to just compile the whole lot as one project with no fuss or muss).

Best Regards,

Mike

[Jason Zaugg]

On Wed, Mar 27, 2013 at 5:49 PM, Mike Allen <mi...@hindsight-consulting.com> wrote:

Hi Scott,

The macros I'm planning to implement are intended for both internal and external use - it makes a lot of sense for them to be part of the same jar file, IMHO.

However, if you refer to my post above, you'll see that it's possible to have some macros that utilize other macros, which in turn could utilize yet other macros - so are you recommending a whole tree of projects/jar files?  As I pointed out, that's a lot of effort on my part for very little perceived reward.  Furthermore, since these macros can utilize regular elements within the libraries, there's potentially a huge code management issue - which project/jar do I put class X into (given that it might or might not be referenced from zero or more macros)?  Finally, some of the code and macros logically belong in the same package - but if they're spread across different jar files, then I can't lock them and prevent user code from adding itself to those packages and by-passing element access protections (I've worked hard to make the user interface as simple as possible and don't want to lose all of that).

As has been mentioned, you can hide the artificial sub-projects from your users by bundling together the classes in a single JAR during packaging. SBT also supports custom configurations, so rather than just "test" and "compile", you could have "compile1", "compile2", "compile3", and configure which sources are in each phase. It will be a bit fiddly setting this up the first time, but I'm sure you could get some help on the SBT mailing list.

But could you also share the motivating factors for macros-built-with-macros (-....) ? Maybe we can help find an alternative, for example, macros implementations in the same module can share libraries of AST manipulations.

There has been some work done to lift this restriction (the macro JIT Compiler), but it is still quite preliminary.

-jason

[Rintcius Blok]

Hi Mike,

I am still wondering how much of your reluctance to separate in multiple projects is build tool related.
I know when I was using Maven, projects where indeed something that I wouldn't set up that easily.
But SBT definately changed that for me. Now the threshold to start a new project, change a project etc. is much lower.

Just to give an example, maybe you have seen it already: https://github.com/scalamacros/sbt-example/blob/master/project/Build.scala
This little build file defines 3 projects (compare that to what you can express in a Maven file of this size).
What is also pretty nice is that I do just "sbt gen-idea" and my IntelliJ workspace is set up with the 3 projects in it.
To me it feels more that I am working with 1 repository than with 3 different projects.

Compared to Maven, it really becomes a lot easier to switch, separate and in general *do* stuff with your projects in this way.

I don't want to imply that it's better to migrate your projects (assuming you are using Maven).
It's just meant to make clear that SBT is a much better fit for this kind of projects than Maven (although SBT's learning curve is quite steep).

That said, the JIT compiler feature that Jason mentioned looks also interesting. Hopefully it will work out for you!

- Rintcius

[Mike Allen]

Hi Rintcius,

Sure, Maven isn't perfect - but neither is SBT.  Also, it's not a trivial task for me to switch, although I'm considering it.  All of which is besides the point.

Here's what I'm proposing: the Scala compiler can handle macros in one run.  It's possible.  SLX - a very simple language developed by a single person - can do it, so I'd be astounded if Scala couldn't do it too.  How sophisticated that implementation would need to be and whether there would be any changes to the language to implement it, I don't know.  A debate about that is what I'm trying to get going.  In the meantime, I've no option but to find workarounds - but that wasn't the focus of my thread.  So far as I can tell, no-one has offered suggestions how that might be done, although Sean did some interesting research on this issue in other languages (and having read the articles he linked to, I remain convinced that the way I outlined things is possible - although, I still don't know if it's feasible), and Jason's mention of the Macro JIT Compiler gives me some hope.

Assuming that Scala can compile and expand macros in one project, here's what I'd need to do to my project to add macros:

1. Possibly add some keywords to my sources to identify what needs to be compiled and loaded into the compiler, as opposed to being just compiled.  This isn't much of an overhead if I add it when writing my macros.
2. Er...
3. That's it!

Here's what everybody seems to be suggesting I do in the meantime:

1. Switch from Maven to SBT.  Even then - as Jason points out - I still have to write custom configurations in SBT.
2. Sort my sources into groups to be compiled separately, either as separate projects, or as separate compile phases in SBT.  Each group would correspond to a tier of macro implementation, of which there could be many.  Regardless of which route I go, I need to either partition my sources into different dependent projects, or set up a series of custom configurations that filter different sources for separate compilation phases.  
3. Write some logic to package up all of the separate projects/compilations as a single jar file, or consider shipping a number of closely inter-related jar files without any lock protection.

So, a simple thought experiment: whether you're writing a simple program or a large project, if you would like to use macros, which would you prefer to use?  (Forget what's involved in getting the compiler to do that.)

Now I'm sure that when someone suggested adding local type inference to Java to make it easier for users, there were (are) people who resisted it.  As Scala users, we know it's a time-saving concept that makes us more productive and reduces programming maintenance chores.  I'm sure that it wasn't all that easy to implement in the compiler either (compared to explicitly specified types).  Similarly, as Scala users, we don't have to add the same amount of boilerplate code that a Java coder does, etc. because the language makes our jobs easier.  My point is that if a language supports what you're trying to do and makes your life easier, you're going to adopt it and use it extensively.  (That's how I feel about Scala!)  If it doesn't support what you're trying to do, and makes you jump through hoops just for the hell of it, well, it can become yesterday's programming language.  (That's how I feel about Java, C++, C and - I hope this doesn't date me - FORTRAN.)

I used macros in SLX quite extensively.  One interesting thing about SLX is that you don't have libraries that you link to - in effect, if you use library code, you have to compile that library code with your own program, so macros HAVE to be compiled with the rest of the source.  You can't split a job into multiple projects.  Funnily enough, that's not a problem.

Another interesting point is that SLX internally uses macros extensively.  In fact, most of the constructs in SLX are implemented as macros by using a very small sub-set of the language (what is called the "kernel layer" of the language).  As layers of features are added (or, as macros are used within macros), the out-of-the-box SLX language appears very much bigger and more sophisticated than it actually is.  In fact, there's an SLX-based DSL for an even older simulation language, called GPSS/H, than can execute GPSS/H programs unmodified!  DSLs can be produced very easily in SLX.  (For those interest in more details, there's an overview of SLX here: http://www.wolverinesoftware.com/SLX00.pdf)

There's a big parallel to how Scala provides its DSL features (currently, without needing macros) - for example, to the extent that people often think of constructs such as "react {}" as built-in Scala features.  Over time, if macros in Scala are sufficiently easy to use, I have no doubt that many Scala features will be re-written as macros in a similar manner.  If you use Scala to build DSLs, don't be surprised if you find yourself building macros with other macro constructs (that are in turn built with macro constructs) - many of which are internal to your project.

On the other hand, if you have to do 5x more work just to use macros, then you likely won't bother using them.

I hope this clears that up.  Please no more questions such as "why can't you split your project?" or "why can't you ship multiple jars?" or "why don't you use SBT?" or "why do your macros contain macros?" - I think I've answered all of those questions and I hope I've satisfied your curiosity, but if I haven't, then I don't think I'll be able to.  I know I have to do some of those things right now, because there's no alternative, but there's got to be a better way...

Best Regards,

Mike

[Scott Carey]

On Wednesday, March 27, 2013 9:49:50 AM UTC-7, Mike Allen wrote:

Hi Scott,

The macros I'm planning to implement are intended for both internal and external use - it makes a lot of sense for them to be part of the same jar file, IMHO.

Ok, so lets say that Alfred is writing a library that uses your library for both macros and other purposes, but it does not expose a public API for macros itself.  Alfred adds a dependency on your jar artifact, which has transitive dependencies on all the macro stuff.   Now, Bruce wants to use Alfred's library, and adds that library to his dependencies.

Now Bruce has a project that implicitly declares transitive dependency requirements on all the Scala macro stuff that are false and lead to bloat or in the worst case, duplicate or conflicting versions if some of his other dependencies have overlapping transitive dependencies.  Bruce or Alfred could set up transitive exclusions to stop this, but the default behavior would be to include a large number of likely not too small jars that your library and Alfred's need at compile time but Bruce does not.

If your library did not include macros for external use, you could declare the dependency on the macro machinery optional or provided scope and prevent leakage, only having a few hidden unused class files in your jar.

However, if you refer to my post above, you'll see that it's possible to have some macros that utilize other macros, which in turn could utilize yet other macros - so are you recommending a whole tree of projects/jar files?  

Yes, this is analogous with test artifacts.  Some tests need other tests artifacts, and people publish jars with test classifiers.   I'd recommend keeping the same groupid:artifactId but changing the classifier for items that are for compile time usage and not runtime in most cases.  This could be wired up with the generate-sources phase and custom configuration of the jar plugin.
 

As I pointed out, that's a lot of effort on my part for very little perceived reward.

Is your goal to make your life easier or those that consume your artifacts?  IMO this comes with being a library designer.  If you are only writing an application, how you bundle up your jar and configure dependencies is your business alone. 
I use some libraries that require me to add over 20 exclusions to prevent pulling in bloat because the project developers are making their own lives easier and treat maven/sbt/whatever as a build tool only, and not what they also are:  tools for publishing artifacts for use by their users with metadata about transitive dependencies.  Such libraries often declare transitive dependencies that are used by the library for purposes that no user of the public API would ever need, and they largely do because the developers are not paying attention to their transitive dependency hull and the downstream impact that has on their users.
 

 Furthermore, since these macros can utilize regular elements within the libraries, there's potentially a huge code management issue - which project/jar do I put class X into (given that it might or might not be referenced from zero or more macros)?  Finally, some of the code and macros logically belong in the same package - but if they're spread across different jar files, then I can't lock them and prevent user code from adding itself to those packages and by-passing element access protections (I've worked hard to make the user interface as simple as possible and don't want to lose all of that).

Yes, it can get messy, all dependency management and modularization is.  Most of it is unavoidable and the nature of the beast.  If you want a class X to be used by a macro, the macro depends on it, whether it is in the same jar or not.  I am not saying that all non-macro code should be somewhere else, but if other code also depends on X, X will need a home separate from both.  I have found that modularization forces me to think more clearly about my code and how it fits together, leading to teasing apart bits that should have been separate in the first place, or introducing interfaces or traits that should have been there to start to cleanly encapsulate.

[Scott Carey]

On Wednesday, March 27, 2013 8:34:11 AM UTC-7, Paul Phillips wrote:

On Wed, Mar 27, 2013 at 8:01 AM, Scott Carey <scott...@gmail.com> wrote:

Separating peojects with macros is the right thing to do.  Your users will appreciate the modularization.  Projects that bundle everything into one jar are oftwn my worst enemies as a consumer of many libraries.

This is a non-sequitur. I could as easily say "separating projects with classes named Bippy is the right thing to do. Your users will appreciate the modularization. Projects that bundle Bippies into the same jar are my worst enemies."

"separating projects with unit test code is the right thing to do.  Your users will appreciate the modularization. Projects that bundle unit test code in the same jar are my worst enemies".

The only thing not true about the above is that it is hard to find people who mash test code and dependencies into their primary artifact, so I have no such enemies.  My first statement was too strong however, it should be "separating macro code into a different maven artifact is the right thing to do" -- whether the artifact separation is via a different maven project or not does not matter.  That a user can include each in different scopes is what matters.
 

Modularization is only an asset to the extent that it is consonant with useful code partitions. Macros are a cross-cutting concern. They're completely decoupled from meaningful module boundaries.

Testing is a cross-cutting concern.  I fail to see how a concern being cross-cutting means it should not be separated.  For maven artifacts, the important element to partition by is the usage context of the code:  under what use cases do I need to include the code into my classpath?  The use case for including macros in a classpath differs significantly from other code.

It is accepted that unit test code and all of its transitive dependencies should not be tied to production code and that it belongs in its own dependency tree and scope.  Likewise, code that is used by the compiler but not at runtime needs its own artifact so it can be scoped into only the classpaths necessary.  Even if downstream users will use it, they may only want to use it in their compilation and not bundle it in their runtime.

[Rintcius Blok]

So, a simple thought experiment: whether you're writing a simple program or a large project, if you would like to use macros, which would you prefer to use?  (Forget what's involved in getting the compiler to do that.)

Yes, that's a good one! Normally, I *want* to separate these artifacts and I think Scott reflects my thoughts as to why well in his post here https://groups.google.com/d/msg/scala-language/k02dLx_Bxtc/KGwtk4ePRZIJ
But I see your use case and can imagine that if you are writing a lot of DSL's it becomes a tedious task.
You want to reduce the technical overhead of this task, preferably to the extend that it becomes just a matter of specification.

What you mentioned "possibly adding some keywords" is one of the ways how you could specify this.
Just to stop avoiding the possibility of religious wars, let's call this keyword "project" ;)
What would this keyword need? A description how it needs to be built (how/where do you want to specify *this*?)

This could be one approach. But you could also approach it from the build tool side with eventually the same effect.
Just that you would specify this description in the build file in that case.

Either way, no matter how you approach it, let's see *what* we are approaching here.
To me it's exactly the concept of a *project*. It's all about making a project more "fluid".
I don't know what's the best approach, but I do know that it's an interesting discussion :)

[Rex Kerr]

On Wed, Mar 27, 2013 at 10:24 PM, Mike Allen <mi...@hindsight-consulting.com> wrote:

Hi Rintcius,

Sure, Maven isn't perfect - but neither is SBT.  Also, it's not a trivial task for me to switch, although I'm considering it.  All of which is besides the point.

Here's what I'm proposing: the Scala compiler can handle macros in one run.  It's possible.  SLX - a very simple language developed by a single person - can do it, so I'd be astounded if Scala couldn't do it too.

Keep in mind that if what you want is simple enough, you can simply _do it yourself_ by preprocessing the code.

I take that approach quite frequently with both code generators and text-expansion macros, and although the most sophisticated code generator is rather difficult to comprehend, I mostly use simple ones.  Text-expansion, in code you control yourself, is also pretty straighforward.  For example, I have something that looks like

////####BEGIN inserty #T ` ' #(f) #setw #w
  def inserty(a: Array[#T], start: Int = 0, end: Int = -1)#(f) {
    val i0 = if (start < 0) a.length+start else start
    val i1 = if (end < 0) a.length+end+1 else end
    var i=i0+1
    while (i<i1) {
      val x = `a(i)'#setw
      var j = i
      while (j>i0 && x < `a(j-1)') { a(j) = a(j-1); j -= 1 }
      if (j<i) a(j) = #w
      i += 1
    }
  }
////####_END_
////####MACRO inserty=insSortF #T=Float `=<..> '=<..> #(f)=<..> #setw=<..> #w=x
////####MACRO inserty=insSortD #T=Double `=<..> '=<..> #(f)=<..> #setw=<..> #w=x
////####MACRO inserty=<.insSort[@specialized(Float, Double) A].> #T=A `=f( '=) #(f)=<.(implicit f: Fn1D[A]).> #setw=<.; val y = a(i).> #w=y

which defines a code block where symbols #T ` ' #(f) #setw and #w are expanded, and there are three different expansions given (<. and .> are delimiters for strings that are empty or may contain spaces).  The code that runs the preprocessing is only about 200 lines, and it's not particularly tricky code, either.

So, yes, I have a two-step build, (preprocess, then scalac/zinc/fsc/whatever), but I can use insSortF elsewhere in that file (even in other preprocessor macros).

Between this and quasiquoting (once it's available), I will consider my macroing needs met.

I suggest that if you find yourself sorely in need of text-expansion macros (rather than just finding that they'd be kind of nice), that you roll your own preprocessor.  If the preprocessor can be as dumb as the C one (or dumber), it's really not that much work.

  --Rex

[Ivan Todoroski]

On 28.03.2013 10:02, Scott Carey wrote:
> Testing is a cross-cutting concern.

Actually, no. Testing is not a cross-cutting concern. You don't freely
mix test code with your application code. Your unit tests are always a
separate group of classes, sitting to the side and doing their own
thing. They do call upon your application classes, but your application
code never calls your unit tests. This makes them an ideal candidate for
modularization.

With macros it's different. Any random part of your application code can
call a macro, and vice-versa, macros could call your application code
(e.g. some shared logic or whatever).

At any point during your project evolution, you may decide that this or
that method is better re-implemented as a macro, and suddenly another
random part of your code uses a macro, regardless of any module
boundaries. Hence the cross-cutting concern.

Same with any exposed public APIs you have. At any point during your
project evolution you might decide that this or that part of your API
might be better implemented as a macro, but now you have to pay a
penalty for that technical improvement - you have to rework your project
structure, and cause additional inconvenience for your downstream users
as they might need to rework projects on their end too.

This represents a barrier to entry for adopting macros in your code.
Whether it's a big or small barrier can certainly be debated, but it's a
barrier nonetheless.

[Rintcius Blok]

Thinking more about the idea above. Here is something what *could* work:

1. define a macro called "project" with a name parameter
2. define a build definition resolver that returns build definitions for each project that is defined
3. specify the build definitions (including the dependencies for example)

Each build tool would typically have a default build definition resolver.
In SBT for example projects already have a name and you won't have to change much.
For Maven the default could be to use the central pom.xml and add the project name to the central artifact and add an option to bundle them in 1 jar.

--
You received this message because you are subscribed to a topic in the Google Groups "scala-language" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/scala-language/k02dLx_Bxtc/unsubscribe?hl=en.
To unsubscribe from this group and all its topics, send an email to scala-languag...@googlegroups.com.

[Ivan Todoroski]

On 28.03.2013 14:45, Rex Kerr wrote:
> On Wed, Mar 27, 2013 at 10:24 PM, Mike Allen

> <mi...@hindsight-consulting.com <mailto:mi...@hindsight-consulting.com>>

Is it April 1st already??? :)

You have got to be kidding. The whole point of the smart, AST-based,
type-sensitive, hygienic macros in Scala was to *avoid* the
monstrosities of the dumb C-style preprocessor, while getting some of
the same power and functionality without the downsides.

And now you are proposing a C-style preprocessor to solve problems with
deployment of Scala macros? :)

Yeah, I'm sure a Scala IDE would love the "challenge" of trying to make
sense of the code you pasted above and implement refactoring tools for
it or things like Find Usages, where any part of the code is subject to
random string replacements that might not even be syntactically correct.

[Rex Kerr]

On Thu, Mar 28, 2013 at 10:19 AM, Ivan Todoroski <grnch...@gmx.net> wrote:

On 28.03.2013 14:45, Rex Kerr wrote:

I am completely serious, and you would be too if you needed to write a large amount of repetitive high-performance code quickly and had been burned on dozens of occasions with bugs or limitations in specializations, value classes, and a clunky and poorly-documented interface to Scala macros.  I use all those things when they do what I need (and am very glad to have them!), but they don't cover everything and aren't yet perfect.

This is not to say that AST-based type-sensitive macros are not wonderful--they are!  They're new, so they're a bit rough around the edges in terms of usage.  And I very much look forward to quasiquoted macros being in the main distribution, as that will make deployment dramatically easier in many cases.  But they _do not target exactly the same need_ in their current form.
 

And now you are proposing a C-style preprocessor to solve problems with deployment of Scala macros? :)

I am proposing that waiting around for Scala macros to deploy the way you want is, if you're happy with something at the level of complexity of SLX macros, likely not the way to go.  Build what you need instead.  It will be messy, of course, but it will do what you need.  (Keep in mind that accruing massive technical debt is easy if you do this--which is why I made my preprocessing as ugly as possible so that my aesthetic sense would align with my long-term interests.)
 

Yeah, I'm sure a Scala IDE would love the "challenge" of trying to make sense of the code you pasted above and implement refactoring tools for it or things like Find Usages, where any part of the code is subject to random string replacements that might not even be syntactically correct.

Well, obviously the IDE can't make sense of anything that it doesn't know about.  Teaching an IDE about simple text replacements should be a heck of a lot easier than teaching it about AST rewriting, however.  And you don't even need to do that--just edit the expanded version.  A reverse processor from an edited expanded version back to what I wrote is also relatively short and easy to code (assuming the document isn't dramatically rearranged).  When you need to change the macroed bits, your IDE has a reload hotkey or button, right?

Let me be clear: I would very much like Scala macros to work within the same compilation unit that they're defined.  But I am not about to pretend that until this is implemented there aren't other reasonable ways to save repetitive work.

  --Rex

[Scott Carey]

On Thursday, March 28, 2013 7:48:16 AM UTC-7, Rex Kerr wrote:

Let me be clear: I would very much like Scala macros to work within the same compilation unit that they're defined.  But I am not about to pretend that until this is implemented there aren't other reasonable ways to save repetitive work.

  --Rex

I am also not saying that having macros and code run in the same compiler pass is a bad idea.  I am however trying to make it clear that in a world where the maven dependency resolution system is used for nearly all projects, the distinction that macro code is executed by the compiler and not at runtime needs to be taken into account.   It is easier to produce artifacts with inappropriate transitive dependencies and classfile bloat/cruft if the two are intermingled.  Ideally, the scala maven plugin and SBT would automatically produce two artifacts when macros are involved so that consumers can easily keep those artifacts in different scope.  Until then, artifacts in separate projects is probably easier than contortions with maven lifecycles and additional custom artifact classifiers.  SBT is a little more flexible but you'll still need to set up two compiler passes.  If you have to deal with understanding what code must be compiled before macros, and after macros, it is not a bad idea to have these three things in separate locations.