scala-macros and scala-virtualized
They are both mechanisms that allow you to write normal-looking Scala
code which does something else in the background, like generating SQL
queries and what not. In other words, both scala-macros and
scala-virtualized are mechanisms for creating domain specific languages
in Scala.
Having two distinct and complex mechanisms for doing essentially the
same thing seems wasteful and redundant. Not to mention the possible
unintended interactions between two such complex features.
Now, if you accept that the two mechanisms are somewhat redundant (which
you might disagree with of course), the question turns to which one is
better for Scala programmers...
1) Scala-virtualized is much more in the spirit of Scala
Even before scala-virtualized you could implement e.g. flatMap() in your
classes and have them participate in Scala's for/yield protocol, or you
could override various operators and such. Using methods to override
behavior of Scala's builtin constructs is nothing new. Scala-virtualized
merely expands that existing precedent to all language constructs and
brings it to its logical conclusion.
On the other hand, Scala had nothing like scala-macros before. It's a
completely new concept and somewhat alien in the Scala context. I've
used macros in Lisp to great effect, but Lisp is a language that is very
amenable to macros because Lisp code and data look the same. Scala code
and data look nothing alike, which brings me to the second point...
2) Scala-virtualized is much more intuitive to use for the library writer
With scala-virtualized you write your DSL implementations in
straightforward Scala code, similar to how you would write e.g. implicit
conversions to pimp a library or things of that nature.
With scala-macros you are forced to write this weird AST meta-language
that looks nothing like normal Scala code. It's almost like having to
learn another language on top of Scala.
3) How do you even debug scala-macros?
In scala-virtualized the overridden behavior sits in normal methods that
a debugger could interactively step through at run time, whenever you
hit an overridden construct as you step through your DSL.
With scala-macros, the code inside the macro is executed in some
precompilation phase before the code runs. When you actually run the
program it's no longer the macro code that gets executed, it's the
*result* produced by the AST manipulations in the macro code. So when
stepping interactively with a debugger you can't really step through the
original macro code that generated the code you are currently running.
This makes finding bugs in the macros much more difficult as your macro
codebase grows.
4) Scala-macros might be more powerful
Not to be completely one-sided, I will concede that scala-macros might
be strictly more powerful than scala-virtualized. I have no formal proof
of this, but it sounds like rewriting/generating ASTs from scala-macros
might enable some crazy things that might not be possible (or easy) in
scala-virtualized. But are these corner cases where scala-macros is more
powerful than scala-virtualized really something that you would often
use in practice?
In summary, compared to scala-virtualized, scala-macros look like a
low-level hackish solution in search of a problem. They are complex to
write, un-scalaish, difficult to debug, and will probably just add to
Scala's image of "too much complex inscrutable magic".
Finally, my question to the Scala Community is:
What *practical* problems can you solve with scala-macros that you
couldn't solve just as easily with scala-virtualized?
On Tue, Feb 14, 2012 at 04:23:18PM +0100, Ivan Todoroski wrote:
> What *practical* problems can you solve with scala-macros that you
> couldn't solve just as easily with scala-virtualized?
Like you, I am not an expert at either the work on macros or on
virtualization. But given that macros are essentially "mini compiler
plugins" which do tree transformations there are some problems I face
which I bet they can solve.
The big one is optimizing away intermediate objects which are created
through use of the implicit enrichment pattern, but which aren't
needed. I talked about this a bit on the scala-sips mailing list
recently but here's a summary in code form:
import scala.{specialized => spec}
import com.azavea.math.Numeric
import com.azavea.math.FastImplicits._
// code user writes
def foo1[@spec A:Numeric](x:A, y:A) = x + y
// code with sugar removed
def foo2[@spec A](x:A, y:A)(implicit ev:Numeric[A]) = x + y
// code after implicit resolution
def foo3[@spec A](x:A, y:A)(implicit ev:Numeric[A]) = infixOps(x)(ev).+(y)
// code with infixOps method inlined
def foo4[@spec A](x:A, y:A)(implicit ev:Numeric[A]) = new FastNumericOps(x)(ev).+(y)
// code i wish could get generated, basically inlining the
// implementation of FastNumericOps#+ without creating a new object.
def foo5[@spec A](x:A, y:A)(implicit ev:Numeric[A]) = ev.plus(x, y)
This might seem a bit esoteric but this pattern also gets used with
Ordering and other type classes. It's really the only performance
problem left with this pattern, after specialization.
There are other cases too (until recently, for-loops) where after
profiling you can identify certain places where a (small) tree
transformation will yield huge gains. Virtualization may help in the
case of loops, but my sense is that it isn't designed to handle this
general class of problem.
-- Erik
P.S. If you're curious the version of Numeric I'm using can be found at:
https://github.com/azavea/numeric.
That is also something found recently in discussion on Scala User Groups
(well, at least PSUG), where user start to ask when Scala will start to
*remove* features, not add (huge) one like this - moreover, two as big
as that, and kind of far from the orthogonal-feature set that Scala
adverts (you know, "deep not broad" - well, it will become hard to
explain what is not broad in Scala)
Thanks,
--
Francois ARMAND
http://fanf42.blogspot.com
http://www.normation.com
As you, I might be wrong of course. I see scala-virtualized as
generating code at run-time, whereas scala-macros generates code as
compile-time. They are both code generators, which, together with
string interpolators and virtpatmat makes it four different new code
generators in the next version of Scala!
Some use cases particular to macros are ad-hoc performance
optimizations, type-safe evaluation of strings (eg: formatting
strings, regex, json, and even xml if its literals are ever replaced
with string interpolation), and generation of code that needs to be
present as libraries.
--
Daniel C. Sobral
I travel to the future all the time.
Macros + Interpolation make it possible to remove XML while, at the
same time, providing it as a library. And, like XML, one could
likewise provide JSON, etc.
[...]
Macros + Interpolation make it possible to remove XML while, at the same time, providing it as a library. And, like XML, one could likewise provide JSON, etc.
Is string interpolation intimately tied with macros though? I thought
(perhaps wrongly) that they were orthogonal features. Would it be
possible to make XML an optional library by using the interpolation
support together with scala-virtualized or whatever other Scala features
aside from macros?
I'm glad you asked!
After an discussing this on the scala-sip mailing list we determined
that value classes would not support this feature. You may want to
revisit that thread for more information:
http://groups.google.com/group/scala-sips/browse_thread/thread/ad21c133a2d224b
-- Erik
I understand this, but I consider it a mere implementation detail. What
do I care when the code is generated?
Maybe I am approaching this from a different perspective than you. I am
not a compiler developer, I am just a Scala programmer who wants to use
the language productively.
I am interested in things like expressiveness, conciseness and readability.
I am interested in removing boiler plate as much as possible (which is
why I am interested in scala-virtualized and scala-macros).
Finally, I am interested in doing all this in a statically checked
typesafe manner that eliminates runtime errors as much as possible.
As long as I get these features, I don't care how they are accomplished.
Whether through compile-time or run-time code generation, I honestly
don't care as a regular Scala programmer.
What I do care about is the severely increased cognitive load of using
scala-macros. I need to learn a whole new AST sub-language just to
eliminate some boilerplate here and there. Plus all the other things
mentioned in my original email that started this thread.
> Some use cases particular to macros are ad-hoc performance
> optimizations
I believe low-level optimizations are something that the compiler and
libraries should take care of. I still try to write as optimal code as
possible without sacrificing readability too much, but expecting me to
write what amounts to mini compiler plugins to help the compiler with
low-level optimizations is not what I would consider a common use case
for a regular Scala programmer.
The optimization use case on its own is certainly not enough to justify
inclusion of a complex feature like scala-macros. In the rare occasions
where you really needed to optimize some critical bit of code and the
Scala compiler was letting you down, you could also write an actual
Scala compiler plugin, or just write it in C and call it from JNI.
> type-safe evaluation of strings (eg: formatting
> strings, regex, json, and even xml if its literals are ever replaced
> with string interpolation), and generation of code that needs to be
> present as libraries.
This is a very interesting point. I already asked you about the
relationship between interpolation and macros in another sub-thread, I
would appreciate if you or anyone else could shed some light on this.
Macros can synthesize new class declarations from thin air rather than
just using existing classes you defined elsewhere? Now that does sound
pretty cool, but again I feel like I must insist on putting things in
perspective regarding the practical utility of all this.
What *practical* problem can be solved by synthesizing new class
declarations from a macro, that couldn't be solved using
scala-virtualized in a different yet still satisfactory way?
I am not trying to be contrary, I honestly wish to be educated on the
new possibilities that are opened by scala-macros and scala-virtualized,
and in particular the practical limits on what can be accomplished with
each of those features.
No, they are not tied. Well, macros have a dependency on string
interpolation to provide quasi-quotations.
But while string interpolation will give you XML literals and
matching, it won't check syntax at compile time. Using macros for the
interpolator makes that possible.
Is it? Do you see no difference between a statically typed language
and a dynamically typed language? The only difference is when the type
check is made. Is it an implementation detail?
>> Some use cases particular to macros are ad-hoc performance
>> optimizations
>
> I believe low-level optimizations are something that the compiler and
> libraries should take care of. I still try to write as optimal code as
> possible without sacrificing readability too much, but expecting me to write
> what amounts to mini compiler plugins to help the compiler with low-level
> optimizations is not what I would consider a common use case for a regular
> Scala programmer.
That's where "ad-hoc" comes in, really. LIBRARIES CANNOT TAKE CARE OF
IT with the present support: they can't get rid of instance creation
and boxing. Yes, as a regular Scala programmer you won't be _creating_
macros, but if your for-loops suddenly become as fast as Java's, that
will be a library-provided macro. Neither can the compiler solve it,
because it doesn't have the knowledge that the library has. Or you
think Scala has the performance issues it has simply because no one
tried to fix them?
> The optimization use case on its own is certainly not enough to justify
> inclusion of a complex feature like scala-macros. In the rare occasions
> where you really needed to optimize some critical bit of code and the Scala
> compiler was letting you down, you could also write an actual Scala compiler
> plugin, or just write it in C and call it from JNI.
Talk to Yammer about that. If your "critical bit" means 80% of your
code, what use is it?
Now, compiler plugins suffer from three problems:
1. They are too hard to write.
2. They depend on compiler internals, which are too volatile.
3. You can't distribute them as a JAR.
So, what happens if you handle that? You get a macro. A macro is
really a specialized compiler plugin that doesn't suffer from those
three problems.
C and JNI? Unless your code spends most of its time on C, that won't
help you. The cost of calling JNI is often much higher than the gains
in performance that C will bring. But that's not a solution: you are
throwing away all of Scala's power when you do that. Consider:
Scala-with-macros:
def fatorial(n: Int) = {
var result = 1
for (n <- 2 to n) result *= n
result
}
Scala-without-macros:
def fatorial(n: Int) = {
// declare JNI binding and call C code that computes fatorial
}
That is what you proposed.
>> type-safe evaluation of strings (eg: formatting
>> strings, regex, json, and even xml if its literals are ever replaced
>> with string interpolation), and generation of code that needs to be
>> present as libraries.
>
> This is a very interesting point. I already asked you about the relationship
> between interpolation and macros in another sub-thread, I would appreciate
> if you or anyone else could shed some light on this.
That is pretty simple, and it comes down to that "implementation
detail" you mentioned. Say you have this:
printf("%d: %f%n", a, b)
At run time you can check whether "a" is an Int and "b" is a Double,
and throw an exception if they are not. At compile time you can check
this and produce a compilation error if they are not. The compiler
can't handle this unless it has knowledge of string formats used with
"printf", and, then, it won't be able to handle anything *like* this.
A compiler plugin can handle this, but we are back to the point that a
macro is nothing more than a light-weight plugin that's easier to
create, maintain and distribute.
Of course, Paul wants compiler plugins to be easier to create,
maintain and distribute, and *that* would represent a more direct
competition to macros.
That isn't entirely true. Paul did manage to get Range's foreach to
the same speed as Java's (after warm up, mind you), and the complexity
associated with it can't be ridden of even in macros, except for cases
where the indices are literals or known constants.
Just to echo Daniel here, I am a library author who wants to do
optimizations which my users won't have to see to benefit from. I have
written a compiler plugin but it's a bit fragile and not as advanced as
I 'd like. Also, many users try to avoid compiler plugins.
I certainly like the idea of being able to use macros instead, since
macros will probably be more "future-proof" than compiler plugins
currently are (or at least, have been).
That said, if the compiler API is stable, compiler plugins get more
advanced, and the community is ready to embrace them, then I agree that
maybe macros aren't needed.
-- Erik
Play! for example has a pretty extensive/complex custom compilation
process to compile your templates into class files, so your
controller->template calls can be verified at compile time, or even
before, during code completion! It also compiles your
LESS/Coffeescript. Being able to treat non-scala code as part of the
program (i.e. statically checked and verified) rather than as data
(dynamically load & pray) would be a pretty big plus.
Being able to statically check more things, e.g:
CSS files
HTML templates
Coffeescript/Javascript
XML Config Files
non-XML config files
Database-schema (a.l.a. F# Type Providers)
Regex-Literals
which are de-facto part of your "program" rather than the "data" would
be pretty awesome (naturally sometimes these things are dynamic and
need to be treated as such, but quite often they are not). One problem
with static languages is that when you integrate with external things
(config files, databases etc.) you lose all static checking and end up
getting lots of silly runtime-errors anyway. Being able to check at
least some of these things at compile-time would be sweet. You can do
all these as compiler plugins, and in a way macros are just a way of
making it easier and more regular.
-Haoyi
No, I haven't got back to it -- there was no Caliper artifact
available last time I tried to -- but my own code was while-loop-fast,
though buggy. Paulp said he also got there, and I believe him.
However, the commit message on the last change to Range indicated he
wasn't done yet. Maybe that's true, maybe not, but I haven't been able
to check how fast that is due to the problem I mentioned.
> Also, I don't know the limitations of Scala's macros, but presumably
> with them I could easily right a construct like:
>
>
> cFor(var i = 0; i < n; i += 1) { body ... }
>
> that would compile down to:
>
> {
> var i = 0
> while(i < n)
> {
> body ...
> i += 1
> }
> }
Well, the start index is constant, and the loop increment is 1, which
makes that code correct. But say, for example, that the increment is
2, and it suddenly stops being correct. Oh, I'll grant that it is
_probably_ correct, but only the programmer can know that, not the
compiler. Specifically, it would fail for n == Int.MaxValue. Now
change the test to <= instead of <, and keep the increment equal to 1,
and you'll get another broken code, for the same value of n.
Range has two problems: one, it doesn't know the step (1 to 5 by 1 --
the step isn't know at the time "1 to 5" is initialized), and, two, it
has to deal with boundary conditions. Macros can help with the former
if the step is known at compile time, which is not always the case
either. There's a LOT of code required to handle all these conditions,
and that's what gets in the way.
JVM can optimize those away, however, which get us back on the same
footing as simple while loops -- once the optimization has kicked in.
> much bigger problem, but also hopefully doable. The point here is that
> with macros your performance is going to be much more predictable and
> less reliant on your JIT, the size of your method bodies, and the
> current phase of the moon.
If you don't trust JIT, you can't rely on your performance. :-) Code
that looks pretty simple and fast can fail to be JITted if you don't
pay attention to it -- which is what happened to Scala's 2.8 and 2.9's
foreach.
Not that I don't agree with you on the general issue of macros and
performance, but it's not that simple either.
Thank you for taking the time to respond in depth.
On 14.02.2012 18:22, Daniel Sobral wrote:
>>> I see scala-virtualized as
>>> generating code at run-time, whereas scala-macros generates code as
>>> compile-time.
>> I understand this, but I consider it a mere implementation detail. What do I
>> care when the code is generated?
>
> Is it? Do you see no difference between a statically typed language
> and a dynamically typed language? The only difference is when the type
> check is made. Is it an implementation detail?
I was under the impression that DSLs written using scala-virtualized are
still statically type checked at compile time. Therefore the fact that
the scala-virtualized "generates code at run time" has no bearing on the
type check. The difference between statically and dynamically typed
language is a red herring, it has no bearing on this discussion.
(Of course, scala-virtualized doesn't really generate new code at
runtime, that's why I put it in quotes above, but I guess you were
making an analogy so it's close enough)
>>> Some use cases particular to macros are ad-hoc performance
>>> optimizations
>> I believe low-level optimizations are something that the compiler and
>> libraries should take care of.
>
> That's where "ad-hoc" comes in, really. LIBRARIES CANNOT TAKE CARE OF
> IT with the present support: they can't get rid of instance creation
> and boxing. Yes, as a regular Scala programmer you won't be _creating_
> macros, but if your for-loops suddenly become as fast as Java's, that
> will be a library-provided macro. Neither can the compiler solve it,
> because it doesn't have the knowledge that the library has. Or you
> think Scala has the performance issues it has simply because no one
> tried to fix them?
I don't understand... if a library has the knowledge how to make general
for-loops as fast as Java, why can't the compiler be enhanced to use the
same knowledge?
>> The optimization use case on its own is certainly not enough to justify
>> inclusion of a complex feature like scala-macros. In the rare occasions
>> where you really needed to optimize some critical bit of code and the Scala
>> compiler was letting you down, you could also write an actual Scala compiler
>> plugin, or just write it in C and call it from JNI.
>
> Talk to Yammer about that. If your "critical bit" means 80% of your
> code, what use is it?
So if macros were available to them, the Yammer programmers would have
spent their time writing complicated optimization macros instead of
working on their problem domain?
Writing code optimizers is a difficult task, especially on the JVM where
you have to second guess what the JVM might or might not do with your
code, and what shape of bytecode is most palatable to the JIT.
Don't get me wrong, I see what you are saying here, I understand that
macros can be used to optimize code specific to your problem, but is
that really the main use case for macros?
It boils down to saying "well, the compiler sucks and can't optimize
away instance creation or other bottlenecks I'm running into, so I am
going to use a complicated macro feature to work around compiler's
deficiencies".
It's a valid use, but it somehow seems less satisfying as a rationale
for such a feature.
I would prefer to see features that enhance expressiveness and reduce
boiler plate, yet are still easily accessible to regular day-to-day
programmers without too much cognitive load.
> Now, compiler plugins suffer from three problems:
>
> 1. They are too hard to write.
> 2. They depend on compiler internals, which are too volatile.
> 3. You can't distribute them as a JAR.
>
> So, what happens if you handle that? You get a macro. A macro is
> really a specialized compiler plugin that doesn't suffer from those
> three problems.
If I understand you correctly, you are saying that macros are basically
compiler plugins that are easier to create and distribute. But it seems
they are still too difficult for general programmers to use to reduce
various boilerplate here and there.
Scala-virtualized seems a much lighter weight feature for reducing
boiler plate and creating DSLs by regular programmers, without having to
learn arcane AST trees.
> C and JNI?
Yeah, I went a bit overboard with JNI, sorry about that.
>> This is a very interesting point. I already asked you about the relationship
>> between interpolation and macros in another sub-thread, I would appreciate
>> if you or anyone else could shed some light on this.
>
> That is pretty simple, and it comes down to that "implementation
> detail" you mentioned. Say you have this:
>
> printf("%d: %f%n", a, b)
>
> At run time you can check whether "a" is an Int and "b" is a Double,
> and throw an exception if they are not. At compile time you can check
> this and produce a compilation error if they are not. The compiler
> can't handle this unless it has knowledge of string formats used with
> "printf", and, then, it won't be able to handle anything *like* this.
> A compiler plugin can handle this, but we are back to the point that a
> macro is nothing more than a light-weight plugin that's easier to
> create, maintain and distribute.
But why do you need *macros* for this? This looks more like a job for
pluggable type providers, in combination with string interpolation.
It does come down to an "implementation detail", i.e. whether you
implement something like this with a specific feature designed for it
(pluggable type providers) or you use some general blunt tool (macros)
which seems like overkill.
The very first use case on that page about Advanced DSLs compares how
macros can achieve much more than Squeryl for example, and they list
limitations of Squeryl such as its inability to use the == operator and
must use === for its syntax (which is expected, since its based on plain
Scala).
Yet they completely ignore the existence of Scala Integrated Query[1]
which is actually built on top of scala-virtualized and solves that
particular problem in Squeryl, along with many more. So that comparison
is misleading. I would love to see a more realistic comparison of SQL
generation by macros vs SIQ.
In the other use cases they mention type providers which are an
orthogonal concept not intrinsically tied to macros (even though their
current implementation might be tightly coupled to macros, I don't know).
Code generation for things like FunctionN and TupleN is certainly
compelling, but I can't help but feel that maybe a language feature for
abstracting over the number of method parameters might be a better
solution to that.
[1]
http://scala-integrated-query.googlecode.com/files/SIQ-Scala-Days-final.pdf
On 14.02.2012 18:05, Rex Kerr wrote:
> The scalamacros.org <http://scalamacros.org> site has a bunch of desired
Then you are under an incorrect assumption. By "generate code" I mean
"compile", with all that it entails, *including* typing.
Here's a simple comparison:
Scala-virtualized (rough example, based on vague memories from long ago):
val s = VString[Regex]("\d+$")
This will type check if Regex is a valid parameter to VString,
indicating it will be able to process the string in question. Also,
"s" will be known as a VString, and will be type-checked through-out.
Scala-macros:
val s = regex("\d+$")
The same guarantees made by Scala-virtualized are valid here, BUT, the
code for "regex" will be run at compile-time, and it will be able not
only to compile the regex itself at compile time, but verify the regex
is indeed valid.
>> That's where "ad-hoc" comes in, really. LIBRARIES CANNOT TAKE CARE OF
>> IT with the present support: they can't get rid of instance creation
>> and boxing. Yes, as a regular Scala programmer you won't be _creating_
>> macros, but if your for-loops suddenly become as fast as Java's, that
>> will be a library-provided macro. Neither can the compiler solve it,
>> because it doesn't have the knowledge that the library has. Or you
>> think Scala has the performance issues it has simply because no one
>> tried to fix them?
>
> I don't understand... if a library has the knowledge how to make general
> for-loops as fast as Java, why can't the compiler be enhanced to use the
> same knowledge?
Does Scala, the compiler, know what Scalaz does? Specs? ScalaTest?
Dispatch? BlueEyes? Anti-xml? The code you'll be writing tomorrow?
It's not that the library knows enough about how to make a fast
for-loop, it is that it knows enough about ITSELF, and that knowledge
would let it perform optimizations the compile can't or shouldn't. Or
even make such optimizations _available_, so that the *user*, who has
way more information about the problem, may choose to do them.
Let's give an example to make this more clear. Let's pick Range's
foreach. One problem with it is that it can't do this:
while(i < n) {
...
i += inc
}
That may wrap-around. However, in most cases "inc" is 1, for which
this idiom, this translation, is ok. How could the compiler possible
know such details about Range's implementation and how it could be
optimized?
>> Talk to Yammer about that. If your "critical bit" means 80% of your
>> code, what use is it?
>
> So if macros were available to them, the Yammer programmers would have spent
> their time writing complicated optimization macros instead of working on
> their problem domain?
Why do you assume macros are complicated? Here's a CFor:
macro def cfor(_this, start, cond, incr)(body) = c"""
${start}
while(${cond}) {
${body}
${incr}
}"""
for(val i = 0, i < 10, i++) {
println(i)
}
Is that complicated? Would it waste Yammer's time? What that would let
is make it possible for Yammer to keep the expressiveness of the
language while enjoying all the speed it could have, at the same time
it avoids having to maintain complex compiler plugins.
Moreover, Yammer might not to have to do that at all, because the
library themselves could take advantage of it, so that Yammer would
not have to go around the libraries. See their comments: they resorted
to Java collections to avoid performance problems with Scala
collections.
> Writing code optimizers is a difficult task, especially on the JVM where you
> have to second guess what the JVM might or might not do with your code, and
> what shape of bytecode is most palatable to the JIT.
Yes, writing code optimizers is a difficult task. Writing optimized
code isn't, which is what macros let you do.
> Don't get me wrong, I see what you are saying here, I understand that macros
> can be used to optimize code specific to your problem, but is that really
> the main use case for macros?
No, it is not the main case. It's one of them. If that was *all*
macros offered, I don't think they'd have much of a chance of getting
in the language. On the other hand, offering the other stuff they do
*and* making it faster, that's a real trick.
> It boils down to saying "well, the compiler sucks and can't optimize away
> instance creation or other bottlenecks I'm running into, so I am going to
> use a complicated macro feature to work around compiler's deficiencies".
It's not that the compiler sucks. JIT is one of the most impressive
piece of technologies out there, it has the benefit of knowing what
happens at run-time, and *it can't do anything about it either*!
Honestly, there isn't much more that Scala knows that JIT doesn't, so
it doesn't have much more opportunity to optimize than JIT. Some,
granted, but not all that much.
Optimizing turning-complete code is hard. It's NP-hard. Ad-hoc
optimization -- optimizing individual cases -- is quite possible, but
you have to special-case them. In fact, JIT does that a lot for Java
-- that's why Java's for-loops are so fast: JIT special-cases them.
But a compiler, a vm, can only work on most common cases. A library
can handle what it knows about itself.
>> Now, compiler plugins suffer from three problems:
>>
>> 1. They are too hard to write.
>> 2. They depend on compiler internals, which are too volatile.
>> 3. You can't distribute them as a JAR.
>>
>> So, what happens if you handle that? You get a macro. A macro is
>> really a specialized compiler plugin that doesn't suffer from those
>> three problems.
>
> If I understand you correctly, you are saying that macros are basically
> compiler plugins that are easier to create and distribute. But it seems they
> are still too difficult for general programmers to use to reduce various
> boilerplate here and there.
And Scala's type system full power is difficult to use as well, but
one can grab Scalaz and Shapeless and make use of some very advanced
type trickery with little trouble. Not everything is targeted at
"Hello World".
> Scala-virtualized seems a much lighter weight feature for reducing boiler
> plate and creating DSLs by regular programmers, without having to learn
> arcane AST trees.
You are much more optimistic about Scala-virtualized than I am.
> But why do you need *macros* for this? This looks more like a job for
> pluggable type providers, in combination with string interpolation.
Pluggable type providers are limited-use macros. They don't reduce
their complexity -- which seems to be your main problem with macros --
in any way.
> It does come down to an "implementation detail", i.e. whether you implement
> something like this with a specific feature designed for it (pluggable type
> providers) or you use some general blunt tool (macros) which seems like
> overkill.
As I said above, I don't see pluggable type providers decreasing
complexity related to macros in any way.
If macros get used (they might), the AST processing will be done at
compile time. Otherwise, it will be done at run-time.
- Macros are good for local, context-free rewrites on Scala ASTs that are possibly untyped and can be manipulated before type checking, and for generating boilerplate code at compile time which seamlessly integrates with the rest of the program (think type providers).
- Staging and virtualization are good for embedded DSLs that need modularity, sophisticated global analysis and compilation, computation at staging time, specialization to runtime data etc., in particular when the DSL expression trees do not exactly correspond to Scala trees.
The static/dynamic checking aspect is the other way round: Virtualization maintains type safety and relative evaluation order of expressions across the stage boundary, whereas macros allow free composition of untyped trees. Both is useful for different purposes.
Do the benefits of macros outweigh the cost of adding them to the language, in particular given the 'Scala is complex debate'? Nobody knows. The only way to find out is to go ahead and implement them ...
As with any new technology, there will certainly be many 'shiny new hammer, looking for nails' effects, so the challenge will be to provide guidance and empower users to pick the right tool for the job. Implementing an 'advanced' DSLs exclusively using macros is almost certainly not a good idea, as experience in other languages shows. In the end I believe both technologies can benefit each other: macros can help reduce some of the boilerplate that is currently needed to define staged DSLs and virtualization technology can help defining more powerful macros, e.g. infix_ methods from Scala-Virtualized would allow to redirect arbitrary method calls to macros, within a local scope.
Cheers,
- Tiark
Scala-virtualized is a research project, done at EPFL and Stanford. We
hope that some number of features will make it into main Scala.
Adriaan's pattern matcher looks like an excellent first candidate, and
other elements might follow. But there's no a priori intention to
migrate all elements of scala virtualized.
Macros have a shorter term horizon. There will be a SIP soon, and if
things go well we might see them soon in trunk, probably enabled by
some flag.
The intention of macros, and of Scala's language design in general, is
to simplify things. We have already managed to replace code lifting by
macros, and hopefully other features will follow. For instance, there
was a strong push to somehow eliminate the implicit parameter in
atomic { implicit transaction => ... }
and several other related situations. With macros this is trivial.
Without them, it requires a sophisticated dance with scoping rules.
Optimizations such as on Range.foreach are another use case. I believe
that in the long run macros can be a simplifying factor. So, in my
mind, there is enough evidence to try out the idea. But the
implementation is considered experimental at present, and we do
welcome critical discussion around the SIP once it appears.
Cheers
-- Martin
On Tuesday, 14. February 2012 at 19:09, Erik Osheim wrote:
That said, if the compiler API is stable, compiler plugins get moreadvanced, and the community is ready to embrace them, then I agree thatmaybe macros aren't needed.
I totally agree! I was just trying to be totally explicit (in the
context of Ivan's skepticism that macros are needed) about what I felt
like they added over compiler plugins, and why I feel like I need them.
-- Erik
It seems to me it's not entirely true that staging and macros don't overlap. For instance, you can achieve in template haskell (macros) what can be done in meta ocaml (staging). Similarity you can achieve with lisp/scheme macros what can be done with quote/unquote/eval.
But I agree that this discussion overlooks the fact that multi staged programming is just a special case of scala virtualized (if I understand correctly, the LMS paper is a special application of scala virtualized).
Who do you see as the target audience for macros?
a) regular rank-and-file developers looking to reduce repeated boiler
plate and create simple DSLs specific to their projects
b) hardcore library designers who in the absence of macros would have
resorted to compiler plugins for their advanced tricks, but will now use
macros for that purpose
If it's A, what is your view on the problems raised in the first 3
points of the original email that started this thread, namely having to
learn an arcane AST sub-language to create macros and the difficulty of
debugging macros?
This would seem to raise the threshold on the amount of boilerplate and
duplication a regular developer would tolerate before finally sitting
down to learn the vagaries of macros to reduce it.
If it's B, then what should the busy regular developer use for reducing
boiler plate and creating simple DSLs for their work, if macros remain
generally difficult for them to use?
I do appreciate your response regarding plans for inclusion of
scala-macros vs scala-virtualized, it clarifies the direction of Scala
somewhat.
I see macros as a useful middle road. Every language designer is faced
with a constant stream of suggestions to enlarge the language. And a
lot of these make sense. But following them would make the language
bigger and that's something I am very reluctant to do. If at all
possible, I'd like to make it smaller! Macros are a more stable and
systematic way to achieve more convenient syntax than compiler
plugins. That's why I think they are promising. The thing that scares
me deeply is that some people will inevitably go overboard with them
and misuse them.
It's a conundrum. Scala's philosophy is to make things possible, and
not erect walls against misuse. Trust the competency of your
programmers. But posts like this one make me realize the danger of
doing this:
http://yz.mit.edu/wp/true-scala-complexity/
It seems that everyone will at some point in their development as a
developer reach "peak abstraction",
http://lambda-the-ultimate.org/node/4442
and Scala's problem is that peak abstraction is so much higher (and
therefore more scary) than in other languages. The question I am
grappling with is, how can we get the advantages of macros without the
potential for misuse?
Cheers
-- Martin
--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967
Now that's interesting. At the time when I was researching scala-macros,
all the examples I could find were very similar to this one from the
Macros SIP:
class Queryable[T, Repr](query: Query) {
macro def filter(p: T => Boolean)(implicit ctx: CompilerContext):
Repr = {
val ast = Block()
val b = ctx.gensym(b, classOf[QueryableBuilder])
ast += Assign(b, Call(newBuilder, this))
ast += Call(query=, b, Create(classOf[Filter], Call(query, this),
reify(p)))
ast += Call(result, b)
ast
}
}
It even praises what a nice example of macros this is in the very next
sentence, which I barely read because my eyes were already bleeding. :)
Now I see in the Quasiquotations SIP this macro being rewritten as:
class Queryable[T, Repr](query: Query) {
macro def filter(p: T => Boolean): Repr = c
val b = $this.newBuilder
b.query = Filter($this.query, $reify(p))
b.result
}
This certainly looks much nicer, but I still don't really understand it
fully and the rest of the SIPs are not much help there.
Maybe it's just a matter of lack of documentation and communication
about what these shiny new things are and how to use them. Not so much
how they are implemented and how cool they are, but more about how a
busy developer who is not a Scala wizard can use them to solve practical
problems like refactoring, reducing boiler plate, and raising the
general level of abstractions.
On the other hand, I didn't have nearly as much trouble understanding
and following the scala-virtualized stuff. It felt easier and more
natural for someone who already understood existing Scala functionality
like implicits, pimping and operator overloading. It felt like a logical
extension of those concepts.
You can certainly dismiss me as not trying hard enough, but I am a Java
developer who really *really* likes the elegance of Scala and is trying
his best to learn about it in his very limited free time, and is even
already trying to use it in non-critical projects to see how it goes. I
think many developers are in a similar situation.
And for the record, I disagree 100% with the "Scala is too complex"
camp. I am not whining about complexity here, I am seriously trying to
learn how to use Scala to its fullest extent, but the cognitive load of
using advanced features is a very real factor in determining how much
boiler plate and code duplication one would tolerate before reaching for
an advanced Scala feature to eliminate it.
Oh absolutely! That is in fact the single biggest reason that drew me to
Scala. The level of abstraction achievable even with the current stable
version of Scala is astounding.
However there are still some warts here and there, such as the inability
to use things like == in DSLs and having to come up with ===, which I
hoped to be able to get around using something like scala-virtualized
that's relatively easy to use.
Still, I appreciate the difficulty faced by language designers and the
various trade offs that have to be made.
Macros are a more stable and
systematic way to achieve more convenient syntax than compiler
plugins. That's why I think they are promising. The thing that scares
me deeply is that some people will inevitably go overboard with them
and misuse them.
The question I am
grappling with is, how can we get the advantages of macros without the
potential for misuse?
In the end I think it would be more reasonable to go ahead and release 2.10 without macros (or behind a compiler switch like it currently is) and officially ship macros when they are ready. (Ready == "there is a real library solving a real problem using it, which can be recommended to developers without having to worry")
The name "macro" is a 100% guarantee for a marketing nightmare, so it shouldn't be made worse by shipping it without any substantial example which can be explained to developers.
--Rex
On Wed, Feb 15, 2012 at 3:25 PM, martin odersky <martin....@epfl.ch> wrote:Macros are a more stable and
systematic way to achieve more convenient syntax than compiler
plugins. That's why I think they are promising. The thing that scares
me deeply is that some people will inevitably go overboard with them
and misuse them.
Code generators are even more scary, though, but right now the options are sometimes
1. Have code that runs 30x slower than C++
2. Repeat yourself a gazillion times
3. Write a code generator
But picking (3) leaves one with generator code like
It's 2012. You can write code generators like this:On Feb 15, 2012, at 9:57 PM, Rex Kerr wrote:On Wed, Feb 15, 2012 at 3:25 PM, martin odersky <martin....@epfl.ch> wrote:
Macros are a more stable and
systematic way to achieve more convenient syntax than compiler
plugins. That's why I think they are promising. The thing that scares
me deeply is that some people will inevitably go overboard with them
and misuse them.
Code generators are even more scary, though, but right now the options are sometimes
1. Have code that runs 30x slower than C++
2. Repeat yourself a gazillion times
3. Write a code generator
But picking (3) leaves one with generator code likeval distances = Stream[Double](data.numRows, data.numRows){ (i,j) => dist(data(i),data(j)) }val densities = DenseVector[Int](data.numRows, true)
for (row <- distances.rows) {if(densities(row.index) == 0) {
val neighbors = row find { _ < apprxWidth }densities(neighbors) = row count { _ < kernelWidth }
}}and emit Scala code that runs as fast as C [1]- Tiark
I don't really see the issue with having 2 or more projects that tackle a similar problem or set of problems. More projects equals more options and that's a good thing in my opinion.
Am Mittwoch, 15. Februar 2012, 22:16:37 schrieb Simon Ochsenreither:
> The name "macro" is a 100% guarantee for a marketing nightmare, so it
> shouldn't be made worse by shipping it without any substantial example
> which can be explained to developers.
I can already imagine the headlines:
"after the longest suicide note in history scala has finally managed to fully erupt in insanity by adding #MACROS"
Well... to be honest - the first thing I thought when I heard "#MACROS" was "Oh dear - they won't do that - will they?". Now I know that we are talking about some substantially different things than those macros I was thinking about (flashback in the 80's... shudder).
But to me "macro" surely has a real negative bias. And I think I am not alone. So maybe its really worth thinking about names here. It won't be the first time in history where something is shipped with a label that's not as clear and technical as it could be (but most often for good reason - marketing is also a part of a product).
Greetings
Bernd
I think “macro” is fine, personally. Yes there is some bad connotations associated with the word in some communities, but then there’s Lisp (Scheme, Clojure) and other languages that use the word to mean something quite different than what it means to C/C++ people. We would just be joining the ranks of those other languages. No problem.
Peter
2012/2/18 √iktor Ҡlang <viktor...@gmail.com>:
--
"source transformers"?On Sat, Feb 18, 2012 at 11:09 PM, Chapin, Peter @ VTC <PCh...@vtc.vsc.edu> wrote:
I think “macro” is fine, personally. Yes there is some bad connotations associated with the word in some communities, but then there’s Lisp (Scheme, Clojure) and other languages that use the word to mean something quite different than what it means to C/C++ people. We would just be joining the ranks of those other languages. No problem.
Peter
From: scala-...@googlegroups.com [mailto:scala-...@googlegroups.com] On Behalf Of Bernd Johannes
Sent: Saturday, February 18, 2012 16:16
To: scala-...@googlegroups.com
Cc: Simon Ochsenreither
Subject: Re: [scala-debate] scala-macros vs scala-virtualized
Am Mittwoch, 15. Februar 2012, 22:16:37 schrieb Simon Ochsenreither:
> The name "macro" is a 100% guarantee for a marketing nightmare, so it
> shouldn't be made worse by shipping it without any substantial example
> which can be explained to developers.
I can already imagine the headlines:
"after the longest suicide note in history scala has finally managed to fully erupt in insanity by adding #MACROS"
Well... to be honest - the first thing I thought when I heard "#MACROS" was "Oh dear - they won't do that - will they?". Now I know that we are talking about some substantially different things than those macros I was thinking about (flashback in the 80's... shudder).
But to me "macro" surely has a real negative bias. And I think I am not alone. So maybe its really worth thinking about names here. It won't be the first time in history where something is shipped with a label that's not as
--
“type loaders and language-quotations.” (http://confluence.jetbrains.net/display/Kotlin/FAQ)
Maybe the name could focus more on the benefit for users and less on the implementation details.
So is that a vote for transmogrifier [1] then? ;)
Sorry, couldn't resist.
-- Erik
Don't bend over trying to appease people who will judge you based on
their misconceptions. If they decide to misjudge it, THEY will call it
macros, no matter name we chose.
But there is another issue, probably a more important one. A name like "macro" has the potential to easily scare away many people who simply don't know better, people who don't have much of an impression of scala yet and will now have a negative one simply due to misunderstanding.
I like the term "typesafe macros" which implies more magikz and awesomez than just "macro"
Which is a bit ridiculous, since they are pre-processor directives.
C++ templates are much more like real macros.
I like the term "typesafe macros" which implies more magikz and awesomez than just "macro"
"typesafe macros" makes me immediately think of things like meta ocaml or lightweight multi staged scala: macros that will be executed at runtime but for which we have a proof they will generate well-typed (and valid) code. This is no surprise compile time macros are typesafe: they just generate code that will be fed to the typechecker.
That's right. But since we're in debate I can add noise...
The point was raised that FUD preacher will call it "macro" no matter how scala is calling it.
But most of the people (supposed to be initially interested) will ask themself "why don't they call it 'macro' if it is a macro?" and will try to find an answer.
Based on their prior knowledge they will either come to the conclusion: "well, it's about macros" because they expect macros to be the scala-way. Then they won't misjudge macros because they don't mistake macros for text preprocessing.
But if they adhere to the concept of macros to be "source text mangling" they will discover (for good) that scala macros are something else entirely.
So we will have a fraction which complaints about "old wine in new flasks" - which is not too bad. And we have a fraction which is NOT scared away by a prejudice without revalidation (which most often won't happen).
And we don't have to write another ton of block posts "why scala macros are no c-style macros"... which will receive the responses "why scala does everything different just to bewilder everyone".
Just my 5 cents
Bernd
-------- Original-Nachricht --------
> Datum: Fri, 24 Feb 2012 09:16:02 +0100
> Von: Bernd Johannes <bjoh...@bacon.de>
> An: scala-...@googlegroups.com
> CC: Daniel Sobral <dcso...@gmail.com>, "Cédric Beust ♔" <ced...@beust.com>, Naftoli Gugenheim <nafto...@gmail.com>
> Betreff: Re: [scala-debate] scala-macros vs scala-virtualized
--
Surely you meant hygienic macros? :-)
-------- Original-Nachricht --------
> Datum: Fri, 24 Feb 2012 08:35:56 -0200
> Von: Daniel Sobral <dcso...@gmail.com>
> An: Josh Suereth <joshua....@gmail.com>
> CC: Naftoli Gugenheim <nafto...@gmail.com>, "scala-...@googlegroups.com" <scala-...@googlegroups.com>, "√iktor Ҡlang" <viktor...@gmail.com>, "Chapin, Peter @ VTC" <PCh...@vtc.vsc.edu>, Lukas Rytz <lukas...@epfl.ch>
> Betreff: Re: [scala-debate] scala-macros vs scala-virtualized
Expression Trees are a runtime thing though.
While on the topic of C# expressions. I really like to have the feature available but in my experience they have one major flaw (besides being severely limited) that contaminates the entire language in that they look like ordinary code. This makes it easy to fall in the trap of assuming ordinary semantics and then being surprised when it's not. Unfortunately it means that you must be suspicious of any code you look at which creates somewhat of a mental burden.
Now Scala allready have this problem with its uniform and free syntax but adding this feature to the mix can potentially make the issue worse. It's like the operator overloading problem on steroids.
So some thought should be directed at mitigating this problem when introducing the feature. Either som syntactic delimiter, or convention, to signal "here be dragons", or maybe some IDE feature to highlight such code, maybe change font or letter spacing.
BR,
John
Expression Trees are a runtime thing though.
While on the topic of C# expressions. I really like to have the feature available but in my experience they have one major flaw (besides being severely limited) that contaminates the entire language in that they look like ordinary code. This makes it easy to fall in the trap of assuming ordinary semantics and then being surprised when it's not. Unfortunately it means that you must be suspicious of any code you look at which creates somewhat of a mental burden.
Now Scala allready have this problem with its uniform and free syntax but adding this feature to the mix can potentially make the issue worse. It's like the operator overloading problem on steroids.
So some thought should be directed at mitigating this problem when introducing the feature. Either som syntactic delimiter, or convention, to signal "here be dragons", or maybe some IDE feature to highlight such code, maybe change font or letter spacing.
CATs! Nobody could object to Scala supporting cats, right?
Cheers,
Mirko
--
Mirko Stocker | m...@misto.ch
Work: http://ifs.hsr.ch | http://infoq.com
Personal: http://misto.ch | http://twitter.com/m_st
That gives a whole new meaning to
http://spl.smugmug.com/Humor/Lambdacats/catt/960526154_fqpKB-O-1.jpg
-sz
John, where did our LOLCODE Scala combinator parser go?
On Wed, Feb 15, 2012 at 9:00 PM, Ivan Todoroski <grnch...@gmx.net> wrote:
> Hi Martin,
>
> Who do you see as the target audience for macros?
>
> a) regular rank-and-file developers looking to reduce repeated boiler plate
> and create simple DSLs specific to their projects
>
> b) hardcore library designers who in the absence of macros would have
> resorted to compiler plugins for their advanced tricks, but will now use
> macros for that purpose
>
>
> If it's A, what is your view on the problems raised in the first 3 points of
> the original email that started this thread, namely having to learn an
> arcane AST sub-language to create macros and the difficulty of debugging
> macros?
>
> This would seem to raise the threshold on the amount of boilerplate and
> duplication a regular developer would tolerate before finally sitting down
> to learn the vagaries of macros to reduce it.
>
>
> If it's B, then what should the busy regular developer use for reducing
> boiler plate and creating simple DSLs for their work, if macros remain
> generally difficult for them to use?
>
I think it's rather B. And, Scala already has a lot of mechanisms that
reduce boilerplate even without resorting to macros and scala
virtualized. So I am not at all concerned that the bar for using
either is high. In fact, I'd prefer it that way.I see macros as a useful middle road. Every language designer is faced
with a constant stream of suggestions to enlarge the language. And a
lot of these make sense. But following them would make the language
bigger and that's something I am very reluctant to do. If at all
possible, I'd like to make it smaller! Macros are a more stable and
systematic way to achieve more convenient syntax than compiler
plugins. That's why I think they are promising. The thing that scares
me deeply is that some people will inevitably go overboard with them
and misuse them.It's a conundrum. Scala's philosophy is to make things possible, and
not erect walls against misuse. Trust the competency of your
programmers. But posts like this one make me realize the danger of
doing this:http://yz.mit.edu/wp/true-scala-complexity/
It seems that everyone will at some point in their development as a
developer reach "peak abstraction",http://lambda-the-ultimate.org/node/4442
and Scala's problem is that peak abstraction is so much higher (and
therefore more scary) than in other languages. The question I am
grappling with is, how can we get the advantages of macros without the
potential for misuse?
Cheers
-- Martin
>
> I do appreciate your response regarding plans for inclusion of scala-macros
> vs scala-virtualized, it clarifies the direction of Scala somewhat.
>
>
>
>
> On 14.02.2012 21:56, martin odersky wrote:
>>
>> Some perspective: First, scala-virtualized and Scala macros are both
>> experimental at the present stage. The discussion on this thread has
>> already worked out the main points of differences, so I won't go into
>> that.
>>
>> Scala-virtualized is a research project, done at EPFL and Stanford. We
>> hope that some number of features will make it into main Scala.
>> Adriaan's pattern matcher looks like an excellent first candidate, and
>> other elements might follow. But there's no a priori intention to
>> migrate all elements of scala virtualized.
>>
>> Macros have a shorter term horizon. There will be a SIP soon, and if
>> things go well we might see them soon in trunk, probably enabled by
>> some flag.
>>
>> The intention of macros, and of Scala's language design in general, is
>> to simplify things. We have already managed to replace code lifting by
>> macros, and hopefully other features will follow. For instance, there
>> was a strong push to somehow eliminate the implicit parameter in
>>
>> atomic { implicit transaction => ... }
>>
>> and several other related situations. With macros this is trivial.
>> Without them, it requires a sophisticated dance with scoping rules.
>> Optimizations such as on Range.foreach are another use case. I believe
>> that in the long run macros can be a simplifying factor. So, in my
>> mind, there is enough evidence to try out the idea. But the
>> implementation is considered experimental at present, and we do
>> welcome critical discussion around the SIP once it appears.
>>
>> Cheers
>>
>> -- Martin
>>
>--
Martin Odersky
Prof., EPFL and Chairman, Typesafe
PSED, 1015 Lausanne, Switzerland
Tel. EPFL: +41 21 693 6863
Tel. Typesafe: +41 21 691 4967