implementing macros for Scala, looking for feedback!

[Eugene Burmako]

Hi folks! My name is Eugene Burmako. I'm a first-year PhD student from Martin's lab at EPFL. During the first year of studies all doctoral students here at EPFL have to carry out a couple of semester projects, and, of course, my projects will involve something Scala-related.

The idea, which I like the most at the moment, is to implement a metaprogramming facility for Scala. By that I mean a full-fledged system with quasiquotations and hygienic compile-time macros. My current inspiration is the macro system of Nemerle (http://nemerle.org/metaprogramming.ps). Sure, it's a huge amount of work, so I don't expect finishing all that stuff soon, but it can be done gradually.

This macro project somewhat intersects with another research direction of our lab, namely, with lightweight modular staging (http://www.stanford.edu/class/cs442/lectures_unrestricted/cs442-lms.pdf). However, despite of being applicable to similar problems, both LMS and macros each have distinctive scenarios where they really shine. That's why we decided that it'd be practical to explore both of these approaches to metaprogramming.

Having said that, I'd like to find compelling use-cases for macros, so that the functionality implemented will be immediately useful. Of course, macros are very elegant in themselves, and, from the standpoint of a language designer, I can name at least a handful of cool features that can be enabled by macros. However, what I'd really like to know is what scenarios are the most important to you in your everyday jobs. I would be very glad to receive your feedback!

P.S. The usual disclaimer applies here. No promises/commitments so far, it's just a research project. However, if it deems useful, I think, it might end up in trunk in some shape or another.

[Roland Kuhn]

Hi Eugene,

this will probably be interesting for a lot of people, glad you tackle it! One little thing I find myself writing very often is a sealed trait hierarchy aka “the better Enum”:

sealed trait State

case object Idle extends State

case object Active extends State

sealed trait Message

case object Kill extends Message

case class DoIt(whatever: String) extends Message

It would be really great if the signal to noise ratio could be reduced: in the above only 6 out of 28 words are actually interesting. If my understanding of macros matches your proposal, this could be reduced to something like

Enum(State, Idle, Active)

Enum(Message, Kill, class DoIt(whatever: String))

(if I did not completely miss the scope of macros from a very cursory glance over the paper.)

Regards,

Roland

Roland Kuhn

Typesafe – Enterprise-Grade Scala from the Experts

twitter: @rolandkuhn

[Debasish Ghosh]

Hi Eugene -

Thanks for giving time towards macros in Scala. Despite having a rich type system, I still think there are valid use cases for compile time macros in Scala. Two such cases come straight to my mind :

1. code generation. No language is boilerplate free and there occurs situations where you feel the necessity to generate code. In one of my recent projects I was writing a Redis client in Scala. Besides the basic client I also implemented a Redis Cluster, which to the end user is also yet another client. The user invokes same operations on the cluster as on a single connection. Most operations are a simple delegate to the appropriate shard while some operations need additional logic. Here I felt the necessity of macros in Scala which would have generated lots of code that I needed to write manually.

2. Inlining code is a good optimization. But often even though the user wants to inline, the compiler would not. Or even the user is not aware of an inline optimization. In such cases if he implements using a macro, it will be more efficient as the inlining is guaranteed.

Will let u know more such use cases as they come up.

Thanks.

--
Debasish Ghosh
http://manning.com/ghosh

Twttr: @debasishg
Blog: http://debasishg.blogspot.com
Code: http://github.com/debasishg

[missingfaktor]

data Message(header: String, contents: String) deriving (Equal, Show)

could expand to:

case class Message(header: String, contents: String)

object Message extends ((String, String) => Message) {

  implicit def messageShow: Show[Message] = showA

  implicit val messageEqual: Equal[Message] = equalA

  def header = (_: Message).header

  def contents = (_: Message).contents

}

And maybe lenses too.

--

Cheers,

missingfaktor.

When you stand for what you believe in, you can change the world.

[Daniel Sobral]

+1 on both. I'd be mentioning them if no one had done so!

I'll put forward conditional compilation based on Scala version. This
is something that has direct impact on 3rd party libraries. Of course,
it wouldn't help them with code *pre*-macros, so it wouldn't be of
immediate use.

--
Daniel C. Sobral

I travel to the future all the time.

[Philippe Lhoste]

On 17/09/2011 13:13, Eugene Burmako wrote:
> The idea, which I like the most at the moment, is to implement a
> metaprogramming facility for Scala. By that I mean a full-fledged system
> with quasiquotations and hygienic compile-time macros. My current
> inspiration is the macro system of Nemerle
> (http://nemerle.org/metaprogramming.ps). Sure, it's a huge amount of
> work, so I don't expect finishing all that stuff soon, but it can be
> done gradually.

Excellent project, taking inspiration from an excellent (from what I
have read) source.

> Having said that, I'd like to find compelling use-cases for macros, so
> that the functionality implemented will be immediately useful. Of
> course, macros are very elegant in themselves, and, from the standpoint
> of a language designer, I can name at least a handful of cool features
> that can be enabled by macros. However, what I'd really like to know is
> what scenarios are the most important to you in your everyday jobs. I
> would be very glad to receive your feedback!

One obvious point it in Scala itself, to avoid generating
Function1.scala to Function22.scala, idem for Product<nn>.scala and
Tuple<nn>.scala

Enum is another good case, eg. to automatically generate an index number
from a set of enums. I made a workaround by putting the case object
names in a list, but I dislike duplicating these names.

I can see them useful to make DSLs, too.
There is a little intersection with partial functions, for example, but
still I think it can be useful there.

--
Philippe Lhoste
-- (near) Paris -- France
-- http://Phi.Lho.free.fr
-- -- -- -- -- -- -- -- -- -- -- -- -- --

[Ken McDonald]

Just as a side note, there should ideally be something in the syntax of macro invocations that makes them easily distinguishable from standard Scala calls. Scala is already so semantically rich that adding in another feature that _looks_ like Scala calls could really muddy the waters when it came to understanding code.

Ideally this should be done by syntax-aware editors, but they're still struggling to get up to Scala's current level, so I think it's safe to assume they wouldn't highlight macros for quite some time.

Cheers,

Ken

[Daniel Sobral]

That assumption is not really supported. Eclipse's ScalaIDE 2.0 used
*the compiler* to figure out Scala code. It, therefore, knows as much
as the compiler does. Its problems lie elsewhere.

It would be true of IDEA, presumably, but we'll have to wait and see.
It's not like such a feature could hit trunk tomorrow.

[William la Forge]

My code never quite seems to fit the language I'm programming into and I always end up with too much boilerplate, even with a language as rich as Scala. Macros would help a lot.

I'm always creating factory classes which contain meta data about another class, with possibly multiple factories (supporting different uses) per class. Add to this I do aggregates of classes and I often end up with 3 or more classes of boilerplate.

Bill

On Sat, Sep 17, 2011 at 4:43 PM, Eugene Burmako <eugene....@epfl.ch> wrote:

[Eric Kolotyluk]

When I first learned IBM 360 Assembler macros were a critical function
of the assembler because you were so close to the machine. You could
even write complete programs in the macro language itself. Eventually
when I was an undergraduate in the late 70's the conventional wisdom
seemed to be that Algol 68 was not much of a success, and people were
hinting that even macros were not such a great idea. In the 80's one of
my colleagues had done his master's thesis on extensibility in
programming languages, and his conclusion was that it was a bad idea. In
early 90's when I went back to grad school the conventional wisdom still
seemed to be extensibility was not such a great idea, and proponents of
Java seemed to believe that macros were such a failure in C & C++ that
Java was much better off without them.

These days I am able to see language extensibility working in Scala, and
it actually seems to work, but not in the way I would have dreamed of
even 15 years ago.

Now we are talking about macros again, and I naturally wonder about such
things as how do you know when there is an error in the macro vs. the
program, or how does the debugger work properly. I'm not saying macros
are a bad idea, but what has fundamentally changed to make people so
positive about them now? Is it that computing science has advanced to
the point where where now know how to do them better?

Cheers, Eric

[Philippe Lhoste]

On 18/09/2011 03:36, Eric Kolotyluk wrote:
> early 90's when I went back to grad school the conventional wisdom still
> seemed to be extensibility was not such a great idea, and proponents of
> Java seemed to believe that macros were such a failure in C & C++ that
> Java was much better off without them.

It is a failure in C because it is unsafe, disregards types, can lead to
incorrect code (eg. #define M x*x instead of (x*x)) and so on.
Not sure about macros in C++, as I never used them.

> I'm not saying macros
> are a bad idea, but what has fundamentally changed to make people so
> positive about them now? Is it that computing science has advanced to
> the point where where now know how to do them better?

You mentioned only a part of computer history, but to my knowledge,
macros in Lisp (and derivatives) and some other languages, so called
hygienic macros, are safe to use and are seen as powerful features. And
it goes back far in the said history.

There might be downsides to the use of macros, perhaps debugging,
readability, I don't really know (I have yet to use a PL with good macro
support, but I like the idea).
I believe that macro support would strongly reduce the need for compiler
plugins, which need to be installed in your Scala distribution and thus
are harder to use in a project.

Scala might need macros less than some other languages (eg. you can
define a repeat(n) partial function, while other languages would have
used a macro to make new structures like that) but I believe it can
still be useful. I might be interesting to take a look at the existing
compiler plugins and see if a macro system can really replace them (or
simplify them).
Macros are about manipulating the AST, to generate new code.

[Philippe Lhoste]

On 17/09/2011 13:13, Eugene Burmako wrote:

> Having said that, I'd like to find compelling use-cases for macros, so
> that the functionality implemented will be immediately useful.

Another interesting feature I though of:
Parse a format string and check it at compile time, instead of run-time.
We then go a step ahead in type safeness.

Ie., define format as a macro, and verify that in:
format("%s value is %d", name, name.value)
name is really a string, and name.value is an int.
And generate directly the StringBuilder stuff, instead of relying in
runtime code walking the string in search of %

Idem for a PHP-like templating system:
"Foo $name bar ${x map (_.toUpperCase)}" or similar: compile time check,
optimized code.
Idem for regular expressions.

Along with Scala's multiline strings, it can be powerful and efficient.

[William la Forge]

Very fond I was of IBM 360 Assembler macros. You could effectively write your own language, it was so powerful.

Bill La Forge

[Alex Repain]

2011/9/18 Philippe Lhoste <Phi...@gmx.net>

On 17/09/2011 13:13, Eugene Burmako wrote:

Having said that, I'd like to find compelling use-cases for macros, so
that the functionality implemented will be immediately useful.

Another interesting feature I though of:
Parse a format string and check it at compile time, instead of run-time. We then go a step ahead in type safeness.

I would add we also go a step ahead in performances. Format strings are about 100 times slower than StringBuilders, even used implicitly in concatenated stuff (in 2.8.0, maybe not anymore)

Ie., define format as a macro, and verify that in:
format("%s value is %d", name, name.value)
name is really a string, and name.value is an int.
And generate directly the StringBuilder stuff, instead of relying in runtime code walking the string in search of %

Idem for a PHP-like templating system:
"Foo $name bar ${x map (_.toUpperCase)}" or similar: compile time check, optimized code.
Idem for regular expressions.

Along with Scala's multiline strings, it can be powerful and efficient.

--
Philippe Lhoste
--  (near) Paris -- France
--  http://Phi.Lho.free.fr
--  --  --  --  --  --  --  --  --  --  --  --  --  --

--
Alex REPAIN
ENSEIRB-MATMECA - student
TECHNICOLOR R&D - intern
BORDEAUX I      - master's student
SCALA           - enthusiast

[Eugene Burmako]

Thanks for all the feedback - please, keep it going!

@All. I'm not really sure whether it will be possible to implement
macros that change syntax (I have yet to dig into scalac parser to
find that out), though I'd really like to implement unrestricted code
generation (i.e. to support not only expressions, but also class/
method/implicit/whatever definitions).

@Daniel. Conditional compilation for different Scala versions is a
brilliant idea!

@Eric. Well, error reporting and debugging are not that bad. There are
ways to retain most of the debug info, and there are ways to keep
macros hygienic and type-safe. You can find more info on that in my
today's blog (see the link below).

@Ken. As I see that, there is a trade-off that involves transparency
of macro invocations. Either we have transparent but not first-class
macros (as it's done in Nemerle), or we have first-class macros that
require explicit triggers (as it's done in Template Haskell). Both
approaches have their merits, so it's not an easy choice.

@Philippe, Alex. Yep, string interpolation, along with Perl-like regex
matching, is near the top of my favorite macro applications in the
small. The top being syntactic sugar for computation expressions. I
always loved the concept of monads, but was scared away by the amount
of boilerplate required to use them in most mainstream languages.

This very semester I will, most likely, go for quasiquotations that
will bring introspection capabilities of Scala to a new level (kinda
what LINQ and expression trees did to .NET, but even better). In the
meanwhile, I'll be posting about the progress in my blog: http://blog-en.xeno.by.
Here's the today's installation: http://xeno-by.blogspot.com/2011/09/macros-state-of-art.html.

[Rex Kerr]

(1) I waste a lot of time on library pimping to add a method:

class XPimpsY[A](a: A) {
  def doSomething[B](b: B) = { ... }
}
implicit def y_to_x[A](y: Y[A]) = new XPimpsY[A]

when all I really want to write is doSomething[A,B](y: Y[A], b: B) = {...}

If Scala gets some independent lightweight extension capability this would be a moot point, but otherwise being able to macro this would be my most important use case.

(2) Using the ordinary PML pattern for collections is often arduous because one needs to write in highly generic CanBuildFroms and pass in a pile of implicits.  If I just want to do something like

   xs.filter(p).map(f).filter(q)

over and over again, it would be nice to be able to specify that xs.fmf(p)(f)(q) expanded to the above, and let the compiler worry about whether that was possible given the types of xs and f, not me when I'm writing the implicit conversions and such.

(3) The type system doesn't make it easy to deal in semi-programmatic ways with primitives.  For example, if you're doing vector math, you might want
  longVector + longVector => longVector
  floatVector + longVector => doubleVector
  floatVector + floatVector => floatVector
and you might want your type system to know this.  Actually telling the type system about this is painful, because you have to duplicate code for the addition each time.  Given how the JVM works, the code _must_ be duplicated (it's different bytecode each time), but if a macroing system could do the work instead of me, it would save a lot of time and potential for error.

(4) For loops are still painfully slow for certain operations; defining equivalent methods only gives equivalent performance if multiple dispatch can be avoided (which it often can't e.g. when nested).  If macroing could, as in the Nemerle example, allow
  For(i = 0; i<N; i+=1) { a(i) = b(i+1)-b(i) }
to be converted into the appropriate while loop (with variable hiding)
  {
    var i = 0
    while (i < N) {
      a(i) = b(i+1) - b(i)
      i += 1
    }
  }
it would save quite a bit of typing and potential for error in high-performance code.  This is exactly the sort of thing one can make DSLs for, but when it comes to primitives, the speed usually leaves much to be desired.

  --Rex

[etorreborre]

> I'll put forward conditional compilation based on Scala version.

I think that would still be helpful in the future. For example to avoid deprecation warnings like error/sys.error:

•  error was deprecated in 2.9.0 in favor of sys.error
  
• if you want to do a cross-compilation with 2.8.1 and 2.9.0 you had to put up with all the warnings.

Eric.

[Daniel Sobral]

Yes, but, to illustrate my point, say macros were introduced in 2.9.
You wouldn't be able to use them to avoid this problem, because 2.8
wouldn't accept the macro.

[etorreborre]

Right, I got that but I'm thinking that this may still be useful in the future to absorb deprecation cycles. I'd be surprised if there weren't any :-).

E.

[Daniel Sobral]

Oh, I *hope* it will be useful for that! I know Odersky wants to put
breaks on wild changes -- and that's good -- but there's still much to
be improved on Scala.

[Adam Jorgensen]

+1 to this.

I think it would be nice and I imagine it's quite possible to avoid the pitfalls of the C macro system.

A totally random and possibly incorrect thought: Macros in C are fairly comparable to a Templating System in a modern Web App

For applications that want to perform some scaffolding at run-time and use the output this might also be useful?

[nicola...@gmail.com]

I think macros are a great idea. I used to code a bit in LISP and find miss them really much.

I used them for DSLs.  (My particular case was to read generalised chemical reactions and output 2 functions : one to compute the activity

of the reaction and another to execute the reaction when it is triggered)  

Contrary to one of the preceding mail, I think it should be a similar syntax for macro call and function call. However, it can be useful to have some of its arguments parsed with another grammar than the scala grammar. (I don't know how hard it is)

For example, with the classical example of a Prolog-like rules as macros, it would be easier to parse it as a list than as a scala AST, and then deconstruct the scala AST to its list meaning. Some DSL might not be even writable as syntictally correct scala. 

So suggestion 1: At macro definition: annotate the arguments of the macro by how it is to be parsed. (I know it is probably difficult because of compiler phases, but one of the strength

 of macros in LISP comes from the fact than LISP syntax is very generic: everything that is well bracketed is a LISP term and can be an argument to a macro)

There is also some interesting relation between macros and typing. Depending of the position where the macro is called, you might have a type (or part of a type) to push into the macro. (Then again, compiler phase issue...). So suggestion 2: allowing some macros to request an access to type information: getting (partial) informations about the type they must return.

(or even type of a subterm?)

Last, I think hygienic macros are certainly a good default, but with macros, it can be good to allow things that are really hackish and inelegant. 

So making sure there is a way of any term generating program is probably a good idea too.

I hope you will do his project and have a lot of fun.

Best regards,

Nicolas.

[Alan Burlison]

On 09/19/11 06:01, Adam Jorgensen wrote:

> A totally random and possibly incorrect thought: Macros in C are fairly
> comparable to a Templating System in a modern Web App

Umm, no they aren't. C macros provide static compile-time textual
substitution, templating systems provide run-time dynamic output formatting.

I'm not at all convinced that macros are a good idea for Scala to be
honest, I certainly don't miss them in Java. However, that's probably
due to the scars they have inflicted on me in C/C++, so perhaps
something less lethal can be devised for Scala.

--
Alan Burlison
--

[Philippe Lhoste]

On 19/09/2011 09:22, nicola...@gmail.com wrote:
> Contrary to one of the preceding mail, I think it should be a similar syntax for macro
> call and function call. However, it can be useful to have some of its arguments parsed
> with another grammar than the scala grammar. (I don't know how hard it is)

I am not sure how to conflate the two sentences.
In a number of languages using (hygienic) macros I looked at (Nermele, Seed7, some others
I don't recall), macro usage doesn't look like a function call in general, but rather
allows to use them to make some kind of DSL, eg. to make new operators, new loop
structures, etc.

Now, perhaps one problem is to stumble upon such macro in some code and to be stumped
because it might precisely not look like the regular language, but offers no hints it uses
some macro defined somewhere...

[√iktor Ҡlang]

Fwd to list

---------- Forwarded message ----------
From: "√iktor Ҡlang" <viktor...@gmail.com>
Date: Sep 19, 2011 11:39 AM
Subject: Re: [scala-user] implementing macros for Scala, looking for feedback!
To: "Alan Burlison" <alan.b...@gmail.com>

Tbh I'd much rather have unification of tuples and parameter lists, abstraction over arity and abstraction over constructors.

Cheers,
V

[Adam Jorgensen]

Thanks for the correction, although I guess what I really was trying to say both are basically textual substitution systems. It seems that

the macro systems in LISP, Scheme, etc are a bit less braindead and hence might actually be a neat idea in Scala :-)

[Tony Morris]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Please see Template Haskell. I would hate for us to repeat mistakes out
of ignorance of lessons already learned.

(not directed at any comment in particular)

- --
Tony Morris
http://tmorris.net/

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOdzBcAAoJEPxHMY3rBz0PIjAH/0YwIH3b+CHtjRyWvC5xg2FP
cvL2rTwELj52tHDy5e7+4/1um2nmy1T6xTdaDF+T3UdofKRnoAYDK3tUKpX9Tv/j
m5er0Te6rqR+bw3x0B2wdZEpCwXuMlaNWnfn74uONs7hpB1EMkx0h6BmHtKYJtdN
iS3bP8hgrMcThchK+VBmWjoJkup3doulbttEZSwa2w4gNc+T6XwSIDQoJ+DOYdgA
ESrJu27Vc5rrctVb4De2iuspucp/GlHLq+rgbCie7EfQWkXMzB1wD0vuN9MHB1mk
gqPphx0HKtrsNPwDIk0yxj+q08v2yfPybdf0J3b2MKJBdgRu+7rAzzsAhIFCb0Y=
=rkgK
-----END PGP SIGNATURE-----

[nicola...@gmail.com]

I am not sure how to conflate the two sentences.
In a number of languages using (hygienic) macros I looked at (Nermele, Seed7, some others I don't recall), macro usage doesn't look like a function call in general, but rather allows to use them to make some kind of DSL, eg. to make new operators, new loop structures, etc.

Now, perhaps one problem is to stumble upon such macro in some code and to be stumped because it might precisely not look like the regular language, but offers no hints it uses some macro defined somewhere...

I am not sure about other languages, but from my macro in LISP experience, macro call looks like a function call, except that the arguments

do not have to make sense in the language itself.

eg: In a Lisp, you can define a macro prolog, that takes a series of rule and compile them to some data structures representing the code or some program to solve the system.

This would look like:

(prolog  (Foo (X) <-  Bar(x))

             (Bar(foo) <-))

Here, the call looks like a function call but the argument does not look like an argument written in Lisp.

The macro will parse the list and makes sense of it. It surely is easier to parse a list than scala AST, when the argument is not a scala program.

Hopes tat makes my point clearer,

Nicolas.

 

[nicola...@gmail.com]

Please see Template Haskell. 

I +1 this.  If I remember correctly, there was some interesting relation to compiler phases, with some interleaving between templating

and typing, which might be of interest here.

[Daniel Sobral]

On Mon, Sep 19, 2011 at 07:42, Philippe Lhoste <Phi...@gmx.net> wrote:
> On 19/09/2011 09:22, nicola...@gmail.com wrote:
>>
>> Contrary to one of the preceding mail, I think it should be a similar
>> syntax for macro
>> call and function call. However, it can be useful to have some of its
>> arguments parsed
>> with another grammar than the scala grammar. (I don't know how hard it is)
>
> I am not sure how to conflate the two sentences.
> In a number of languages using (hygienic) macros I looked at (Nermele,
> Seed7, some others I don't recall), macro usage doesn't look like a function
> call in general, but rather allows to use them to make some kind of DSL, eg.
> to make new operators, new loop structures, etc.

On Nemerle macros normally appear as either functional calls or
annotations -- which seems ideal for Scala. They can _also_ add
syntax, by defining how the parameters are separated by literal
tokens, but I doubt that can be achieved.

See http://nemerle.org/macros.html. One of the first examples show the
macro being used function-call like, and the first for-loop example
shows that as well. And, then, there's the serializable example, which
is interesting because it shows the (same) macro being inserted at two
points in the compiler phases, first to modify the type, and, then, to
add a method definition when all types are already settled.

>
> Now, perhaps one problem is to stumble upon such macro in some code and to
> be stumped because it might precisely not look like the regular language,
> but offers no hints it uses some macro defined somewhere...
>
> --
> Philippe Lhoste
> --  (near) Paris -- France
> --  http://Phi.Lho.free.fr
> --  --  --  --  --  --  --  --  --  --  --  --  --  --
>
>

--

[Arno Haase]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Love to see you tackle this - I have been wanting this for a long time!

Usage examples:

* adding change notification support to classes

* 'cached' semantics for defs - store each result in some sort of cache
(that can be passed in as an additional implicit parameter) and evaluate
the def exactly once for every combination of parameters

* the @beanGetter / @beanSetter stuff from the compiler itself

* DSLs, especially for interop with typical Java frameworks, require
code to be generated rather than wrapping stuff at runtime

- - Arno

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAk53XRoACgkQbmZsMyUPuXT+PQCg7t4zqqy1JKUCRGZ0pImInKOY
tIYAmwSGToKU9o32HiI/SnZ7iwUQu4vN
=wGXv
-----END PGP SIGNATURE-----

[Joshua Gooding]

I have been occasionally than king that that maybe there is a
sufficientately generaic that can be macro system that can be applied
across programing languages. For example, for each new programing
language you can write plugins to make it work, I have in Lisp from
because that is particularly good for macros. So the plugins convert
X -> Lisp AST -> User Transforms -> Lisp ASP -> X. The Lisp (or want
ever) from is so that it can be as as flexible as prospiele and make
many macros against many languages. So they don't have to make one
macro accoros many languages. I'm sorry my communications skills are
so bad.

[Ken McDonald]

On Monday, September 19, 2011 7:04:42 AM UTC-5, Adam Jorgensen wrote:

Thanks for the correction, although I guess what I really was trying to say both are basically textual substitution systems. It seems that

the macro systems in LISP, Scheme, etc are a bit less braindead and hence might actually be a neat idea in Scala :-)

Well, I've used LISP macros a reasonable amount a _long_ time ago, and what I remember from the experience is that it always felt ugly and unreliable, compared to programming in pure Lisp. And needless to say, macros in C/C++ are a godawful mess that, depending on how extensively they are used, can basically require developers to program and debug in two languages at once, one of which is not very amenable to IDE support.

So out of the two macro systems I have some experience with, one has been less than ideal and one has been just awful. This isn't to say that working on a macro system for Scala isn't worthwhile, but I think it will need some new ideas as to the nature of what "macro" means, and very careful design. Recently, I had to generate a lot of boilerplate (many setter/getter type functions which only differed in the name and type of the property they operated on), and to do so, wrote a little Scala program than actually generated them. It would've been nice to have a macro system that could've done that.

Cheers,

Ken 

[Tony Morris]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 09/20/2011 07:40 AM, Ken McDonald wrote:
> (many setter/getter type
> functions which only differed in the name and type of the property they
> operated on), and to do so, wrote a little Scala program than actually
> generated them. It would've been nice to have a macro system that could've
> done that.

:(

- --
Tony Morris
http://tmorris.net/

-----BEGIN PGP SIGNATURE-----

Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOd7c6AAoJEPxHMY3rBz0P8WAIAMovYGO3/yqLkCBGWdBvWsAx
5jclBR1AayY4xAlrCbclQFL0pfinKaFM6+jFQAhqWoScSBzfszLiMVejfnXcexJo
b+cS53Cs0oPHN0DWl3iu0CIl0gqxzsjTrMGrU6emlDvDiv1ZwfLx+ibUkjEl8AQg
QEghgei6sWMAGCrO5ENkdPlUBiTNTIVRATgLfJc2bca6iH/TvGaRBvfI0wKfKPEW
5mHFkfFHO4cgvNRIUaVFf3GcX06AWCqHLxx5/k47Ud4IRNx0niLXDltp4iPnVQMb
dTMPV5IL0rNvXxyFXgdUu9HBa9qw6P1tjFLGnYm8yKCk543+RQlsdrwujdtPooo=
=qS+s
-----END PGP SIGNATURE-----

[Ken McDonald]

On Sunday, September 18, 2011 1:19:56 PM UTC-5, Eugene Burmako wrote:

Thanks for all the feedback - please, keep it going!

@All. I'm not really sure whether it will be possible to implement
macros that change syntax (I have yet to dig into scalac parser to
find that out), though I'd really like to implement unrestricted code
generation (i.e. to support not only expressions, but also class/
method/implicit/whatever definitions).

Unless there is a compelling use case for doing the above, please don't do it! Scala syntax is already highly flexible due to a number of features; interchangeability of method invocations and operators, use of implicits, and several other things. Unless others can give a real use case for needing further syntactic flexibility, I think that additional such flexibility would make Scala harder to learn without adding much real value.

Ken 

[Daniel Sobral]

On Mon, Sep 19, 2011 at 18:40, Ken McDonald <ykke...@gmail.com> wrote:
>
> Well, I've used LISP macros a reasonable amount a _long_ time ago, and what
> I remember from the experience is that it always felt ugly and unreliable,
> compared to programming in pure Lisp. And needless to say, macros in C/C++
> are a godawful mess that, depending on how extensively they are used, can
> basically require developers to program and debug in two languages at once,
> one of which is not very amenable to IDE support.

One difference is that Scala is typed, and so would the macros. That,
plus hygiene, will go a long way towards increasing their safety.

[Dave]

I have a use case for a template macro that is for automatic creation
of plumbing code for work arounds.

I am not that familiar with Nemerle templating. I only read some
documentation about it. Is it supposed after compiling to binary as a
plugin for the compiler and still inserts automatically code when
someone is extending some class?

I have here some code:
This code has to abstract methods
def isV(v: Visitor):Boolean
def asV(v: Visitor):V

that need to be implemented when down the inheritance tree
For instance in:


class Lit(val value: Int) extends Node[AleVisitor] with PrintExp {
def print(print: Print) { Console.print(value) }
def accept(v: AleVisitor) { v.visitLit(this) }
def isV(v: Visitor):Boolean = v.isInstanceOf[AleVisitor]
def asV(v: Visitor) = v.asInstanceOf[AleVisitor]
}

It is always the pattern:
def isV(v: Visitor):Boolean = v.isInstanceOf[V]
def asV(v: Visitor) = v.asInstanceOf[V]
that V as to be substituted with the concrete Visitor class in this
case AleVisitor

It would be handy that this plumbing code would be automatically
generated by the macro even when the macro in the abstract class in
the framework package is already compiled so that it works like a
plugin for the compiler and that the one that extends a class can
focus on its own domain problem and does not have to bother about
extending plumbing code.

expressionproblem.scala
=======================
package exprframework {

trait Exp {
def handle(v: Visitor)
}

trait Visitor {
def apply(e: Exp)
def default(e: Exp)
}

abstract class Node[V <: Visitor] extends Exp {
final def handle(v: Visitor) {
if (isV(v)) accept(asV(v))
else v.default(this)
}
private[exprframework] def accept(v: V)
def isV(v: Visitor):Boolean
def asV(v: Visitor):V
}

abstract class Op[E <: Exp] extends Visitor {
final def apply(e: Exp) {
if (isE(e)) call(asE(e))
else e.handle(this)
}
private[exprframework] def call(e: E)
def default(e: Exp) {
throw new IllegalArgumentException("Expression problem
occurred!")
}
def isE(e: Exp):Boolean
def asE(e: Exp):E
}
}
//implementation
package impl{
import exprframework._
trait AleVisitor extends Visitor {
def visitLit(lit: Lit)
def visitAdd(add: Add)
}

trait PrintExp extends Exp {
def print(print: Print)
}

class Print extends Op[PrintExp] with Visitor {
def call(e: PrintExp) { e.print(this) }
def isE(e: Exp):Boolean = e.isInstanceOf[PrintExp]
def asE(e: Exp) = e.asInstanceOf[PrintExp]
}


class Lit(val value: Int) extends Node[AleVisitor] with PrintExp {
def print(print: Print) { Console.print(value) }
def accept(v: AleVisitor) { v.visitLit(this) }
def isV(v: Visitor):Boolean = v.isInstanceOf[AleVisitor]
def asV(v: Visitor) = v.asInstanceOf[AleVisitor]
}

class Add(val left: Exp, val right: Exp) extends Node[AleVisitor]
with PrintExp {
def print(print: Print) {
print.apply(left); Console.print("+"); print.apply(right)
}
def accept(v: AleVisitor) { v.visitAdd(this) }
def isV(v: Visitor):Boolean = v.isInstanceOf[AleVisitor]
def asV(v: Visitor) = v.asInstanceOf[AleVisitor]

}
}
//extension
package extension {
import exprframework._
import impl._

trait EvalExp extends PrintExp {
def eval(eval: Eval): Int
}

class Eval extends Op[EvalExp] with AleVisitor {
private [this] var result = 0
final def eval(e: Exp) = { apply(e); result }
def call(e: EvalExp) { result = e.eval(this) }
def visitLit(lit: Lit) { result = lit.value }
def visitAdd(add: Add) { result = eval(add.left) +
eval(add.right) }
def isE(e: Exp):Boolean = e.isInstanceOf[EvalExp]
def asE(e: Exp) = e.asInstanceOf[EvalExp]
}
}
//further extension
package impl {
import exprframework._
import extension._

trait NaleVisitor extends AleVisitor {
def visitNeg(neg: Neg)
}

class Neg(val exp: Exp) extends Node[NaleVisitor] with EvalExp {
def print(print: Print) {
Console.print("-("); print.apply(exp); Console.print(")")
}
def eval(eval: Eval) = { -eval.eval(exp) }
def accept(v: NaleVisitor) { v.visitNeg(this) }
def isV(v: Visitor):Boolean = v.isInstanceOf[NaleVisitor]
def asV(v: Visitor) = v.asInstanceOf[NaleVisitor]

}
}
package main {
import exprframework._
import impl._
import extension._
object Main extends App {
val lit = new Lit(5)
val neg = new Neg(lit)
val eval = new Eval
println(eval.eval(neg))
}
}

[Tiark Rompf]

Hey Tony,

> Please see Template Haskell. I would hate for us to repeat mistakes out
> of ignorance of lessons already learned.

what are (in your view) the lessons to be learned from Template Haskell?

- Tiark

[Dave]

Hi Andreas,

thanks for the link.
yes I think it is certainly possible although my level is not there
yet.
I learned about datatype-generic programming (DGP), origami patterns
and higher kinded shapes of data a month or so ago via
these links:
http://www.cs.ox.ac.uk/jeremy.gibbons/publications/scalagp.pdf
http://vasilrem.com/blog/software-development/effective-scala-programming-patterns/
http://www.slideshare.net/remeniuk/algebraic-data-types-and-origami-patterns
http://www.slideshare.net/remeniuk/effective-scala-programming-patterns

Although this approach is certainly better I think a pragmatic macro/
template way to solve a pattern would also be welcome.
because everybody understands the concept of a macro/template because
of Excel and Word, but a few understand what a bifunctor/bimap
etcetera is.

> Date: Tue, 20 Sep 2011 01:02:10 -0700
> Subject: Re: Fwd: Re: [scala-user] implementing macros for Scala, looking for feedback!
>
> Hi Dave!
> It actually possible to encode some OO patterns in a Library. So you
> wouldn't need macros but could use that lib. For further reference see
> this post:http://debasishg.blogspot.com/2011/07/datatype-generic-
> programming-in-scala.html
> And the corresponding paper - ORIGAMI Pattern
>
> Regards Andreas

[nicola...@gmail.com]

Unless there is a compelling use case for doing the above, please don't do it! Scaisla syntax is already highly flexible due to a number of features; interchangeability of method invocations and operators, use of implicits, and several other things. Unless others can give a real use case for needing further syntactic flexibility, I think that additional such flexibility would make Scala harder to learn without adding much real value.

Ken 

Some DSLs would not play well with scala syntax. 

In Graham's On Lisp, he builds a syntax for Prolog in Lisp using macros. ( http://lib.store.yahoo.net/lib/paulgraham/onlisp.pdf chapter 24)

In practice, this allows to build macro-based DSLs that are directly usable by domain expert with little to no knowledge of lisp.

It is more difficult to do so, and sometimes impossible, without a flexible syntax. Scala syntax is flexible but still has numerous assumptions on the 

language which make creating some DSLs quite difficult.

On top of that, there is no syntax to generate trait, type or classes at compile-time.

Macro would be a great plus too.

On a different note, I have a question: 

how much of something like the continuation plugin cold be implemented as a set of Nemerle macro?

It would be a good question to ask for different plugins.

Best,

Nicolas.

[Stephen Haberman]

Awesome to hear this!

I've been hobbling along doing "kinda sorta" metaprogramming in Java with annotation processors, but the limitations are painful. It would be great to finally have real compile-time metaprogramming.

If you're looking for more inspiration, I'm a fan of boo, another .NET language that has metaprogramming support:

http://boo.codehaus.org/

- Stephen

[Ken McDonald]

On Tuesday, September 20, 2011 9:44:04 AM UTC-5, Nicolas Oury wrote:

Unless there is a compelling use case for doing the above, please don't do it! Scaisla syntax is already highly flexible due to a number of features; interchangeability of method invocations and operators, use of implicits, and several other things. Unless others can give a real use case for needing further syntactic flexibility, I think that additional such flexibility would make Scala harder to learn without adding much real value.

Ken 

Some DSLs would not play well with scala syntax. 

In Graham's On Lisp, he builds a syntax for Prolog in Lisp using macros. ( http://lib.store.yahoo.net/lib/paulgraham/onlisp.pdf chapter 24)

In practice, this allows to build macro-based DSLs that are directly usable by domain expert with little to no knowledge of lisp.

It is more difficult to do so, and sometimes impossible, without a flexible syntax. Scala syntax is flexible but still has numerous assumptions on the 

language which make creating some DSLs quite difficult.

That's true of course, but that's the nature of a formal language; it is defined with syntactic and semantic rules, and those rules will naturally preclude doing some things. I wouldn't be surprised if building a "perfectly expressive" language is equivalent to solving the Halting Problem :-)

Please don't think that I'm objecting to macros--I just think they should be approached slowly. Implement macro processing with a few small features, see if everything works alright, and go from there. Perhaps it's just that I'm still traumatized by having to work with C macros--it does sound like they've been put on a better formal footing now.

And I certainly wouldn't mind the ability to automatically generate boilerplate at compile time.

Ken

[Bakos Gábor]

Hi Eugene,

I am not sure how important what will be in F#, but it might worth
looking at it[0]:
- I think type providers-like functionality were already asked for
- Auto-properties might be a nice addition
- Maybe some people get familiar with LINQ, so being able to use a
similar construct for various tasks would be welcome
(- I think the measure types can be implemented as a library, but the
precedence rules can be offensive.)

Personally I do not see these as must have features, but I guess
Microsoft were made researches on the most wanted functionalities.
Cheers, gabor

[0]: http://msdn.microsoft.com/en-us/library/hh370982(v=VS.110).aspx

[HamsterofDeath]

DYNAMIC MIXINS!!!!1111

aka

subclass, proxy, and add another trait