Enum Paradise: An experimentation of Enumeration implementation using type macro

[Alois Cochard]

Hey hackers,

I've just pushed to github my recent experimentation about implementing Enumeration using type macro:

https://github.com/aloiscochard/enum-paradise

It's still very WIP, for the moment you can't define custom Value trait or even change values name, but it should give you an idea.

You will find in the test the syntax I want to support (what is not currently implemented is in comment, still not sure about the syntax yet):

https://github.com/aloiscochard/enum-paradise/blob/master/core/src/main/scala/test.scala

Feel free to discuss/contribute/blame ;)

Cheers

Alois Cochard

http;//twitter.com/aloiscochard

http://github.com/aloiscochard

[Eugene Burmako]

Would be great to have untyped macros to get rid of quotes in symbol names. I thought untyped macros will be quite an esoteric feature, but as of late they come up surprisingly often.

[Simon Ochsenreither]

Hi Alois!

This looks really nice!

There have been some discussions about enums lately: https://groups.google.com/d/topic/scala-language/IcDNzjhxV3E/discussion, https://groups.google.com/d/topic/scala-language/1bzZz14j87o/discussion and https://groups.google.com/d/topic/scala-internals/8RWkccSRBxQ/discussion

Considering the details, I came up with a possible syntax like this ...

object Direction extends scala.Enum {
  object North
  object East
  object South
  object West
}

... which would be desugared to this:

final class Direction private(val name: String, ordinal: Int) extends java.lang.Enum[Direction](name, ordinal)
object Direction {
  val North = new Direction("North", 0)
  val East  = new Direction("East", 0)
  val South = new Direction("South", 0)
  val West  = new Direction("West", 0)

  private val $VALUES = Array(North, East, South, West)
  def values = $VALUES.clone
  def valueOf(name: String) = java.lang.Enum.valueOf(classOf[Stuff], name)
}

It uses objects to allow the definition of methods for enum items. I think this would cover all features of Java's enums, as far as I understand it.

From my POV, being compatible with the platform's idea of enums is absolutely crucial. scala.Enumeration is not, and absolutely no one is using it. Emitting JVM-compatible enums probably requires some fixes/changes to the compiler itself (e. g. setting the appropriate flags for the class/the individual enum items when the compiler discovers the Enum type).
I mentioned some of the preliminary work here: https://groups.google.com/forum/?fromgroups=&hl=en#!searchin/scala-internals/enum/scala-internals/-0vjKs3RecE/XawFkdSKqZwJ

Thanks and bye,

Simon

[Alois Cochard]

Hi Simon

On Thursday, 10 January 2013 19:35:04 UTC, Simon Ochsenreither wrote:

Hi Alois!

This looks really nice!

There have been some discussions about enums lately: https://groups.google.com/d/topic/scala-language/IcDNzjhxV3E/discussion, https://groups.google.com/d/topic/scala-language/1bzZz14j87o/discussion and https://groups.google.com/d/topic/scala-internals/8RWkccSRBxQ/discussion

Great stuff, I'll look into them, thanks for taking time to compile them here :)

 

Considering the details, I came up with a possible syntax like this ...

object Direction extends scala.Enum {
  object North
  object East
  object South
  object West
}

Too verbose imo,  I don't see the point of repeating object here.

... which would be desugared to this:

final class Direction private(val name: String, ordinal: Int) extends java.lang.Enum[Direction](name, ordinal)
object Direction {
  val North = new Direction("North", 0)
  val East  = new Direction("East", 0)
  val South = new Direction("South", 0)
  val West  = new Direction("West", 0)

  private val $VALUES = Array(North, East, South, West)
  def values = $VALUES.clone
  def valueOf(name: String) = java.lang.Enum.valueOf(classOf[Stuff], name)
}

It uses objects to allow the definition of methods for enum items. I think this would cover all features of Java's enums, as far as I understand it.

I never faced case where I wanted to define method on Value instance, but adding methods on Value type definitely make sense and it's why I would like to support like this:

https://github.com/aloiscochard/enum-paradise/blob/master/core/src/main/scala/test.scala#L27-L37 

From my POV, being compatible with the platform's idea of enums is absolutely crucial. scala.Enumeration is not, and absolutely no one is using it. Emitting JVM-compatible enums probably requires some fixes/changes to the compiler itself (e. g. setting the appropriate flags for the class/the individual enum items when the compiler discovers the Enum type).
I mentioned some of the preliminary work here: https://groups.google.com/forum/?fromgroups=&hl=en#!searchin/scala-internals/enum/scala-internals/-0vjKs3RecE/XawFkdSKqZwJ

I definitely think that scala.Enumeration not being compatible with JVM Enum isn't the reason people don't use it, as I said on twitter having JVM-compatible would be a nice to have, but I think having specific construct for them in the language spec is a bad thing. It should be done as a lib imo, if we are able to have some feature in macro to allow us to set the needed flag, then fine I would be happy to make them java interopabe.

The main reason folks don't use scala.Enumeration, is imo because they don't give you exhaustive matching (maybe I missed a fix about that?).

When one design API that need to be compatible with Java, we end writing some Java code... I think it could be the same with Enum, if you create a java compatible API use java Enum. 

Thanks and bye,

Simon

[Simon Ochsenreither]

Hi Alois,

 

Too verbose imo,  I don't see the point of repeating object here.

 

The idea is that you can define methods on the objects if you define them this way ...

object Direction extends scala.Enum with HasFoo {
  object North { def foo = 42 }
  object East  { def foo = 43 }
  object South { def foo = 44 }
  object West  { def foo = 45 }
}

... and it would all be bog-standard Scala syntax.

I never faced case where I wanted to define method on Value instance, but adding methods on Value type definitely make sense and it's why I would like to support like this:

https://github.com/aloiscochard/enum-paradise/blob/master/core/src/main/scala/test.scala#L27-L37 

Looks interesting. I preferred my approach because it feels like it needs less tree rewriting, but if it works in your case, I'll be happy too.

I definitely think that scala.Enumeration not being compatible with JVM Enum isn't the reason people don't use it, as I said on twitter having JVM-compatible would be a nice to have, but I think having specific construct for them in the language spec is a bad thing. It should be done as a lib imo, if we are able to have some feature in macro to allow us to set the needed flag, then fine I would be happy to make them java interopabe.

While I partially disagree about the reason s.Enumeration is not used, I think we are absolutely on the same page here. We don't need any specific construct in the language. The change to the backend is purely a check for setting the right flags which we probably need anyway in one way or another, because it is forbidden to extend java.lang.Enum per language spec in Java, but Scala let's you happily do that, which can be very confusing to users.

The main reason folks don't use scala.Enumeration, is imo because they don't give you exhaustive matching (maybe I missed a fix about that?).

When one design API that need to be compatible with Java, we end writing some Java code... I think it could be the same with Enum, if you create a java compatible API use java Enum. 

Is there any benefit in not being JVM compatible? We loose a ton of interoperability with basically no advantage at all.
We should either get rid of all Enumeration constructs/features/libraries in Scala and point people to Java, or fix it.
I see very limited benefits in having a rewritten Enum if it is still not JVM compatible, especially when it's done via macros where making it compatible would be no big magic anymore.

Bye,

Simon

[Alois Cochard]

HI mate,

On Friday, 11 January 2013 10:54:00 UTC, Simon Ochsenreither wrote:

Hi Alois,

 

Too verbose imo,  I don't see the point of repeating object here.

 

The idea is that you can define methods on the objects if you define them this way ...

object Direction extends scala.Enum with HasFoo {
  object North { def foo = 42 }
  object East  { def foo = 43 }
  object South { def foo = 44 }
  object West  { def foo = 45 }
}

... and it would all be bog-standard Scala syntax.

I see, I may take this path if the syntax I'll try to impl. is not feasible. 

I never faced case where I wanted to define method on Value instance, but adding methods on Value type definitely make sense and it's why I would like to support like this:

https://github.com/aloiscochard/enum-paradise/blob/master/core/src/main/scala/test.scala#L27-L37 

Looks interesting. I preferred my approach because it feels like it needs less tree rewriting, but if it works in your case, I'll be happy too.

Make sense, but in case of 'more tree rewriting' vs 'more compact syntax', I prefer to choose the later

 

I definitely think that scala.Enumeration not being compatible with JVM Enum isn't the reason people don't use it, as I said on twitter having JVM-compatible would be a nice to have, but I think having specific construct for them in the language spec is a bad thing. It should be done as a lib imo, if we are able to have some feature in macro to allow us to set the needed flag, then fine I would be happy to make them java interopabe.

While I partially disagree about the reason s.Enumeration is not used, I think we are absolutely on the same page here. We don't need any specific construct in the language. The change to the backend is purely a check for setting the right flags which we probably need anyway in one way or another, because it is forbidden to extend java.lang.Enum per language spec in Java, but Scala let's you happily do that, which can be very confusing to users.

Cool, I'm not that much comfortable with this flags stuff, but for sure, I'll definitely investigate how this can be achieved, and fill issues in case something is missing, I'll probably come back to you about that.

 

The main reason folks don't use scala.Enumeration, is imo because they don't give you exhaustive matching (maybe I missed a fix about that?).

When one design API that need to be compatible with Java, we end writing some Java code... I think it could be the same with Enum, if you create a java compatible API use java Enum. 

Is there any benefit in not being JVM compatible? We loose a ton of interoperability with basically no advantage at all.
We should either get rid of all Enumeration constructs/features/libraries in Scala and point people to Java, or fix it.
I see very limited benefits in having a rewritten Enum if it is still not JVM compatible, especially when it's done via macros where making it compatible would be no big magic anymore.

Agreed, there is clearly no benefit *not* being JVM compatible!

 

Bye,

Thanks for your detailed explanation! 

 

Simon

[Simon Ochsenreither]

Hi Alois,

I see, I may take this path if the syntax I'll try to impl. is not feasible.

I'd be interested about your ideas of combining symbols and method definitions.

Cool, I'm not that much comfortable with this flags stuff, but for sure, I'll definitely investigate how this can be achieved, and fill issues in case something is missing, I'll probably come back to you about that.

Agreed, there is clearly no benefit *not* being JVM compatible!

I'll just write down the points we mentioned on IRC, so it isn't forgotten:

• An enum basically needs to
• extend java.lang.Enum
• set the ACC_ENUM flag
• is it possible to set it via macros or do we need to fix it in the compiler?
• emit the appropriate, static methods (values, valueOf, ...)
  
• Is it possible for type macros to introduce a new class where the object already exists? (E. g. object Days exists, class Days needs to be generated)

Regarding the flag business and touching the compiler in general: I think it makes sense to add a check to scalac to prevent people from mistakenly extending java.lang.Enum (like javac does) anyway. So adding an error message in the sense of “It seems you are trying to declare an enum by extending java.lang.Enum. Extend scala.Enum instead.” and making sure that the right flags are set should be “no big deal”.

Thanks,

Simon

[Eugene Burmako]

1) You can only set flags if you cast to internal API. I believe there's an issue which asks to make that API public.

2) Unfortunately c.introduceTopLevel cannot be used to generate companions, because its argument is put into a virtual source file, which is deemed to be different from the source file, which hosts a manually written companion, which triggers a compilation errors.

[Klaus Havelund]

> I've just pushed to github my recent experimentation about implementing
> Enumeration using type macro:
> https://github.com/aloiscochard/enum-paradise
>
> It's still very WIP, for the moment you can't define custom Value trait or
> even change values name, but it should give you an idea.
>
> You will find in the test the syntax I want to support (what is not
> currently implemented is in comment, still not sure about the syntax yet):
> https://github.com/aloiscochard/enum-paradise/blob/master/core/src/main/scala/test.scala

I am not sure I like the quotes in:

object Days extends Enum('Monday, 'Tuesday, 'Wednesday, 'Thursday, 'Friday)

and the need to write Days.Value in:

type X = Days.Value
def xs: Seq[Days.Value] = Days.values

It is also not quite clean that there are quotes above but not here:

import Days._
def f(x: Value) = x match {
case Monday =>
case Tuesday =>
case Wednesday =>
case Thursday =>
case Friday =>
}

Beyond these comments I would like to point to one of my previous
comments that perhaps
enumerated types should be considered as a special case of what is
normally called algebraic
data types (or variant types) in functional programming languages. My
previous comment is included below.

Best,

Klaus Havelund

--- my previous comment: ---

Let me reconsider what I wrote a few days ago wrt. enumerated types.

I wrote this:

>> This is (approximately) how one should write an enumerated type:
>>
>> enum Direction {North,East,South,West}
>>
>> Anything longer than that is a challenge to keep working.

as a response to the following way of writing "the same" enumerated type:

object Direction extends scala.Enum {
object North
object East
object South
object West
}

After re-consideration, perhaps this is ok. The reason I say this is
that one probably should see enumerated
types a special case of algebraic datatypes:

http://en.wikipedia.org/wiki/Algebraic_data_type

For example, in Haskell we would write a tree as follows:

data Tree = Empty
| Leaf Int
| Node Tree Tree

In Scala we currently could write this as:

trait Tree
case object Empty extends Tree
case class Leaf(x:Int) extends Tree
case class Node(left: Tree, right: Tree) extends Tree

If we count letters, Haskell uses 35 non-blank characters, while Scala
uses 112, more than 3 times
as many characters. Granted, once you start typing, a few characters
more do not really do much harm.
It is not a big deal, but still.

I think I understand why it cannot be as succinct as in Haskell:
because Scala is charged with
the additional demanding task of merging functional and object
oriented programming, and we might
want for example to add methods to the alternatives, as in:

trait Tree

case object Empty extends Tree {
override def toString = "*"
}

case class Leaf(x:Int) extends Tree {
override def toString = x.toString
}

case class Node(left: Tree, right: Tree) extends Tree {
override def toString = "<" + left + "," + right + ">"
}

Hence, the syntax should support adding methods to the alternatives.
So one can probably not get
away writing it the Haskel way. Instead, what might be desirable
(ignoring the added methods) is something like this:

datatype Tree {
object Empty
class Leaf(x:Int)
class Node(left: Tree, right: Tree)
}

(with the slight annoyance that one has to choose between 'object' and
'class') and this does not seem to
be so far away from:

object Direction extends Enum {

[Simon Ochsenreither]

Hi Klaus,

Beyond these comments I would like to point to one of my previous
comments that perhaps
enumerated types should be considered as a special case of what is
normally called algebraic
data types (or variant types) in functional programming languages. My
previous comment is included below.

Yes, I agree with that notion.
The issue is that the usual GADT approach is considered to be too heavyweight in terms of resulting amount of class files by some people. (And they are not compatible with JVM enums, btw. :-))
So basically, we are looking for an approach which combines all the benefits but no drawbacks.

I could imagine that we could also have a Enum macro which would try to bridge the gap between GADTs and JVM enums.
 

  datatype Tree {
    object Empty
    class Leaf(x:Int)
    class Node(left: Tree, right: Tree)
  }

Requiring a language change reduces the chance of getting it merged close to zero. That's why Alois and I are trying to implement enums with existing language syntax without breaking too many assumptions users could have about the code.

Thanks and bye,

Simon

[Klaus Havelund]

Thanks for the explanation!

Concerning this argument:

> The issue is that the usual GADT approach is considered to be too
> heavyweight in terms of resulting amount of class files by some people.

Sure. Is it a strong argument though?

Klaus

[Alois Cochard]

Hi Eugene,

On Friday, 11 January 2013 16:36:22 UTC+1, Eugene Burmako wrote:

1) You can only set flags if you cast to internal API. I believe there's an issue which asks to make that API public.

Would be nice if you find the issue number, I'll try to search for it but you will be probably more efficient at this ;)

 

2) Unfortunately c.introduceTopLevel cannot be used to generate companions, because its argument is put into a virtual source file, which is deemed to be different from the source file, which hosts a manually written companion, which triggers a compilation errors.

In fact I don't think we need to generate a companion, but just introduce a sealed trait named according to enum name and adding a prefix, I'm pretty sure this is feasible using introduceTopLevel from what I seen in your type provider example.

I'll try to implement that, so then I can change: Enumaration { type Value } to Enumeration[T] { type Value = T }, and extends java Enum too.

Thanks

[Eugene Burmako]

https://issues.scala-lang.org/browse/SI-6785

[Alois Cochard]

On Friday, 11 January 2013 18:07:09 UTC+1, Klaus wrote:

> I've just pushed to github my recent experimentation about implementing
> Enumeration using type macro:
> https://github.com/aloiscochard/enum-paradise
>
> It's still very WIP, for the moment you can't define custom Value trait or
> even change values name, but it should give you an idea.
>
> You will find in the test the syntax I want to support (what is not
> currently implemented is in comment, still not sure about the syntax yet):
> https://github.com/aloiscochard/enum-paradise/blob/master/core/src/main/scala/test.scala

I am not sure I like the quotes in:

  object Days extends Enum('Monday, 'Tuesday, 'Wednesday, 'Thursday, 'Friday)

Quote could be avoided by what Eugene call 'untyped macros', in the meantime it's the cleanest way I think. 

and the need to write Days.Value in:

This unfortunately impossible to avoid that without being able to define companion with type macro, that the reason I named the enum 'Days' and not 'Day', to make clear that Days is not the type of the values ...

 

  type X = Days.Value
  def xs: Seq[Days.Value] = Days.values

It is also not quite clean that there are quotes above but not here:

Same as first comment, quoted names are in fact symbol:

http://www.scala-lang.org/api/current/index.html#scala.Symbol

Ok good point, was asking for an example like that because I wasn't able to find a use-case myself.

We can surely have an alternative syntax to allow declaration of such ADT! I'll work on it.

Thanks for your feedback! 

[Alois Cochard]

@Eugene great ty, I can live with the casting for now :)

@Simon thank for summarizing it here, I just created an issue with all the info I need to implement it: https://github.com/aloiscochard/enum-paradise/issues/1 according to the technic described in SI-6795 I'm pretty sure this can be done now! 

[Simon Ochsenreither]

Hi Alois,

In fact I don't think we need to generate a companion, but just introduce a sealed trait named according to enum name and adding a prefix, I'm pretty sure this is feasible using introduceTopLevel from what I seen in your type provider example.

You'll need at least a class because every enum item is instantiated, remember EnumItem ---> public static final EnumItem = new EnumClass("EnumItem", 42)

If we need different names, I think having singular/plural would be a nice approach (e. g. append an "s" to the generated type).

Thanks and bye,

Simon

[Alois Cochard]

Thanks to Eugene for the integration of untyped macro into 'paradise', I was able to avoid symbol and add support for defining custom name:

https://github.com/aloiscochard/enum-paradise/blob/master/core/src/main/scala/enum.scala

I added as well support for 'EnumOf' which receive a user defined value trait:

https://github.com/aloiscochard/enum-paradise/blob/master/core/src/main/scala/enumOf.scala

Next step is supporting syntax in comments of enumOf.scala file, that really shouldn't be a big deal... but I have to go sleeping unfortunately ;)

As a side note, I had a idea, (maybe a terrible one?) kind of support for type class definition:

http://pastie.org/5690886

Shouldn't be hard to implement as well...

Cheers

[Simon Ochsenreither]

Hi,

I'm currently trying to figure out how to support most of the standard functionality like

1. User-defined constructors
2. User-defined methods
3. User-defined overridden methods in enum items
4. Adding/implementing traits
   

To enable it on a syntactic level, I'm considering something like:

class Days(val inGerman: String) /* 1. */ extends Enum with HasName /* 4. */ (
  Monday("Montag"),
  Tuesday("Dienstag"),
  Wednesday("Mittwoch"),
  Thursday("Donnerstag"),
  Friday("Freitag"),
  Saturday("Samstag") { override def workingDay: Boolean = false }, // 3.
  Sunday("Sonntag") { override def workingDay: Boolean = false }
) {
  def abbreviation = name take 3 // 2.
  def workingDay: Boolean = true // 3.
}

trait HasName { def name: String }

Additional rules would be:

• Constructor is implicitly private
• Class is abstract/sealed or final
  
• Enum items need to implement traits and the methods in the class body
  

I think it would be possible to avoid the restriction regarding companions by not emitting a companion object at all and instead setting the proper static modifiers for values/methods (too bad that the @static annotation didn't make it into 2.10).

I'm not sure whether type macros would be powerful enough to guarantee all the constraints related to the methods defined in the class body. I tried to use multiple argument lists for Enum, like defining Enum(values: _*)(definitions: _*) and using it like Enum(/* values */) { /* definitions */ } to get more control over it, but that doesn't seem to be supported.

Alois, Eugene, what do you think?

Thanks and bye,

Simon

[Simon Ochsenreither]

I worked a bit on an implementation here: https://github.com/soc/enum-paradise/blob/java-enums/macros/src/main/scala/scalax.scala
I'm experiencing one issue, though:

Compilation fails with

[error] /home/soc/Entwicklung/enum-paradise/core/src/main/scala/enum.scala:3: class Any is abstract; cannot be instantiated
[error] class Days extends Enum(
[error]                    ^
[error] one error found

I think the issue is that I'm not setting up the constructor call properly in line 19: https://github.com/soc/enum-paradise/blob/java-enums/macros/src/main/scala/scalax.scala#L19

Any ideas?

Maybe someone can help me with a few questions:

• Is c.enclosingClass.tpe the right way to refer to the type of the class extending my macro type?
• What's the right way to set modifiers for the class I'm extending? Return a ClassDef instead of a Template from the type macro?
• Eugene mentioned that there was a way to set the STATIC flag by casting to the right type, could you give me a hint here?
  

[Simon Ochsenreither]

Ok, I made some progress and added “java.lang.Enum[<EnumClass>]” to the list of parents and the appropriate constructors to the list of methods: https://github.com/soc/enum-paradise/blob/1b3fd324b3d99e009bcc6f74c5afacda94b82eda/macros/src/main/scala/scalax.scala#L78

Now, I'm experiencing the following issue:

[error] /home/soc/Entwicklung/enum-paradise/core/src/main/scala/enum.scala:3: illegal cyclic reference involving class Days
[error] class Days() extends Enum(

[error]       ^
[error] one error found

I think this is because java.lang.Enum needs to take Days as an type argument, i.e. class Days extends java.lang.Enum[Days].

Any ideas how this can be fixed?

Thanks,

Simon

[Klaus Havelund]

Where is "name" bound in Days? :

trait HasName {
def name: String
}

class Days(val inGerman: String) extends Enum with HasName (

Monday("Montag"),
Tuesday("Dienstag"),
Wednesday("Mittwoch"),
Thursday("Donnerstag"),
Friday("Freitag"),
Saturday("Samstag") {
override def workingDay: Boolean = false
},

Sunday("Sonntag") {
override def workingDay: Boolean = false
}
) {
def abbreviation = name take 3

def workingDay: Boolean = true
}

Klaus

[Simon Ochsenreither]

Where is "name" bound in Days?

It's defined in java.lang.Enum.

[Simon Ochsenreither]

I made a lot of progress, and I fixed the issue with extending java.lang.Enum.

I also managed to implement all the required fields and methods (except the enum fields itself): https://github.com/soc/enum-paradise/blob/88f96735920f0ae810d3e0a020108aa989c27482/macros/src/main/scala/scalax.scala

Adding the enum fields to the template creates the following error:
scalax.Days does not take parameters

No idea where it comes from, because instantiating Days manually (new Days("Monday", 0)) works perfectly well...

[Simon Ochsenreither]

Hi Eugene,

this is where I am at the moment:

println(show(template)):

java.lang.Enum[Days] {
  private def <init>(name: String, ordinal: Int) = super.<init>(name, ordinal);
  private val Monday = new Days("Monday", 0);
  private val Tuesday = new Days("Tuesday", 1);
  private val Wednesday = new Days("Wednesday", 2);
  private val Thursday = new Days("Thursday", 3);
  private val Friday = new Days("Friday", 4);
  private val $VALUES: scala.Array[Days] = scala.Array.apply[Days]()(Predef.implicitly);
  def values: scala.Array[Days] = $VALUES.clone();
  def valueOf(name: String) = java.lang.Enum.valueOf(null, name)
}

I have made the following observations:

• The STATIC flag is already set. Setting the STATIC flag doesn't have any effect.
  Is this a bug?
• c.enclosingClass.tpe is null.
  Is this a bug?
• How can I refer to c.enclosingClass as a class literal (classOf[])? (I need to call a method defined as def valueOf(enumType: Class[T], name: String).)
  I printed show and showRaw, but both cleanup their output too much, so I can't tell anymore what “Literal(Constant(Days))” is really.
  I tried increasingly complex stuff like Literal(Constant(c.enclosingClass.symbol.typeSignature.typeSymbol.asType.toType)) or TypeTree(c.enclosingClass.symbol.typeSignature.typeSymbol.asType.toType) without success.

Eugene, could you give me a few hints into the right direction?

Thanks a lot!

Simon

[Eugene Burmako]

1) Sorry I haven't been following the thread for quite a while. Where would you like to put the STATIC flag?

2) I don't think it's a bug - it's just a timing issue, because type macros expand quite early in the typechecking pipeline.

3) c.reifyRuntimeClass should do the trick

[Simon Ochsenreither]

Hi Eugene,

1) Sorry I haven't been following the thread for quite a while. Where would you like to put the STATIC flag?

No problem and thanks for all your help!
I'd like to put the STATIC flag on all members of the class.
This is how I'm trying to set it:

      val STATIC = 1 << 23
      val PARAMACCESSOR = 1 << 29
      def setFlag(symbol: Symbol, flag: Long) {
        val compilerSymbol = symbol.asInstanceOf[scala.tools.nsc.Global#Symbol]
        compilerSymbol.setFlag(flag)
      }
      def printFlags(symbol: Symbol) {
        println(symbol.asInstanceOf[scala.tools.nsc.Global#Symbol].flagString)
      }

You can see the actual (commented) usages here:
https://github.com/soc/enum-paradise/blob/java-enums/macros/src/main/scala/scalax.scala
 

2) I don't think it's a bug - it's just a timing issue, because type macros expand quite early in the typechecking pipeline.

Ok, thanks, good to know.
 

3) c.reifyRuntimeClass should do the trick

Mhh. Not sure about the parameters ... I tried c.reifyEnclosingRuntimeClass instead and I got the usual “illegal cyclic reference involving class Days” issue which I have experienced a lot while experimenting.

This is the approach which seems to work and which I'm using now:

  Apply(
    Select(Select(Select(Ident(TermName("java")), TermName("lang")), TermName("Enum")), TermName("valueOf")),
    List(TypeApply(Ident(TermName("classOf")),List(Ident(className))), Ident(TermName("name")))
  )
Anything to improve?


==IGNORE (see end)================================================================
Another question ... compilation fails with a MatchError when I want to emit the constructor body like this:

  private def <init>(name: String, ordinal: Int) =
    super.<init>(name, ordinal);

E. g. rhs = Apply(Select(Super(This(tpnme.EMPTY), tpnme.EMPTY), nme.CONSTRUCTOR), List(Ident(TermName("name")), Ident(TermName("ordinal"))))

I basically need to wrap the Expression into a Block and have a literal unit value at the end, and then it works:

  private def <init>(name: String, ordinal: Int) = {
    super.<init>(name, ordinal);

    ()
  };

E. g.
  rhs  =
    Block(
      List(
        Apply(Select(Super(This(tpnme.EMPTY), tpnme.EMPTY), nme.CONSTRUCTOR), List(Ident(TermName("name")), Ident(TermName("ordinal"))))
      ),
      Literal(Constant(()))
    )

I assumed that the issue was that the wrong result type was inferred, but even setting it explicitly to Unit doesn't fix it.Is there a reason for that? Or is it just that the “let's add an Unit value at the end”-step happens early and the type-checker doesn't know about it?

Edit: WOW, now I'm confused ... I tried to recreate the exception and now it works. Weird. Ignore this ...
==================================================================

Yet another question (sorry) ...
If my type-macro is named scala.Enum, is there a way to make the extending class of the type-macro extend a marker trait with the same name?
E.g.
  class Days extends scala.Enum // type-macro
  assert(dayInstance.isInstanceOf[scala.Enum])

Thanks a lot!

Simon

[Simon Ochsenreither]

Oooops, looks like I forgot the most interesting issue.

This is the issue on which I'm stuck for days now:

If I generate the following template:

/*class Days extends*/ java.lang.Enum[Days] {

  private def <init>(name: String, ordinal: Int) = super.<init>(name, ordinal);

  private val Monday: Days = new Days("Monday", 0);
  private val Tuesday: Days = new Days("Tuesday", 1);
  private val Wednesday: Days = new Days("Wednesday", 2);
  private val Thursday: Days = new Days("Thursday", 3);
  private val Friday: Days = new Days("Friday", 4);
  private[this] val $VALUES: scala.Array[Days] = scala.Array.apply[Days]()(Predef.implicitly);

  def values: scala.Array[Days] = $VALUES.clone();

  def valueOf(name: String): Days = java.lang.Enum.valueOf(classOf[Days], name)
}

I'm getting the following compiler error:

[error] /home/soc/Entwicklung/enum-paradise/core/src/main/scala/enum.scala:3: scalax.Days does not take parameters

[error] class Days extends Enum(
[error]                    ^

If I don't comment out the static fields (staticEnumFields) and I use the shorter version of the constructor (without Unit) I somehow get the MatchError I described earlier (but wasn't able to figure out how it was triggered):

[info] Compiling 1 Scala source to /home/soc/Entwicklung/enum-paradise/core/target/scala-2.11/classes...

java.lang.Enum[Days] {
  private def <init>(name: String, ordinal: Int) = super.<init>(name, ordinal);

  private[this] val $VALUES: scala.Array[Days] = scala.Array.apply[Days]()(Predef.implicitly);

  def values: scala.Array[Days] = $VALUES.clone();

  def valueOf(name: String): Days = java.lang.Enum.valueOf(classOf[Days], name)
}

     while compiling: /home/soc/Entwicklung/enum-paradise/core/src/main/scala/enum.scala
        during phase: uncurry
     library version: version 2.11.0-20130119-154755-35604566d7
    compiler version: version 2.11.0-20130119-154755-35604566d7
  reconstructed args: -d /home/soc/Entwicklung/enum-paradise/core/target/scala-2.11/classes -classpath /home/soc/Entwicklung/enum-paradise/core/target/scala-2.11/classes:/home/soc/Entwicklung/enum-paradise/macros/target/scala-2.11/classes:/home/soc/.ivy2/cache/org.scala-lang.macro-paradise/scala-reflect/jars/scala-reflect-2.11.0-SNAPSHOT.jar:/home/soc/.ivy2/cache/org.scala-lang.macro-paradise/scala-compiler/jars/scala-compiler-2.11.0-SNAPSHOT.jar -bootclasspath /usr/lib/jvm/java-7-openjdk-amd64/jre/lib/resources.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rt.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/sunrsasign.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jsse.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jce.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/charsets.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/netx.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/plugin.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/rhino.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/lib/jfr.jar:/usr/lib/jvm/java-7-openjdk-amd64/jre/classes:/home/soc/.sbt/0.12.1/boot/org.scala-lang.macro-paradise.scala-2.11.0-SNAPSHOT/lib/scala-library.jar

  last tree to typer: Ident(name)
              symbol: value name (flags: <param> <triedcooking>)
   symbol definition: name: String
                 tpe: String
       symbol owners: value name -> method valueOf -> class Days -> package scalax
      context owners: constructor Days -> class Days -> package scalax

== Enclosing template or block ==

<tpt> // tree.tpe=Enum[scalax.Days]

== Expanded type of tree ==

AliasTypeRef(
  pre = SingleType(pre = ThisType(package scala), object Predef)
  Alias(type String = String)
  normalize = TypeRef(
    TypeSymbol(
      final class String extends Serializable with Comparable[String] with CharSequence
     
    )
  )
)

unhandled exception while transforming enum.scala
[error] uncaught exception during compilation: MatchError("Days.super.<init>(name, ordinal) (of class scala.reflect.internal.Trees$Apply)") @ scala.tools.nsc.transform.UnCurry$UnCurryTransformer$$anonfun$5$$anonfun$apply$5.apply(UnCurry.scala:434)
scala.MatchError: Days.super.<init>(name, ordinal) (of class scala.reflect.internal.Trees$Apply)
    at scala.tools.nsc.transform.UnCurry$UnCurryTransformer$$anonfun$5$$anonfun$apply$5.apply(UnCurry.scala:434)
    at scala.tools.nsc.transform.UnCurry$UnCurryTransformer$$anonfun$5$$anonfun$apply$5.apply(UnCurry.scala:433)
    at scala.reflect.api.Trees$Transformer.atOwner(Trees.scala:3057)
    at scala.tools.nsc.transform.TypingTransformers$TypingTransformer.atOwner(TypingTransformers.scala:30)
    at scala.tools.nsc.transform.UnCurry$UnCurryTransformer.mainTransform(UnCurry.scala:431)
    at scala.tools.nsc.transform.UnCurry$UnCurryTransformer.transform(UnCurry.scala:102)
    at scala.tools.nsc.transform.UnCurry$UnCurryTransformer.transform(UnCurry.scala:64)
    at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:3046)
    at scala.reflect.api.Trees$Transformer$$anonfun$transformStats$1.apply(Trees.scala:3044)
    at scala.collection.immutable.List.loop$1(List.scala:164)
    at scala.collection.immutable.List.mapConserve(List.scala:180)
    at scala.reflect.api.Trees$Transformer.transformStats(Trees.scala:3044)
    at scala.reflect.internal.Trees$class.itransform(Trees.scala:1332)
    at scala.reflect.internal.SymbolTable.itransform(SymbolTable.scala:13)
[error] (core/compile:compile) scala.MatchError: Days.super.<init>(name, ordinal) (of class scala.reflect.internal.Trees$Apply)

... adding the longer version of the constructor while keeping the static enum fields commented out makes the MatchError go away and everything compiles successfully.
I don't care that much about the MatchError, but I'd like to understand the scalax.Days does not take parameters-issue.

Any ideas?

Thanks,

Simon

[Eugene Burmako]

Hi. Sorry for taking long to respond.

1) I don't think that will work. From what I can see, STATIC only has effect on members of objects.

3) Right, reifyRuntimeClass and its cousin won't work, because they try to immediately calculate the j.l.Class, which corresponds to the type you provide. To do that, you need to calculate erasure, but that's impossible and here's why:
* At the moment when a type macro expands, the signature of the enclosing class is unknown.
* This means that we don't know its supertypes.
* Consequently, we don't know whether the class synthesized by the type macro is going to be a value class or not.

* Therefore computing erasure right away is out of the question.

Interestingly enough, if I remove isDerivedValueClass checks from Erasure.scala, reifyRuntimeClass works without problems. We should definitely blame SIP-15 for breaking macros!

On a serious note, the right approach here would probably involve generating a stub, which would expand into j.l.Class later on, when the synthesized Template is typechecked. That's exactly what you did. Sorry for misleading you into reifyRuntimeClass back then.

4a) So it did work when you explicitly specified Unit as the return type without appending Literal(Constant(())) to the block?

4b) I don't think so. That's because type macros consume a typename slot in their owner's namespace. Just as you cannot define a type alias and a class with the same name, you can't define an homonymous trait alongside a type macro.

[Simon Ochsenreither]

Hi Eugene,

1) I don't think that will work. From what I can see, STATIC only has effect on members of objects.

Do you know whether this is an implementation restriction or because STATIC is used in the compiler to mark “stable” paths (instead of Java's definition of static)?
So what is the supported way to emit static members into classes then?
 

3) [...] the right approach here would probably involve generating a stub, which would expand into j.l.Class later on, when the synthesized Template is typechecked. That's exactly what you did.

Ok, thanks!
 

4a) So it did work when you explicitly specified Unit as the return type without appending Literal(Constant(())) to the block?

Exactly, but only if the static enum fields where commented out. If I use the shorter variant and the enum fields are not commented, it fails with the MatchError.
I wonder what's the right thing here, in terms of a) return type and b) having to add b) () to the end.

4b) I don't think so. That's because type macros consume a typename slot in their owner's namespace. Just as you cannot define a type alias and a class with the same name, you can't define an homonymous trait alongside a type macro.

Would it make sense to implement that as a general macro rule? I think it's easier to understand if “extends Foo” ends up extending “Foo” even if it is just a marker trait, imho.

Do you have an idea about the causes of the scalax.Days does not take parameters issue?


Thanks a lot!

Bye,

Simon

[Eugene Burmako]

1) Unfortunately there's none. Not until we have the machinery to changed arbitrary classes (something which is attributed to the macro annotation milestone).

4a) The right thing is to append (), because UnCurry seems to expect a Block and, as far as I can guess, if you omit the (), the rhs of the ctor is treated as a plain expr.

4b) Everything works when I change New(...) to Select(New(...), nme.CONSTRUCTOR) in enumInstance. That's very misleading, I agree, but at least I can say that this was mentioned in the docs. I understand though that's not an excuse for exposing a quirky API.

--
 
 

[Simon Ochsenreither]

Hi Eugene,

1) Unfortunately there's none. Not until we have the machinery to changed arbitrary classes (something which is attributed to the macro annotation milestone).

Ok, then I guess it is time to resurrect the @static proposal again. :-)
 

4a) The right thing is to append (), because UnCurry seems to expect a Block and, as far as I can guess, if you omit the (), the rhs of the ctor is treated as a plain expr.

Thanks, I'll do that!
 

4b) Everything works when I change New(...) to Select(New(...), nme.CONSTRUCTOR) in enumInstance. That's very misleading, I agree, but at least I can say that this was mentioned in the docs. I understand though that's not an excuse for exposing a quirky API.

Ooops. Sorry for missing that. I think I just copied the outputs of showRaw here.

Thanks a lot!

Simon

[Simon Ochsenreither]

Except for the static member issue, I can report that it works correctly now! Thanks a lot Eugene!

One weird thing:

If I set
tpt  = Ident(TypeName("Unit")),
instead of
tpt  = TypeTree(),
in enumConstructor, I get the following error:

type mismatch;
[error]  found   : Unit
[error]  required: scalax.Day

That's pretty weird in my opinion, because the () at the end of the constructor body should force the return type to be Unit.
Is this expected?

Bye,

Simon

[Eugene Burmako]

4b) You got me scared for a moment, but I just verified that showRaw actually prints Select(New(...), nme.CONSTRUCTOR). Phew - at least this works as intended.

--
 
 

[Eugene Burmako]

Wow there's something really weird going on. First of all, namers explicitly set tpts of constructors to their enclosing class. Secondly, typers typecheck constructor bodies against WildcardType, not against those tpts. Probably, for now we should avoid touching tpts of constructors.

--
 
 

[Simon Ochsenreither]

4b) You got me scared for a moment, but I just verified that showRaw actually prints Select(New(...), nme.CONSTRUCTOR). Phew - at least this works as intended.

Wow, right. I'm wrong ... no idea where I got that code from.

[Simon Ochsenreither]

Wow there's something really weird going on. First of all, namers explicitly set tpts of constructors to their enclosing class. Secondly, typers typecheck constructor bodies against WildcardType, not against those tpts. Probably, for now we should avoid touching tpts of constructors.

Imho, either constructors ...

• should have the return type of the class
• () should be unnecessary (and wrong)
• the transformations to conform do the JVM requirements should be treated as an implementation detail

... or ...

• should have an return type of Unit
  
• () might be necessary (no idea, seems the automatic Unit-appending transformation happens quite early)
• the transformations to conform do the JVM requirements are an integral part of how reflection works
  

It feels like somehow we are stuck in the middle currently.