Making scala repl's class generation customizable.

[Prashant Sharma]

.... copy pasting the content from Pull Request: 2788 ....

We at spark, have to customize scala repl to support our way of "code wrapping" for input code. The reason we want it that way is spark needs code serializable so that it can be executed on a remote worker.

I wish it was possible to configure rather than port scala repl code for every scala release migration. This pull request mayonly present my intention which is, we may not need a complete overhauling of scala REPL to support it and thus there should be a simpler solution to it. It also has a sample customization which uses class instead of object for wrapping code to be executed. The sample customization is exactly what we would need for spark i.e. serializable. Actually there is more space for making things in the repl customizable, but then this is just a beginning. This has been done in this particular way only to achieve the goal in minimal code change at expense of elegance. Of course suggestions and reviews comments are welcomed for me being on lurker side trying to step into space that might not be welcoming for its technical depth.

For Example

scala> val a = 10
a: Int = 10

scala> val b = a * 2 // show
class $read extends Serializable {
  def <init>() = {
    super.<init>;
    ()
  };
  class $iwC extends Serializable {
    def <init>() = {
      super.<init>;
      ()
    };
    val $VAL1 = $line3.$read.INSTANCE;
    import $VAL1.$iw.$iw.a;
    class $iwC extends Serializable {
      def <init>() = {
        super.<init>;
        ()
      };
      val b = a * 2
    };
    val $iw = new $iwC.<init>
  };
  val $iw = new $iwC.<init>
}
object $read extends scala.AnyRef {
  def <init>() = {
    super.<init>;
    ()
  };
  val INSTANCE = new $read.<init>
}
b: Int = 20

scala> 

An issue on jira to track this: https://issues.scala-lang.org/browse/SI-7747

[Prashant Sharma]

looping in Matei Zaharia in the discussion. I might have missed out a few details that he might want to add.

--
s

[Matei Zaharia]

Thanks for looping me in! Just FYI, I would also be okay if instead of making the wrapper code pluggable, the REPL just changed to one based on classes, as in Prashant's example, rather than singleton objects.

To give you background on this, the problem with the "object" wrappers is that initialization code goes into a static initializer that will have to run on all worker nodes, making the REPL unusable with distributed applications. As an example, consider this:

// file.txt is a local file on just the master

val data = scala.io.Source.fromFile("file.txt").mkString

// now we use the derived string, "data", in a closure that runs on the cluster

spark.textFile.map(line => doStuff(line, data))

The current Scala REPL creates an object Line1 whose static initializer sets data with the code above, then does import Line1.data in the closure, which will cause the static initializer to run *again* on the remote node and fail. This issue definitely affects Spark, but it could also affect other interesting projects that could be built on Scala's REPL, so it may be an interesting thing to consider supporting in the standard interpreter.

Matei

[Som Snytt]

I got a bit up to speed last night and I see that this is just a small part of your REPL customization needs.

For this baby customization step, maybe either a -Y as you suggest or, my last incomplete thought was a -P.  I'll follow up on that idea today.

What is the time frame if I wanted to contribute something?  M5 is close, right?

It would be fun if Spark REPL could extend something built for extension.  Even for something like disabling power mode.

You'd hate to think of downstream REPLs not benefiting from ongoing improvements because of maintenance.

Footnote, now you can step outside the wrapper using raw paste. I'm not sure that addresses the use case.

I also noticed the code to delay <init> and wonder if this would be a good use of DelayedInit.

--
You received this message because you are subscribed to the Google Groups "scala-internals" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-interna...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Prashant Sharma]

There is no urgency except that we just wished this feature in scala 2.11 atleast(to save us the effort for next time). Apart from that I would love to wait for your and others suggestions and having spent some time on it already it might be possible to help out, only in case that makes some sense. 

--
s