Scala as script

[Nils Kilden-Pedersen]

We run a Scala script every minute, and it appears that the compiled class files are generated every time and stored in /tmp.

Are there any strategies to mitigate or change this?

[Michael Schmitz]

Unless it's changed, there's also problems on shared machines. Files
are created in /tmp with the current user's permissions and some of
them collide with other users--permission denied!

Peace. Michael

[Vlad Patryshev]

My scala script just does the delay itself:

def every(dt: Time)(action: => Unit) = {

  val timer = new java.util.Timer()

  timer.scheduleAtFixedRate(new java.util.TimerTask() {

    def run = action

  }, dt, dt)

  timer

}

...

every(13 seconds) {

   print("\'")

  if (certTime > lastCertTime) {

    println("\n" + new java.util.Date() + "] Got new certificates, restarting rabbitMQ")

    copyRabbitStuff

    restartRabbit

    lastCertTime = certTime

  }

}

Thanks,
-Vlad

On Mon, Jul 23, 2012 at 1:07 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

[Vlad Patryshev]

(also, I believe in using prime numbers as periods)

Thanks,
-Vlad

[Nils Kilden-Pedersen]

That should also work, but I prefer to not have the script be a constantly running process, for a couple of reasons.

[Nathaniel Harward]

You can try setting TMPDIR to something like "$HOME/.tmp" as an immediate fix for permissions (maybe setting -Djava.io.tmpdir=$HOME/.tmp does the same thing, haven't tried).

The scala runtime also has a "-save" option for saving the script as a JAR file for future use to avoid regenerating .class files.

Hope that helps.

[Nils Kilden-Pedersen]

On Mon, Jul 23, 2012 at 5:13 PM, Nathaniel Harward <nhar...@gmail.com> wrote:

You can try setting TMPDIR to something like "$HOME/.tmp" as an immediate fix for permissions (maybe setting -Djava.io.tmpdir=$HOME/.tmp does the same thing, haven't tried).

The scala runtime also has a "-save" option for saving the script as a JAR file for future use to avoid regenerating .class files.

If you mean -savecompiled, then yes, that's what I'm doing now.

Thanks.

(Did it used to be -save, since two people now have referred to it like that?)

[Adam Shannon]

Why not package the script and run it via java -jar?

--
Adam Shannon
Developer
University of Northern Iowa
Junior -- Computer Science B.S. & Mathematics
http://ashannon.us

[Tanner Ezell]

I've had good experience with run jar (and the sbt plugin).

[Nathaniel Harward]

I'm using 2.9.2 and "-save" is the only option I get.
I'm sure the name moves around between versions, but it sounds like the same thing to me.

http://www.scala-lang.org/docu/files/tools/scala.html - the way they phrase how "-savecompiled" works, sounds like the JAR file might go someplace in /tmp (or somewhere else static) so you could potentially get clashes if running multiple instances.  Or at least that's how I read it.

[Eric Kolotyluk]

On 2012-07-23 4:49 PM, Nils Kilden-Pedersen wrote:

On Mon, Jul 23, 2012 at 4:15 PM, Eric Kolotyluk <eric.ko...@gmail.com> wrote:

With my technique, it does not have to be a constantly running process.

And what is your technique?

From a previous response...


Last year I developed some code that basically invokes Scala programs from the command line by:

1. Checking to see if the .class files exist or are newer than the .scala files.
2. If the .class files need to be compiled or recompiled, they are.
3. Invoking the .class files directly.
   

If you are running Scala code frequently this technique speeds things up quite a lot, especially when the JVM is all warmed up. Feel free to hack this for your own use. I wish something like this was part of the standard Scala run time. For example, to run a command I use something like

C:\> intersystem backup

Which calls my Backup.class program to backup my intersystem service. Basically I have intersystem.bat and Perform.scala in my bin directory, as well as well as the folders classes, logs, src in my bin directory for my Scala source scripts, class file and logs. In cases where the .scala files have not changed the .class files are invoked directly. However, you still have the flexibility to change your .scala files as you would with any script, and they simply get recompiled. Note: this technique could also be used to make Java look like a scripting language, but Scala is way more fun to 'script' with than Java.

Cheers, Eric

:: @see http://technet.microsoft.com/en-us/library/cc750054.aspx
:: @see http://www.scala-lang.org/docu/files/tools/scala.html
@echo off
pushd %INTERSYSTEM_HOME%\bin
:: intersystem-service.jar needs to preceed classes - EK
call scala -classpath ..\Service\intersystem-service.jar;classes -Dlog4j.debug Perform.scala com.kodak.intersystem.scripts.Intersystem %*
popd

/* file-encoding: UTF-8
 *
 * Copyright © Kodak 2012
 *
 * Perform.scala
 */

import com.kodak.intersystem.common.Properties
import com.kodak.intersystem.common.Utilities
import java.io.File
import scala.sys
import scala.sys.env
import scala.sys.process.Process
import scala.sys.process.ProcessIO

/**
 * <strong>If necessary, compile any Scala source files, then perform the utility.</strong>
 * <p>
 * Search the local directory 'src' and compare the file there against the
 * intersystem-service.jar file. If the jar file is newer than anything in
 * src then have intersystem-service.jar redeploy the scripts. Finally, search
 * the local directory 'classes' an if necessary, compile any that are newer,
 * or that do not have the corresponding class files.
 * @author Eric Kolotyluk
 */
object Perform {
  val properties = Properties.getScriptProperties() // must call before getLogger() - EK
  lazy val logger = Properties.getLogger(getClass())
 
  var exitValue = 0

  var exit : Boolean = false  // must be set before calling getEnv()
  var scalaBat : String = ""
  var scalaLib : String = ""
  var javaExe : String = ""

  val INTERSYSTEM_HOME = getEnv("INTERSYSTEM_HOME")
  val JAVA_HOME = getEnv("JAVA_HOME")
  val SCALA_HOME = getEnv("SCALA_HOME")

  /**
   * arguments.head - the qualified class name to invoke
   * arguments.tail - the rest of the arguments to pass when invoking main(arguments)
   */
  def main(arguments : Array[String]) {
    try {
      perform(arguments)
    } finally {
      // TODO should not need to use this - EK
      com.kodak.intersystem.common.Application.exit(getClass(), exitValue, false)
      //System.exit(exitValue)
    }
  }
 
  def perform(arguments : Array[String]) {
    if (exit) return
   
    // for debugging - EK
//    println(arguments.foldLeft("perform") {(string, argument) => string + " " + argument})
   
//    println("INTERSYSTEM_HOME = " + INTERSYSTEM_HOME)
//    println("JAVA_HOME = " + JAVA_HOME)
//    println("SCALA_HOME = " + SCALA_HOME)
//   
//    println("java.class.path = " + System.getProperty("java.class.path"))
   
    scalaBat = SCALA_HOME + """\bin\scala.bat"""
    scalaLib = SCALA_HOME + """\lib"""
    javaExe = JAVA_HOME + """\bin\java.exe"""
   
    // If our jar file is newer than any of our scripts then re-deploy the scripts.
    // This ensures that we always have the latest version of the scripts.
    // If you have modified any of the scripts, they will be renamed first
    // with the last-modified time appended to the name.

    val src = new File(INTERSYSTEM_HOME + File.separator + "bin" + File.separator + "src")
    val jar = new File(INTERSYSTEM_HOME + File.separator + "Service" + File.separator + "intersystem-service.jar")
    if (!src.exists || jar.exists && newer(jar.lastModified(), src)) Utilities.deployScripts()

    // Some things fail if this directory does not exist, so make sure it does
    new File(INTERSYSTEM_HOME + File.separator + "bin" + File.separator + "classes").mkdirs

    // Now get a list of files where the source files are newer than the object (.class) files
    // Note: newer returns List[Option[String]] and flatten is used to eliminate any None results - EK
    val newerFiles = newer("src", "classes").flatten.filter(_.endsWith(".scala"))
    println    // make the output look a little cleaner
    if (newerFiles.isEmpty) {
      perform(arguments.head, arguments.tail)
      return
    }
   
    val compilingMessage = "compiling " + newerFiles.reduceLeft[String] { (path, file) => path + " " + file} + "\n"
    logger.info(compilingMessage)
    println(compilingMessage)
   
    val libFile = new File(SCALA_HOME + """\lib""")
    val jarFiles = libFile.listFiles.filter(s=>s.getName.endsWith(".jar"))
    val jarPaths = jarFiles.map(f=>f.getPath)
    val classPath = jarPaths.reduceLeft[String] { (path, file) => path + File.pathSeparator + file }
   
    val classPathMessage = "classPath = " + classPath
    logger.info(classPathMessage)
    println(classPathMessage)
   
    val service = File.pathSeparator + ".." + File.separator + "service" + File.separator + "intersystem-service.jar"
   
    val command = List(javaExe,
      "-Dscala.home=" + SCALA_HOME,
      "-Dscala.usejavacp=true",
      "-classpath", classPath + service + File.pathSeparator + "classes",
      "scala.tools.nsc.Main",
      "-d", "classes") ::: newerFiles
   
    val commandMessage = "command = " + command.reduceLeft({ (path, file) => path + "\n" + file })
    logger.info(commandMessage)
    println(commandMessage)
   
    val processBuilder = Process(command)
    val process = processBuilder.run
    exitValue = process.exitValue // Blocks until this process exits and returns the exit code.
   
    if (exitValue == 0) {
      println("\n")
      perform(arguments.head, arguments.tail)
    } else {
      val compilationMessage = "Compilation failed, exitValue = " + exitValue
      logger.error(compilationMessage)
      println(compilationMessage)
    }
  }
 
  /**
   * We invoke the command object via reflection because it may not
   * be on the classpath yet when this code is compiled. However, this
   * code makes sure that everything has been compiled before it invokes
   * the command object and processes the rest of the command line.
   * We have to be a little bit tricky here because when you declare
   * something as an object, then a singleton instance is already created
   * so we have to find it. Under the hood, Scala class names end with "$"
   * We have to use asInstanceOf[{ def main(arguments : Array[String]) }]
   * because asInstanceOf[Command.type] may not exist at compile time,
   * but we do know the signature of the main method so we use the
   * structural equivalence instead - this is one of the sweeter features
   * of Scala.
   */
  def perform(command : String, arguments : Array[String]) = {
    val className = command + "$"
    try {
      val objectInstance = Class.forName(className).getField("MODULE$").get(null).asInstanceOf[{ def main(arguments : Array[String]) }]
      objectInstance.main(arguments)
    } catch {
      case e : ClassNotFoundException =>
        println("ClassNotFoundException: " + className)
        println("Make sure that " + className + " is a declared singleton Scala oject, and not just a class")
    }
  }
 
//  /**
//   * Return true if time is newer than any source files
//   * @author Eric Kolotyluk
//   */
//  def newer(time:Long, sourceName : String) : Boolean = {
//    def newerOption(sourceName : String) : Option[Boolean] = {
//      val sourceFile = new File(sourceName)
//      if (sourceFile.isDirectory())
//        sourceFile.list.foldLeft(Option.empty[Boolean]) {(result, name) =>
//          val sourcePath = sourceName + File.separator + name
//          result match {
//            case None => newerOption(sourcePath)
//            case Some(boolean1:Boolean) => {
//              def boolean3 = newerOption(sourcePath) match {
//                case None => boolean1
//                case Some(boolean2) => boolean2
//              }
//              Some(boolean1 || boolean3)
//            }
//          }
//        }
//      else
//        Some(sourceName.endsWith(".scala") && sourceFile.lastModified() < time)
//    }
//    newerOption(sourceName).getOrElse(true)
//  }

  /**
   * Return true if time is newer than any source files
   * @author joshua....@gmail.com
   */
  def newer(time : Long, source : File) : Boolean = {
    // (1) We create a 'collection' that can iterate over files.
    object files extends Traversable[File] {
      override def foreach[U](f: File => U) : Unit = {
        def iter(file: File) : Unit =
          if (file.isDirectory) file.listFiles foreach iter  
          else f(file)
        iter(source)
       }
    }
    // (2) This should lazily read file names, using only those that end with .scala
    // and then bail early if it finds *any* of them with time > lastModifed.
    if (files.isEmpty) true
    else files.view filter (_.getName endsWith ".scala") exists (time > _.lastModified)
  }

  /**
   * Return a list of scala source files that are newer than their
   * corresponding class files.
   */
  def newer(sourceName : String, objectName : String) : List[Option[String]] = {
    val sourceFile = new File(sourceName)
    val objectFile = new File(objectName)

    if (sourceFile.isDirectory()) {
        sourceFile.list.foldLeft(List[Option[String]]()) { (result, name) =>
          val sourcePath = sourceName + File.separator + name
          val objectPath = objectName + File.separator + name
          result ::: newer(sourcePath,
            if (sourcePath.endsWith(".scala")) objectPath.replace(".scala", ".class") else objectPath)
        }
    }
    else {
      if (sourceName.endsWith(".scala") && !objectFile.exists() || sourceFile.lastModified() > objectFile.lastModified())
        List(Some(sourceName))
      else
        List(None)
    }
  }

  def getEnv(name : String) = {
    env.get(name) match {
      case Some(s) => s
      case None =>
        println("environment variable " + name + " is not defined!")
        exit = true
        "?"
    }
  }
}

/*    Scala Newbie Notes

    WARNING: there is some functional programming shit going on here,
    so use care if you modify any of the 'newer' methods,
    but this code does not need to change (much) - EK
   
    newer(time : Long, source : File) : Boolean is a good example of Scala
    style in that we separate our concerns by (1) defining a collection-like
    object called 'files' that represents all the files in a given directory,
    and (2) we then use standard Scala primitives that work on collections.
    A Traversable is a trait common to all collections that implement the
    foreach method. The filter and exists methods operate on collections
    because they require foreach.
   
    Note we could remove recursion here with some work. We can make this
    recurse in whatever order we desire for optimum early-exit in the algorithm.
    As our collection of files is typically small we are not worried about
    performance that much.

    newer(sourceName : String, objectName : String) : List[Option[String]]
    is a good example of a neophyte writing functional code. While it is
    not as flexible as the code that separates concerns, it is easier to
    read and write for people with less advanced skills.
   
*/

[Dinotzar Tzar]

On Mon, Jul 23, 2012 at 10:40 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

On Mon, Jul 23, 2012 at 3:10 PM, Dinotzar Tzar <dino...@gmail.com> wrote:

On Mon, Jul 23, 2012 at 10:07 PM, Nils Kilden-Pedersen <nil...@gmail.com> wrote:

We run a Scala script every minute, and it appears that the compiled class files are generated every time and stored in /tmp.

Are there any strategies to mitigate or change this?

Have you tried scala -save your_scrip.scala?

I have not. It prompted me to search the internets, and this page lists the -savecompiled option. Same thing I assume.

Right. Until more recent versions of Scala, it used to be -savecompiled, now it's -save. A very useful but not well advertised feature that speeds up start-up time (after the initial run) by eliminating recompilation of scripts.

D.

 

Thanks.

[Nils Kilden-Pedersen]

On Mon, Jul 23, 2012 at 7:31 PM, Adam Shannon <ad...@ashannon.us> wrote:

Why not package the script and run it via java -jar?

Because one of the reasons to use a script is to change it at will.

[Nils Kilden-Pedersen]

On Mon, Jul 23, 2012 at 9:10 PM, Nathaniel Harward <nhar...@gmail.com> wrote:

I'm using 2.9.2 and "-save" is the only option I get.
I'm sure the name moves around between versions, but it sounds like the same thing to me.

http://www.scala-lang.org/docu/files/tools/scala.html - the way they phrase how "-savecompiled" works, sounds like the JAR file might go someplace in /tmp (or somewhere else static) so you could potentially get clashes if running multiple instances.  Or at least that's how I read it.

Based on my new found experience, the jar goes in the same directory as the script and gets the same name as the Scala script file.

BTW, I'm running 2.9.1-1, so maybe the name changed for 2.9.2.

[Daniel Sobral]

It became -save on 2.10.


--
Daniel C. Sobral

I travel to the future all the time.