configuration error in assembly jar

[Nathan]

I've found a small problem in the assembly (and only the assembly) jar.

Several included libraries have a reference.conf file - notably akka and spray (each has more than one such, though, for instance, both akka reference.conf files may be identical, I haven't checked that).

When the build creates the core assembly jar, it only puts in one reference.conf file.  If this is the akka version, spark seems to run; if the spary one, it errors on startup.

It seems to be somewhat random which one gets in - I've had it work for a while, then do a new build, then it fails for a while.

I don't see an easy solution to this; does anyone else?

And, when I manage to get it in a state where the akka file is the version getting in, what might be the problems from not having the spray version?

             -Nathan

[Nathan]

I guess I should have dug deeper.

It seems to be concattenating all the reference.conf files.

In that case, I don'e know how to explain my error:

com.typesafe.config.ConfigException$Missing: No configuration setting found for

key 'akka.version'

        at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:126)

        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:146)

        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:151)

        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:119)

        at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:157)

        at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:199

)

        at akka.actor.ActorSystem$Settings.<init>(ActorSystem.scala:136)

        at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:449)

        at akka.actor.ActorSystem$.apply(ActorSystem.scala:111)

        at spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:55)

        at spark.SparkEnv$.createFromSystemProperties(SparkEnv.scala:72)

        at spark.SparkContext.<init>(SparkContext.scala:98)

        at spark.SparkContext.<init>(SparkContext.scala:82)

Yet I see the akka config in the reference.conf file; just when reading the file, it never gets that far.

[Matei Zaharia]

Is this when building Spark, or when building your own application? Maybe your application somehow gets another reference.conf on the classpath before the one we built.

Matei

[Nathan]

My own application.  spark-shell works just fine.

I put the lines:

   val resources = getClass.getClassLoader.getResources("reference.conf")

   println("resources are:")

   while (resources.hasMoreElements()) {

     println("\t"+resources.nextElement())

   }

in AkkaUtils.createActorSystem, between creating akkaConf and using it to create actorSystem, and it tells me it's getting the file from the assembly jar.:

resources are:

        jar:file:lib/spark-core-assembly-0.7.0-SNAPSHOT.jar!/reference.conf

[Matei Zaharia]

Does your project depend on Akka or Spray too by any chance? Maybe it somehow gets another version of the file on the classpath. In any case, maybe one way to fix it is to unzip that assembly, pull out the reference.conf, and add it to your classpath on your driver program before anything else (even before the JAR).

Matei

[Nathan]

Nope, just the one dependency, the core assembly jar

And from the resource test I wrote, I don't think that it isn't finding the file, it's just not able to use it for some reason.

           =Nathan

[Nathan]

never mind, I'm an idiot.  I wasn't restarting sbt in my project when copying in a new assembly jar, so it wasn't reloading it, so it had bad information on where each file in the jar was located or something.

Restarting sbt seems to have fixed it.