Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Maven and Scala

242 views
Skip to first unread message

Mark Rajcok

unread,
Nov 19, 2016, 9:55:49 PM11/19/16
to Sparkour
Hi Brian,
Thank you very much for your post about Building Spark Applications with Maven.
I've been looking for a Maven pom.xml that I could use to build my Spark-Scala app.

When I first packaged my Scala app using your pom-scala.xml file, it was 7MB.  After adding <scope>provided</scope> to the scala-library artifact, the resulting jar was less than 10k.  (So you might want to add that to your tutorial.)

I'm stuck on Spark 1.5.2 (Scala 2.10) right now.  I'm getting this warning from the build:

[INFO] --- scala-maven-plugin:3.2.2:compile (default) @ test-spark-maven ---
[WARNING]  Expected all dependencies to require Scala version: 2.10.4
[WARNING]  com.att.flood:test-spark-maven:0.0.1 requires scala version: 2.10.4
[WARNING]  com.twitter:chill_2.10:0.5.0 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-remote_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-actor_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  com.typesafe.akka:akka-slf4j_2.10:2.3.11 requires scala version: 2.10.4
[WARNING]  org.apache.spark:spark-core_2.10:1.5.2 requires scala version: 2.10.4
[WARNING]  org.json4s:json4s-jackson_2.10:3.2.10 requires scala version: 2.10.0
[WARNING] Multiple versions of scala libraries detected!

Any ideas how to fix that?
-- Mark


Brian Uri!

unread,
Nov 20, 2016, 8:28:16 AM11/20/16
to Sparkour
Hello Mark,

In your case, the "Multiple versions of scala libraries detected!" error is a false positive. It looks like your effective dependency tree is trying to pull in 2.10.0 and 2.10.4, and all of the 2.10.x versions are supposed to be fully compatible.

There are a few options you can try:

1) If you're running version 3.1.6 or higher of the scala-maven-plugin, you can disable the multiple version check with the checkMultipleScalaVersions property.

http://davidb.github.io/scala-maven-plugin/compile-mojo.html#checkMultipleScalaVersions

2) You can try setting the scalaCompatVersion property to 2.10 so the version check sees both 2.10.x versions as compatible.

http://davidb.github.io/scala-maven-plugin/compile-mojo.html#scalaCompatVersion

3) In the dependency definition for json4s-jackson, you can add an exclusion for Scala so the old 2.10.0 version isn't pulled in. json4s should then use 2.10.4.

<exclusions>
  <exclusion>
    <groupId>org.scala-lang</groupId>
    <artifactId>scala-library</artifactId>
  </exclusion>
</exclusions>

These are all mutually exclusive suggestions, so you should only need to do one of them.

Regards,
BU

Brian Uri!

unread,
Nov 20, 2016, 8:31:01 AM11/20/16
to Sparkour
Mark,

I have also opened SPARKOUR-22 to track the mention of "provided" to reduce the package size. Thanks for the suggestion!

Regards,
BU

Mark Rajcok

unread,
Nov 21, 2016, 11:31:27 AM11/21/16
to Sparkour
Thanks BU,
Thanks for the multiple suggestions. I opted for 2) and this fixed the problem.  Thanks for your help.

  <properties>
      <scala.compat.version>2.10</scala.compat.version>
    </properties>

I'm not directly pulling in json4s-jackson so I didn't want to use 3). 
I wasn't sure if disabling multiple version check would prevent the plugin from catching a case where I was pulling in Scala 2.10 and 2.11, hence I didn't want to use 1).

-- Mark

Brian Uri!

unread,
Nov 21, 2016, 12:48:06 PM11/21/16
to Sparkour
Great, I'm glad the solution worked out!

Regards,
BU
Reply all
Reply to author
Forward
0 new messages