java.lang.NoClassDefFoundError: com/google/protobuf/CodedInputStream$

[Zda Kao]

I got this error "User class threw exception: java.lang.NoClassDefFoundError: com/google/protobuf/CodedInputStream$" when I run on AWS EMR Spark:

Release label:emr-6.0.0

Hadoop distribution:Amazon 3.2.1

Applications:Spark 2.4.4, Hive 3.1.2

My configuration runs fine locally:

scalaVersion := "2.12.10"

build.sbt:

libraryDependencies ++= Seq(

...

// ScalaPB with SparkSQL

"com.thesamet.scalapb" %% "sparksql-scalapb" % "0.9.0"
)
// Hadoop contains an old protobuf runtime that is not binary compatible
// with 3.0.0. We shared ours to prevent runtime issues.
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.protobuf.**" -> "shadeproto.@1").inAll,
ShadeRule.rename("scala.collection.compat.**" -> "shadecompat.@1").inAll
)

PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)

assembly.sbt:

resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)

addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.10")

build.properties:

sbt.version = 1.3.13

scalapb.sbt:

addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.27")

libraryDependencies += "com.thesamet.scalapb" %% "compilerplugin" % "0.9.6"

Class:

sbt: com.google.protobuf:protobuf-java:3.8.0:jar

I tried copying the protobuf-java:3.8.0:jar to /home/hadoop/extrajars folder and reference it in the spark default config:

- sudo vim /etc/spark/conf/spark-defaults.conf

spark.driver.extraClassPath :/home/hadoop/extrajars/*

spark.executor.extraClassPath :/home/hadoop/extrajars/* 

I also reference it in the spark-submit:

--driver-library-path /home/hadoop/extrajars/postgresql-42.2.11.jar,/home/hadoop/extrajars/spark-sql-kafka-0-10_2.12-2.4.4.jar,/home/hadoop/extrajars/spark-streaming-kafka-0-10_2.12-2.4.4.jar,/home/hadoop/extrajars/jdbc-4.50.3.jar,/home/hadoop/extrajars/config-1.4.0.jar,/home/hadoop/extrajars/scala-logging_2.13-3.9.2.jar,/home/hadoop/extrajars/sparksql-scalapb_2.12-0.9.0.jar,/home/hadoop/extrajars/frameless-dataset_2.12-0.8.0.jar,/home/hadoop/extrajars/frameless-core_2.12-0.8.0.jar,/home/hadoop/extrajars/scalapb-runtime_sjs0.6_2.12-0.9.6.jar,/home/hadoop/extrajars/lenses_sjs0.6_2.12-0.9.6.jar,/home/hadoop/extrajars/protobuf-java-3.8.0.jar \

--driver-class-path /home/hadoop/extrajars/postgresql-42.2.11.jar,/home/hadoop/extrajars/spark-sql-kafka-0-10_2.12-2.4.4.jar,/home/hadoop/extrajars/spark-streaming-kafka-0-10_2.12-2.4.4.jar,/home/hadoop/extrajars/jdbc-4.50.3.jar,/home/hadoop/extrajars/config-1.4.0.jar,/home/hadoop/extrajars/scala-logging_2.13-3.9.2.jar,/home/hadoop/extrajars/sparksql-scalapb_2.12-0.9.0.jar,/home/hadoop/extrajars/frameless-dataset_2.12-0.8.0.jar,/home/hadoop/extrajars/frameless-core_2.12-0.8.0.jar,/home/hadoop/extrajars/scalapb-runtime_sjs0.6_2.12-0.9.6.jar,/home/hadoop/extrajars/lenses_sjs0.6_2.12-0.9.6.jar,/home/hadoop/extrajars/protobuf-java-3.8.0.jar \

--jars /home/hadoop/extrajars/postgresql-42.2.11.jar,/home/hadoop/extrajars/spark-sql-kafka-0-10_2.12-2.4.4.jar,/home/hadoop/extrajars/spark-streaming-kafka-0-10_2.12-2.4.4.jar,/home/hadoop/extrajars/jdbc-4.50.3.jar,/home/hadoop/extrajars/config-1.4.0.jar,/home/hadoop/extrajars/scala-logging_2.13-3.9.2.jar,/home/hadoop/extrajars/sparksql-scalapb_2.12-0.9.0.jar,/home/hadoop/extrajars/frameless-dataset_2.12-0.8.0.jar,/home/hadoop/extrajars/frameless-core_2.12-0.8.0.jar,/home/hadoop/extrajars/scalapb-runtime_sjs0.6_2.12-0.9.6.jar,/home/hadoop/extrajars/lenses_sjs0.6_2.12-0.9.6.jar,/home/hadoop/extrajars/protobuf-java-3.8.0.jar \

The default version on AWS spark 2.4.4 is protobuf-java-2.5.0.jar.

Any advise is much appreciated.

[Nadav Samet]

Hi Zda, how do you deploy to EMR? The recommended way is to create a fat jar using sbt-assembly, and pass that as a single dependency to spark-submit. It looks like you are passing individual jars manually, some of which are related to Scala.js (those that have sjs0.6 in their name) which is probably not what you want.

-Nadav

--
You received this message because you are subscribed to the Google Groups "ScalaPB" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalapb+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scalapb/425910b9-ff77-4d09-8f51-57a05c379490n%40googlegroups.com.

--

-Nadav

[Zda Kao]

Hi Nadav,

I'm new to scala and I wasn't familiar with sbt. I thought sbt package was the method of creating the fat jar. I was able to use the assembly plugin to create the fat jar now. However, I ran into many duplicate errors so I reduce to only the essential jars for testing and I was able to deploy successfully to AWS EMR Spark. Now I'm going to read up on how to remove duplicates.

Thanks you very much for your help,

Zda