serialization error

[Marcin Jurek]

Hey,

I'm running the notebook on Mesos v. 0.20.1 with spark 1.2.1 and scala 2.10.4. When I run a simple spark job on a cluster it completes with no problem (the job being, say:

val numbers = sc.parallelize(1 to 1000)

numbers.count()

I'm trying to test the Spark on Mesos using Cassandra notebook shipped with the distribution. I change the configuration to the following (IP changed):

spark.tachyonStore.folderName: spark-d0f3d40d-02ed-4940-ad9c-de1c2e1ad2c8

spark.mesos.executor.home: /home/user01/spark-1.2.1/

spark.mesos.native.library: /usr/local/lib/libmesos.so

spark.jars: 

spark.driver.port: 46762

spark.executor.memory: 2G

spark.executor.id: driver

spark.master: mesos://10.14.56.156:5050

spark.app.id: 20150309-091002-2620919306-5050-1823-0002

spark.fileserver.uri: http://10.14.56.156:37835

spark.repl.class.uri: http://10.14.56.156:44950

spark.driver.host: 10.14.56.156

spark.app.name: Notebook on dnode-3

The job is shipped to the nodes but none of the tasks completes (has status lost) and eventually the whole job fails.

On each of the worker nodes I get the following output on stderr:

Spark assembly has been built with Hive, including Datanucleus jars on classpath

Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties

15/03/10 10:41:42 INFO MesosExecutorBackend: Registered signal handlers for [TERM, HUP, INT]

I0310 10:41:42.468286 26382 exec.cpp:132] Version: 0.20.1

I0310 10:41:42.470656 26398 exec.cpp:206] Executor registered on slave 20150309-091002-2620919306-5050-1823-2

15/03/10 10:41:42 INFO MesosExecutorBackend: Registered with Mesos as executor ID 20150309-091002-2620919306-5050-1823-2 with 1 cpus

java.io.InvalidClassException: scala.Tuple2; local class incompatible: stream classdesc serialVersionUID = 3356420310891166197, local class serialVersionUID = -4864544146559264103

at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:617)

at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1622)

at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1517)

at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1771)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)

at java.io.ObjectInputStream.readArray(ObjectInputStream.java:1706)

at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1344)

at java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)

at org.apache.spark.util.Utils$.deserialize(Utils.scala:78)

at org.apache.spark.executor.MesosExecutorBackend.registered(MesosExecutorBackend.scala:65)

Exception in thread "Thread-1" I0310 10:41:42.538856 26398 exec.cpp:413] Deactivating the executor libprocess

I tried searching for this error online and found the following question:

http://stackoverflow.com/questions/20957572/how-do-i-fix-a-bug-in-the-scala-library-jar

which suggests downgrading the Scala version (below 2.10) which I am unwilling to do because of compatibility with other systems we are running.

Could you guys offer some suggestions how to solve it? Thanks!

Marcin

This email and any attachments thereto may contain private, confidential, and/or privileged material for the sole use of the intended recipient. Any review, copying, or distribution of this email (or any attachments thereto) by others is strictly prohibited. If you are not the intended recipient, please contact the sender immediately and permanently delete the original and any copies of this email and any attachments thereto.

[andy petrella]

Hello Marcin,

It's typically a clash with the scala versions :-/.

I just checked locally the serial for Tuple2 using the master branch with 2.10.4 and 2.11.2 like so:

$ sbt console
[info] Starting scala interpreter...
[info] 
Welcome to Scala version 2.10.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_72).
Type in expressions to have them evaluated.
Type :help for more information.
scala> java.io.ObjectStreamClass.lookup(classOf[scala.Tuple2[_,_]]).getSerialVersionUID
res2: Long = -4864544146559264103

and 

$ sbt -Dscala.version=2.11.2 console
[info] Starting scala interpreter...
[info] 
Welcome to Scala version 2.11.2 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_72).
Type in expressions to have them evaluated.
Type :help for more information.
scala> java.io.ObjectStreamClass.lookup(classOf[scala.Tuple2[_,_]]).getSerialVersionUID
res0: Long = 3356420310891166197

So there is something fishy out-there...

But we can check some stuffs, for instance which version of the notebook you're using: for 2.11.2?

The name of the zip/docker/deb contains the scala version since v0.3.0 so it's easy to double check. To triple check, you can also look into the lib directory in your installed notebook.

If it's the version you'd like, hence it should be in mesos I guess, which executor is deployed? Note that I don't see the spark.executor.uri pointing to the tar.gz in hdfs/fs in the spark conf you gave me, is that ok?

The first check you can do is which tar.gz is deployed by mesos on the slaves. Anf to double check you can head to the <mesos-cwd>/workspace/slaves/<executorid>/<frameworkId>/run/latest to find the tar.gz to be used.

When you have this guy the first thing to check is also the name which should point out which scala version is used. Otherwise, like for the notebook you could simply open the tar and see in its lib directory what's going on.

If you need some help (maybe shall you head to the gitter so the help can be smoother than by mail ;-).

FYI: https://gitter.im/andypetrella/spark-notebook

HTH,

in any case, keep me posted :-D

Cheers,

andy

--
You received this message because you are subscribed to the Google Groups "spark-notebook-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-notebook-user+unsubscribe...@googlegroups.com.
To post to this group, send email to spark-notebook-user@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spark-notebook-user/D12482AB.C668%25marcin.jurek%40fireeye.com.
For more options, visit https://groups.google.com/d/optout.

[Marcin Jurek]

Alright, I'll try to catch you on gitter then.

And thanks, it helped. The version of the notebook was inappropriate for the version of scala I was using. Once I set both to 2.10 it worked! Thanks!

[andy petrella]

Awesome
Actually, scala + spark + hadoop + jetty + jets3 + akka + guava versions → freaking out mess...

:-D

To unsubscribe from this group and stop receiving emails from it, send an email to spark-notebook-...@googlegroups.com.
To post to this group, send email to spark-not...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/spark-notebook-user/D1271323.C72D%25marcin.jurek%40fireeye.com.

[Howard Wang]

Hi, all,

I met with similar problem when trying to modify the "AnomalyDetection.snb" example notebook to be run on HDP YARN cluster:

From the cell including "dataFrame = sqlContext.jsonFile(jsonFileHDFS).cache" and following ones, I got this error message:

ERROR [Remote-akka.actor.default-dispatcher-14] (notebook.kernel.Repl) - Ooops, exception in the cell java.lang.ExceptionInInitializerError org.apache.spark.SparkException: Job aborted due to stage failure: Exception while getting task result: java.io.InvalidClassException: org.apache.spark.unsafe.ty pes.UTF8String; local class incompatible: stream classdesc serialVersionUID = 7786395165093970948, local class serialVersionUID = 7459647620003804432

If I remove ".cache", this cell can run correctly with output, but some of the follow cells such as "labeledNumericFrame.take(1)(0)" still failed with the same error message.

I had run this notebook successfully before, but for unknown reasons it failed to run 2 days later.  The only environment difference on the HDP  cluster was an error report about log folder free space below 1G.  I fixed it by releasing disk space and relocating log file locations, but the notebook still couldn't run correctly.  The other changes might have effects were our attempts to test loading our own jar libs with (:cp, :dp, etc) in another separate snb file, without any global configuration change in application.conf etc.  And I've searched and removed all .ivy2 cached files left over by the other tests.

What do you think is the possible cause?

The Spark Notebook version:

Build: |  buildTime-Tue Dec 22 10:18:45 UTC 2015 | formattedShaVersion-0.6.2-7c7b07797474ce69a7edeee78cd1c1df09bd5730 | sbtVersion-0.13.8 | scalaVersion-2.10.4 | sparkNotebookVersion-0.6.2 | hadoopVersion-2.7.1 | jets3tVersion-0.7.1 | jlineDef-(org.scala-lang,2.10.4) | sparkVersion-1.5.2 | withHive-true | withParquet-true |.

The HDP version: HDP-2.3.4.0

HDFS        2.7.1.2.3
YARN        2.7.1.2.3
Hive        1.2.1.2.3
ZooKeeper        3.4.6.2.3
Spark        1.5.2.2.3

The Spark related configuration in notebook meta data:

    "customLocalRepo": "/tmp/repo",
    "customRepos": null,
    "customDeps": null,
    "customImports": null,
    "customArgs": null,
    "customSparkConf": {
      "hdp.version": "2.3.4.0-3485",
      "spark.app.name": "AnomalyDetectionDemo",
      "spark.master": "yarn-client",
      "spark.driver.memory": "5G",
      "spark.executor.memory": "1G",
      "spark.yarn.jar": "hdfs:///user/spark/spark-assembly-1.5.2.2.3.4.0-3485-hadoop2.7.1.2.3.4.0-3485.jar",
      "spark.serializer.extraDebugInfo": "true"
    }

The complete snb file is attached.

Thanks in advance for any kind help.

Regards,

Howard.