UnsatisfiedLinkError: no mesos in java.library.path

[Arpan Ghosh]

Hi,

I built and installed a new revision of mesos. I have set 'MESOS_HOME' in 'conf/spark-env.sh' to point to my new mesos installation but I am getting the following error when I try and run a job.

Thanks

Arpan

java.lang.UnsatisfiedLinkError: no mesos in java.library.path
    at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1681)
    at java.lang.Runtime.loadLibrary0(Runtime.java:840)
    at java.lang.System.loadLibrary(System.java:1047)
    at spark.SparkContext.<init>(SparkContext.scala:75)
    at spark.EagerOnlineLDAMain$.main(OnlineLDA.scala:652)
    at spark.EagerOnlineLDAMain.main(OnlineLDA.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:616)
    at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)
    at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)
    at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)
    at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)
    at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)
    at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)
    at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)
    at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)
    at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)
    at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)
    at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

[Matei Zaharia]

Which version of Mesos are you using, and which version of Spark?

For the master branch of Spark, you need an old SVN revision of Mesos listed at https://github.com/mesos/spark/wiki/Running-spark-on-mesos.

For Mesos 0.9 (the recent release we made), you need the mesos-0.9 branch of Spark. Also, in that one, there's no longer a MESOS_HOME setting because Mesos can get installed on the system paths. Instead, there's a MESOS_NATIVE_LIBRARY setting that should point to the libmesos.so (whether you installed Mesos with make install or just built it). It's detailed in the Readme and spark-env.sh.template for that branch, but it's not super obvious.

Matei

[Arpan Ghosh]

We forked from Spark on May 20th and wanted to use a version of mesos right before it.

Spark Version:
commit 10716b1766625d7cb5a75b99b399a879eee27aef
Merge: f487426 d0c6e9f
Author: Matei Zaharia <ma...@eecs.berkeley.edu>
Date:   Fri May 18 15:21:30 2012 -0700

    Merge pull request #131 from rxin/master
   
    Return size estimation, cache usage, and cache capacity from slave nodes to CacheTracker


Mesos Version:
r1338863 | benh | 2012-05-15 19:52:30 +0000 (Tue, 15 May 2012) | 1 line

Added popovers and relative dates to the webui (https://reviews.apache.org/r/5127).

Arpan

[Matei Zaharia]

Yeah, you can't use that Mesos version with the master branch of Spark. Either use the old Mesos, or switch to the mesos-0.9 branch of Spark, because the Mesos API has changed. I'm going to merge mesos-0.9 into master pretty soon by the way, so it will become the default. So if you have a choice, I suggest just basing your code on that.

Matei

[Arpan Ghosh]

Hey Matei,

Following up on this issue. I am doing another Mesos + Spark install and I pulled spark (master) and Mesos (from the mirror) couple of days back. I am still getting the link unsatisfied error. I have set the MESOS_NATIVE_LIBRARY variable in conf/spark-env.sh but when I grep the code, it doesn't seem to be used anywhere. Are spark master and the latest mesos in sync now?

[root@node30 spark]# grep -r MESOS_NATIVE_LIBRARY .

./README.md:- `MESOS_NATIVE_LIBRARY`: Your Mesos library, if you want to run on a Mesos

./conf/spark-env.sh:# - MESOS_NATIVE_LIBRARY, to point to your Mesos native library (libmesos.so)

./conf/spark-env.sh:MESOS_NATIVE_LIBRARY=/network20q/arpan/mesos/lib/libmesos.so

./repl/src/test/scala/spark/repl/ReplSuite.scala:  if (System.getenv("MESOS_NATIVE_LIBRARY") != null) {

Here's my conf/spark-env.sh

# Set Spark environment variables for your site in this file. Some useful

# variables to set are:

# - MESOS_NATIVE_LIBRARY, to point to your Mesos native library (libmesos.so)

MESOS_NATIVE_LIBRARY=/network20q/arpan/mesos/lib/libmesos.so

# - SCALA_HOME, to point to your Scala installation

# - SPARK_CLASSPATH, to add elements to Spark's classpath

# - SPARK_JAVA_OPTS, to add JVM options

# - SPARK_MEM, to change the amount of memory used per node (this should

#   be in the same format as the JVM's -Xmx option, e.g. 300m or 1g).

SPARK_MEM=45g

# - SPARK_LIBRARY_PATH, to add extra search paths for native libraries.

And here's the stack trace:

Failed to load native Mesos library from 

java.lang.UnsatisfiedLinkError: no mesos in java.library.path

at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1681)

at java.lang.Runtime.loadLibrary0(Runtime.java:840)

at java.lang.System.loadLibrary(System.java:1047)

at org.apache.mesos.MesosNativeLibrary.load(MesosNativeLibrary.java:46)

at spark.SparkContext.<init>(SparkContext.scala:77)

at spark.examples.SparkPi$.main(SparkPi.scala:15)

at spark.examples.SparkPi.main(SparkPi.scala)

at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:616)

at scala.tools.nsc.util.ScalaClassLoader$$anonfun$run$1.apply(ScalaClassLoader.scala:78)

at scala.tools.nsc.util.ScalaClassLoader$class.asContext(ScalaClassLoader.scala:24)

at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.asContext(ScalaClassLoader.scala:88)

at scala.tools.nsc.util.ScalaClassLoader$class.run(ScalaClassLoader.scala:78)

at scala.tools.nsc.util.ScalaClassLoader$URLClassLoader.run(ScalaClassLoader.scala:101)

at scala.tools.nsc.ObjectRunner$.run(ObjectRunner.scala:33)

at scala.tools.nsc.ObjectRunner$.runAndCatch(ObjectRunner.scala:40)

at scala.tools.nsc.MainGenericRunner.runTarget$1(MainGenericRunner.scala:56)

at scala.tools.nsc.MainGenericRunner.process(MainGenericRunner.scala:80)

at scala.tools.nsc.MainGenericRunner$.main(MainGenericRunner.scala:89)

at scala.tools.nsc.MainGenericRunner.main(MainGenericRunner.scala)

Thanks so much

Arpan

[Matei Zaharia]

You need to add an "export" when you set MESOS_NATIVE_LIBRARY. So

export MESOS_NATIVE_LIBRARY=/network20q/arpan/mesos/lib/libmesos.so

Matei