Using spark to access HDFS failed

[Hao REN]

I am using Cloudera 4.2.0 and Spark.

I just want to try out some examples given by Spark.

// HdfsTest.scala
package spark.examples

import spark._

object HdfsTest {
  def main(args: Array[String]) {
    val sc = new SparkContext(args(0), "HdfsTest",
      System.getenv("SPARK_HOME"), Seq(System.getenv("SPARK_EXAMPLES_JAR")))

    val file = sc.textFile("hdfs://n1.example.com/user/cloudera/data/navi_test.csv")
    val mapped = file.map(s => s.length).cache()
    for (iter <- 1 to 10) {
      val start = System.currentTimeMillis()
      for (x <- mapped) { x + 2 }
      //  println("Processing: " + x)
      val end = System.currentTimeMillis()
      println("Iteration " + iter + " took " + (end-start) + " ms")
    }
    System.exit(0)
  }
}

It's ok for compiling, but there is always some runtime problems:

Exception in thread "main" java.util.ServiceConfigurationError: org.apache.hadoop.fs.FileSystem: Provider org.apache.hadoop.hdfs.HftpFileSystem could not be instantiated: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.DelegationTokenRenewer.<init>(Ljava/lang/Class;)V from class org.apache.hadoop.hdfs.HftpFileSystem
    at java.util.ServiceLoader.fail(ServiceLoader.java:224)
    at java.util.ServiceLoader.access$100(ServiceLoader.java:181)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:377)
    at java.util.ServiceLoader$1.next(ServiceLoader.java:445)
    at org.apache.hadoop.fs.FileSystem.loadFileSystems(FileSystem.java:2229)
    at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2240)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2257)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:86)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2296)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2278)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:316)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:162)
    at org.apache.hadoop.mapred.JobConf.getWorkingDirectory(JobConf.java:587)
    at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:315)
    at org.apache.hadoop.mapred.FileInputFormat.setInputPaths(FileInputFormat.java:288)
    at spark.SparkContext.hadoopFile(SparkContext.scala:263)
    at spark.SparkContext.textFile(SparkContext.scala:235)
    at spark.examples.HdfsTest$.main(HdfsTest.scala:9)
    at spark.examples.HdfsTest.main(HdfsTest.scala)
Caused by: java.lang.IllegalAccessError: tried to access method org.apache.hadoop.fs.DelegationTokenRenewer.<init>(Ljava/lang/Class;)V from class org.apache.hadoop.hdfs.HftpFileSystem
    at org.apache.hadoop.hdfs.HftpFileSystem.<clinit>(HftpFileSystem.java:84)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
    at java.lang.Class.newInstance0(Class.java:374)
    at java.lang.Class.newInstance(Class.java:327)
    at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:373)
... 16 more

val file = sc.textFile("hdfs://n1.example.com/user/cloudera/data/navi_test.csv") is where the problem occurs.

13/04/04 12:20:51 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

And I got this Warning. Maybe I should add some hadoop paths in CLASS_PATH.

Feel free to give any clue. =)

Thank you all.

REN Hao

[Matei Zaharia]

You need to compile Spark against that version of Hadoop. Take a look at http://spark-project.org/docs/latest/#a-note-about-hadoop-versions.

Matei

--
You received this message because you are subscribed to the Google Groups "Spark Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to spark-users...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.