Re: problem reading file from spark

Jiří Šimša

unread,

Mar 1, 2016, 10:07:09 AM3/1/16

to Nguyen khac tu, Alluxio Users

It looks like the Alluxio jar is not in your Spark classpath. Take a look at these instructions.

On Mon, Feb 29, 2016 at 11:39 PM, Nguyen khac tu <tub...@gmail.com> wrote:

i 'm trying to integrate Alluxio with Spark but when i try to read an text file from Alluxio it throws me some errors :

sc.hadoopConfiguration.set("fs.alluxio.impl", "alluxio.hadoop.FileSystem")
sc.textFile("alluxio://192.168.30.128/data/hosts")

res4.first

res0.first

error:

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class alluxio.hadoop.FileSystem not found

at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)

at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2379)

at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)

at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)

at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)

at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)

at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)

at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)

at org.apache.hadoop.mapred.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:256)

at org.apache.hadoop.mapred.FileInputFormat.listStatus(FileInputFormat.java:228)

at org.apache.hadoop.mapred.FileInputFormat.getSplits(FileInputFormat.java:304)

at org.apache.spark.rdd.HadoopRDD.getPartitions(HadoopRDD.scala:199)
.........................

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Jiří Šimša

Alluxio, Inc.

Nguyen khac tu

unread,

Mar 1, 2016, 10:32:12 AM3/1/16

to Alluxio Users, tub...@gmail.com

yep , i fixed it , mistake come from me ( not including client library of alluxio )

Bruno Costa

unread,

Apr 13, 2016, 12:26:17 PM4/13/16

to Alluxio Users, tub...@gmail.com

I have the same error... how can I include such library?

Jiří Šimša

unread,

Apr 14, 2016, 9:49:10 AM4/14/16

to Bruno Costa, Alluxio Users, Nguyen khac tu

Hello Bruno, do you have the following line in your spark-env.sh?

export SPARK_CLASSPATH=/pathToAlluxio/core/client/target/alluxio-core-client-1.0.1-jar-with-dependencies.jar:$SPARK_CLASSPATH

Reply all

Reply to author

Forward