java.io.IOException: No FileSystem for scheme: alluxio

2,476 views
Skip to first unread message

Bruno Costa

unread,
Apr 13, 2016, 2:41:17 PM4/13/16
to Alluxio Users
Hi guys,

I've done a simple example with a Spark Local Mode without DHFS as described here: http://www.alluxio.org/documentation/en/Running-Spark-on-Alluxio.html

All configurations were precisely done.

However, when I run the code, this error appears:

> val s = sc.textFile("alluxio://localhost:19998/LICENSE")
> val double = s.map(line => line + line)
> double.saveAsTextFile("alluxio://localhost:19998/LICENSE2")

Py4JJavaError: An error occurred while calling o26.saveAsTextFile.
: java.io.IOException: No FileSystem for scheme: alluxio
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2584)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2591)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.spark.SparkHadoopWriter$.createPathFromString(SparkHadoopWriter.scala:170)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:1059)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1026)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:1026)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:1026)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:952)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:952)
at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:952)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:951)
at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1443)
at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1422)
at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1422)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:150)
at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:111)
at org.apache.spark.rdd.RDD.withScope(RDD.scala:316)
at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1422)
at org.apache.spark.api.java.JavaRDDLike$class.saveAsTextFile(JavaRDDLike.scala:507)
at org.apache.spark.api.java.AbstractJavaRDDLike.saveAsTextFile(JavaRDDLike.scala:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:381)
at py4j.Gateway.invoke(Gateway.java:259)
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
at py4j.commands.CallCommand.execute(CallCommand.java:79)
at py4j.GatewayConnection.run(GatewayConnection.java:209)
at java.lang.Thread.run(Thread.java:745)

Antonio Si

unread,
Apr 13, 2016, 4:18:01 PM4/13/16
to Alluxio Users
I am having the same problem. I am using spark 1.6.1 and alluxio 1.0.1.

Any help would be helpful.

Thanks.

Antonio.

Calvin Jia

unread,
Apr 13, 2016, 8:29:35 PM4/13/16
to Alluxio Users
Hi,

Could you verify that the SPARK_CLASSPATH is set correctly to the location of your Spark compatible Alluxio client jar?

Thanks,
Calvin

Antonio Si

unread,
Apr 13, 2016, 8:44:52 PM4/13/16
to Alluxio Users
Yes, it is correct. It is set to something like:

export SPARK_CLASSPATH=<absolute path to alluxio>/core/client/target/alluxio-core-client-1.0.1-jar-with-dependencies.jar:$SPARK_CLASSPATH

Thanks.

Antonio.

Antonio Si

unread,
Apr 14, 2016, 6:42:39 PM4/14/16
to Alluxio Users
I also run mvn clean package -Pspark -DskipTests, but still get the same problem.

Thanks.

Antonio.

Gene Pang

unread,
Apr 14, 2016, 11:57:13 PM4/14/16
to Alluxio Users
Hi,

I'm not sure if will solve the problem, but can you try adding this to "spark/conf/core-site.xml"?

  <property>
    <name>fs.alluxio.impl</name>
    <value>alluxio.hadoop.FileSystem</value>
  </property>

Thanks,
Gene

Antonio Si

unread,
Apr 15, 2016, 10:10:43 PM4/15/16
to Alluxio Users
This property setting does not help.

However, I manage to get it to work with Spark-1.5.2, but still cannot get it to work with Spark-1.5.1

However, for Spark 1.5.2, when I run the following spark testing code in spark-shell:

val rdd = sc.textFile("file:///some file")
rdd.saveAsTextFile("alluxio://rws66477fwks:19998/test1")

It does not save to the hdfs underfs. I already passed -Dalluxio.user.file.writetype.default=CACHE_THROUGH

What might I do wrong?

Thanks.

Antonio.

Antonio Si

unread,
Apr 16, 2016, 7:12:20 PM4/16/16
to Alluxio Users
Sorry, typo, I meant I can get it to work for Spark 1.5.2, but not Spark 1.6.1

Thanks.

Antonio.

Antonio Si

unread,
Apr 19, 2016, 3:54:13 PM4/19/16
to Alluxio Users
Finally find out the problem for Spark 1.6.1. In my case, I have a SPARK_HOME set to an older version of spark
and thus causing the it to pick up the wrong spark conf file. So, the classpath is wrong. Clearing the SPARK_HOME
fixes the problem.

Jiří Šimša

unread,
Apr 19, 2016, 6:29:21 PM4/19/16
to Antonio Si, Alluxio Users
Thank you for posting your solution.

Best,

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.



--
Jiří Šimša

Bruno Costa

unread,
Apr 19, 2016, 9:55:06 PM4/19/16
to Jiří Šimša, Wagner Vieira, Antonio Si, Alluxio Users
Thank you!!!

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/fvqMYvwcFMM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to alluxio-user...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.



--
 Bruno.

Arun S

unread,
Apr 28, 2016, 8:49:05 AM4/28/16
to Alluxio Users
Same here -

Spark 1.6.1
Alluxio 1.0.1

Spark is not able to recognize the Alluxio file system. Any help in this regard is highly appreciated.

scala> file.saveAsTextFile("alluxio://localhost:19999/")
java.io.IOException: No FileSystem for scheme: alluxio
        at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2644)
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2651)
        at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
        at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)

Gene Pang

unread,
Apr 28, 2016, 10:17:15 AM4/28/16
to Alluxio Users
Hi,

Could you check to see if adding the properties to core-site.xml helps? (http://alluxio.org/documentation/v1.0.1/en/Running-Spark-on-Alluxio.html)

Also, could you check the SPARK_CLASSPATH?

Thanks,
Gene
Reply all
Reply to author
Forward
0 new messages