Spark: No FileSystem for scheme "alluxio"

36 views
Skip to first unread message

brijesh gautam

unread,
Jun 21, 2023, 9:20:13 AM6/21/23
to Alluxio Users
HI 

I am using Minikube hosted Kubectl cluster for testing out Alluxio and Spark. As mentioned on https://docs.alluxio.io/os/user/stable/en/kubernetes/Spark-On-Kubernetes.html , copied alluxio-2.9.3-client.jar to spark-3.4.0-bin-hadoop3/jars directory  and build image using command which is mentioned on this Alluxio Page.  
Uploaded image to Minikube and then triggered  spark-submit command. But got following exception: 
23/06/21 13:16:07 INFO SharedState: Warehouse path is 'file:/opt/spark/work-dir/spark-warehouse'.
23/06/21 13:16:08 WARN FileStreamSink: Assume no metadata directory. Error while looking for metadata directory in the path: alluxio://alluxio-master-0:19998/LICENSE.
org.apache.hadoop.fs.UnsupportedFileSystemException: No FileSystem for scheme "alluxio"
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:3443)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:3466)
at org.apache.hadoop.fs.FileSystem.access$300(FileSystem.java:174)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:3574)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:3521)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:540)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:365)
at org.apache.spark.sql.execution.streaming.FileStreamSink$.hasMetadata(FileStreamSink.scala:53)
at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:366)

Please let me know how to resolve this exception. 
Regards 
Brijesh 
Reply all
Reply to author
Forward
0 new messages