no under file system factory found for hdfs

[William Callaghan]

I know this has been brought up a few times, and I've tried a number of things to no avail:

Spark 1.6.0

Alluxio 1.1.1 (recompiled for Hadoop 2.7.1)

Followed the following tutorials:

http://www.alluxio.org/docs/master/en/Running-Hadoop-MapReduce-on-Alluxio.html

http://www.alluxio.org/docs/master/en/Configuring-Alluxio-with-HDFS.html

Also tried recompiling Spark Core with the alluxio-underfs-hdfs dependency added in core/pom.xml

The Alluxio tests work fine for underfs and I can see those files in Alluxio and HDFS.

However, from Spark, when writing a dataframe it will not persist to hdfs (CACHE_THROUGH is set but I will get the no under file system factory found error).

[Gene Pang]

Hi,

Could you provide additional details on what is happening?

Could you describe what application you are running? What is the error message you are getting? How many Alluxio workers are you using?

Thanks,

Gene

[William Callaghan]

So after struggling with this for a day or so:

Passing the packages alluxio-underfs-hdfs and alluxio-core-client right to spark-submit seemed to do the trick. Usually I don't like doing this so I will continue to look into the problem and report back if I find another solution.

[William Callaghan]

I should add that I'm using the spark-job-server project and I've noticed a number of issues similar to this (with other packages): I'll build a jar with the dependencies, submit it to the job server and there will be some issue related to a dependency. However passing in those jars directly to spark-submit resolves the issue.

[Gene Pang]

Hi,

What is the error you are seeing?

Did you use the jar-with-dependencies client jar? If you are using Spark, you should be using the Alluxio Spark client jar, which can be found here: http://downloads.alluxio.org/downloads/files/1.1.1/ for your Alluxio version.

Thanks,

Gene

[William Callaghan]

As mentioned above, passing the client jar to spark-submit solves the problem. Thanks.

[Gene Pang]

Ok, thanks for the confirmation!

-Gene