Saving Spark DataFrame in Tachyon

[sara mustafa]

Hi All,
I have Spark-1.3.0 and Tachyon-0.5.0. When I am trying to save RDD in tachyon, it success. But for saving a DataFrame it fails with the following error:

java.lang.IllegalArgumentException: Wrong FS: tachyon://localhost:19998/myres, expected: hdfs://localhost:54310
    at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)

Here is my code:
sc.hadoopConfiguration.set("fs.tachyon.impl", "tachyon.hadoop.TFS")
val region = sc.textFile("tachyon://localhost:19998/user/hduser/region.tbl")
val nation = sc.textFile("tachyon://localhost:19998/user/hduser/nation.tbl")
val regionSchemaString = "R_REGIONKEY R_NAME R_COMMENT"
val nationSchemaString = "N_NATIONKEY N_NAME N_REGIONKEY N_COMMENT"
val regionSchema =
    StructType(
      regionSchemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))
val nationSchema =
    StructType(
      nationSchemaString.split(" ").map(fieldName => StructField(fieldName, StringType, true)))
val regionRowRDD = region.map(_.split("\\|")).map(r => Row(r(0), r(1), r(2)))
val nationRowRDD = nation.map(_.split("\\|")).map(r => Row(r(0), r(1), r(2), r(3)))
val regionDataFrame = sqlContext.createDataFrame(regionRowRDD, regionSchema)
val nationDataFrame = sqlContext.createDataFrame(nationRowRDD, nationSchema)
regionDataFrame.registerTempTable("REGION")
nationDataFrame.registerTempTable("NATION")
 val df = sqlContext.sql("select * from REGION r, NATION n  where r.R_REGIONKEY = n.N_REGIONKEY ")
df.count()
df.save("tachyon://localhost:19998/myres")

Could anyone help me please?

Thanks,
 

[Calvin Jia]

Hi,

I think this might be a bug in Spark 1.3.0, for example see SPARK-6330. If possible, it would be great to use Spark 1.4 with Tachyon 0.6.4, which shouldn't have this problem.

Hope this helps,

Calvin