FileNotFoundException when save spark dataset to alluxio using CACHE_THROUGH

[forest M]

versions:

spark2.0.0

alluxio1.2.0

java1.7.0_45

centos6.7

when create sparksession, use this config:

builder.config("alluxio.user.file.writetype.default", "CACHE_THROUGH");

alluxio-env.sh

ALLUXIO_UNDERFS_ADDRESS="/disk2/alluxio-underFSStorage"

after a dataset created, 

String path = "alluxio://ip:port/dataset/";

ds.write().mode(SaveMode.Overwrite).save(path + "ds1.rec");

and then an exception come out:
Caused by: java.io.FileNotFoundException: /disk2/alluxio-underFSStorage/dataset/ds1.rec/_temporary/0/_temporary/attempt_201611240909_0001_m_000000_3/part-r-00000-0605ea28-283a-48c4-8bae-7fc0a5c6d0e5.snappy.parquet.alluxio.0x7B25791C16070780.tmp (No such file or directory)
        at java.io.FileOutputStream.open(Native Method)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:221)
        at java.io.FileOutputStream.<init>(FileOutputStream.java:110)
        at alluxio.underfs.local.LocalUnderFileSystem.create(LocalUnderFileSystem.java:77)
        at alluxio.client.file.FileOutStream.<init>(FileOutStream.java:124)
        at alluxio.client.file.BaseFileSystem.createFile(BaseFileSystem.java:102)
        at alluxio.hadoop.AbstractFileSystem.create(AbstractFileSystem.java:153)
        at alluxio.hadoop.FileSystem.create(FileSystem.java:25)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
        at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
        at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:176)
        at org.apache.parquet.hadoop.ParquetFileWriter.<init>(ParquetFileWriter.java:160)
        at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:289)
        at org.apache.parquet.hadoop.ParquetOutputFormat.getRecordWriter(ParquetOutputFormat.java:262)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetOutputWriter.<init>(ParquetFileFormat.scala:548)
        at org.apache.spark.sql.execution.datasources.parquet.ParquetFileFormat$$anon$1.newInstance(ParquetFileFormat.scala:138)
        at org.apache.spark.sql.execution.datasources.BaseWriterContainer.newOutputWriter(WriterContainer.scala:131)
        at org.apache.spark.sql.execution.datasources.DefaultWriterContainer.writeRows(WriterContainer.scala:247)
        at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
        at org.apache.spark.sql.execution.datasources.InsertIntoHadoopFsRelationCommand$$anonfun$run$1$$anonfun$apply$mcV$sp$1.apply(InsertIntoHadoopFsRelationCommand.scala:143)
        at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
        at org.apache.spark.scheduler.Task.run(Task.scala:85)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
        ... 3 more

how could I solve it?

[Haoyuan Li]

Any chance to try 1.3.0 release instead?

Haoyuan

Alluxio, Inc. 

--
You received this message because you are subscribed to the Google Groups "Alluxio Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to alluxio-users+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gene Pang]

Hi,

How many nodes are you running? Is your Alluxio Master on the same node as your Alluxio Worker?

It looks like your underfs address is pointing to a local disk path: /disk2/alluxio-underFSStorage . This will not work if your Alluxio master and Alluxio worker are not on the same machine. The UnderFS Address should be accessible from any Alluxio Master, Alluxio Worker, and application client.

Thanks,

Gene

[forest M]

yes, I'll try later, thanks!

在 2016年11月24日星期四 UTC+8下午2:17:04，Haoyuan Li写道：

[forest M]

Hi, gene.

There're 4 nodes -- one master and four workers, and the master is on the same node as the first worker.

When my application deployed on the Alluxio Master node, it does work, and the spark dataset only store on the Master node. 

But on the  Alluxio Worker node, it doesn't!

So,

1. Why should the application be deployed on the Alluxio Master node?

2. Why does the data only store on the Alluxio Master's underfs?

(If the underfs points to a HDFS path, the two questions above will not exist.)

Thank you!

forest

在 2016年11月27日星期日 UTC+8上午3:28:15，Gene Pang写道：

[Gene Pang]

Hi Forest,

The application does not have to be deployed on the master node. Alluxio is typically used with distributed computation frameworks like MapReduce, Spark or Flink so they are run across many nodes in the cluster.

The underfs is not specific to a master or worker. It should be a filesystem for the entire system. That is why the underfs should not point to a local directory of a particular machine, because it would be specific to a the machine you are on. The underfs address should be accessible by any node in the cluster. A local path is only "globally" accessible by every master, worker or application if everything is running on a single node.

Hope that helps,

Gene

[Gene Pang]

Hi Forest,

Were you able to resolve your issue?

Thanks,

Gene

[forest M]

Hi Gene,

Thank you very much. I've understood the mechanism of underfsStorage, 'local' mode is for single node, 'hdfs' (and other distributed filesystem) mode is for multi-node.

在 2016年12月21日星期三 UTC+8下午10:25:19，Gene Pang写道：

[Haoyuan Li]

Forest, "local" mode is for testing only. For production use, please use a distributed persistent storage as Alluxio's under storage: http://www.alluxio.org/docs/master/en/Getting-Started.html ;

 

Best regards,

Haoyuan

Alluxio, Inc. 

--