Spark shell read file on Tachyon error

[chenxu...@gmail.com]

I create issues at jira 

https://tachyon.atlassian.net/browse/TACHYON-751

https://tachyon.atlassian.net/browse/TACHYON-751

there are two issues :

File does not exist the underfs

I'm use tachyon with Spark 1.4.1 for Test, when I stop -> format -> start the master ,and lookup the webui, I can see the file that before restart loaded(copyFromLocal command) , But when I read or open it show me the error log

015-08-05 18:21:48,984 ERROR MASTER_LOGGER (HdfsUnderFileSystem.java:getFileLocations) - Unable to get file location for hdfs://data1:8020/tmp/tachyon/data/49
java.io.FileNotFoundException: File does not exist: hdfs://data1:8020/tmp/tachyon/data/49
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1128)
        at org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1120)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1120)
        at tachyon.underfs.hdfs.HdfsUnderFileSystem.getFileLocations(HdfsUnderFileSystem.java:223)
        at tachyon.master.BlockInfo.getLocations(BlockInfo.java:171)
        at tachyon.master.BlockInfo.generateClientBlockInfo(BlockInfo.java:123)
        at tachyon.master.InodeFile.getClientBlockInfo(InodeFile.java:266)
        at tachyon.master.MasterInfo.getClientBlockInfo(MasterInfo.java:1205)
        at tachyon.master.MasterServiceHandler.user_getClientBlockInfo(MasterServiceHandler.java:183)
        at tachyon.thrift.MasterService$Processor$user_getClientBlockInfo.getResult(MasterService.java:2995)
        at tachyon.thrift.MasterService$Processor$user_getClientBlockInfo.getResult(MasterService.java:2979)
        at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
        at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39)
        at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:225)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

Spark shell read tachyon file infinite circle

Use Spark shell to read tachyon file

// Spark-shell
sc.textFile("tachyon://name1:19998/tpch/nation.tbl").count

it show me log that infinite circle

15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
15/08/06 10:21:30 WARN : Read nothing
......

[chenxu...@gmail.com]

I lookup the log again , Dose spark-shell alway connect to local worker ? or master machine should start a worker ? when I start a worker at master node , read file just fine , But when I stop the worker on master node , read file goes to infinite circle show me the log "WARN : Read nothing" 

I confused for above problems 

How to config cluster ? dose tachyon not works like Namenode / Datanode ? 

[Calvin Jia]

Hi,

What version of Tachyon are you using? Spark 1.4.1 works with Tachyon 0.6.4 out of the box.

Thanks,

Calvin

[Lei Fan]

I'm seeing the same issue too, though for me, I have no problem opening the file in the web UI.

I'm using Tachyon 0.7.0.

Thanks,

Lei

[Calvin Jia]

Hi Lei,

Please use Tachyon 0.6.4 with Spark 1.4.1. Alternatively you can recompile spark with tachyon-client 0.7.0.

Hope this helps,

Calvin

[Haoyuan Li]

You can also find the compatibility table here : http://tachyon-project.org/master/Running-Spark-on-Tachyon.html

Thanks,

Haoyuan

--
You received this message because you are subscribed to the Google Groups "Tachyon Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tachyon-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Haoyuan Li

CEO, Tachyon Nexus

AMPLab, EECS, UC Berkeley http://www.cs.berkeley.edu/~haoyuan/

[chenxu...@gmail.com]

Thanks Haoyuan , I 'll recompile Spark 1.4.1 with tachyon 0.7.0 and try to test again .

在 2015年8月7日星期五 UTC+8上午7:53:03，Haoyuan Li写道：

[Lei Fan]

Thanks Calvin! I'll go back to 0.6.4 and recompile Spark for 0.7.0 in the meantime.

Lei