Exception is happened when running hadoop mapreduce on tachyon

[Zhiwen Sun]

When I followed this guide to run wordcount example on tachyon.

I got following exception:

15/04/03 17:07:40 INFO : Tachyon client (version 0.6.2) is trying to connect master @ h10-051/10.10.8.51:19999
15/04/03 17:07:40 ERROR : Frame size (1213486160) larger than max length (16384000)!
tachyon.org.apache.thrift.transport.TTransportException: Frame size (1213486160) larger than max length (16384000)!
    at tachyon.org.apache.thrift.transport.TFramedTransport.readFrame(TFramedTransport.java:137)
    at tachyon.org.apache.thrift.transport.TFramedTransport.read(TFramedTransport.java:101)
    at tachyon.org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    at tachyon.org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:362)
    at tachyon.org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:284)
    at tachyon.org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:191)
    at tachyon.org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
    at tachyon.thrift.MasterService$Client.recv_user_getUserId(MasterService.java:710)
    at tachyon.thrift.MasterService$Client.user_getUserId(MasterService.java:698)
    at tachyon.master.MasterClient.connect(MasterClient.java:207)
    at tachyon.master.MasterClient.user_getUfsAddress(MasterClient.java:568)
    at tachyon.client.TachyonFS.getUfsAddress(TachyonFS.java:685)
    at tachyon.hadoop.AbstractTFS.initialize(AbstractTFS.java:362)
    at tachyon.hadoop.TFS.initialize(TFS.java:25)
    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2308)
    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:87)
    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2342)
    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2324)
    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:351)
    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:194)
    at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.addInputPath(FileInputFormat.java:368)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:65)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:72)
    at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:64)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:208)

I have searched this exception, but I can not find useful solution. Does anyone meet this exception?

Any suggestion are appreciated.

The stack of my cluster is :

 * CDH 4.5.0 (hadoop 2.0.0)
 * tachyon 0.6.2

[Haoyuan Li]

Seems it's connecting to the wrong port. Please use 19998 in the URI. 19999 is for the web ui.

Best,

Haoyuan

--
You received this message because you are subscribed to the Google Groups "Tachyon Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tachyon-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Haoyuan Li

AMPLab, EECS, UC Berkeley

http://www.cs.berkeley.edu/~haoyuan/

[Zhiwen Sun]

Haoyuan:

Thanks for your reply!

It is my fault, use the wrong port.

It works when I list a directory in tachyon. But new exception is occurred when I try to cat a file use following command:

HADOOP_CLASSPATH=/tmp/tachyon-0.6.2-jar-with-dependencies.jar hadoop dfs -cat tachyon://h10-051:19998/tmp/README.md

SLF4J: Found binding in [jar:file:/tmp/tachyon-0.6.2-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
15/04/04 10:40:20 INFO : initialize(tachyon://h10-051:19998/tmp/README.md, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Tachyon: tachyon://h10-051:19998/tmp/README.md
15/04/04 10:40:20 WARN : tachyon.home is not set. Using /mnt/tachyon_default_home as the default value.
15/04/04 10:40:20 INFO : Tachyon client (version 0.6.2) is trying to connect master @ h10-051/10.10.8.51:19998
15/04/04 10:40:20 INFO : User registered at the master h10-051/10.10.8.51:19998 got UserId 7
15/04/04 10:40:20 INFO : tachyon://h10-051:19998 tachyon://h10-051:19998 /root/tachyon-0.6.2/libexec/../underfs
15/04/04 10:40:20 INFO : getFileStatus(/tmp/README.md): HDFS Path: /root/tachyon-0.6.2/underfs/tmp/README.md TPath: tachyon://h10-051:19998/tmp/README.md
15/04/04 10:40:20 INFO : getWorkingDirectory: /
15/04/04 10:40:20 INFO : open(tachyon://h10-051:19998/tmp/README.md, 65536)
15/04/04 10:40:20 INFO : Trying to get local worker host : h10-040
15/04/04 10:40:20 INFO : No local worker on h10-040
15/04/04 10:40:20 INFO : Connecting remote worker @ /10.10.8.51:29998
15/04/04 10:40:20 WARN : Recache attempt failed.
java.io.IOException: The machine does not have any local worker.
    at tachyon.client.BlockOutStream.<init>(BlockOutStream.java:94)
    at tachyon.client.BlockOutStream.<init>(BlockOutStream.java:65)
    at tachyon.client.RemoteBlockInStream.read(RemoteBlockInStream.java:204)
    at tachyon.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:142)
    at java.io.DataInputStream.read(DataInputStream.java:83)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:78)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
    at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:86)
    at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:81)
    at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
    at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
    at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
    at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
    at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
    at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
    at org.apache.hadoop.fs.FsShell.run(FsShell.java:254)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.fs.FsShell.main(FsShell.java:304)
15/04/04 10:40:20 INFO : Try to find remote worker and read block 111669149696 from 0, with len 1228
15/04/04 10:40:20 INFO : Block locations:[NetAddress(mHost:h10-051, mPort:-1, mSecondaryPort:-1)]
15/04/04 10:40:20 INFO : Block locations:[NetAddress(mHost:h10-051, mPort:-1, mSecondaryPort:-1)]
15/04/04 10:40:20 INFO : Opening stream from underlayer fs: /root/tachyon-0.6.2/underfs/tmp/tachyon/data/104
15/04/04 10:40:20 ERROR : /root/tachyon-0.6.2/underfs/tmp/tachyon/data/104 (Permission denied)
java.io.FileNotFoundException: /root/tachyon-0.6.2/underfs/tmp/tachyon/data/104 (Permission denied)
    at java.io.FileInputStream.open(Native Method)
    at java.io.FileInputStream.<init>(FileInputStream.java:120)
    at java.io.FileInputStream.<init>(FileInputStream.java:79)
    at tachyon.UnderFileSystemSingleLocal.open(UnderFileSystemSingleLocal.java:178)
    at tachyon.client.RemoteBlockInStream.setupStreamFromUnderFs(RemoteBlockInStream.java:371)
    at tachyon.client.RemoteBlockInStream.read(RemoteBlockInStream.java:229)
    at tachyon.hadoop.HdfsFileInputStream.read(HdfsFileInputStream.java:142)
    at java.io.DataInputStream.read(DataInputStream.java:83)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:78)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:52)
    at org.apache.hadoop.io.IOUtils.copyBytes(IOUtils.java:112)
    at org.apache.hadoop.fs.shell.Display$Cat.printToStdout(Display.java:86)
    at org.apache.hadoop.fs.shell.Display$Cat.processPath(Display.java:81)
    at org.apache.hadoop.fs.shell.Command.processPaths(Command.java:306)
    at org.apache.hadoop.fs.shell.Command.processPathArgument(Command.java:278)
    at org.apache.hadoop.fs.shell.Command.processArgument(Command.java:260)
    at org.apache.hadoop.fs.shell.Command.processArguments(Command.java:244)
    at org.apache.hadoop.fs.shell.Command.processRawArguments(Command.java:190)
    at org.apache.hadoop.fs.shell.Command.run(Command.java:154)
    at org.apache.hadoop.fs.FsShell.run(FsShell.java:254)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.fs.FsShell.main(FsShell.java:304)

[Calvin Jia]

Hi,

It seems like there is a permissions issue preventing the client from reading the file. However, its strange that its a local file path when you are using hdfs as the underlying filesystem. What's your UNDERFS_ADDRESS set to in conf/tachyon-env.sh?

Thanks,

Calvin

[Zhiwen Sun]

Hi Calvin:

I use the default settings, the value of TACHYON_UNDERFS_ADDRESS is $TACHYON_HOME/underfs

[Zhiwen Sun]

When the exception is occurred, I find the following exception in tachyon master:

2015-04-07 10:27:43,026 INFO  MASTER_LOGGER (MasterInfo.java:getWorker) - getLocalWorker: no local worker on h10-040

Does it means that I need install worker in client machine (h10-040)?

But I just wanna read a file in tachyon. Not write.

[Calvin Jia]

Hi Zhiwen,

You shouldn't need to run a worker to do reads, the issue is your read type is probably `cache` which requires a worker. Could you try using the Tachyon client to do the read instead?

Also, what's your current setup like? You have HDFS running and Tachyon 0.6.2 running, but Tachyon is not using HDFS as the under filesystem?

Thanks,

Calvin

[Zhiwen Sun]

Hi Calvin:

Have tried to use tachyon client in h10-040, the same problem : 

"/root/tachyon-0.6.2/underfs/tmp/tachyon/data/104 (Permission denied)"

I have setup a tachyon test node in h10-051 and have another HDFS cluster, h10-040 is the gateway of the HDFS cluster.

Finally, I want to setup another tachyon cluster alone, but current applications can access it according  HDFS command.

[Zhiwen Sun]

When I run tachyon under root user, the error info is "No such file or directory", so I think tachyon tried to read local filesystem when there is no worker progress in h10-040.

strace tachyon, I find such system call:

[pid 14866] open("/root/tachyon-0.6.2/underfs/tmp/tachyon/data/104", O_RDONLY) = -1 ENOENT (No such file or directory)

[mengfang shen]

Hi, Zhiwen

I think you had not configured TACHYON_UNDERFS_ADDRESS properly. Do you use hdfs as tachyon under filesystem ? If so,

you need to configure conf/tachyon-env.sh as TACHYON_UNDERFS_ADDRESS=hdfs://[namenodehost]:[namenodeport]

Thanks,

Sun.

在 2015年4月7日星期二 UTC+8上午11:26:31，Zhiwen Sun写道：

[Zhiwen Sun]

Hi, Sun:


Thanks for your reply. When I change UNDERFS from local directory to HDFS, it works.

Does it means that client communicate with HDFS cluster directly to read data?

[Calvin Jia]

Hi,

Glad it worked out!

When you specify the underfs as HDFS, data that Tachyon does not currently have in memory will be read from there.