Tez over Tachyon

[Nishant Agrawal]

Hi All,

I am student from UW-Madison and as part of our course project we are planning to run Tez over Tachyon to evaluate the benefits of Tachyon in such setting.

We are planning to use,

1. HDFS for persistent layer.
2. Ubuntu OS
3. Tez as execution framework and Hive to run some standard TPC-DS & TPCH queries.
4. We will be using latest version of Techyon.

It would be great to know if Tachyon will work in this setting ? and steps to proceed as we didn't find much documentation for running Tez over Tachyon.
We will soon be trying to deploy Tachyon with Map Reduce and Spark to get more idea about setup issues.

Thanks for your help in advance.

Regards,
Nishant

[Gene Pang]

Hi Nishant,

Thanks for your interest in running Tez on Tachyon!

I am not aware of others running Tez on Tachyon, so I don't think there is a guide for running Tez on Tachyon. However, since Tez is a distributed framework to run DAGs for processing data (sounds similar to Spark?), Tachyon should be able to work with Tez. Feel free to ask additional questions as you get into deploying Tez on Tachyon.

Thanks,

Gene

[NISHANT AGRAWAL]

Hi Gene,

We are able to run the MapReduce jobs on top of tachyon but we are not able to run Hive jobs using MR as execution framework.

Do you have any idea if hive is supported with Tachyon or not? Any pointers will be of good help.

Regards,

Nishant

[Jiří Šimša]

Hi Nishant,

Could you please let the mailing list know the steps you followed to run a MapReduce job on top of Tez + Tachyon? As for Hive, could you describe your setup and the problem you are experiencing in more detail? Thanks.

Best,

--

Jiří Šimša

--
You received this message because you are subscribed to the Google Groups "Tachyon Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tachyon-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Nishant Agrawal]

Hi Jiri,

I wasn't able to run Map/Reduce on top of Tez. I was able to run just pure MapReduce jobs.

Current stack look like:

Hive

MR

Tachyon

HDFS

In this stack if I just run the MR jobs it runs perfectly. I just followed the steps given on Tachyon website. However, if I try to run any Hive query it fails.

To make Hive read/write to Tachyon, I have modified core-site.xml of hadoop to make Tachyon as default dfs. But it's causing below error:

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/users/nish191/software/tez-0.7.1-SNAPSHOT/tachyon-client-0.8.3-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/tez-0.7.1-SNAPSHOT/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/hadoop-2.6.0/share/hadoop/common/lib/tachyon-client-0.8.3-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
2015-11-29 22:52:47,882 INFO  [main]  (AbstractTFS.java:initialize(331)) - initialize(tachyon://10.10.1.1:19998/wordcount/input.txt, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Tachyon: tachyon://10.10.1.1:19998/wordcount/input.txt
2015-11-29 22:52:47,893 INFO  [main]  (ConfUtils.java:loadFromHadoopConfiguration(91)) - Loading Tachyon properties from Hadoop configuration: {tachyon.user.file.understoragetype.default=SYNC_PERSIST}
2015-11-29 22:52:47,939 INFO  [main]  (ClientBase.java:connect(112)) - Tachyon client (version 0.8.3-SNAPSHOT) is trying to connect with BlockMaster master @ /10.10.1.1:19998
2015-11-29 22:52:47,962 INFO  [main]  (ClientBase.java:connect(120)) - Client registered with BlockMaster master @ /10.10.1.1:19998
2015-11-29 22:52:48,013 INFO  [main]  (ClientBase.java:connect(112)) - Tachyon client (version 0.8.3-SNAPSHOT) is trying to connect with FileSystemMaster master @ /10.10.1.1:19998
2015-11-29 22:52:48,014 INFO  [main]  (ClientBase.java:connect(120)) - Client registered with FileSystemMaster master @ /10.10.1.1:19998
2015-11-29 22:52:48,030 INFO  [main]  (AbstractTFS.java:initialize(352)) - tachyon://10.10.1.1:19998 tachyon://10.10.1.1:19998 hdfs://10.10.1.1:8020
2015-11-29 22:52:48,031 INFO  [main]  (AbstractTFS.java:getWorkingDirectory(319)) - getWorkingDirectory: /
2015-11-29 22:52:48,031 INFO  [main]  (AbstractTFS.java:getWorkingDirectory(319)) - getWorkingDirectory: /
2015-11-29 22:52:48,881 INFO  [main] impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(285)) - Timeline service address: http://10.10.1.1:8188/ws/v1/timeline/
2015-11-29 22:52:49,011 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /10.10.1.1:8032
2015-11-29 22:52:49,137 INFO  [main] client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at /10.10.1.1:10200
2015-11-29 22:52:49,525 INFO  [main]  (AbstractTFS.java:getFileStatus(271)) - getFileStatus(tachyon://10.10.1.1:19998/wordcount/output): HDFS Path: hdfs://10.10.1.1:8020/wordcount/output TPath: tachyon://10.10.1.1:19998/wordcount/output
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory tachyon://10.10.1.1:19998/wordcount/output already exists
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

With the change in core-site.xml even MR jobs are failing. To run the MR jobs successfully over Tachyon I had to give hdfs as default dfs in core-site.xml. I think, MR jobs reads from Tachyon because it gets hint from I/P & O/P files. But for hive I don't see any other option than changing core-site.xml ( which apparently isn't working ).

Any help to run Hive over MR and Tachyon is greatly appreciated.

Regards,
Nishant

Regards,

Nishant

--
You received this message because you are subscribed to a topic in the Google Groups "Tachyon Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/tachyon-users/-fBVsEAJmuM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to tachyon-user...@googlegroups.com.

[Jiří Šimša]

Hi Nishant,

I don't think that the error you are seeing is related to core-site.xml settings. The issue is that you are trying create a file that already exists in Tachyon. In that case, your job will fail because Tachyon does not allow your job to overwrite an existing file. This could for instance happen if you try to run the same word count example (with the same input / output arguments) multiple times. To run the same word count example again, you first need to remove the "output" file created by the previous run of the job. To do that, you can use the following command: "${TACHYON_HOME}/bin/tachyon tfs rmr /wordcount/output".

Hope this helps.

Best,

--

Jiří Šimša

[Nishant Agrawal]

Hi Jiri,

Sorry for the confusion. The actually error stack trace is pasted below. I mistakenly pasted wrong error.

SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/users/nish191/software/tez-0.7.1-SNAPSHOT/tachyon-client-0.8.3-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/tez-0.7.1-SNAPSHOT/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/hadoop-2.6.0/share/hadoop/common/lib/tachyon-client-0.8.3-SNAPSHOT-jar-with-dependencies.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/users/nish191/software/hadoop-2.6.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]

2015-11-29 23:44:36,217 INFO  [main]  (AbstractTFS.java:initialize(331)) - initialize(tachyon://10.10.1.1:19998/wordcount/input.txt, Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, yarn-default.xml, yarn-site.xml, hdfs-default.xml, hdfs-site.xml). Connecting to Tachyon: tachyon://10.10.1.1:19998/wordcount/input.txt
2015-11-29 23:44:36,222 INFO  [main]  (ConfUtils.java:loadFromHadoopConfiguration(91)) - Loading Tachyon properties from Hadoop configuration: {}
2015-11-29 23:44:36,241 INFO  [main]  (ClientBase.java:connect(112)) - Tachyon client (version 0.8.3-SNAPSHOT) is trying to connect with BlockMaster master @ /10.10.1.1:19998
2015-11-29 23:44:36,251 INFO  [main]  (ClientBase.java:connect(120)) - Client registered with BlockMaster master @ /10.10.1.1:19998
2015-11-29 23:44:36,272 INFO  [main]  (ClientBase.java:connect(112)) - Tachyon client (version 0.8.3-SNAPSHOT) is trying to connect with FileSystemMaster master @ /10.10.1.1:19998
2015-11-29 23:44:36,273 INFO  [main]  (ClientBase.java:connect(120)) - Client registered with FileSystemMaster master @ /10.10.1.1:19998
2015-11-29 23:44:36,280 INFO  [main]  (AbstractTFS.java:initialize(352)) - tachyon://10.10.1.1:19998 tachyon://10.10.1.1:19998 hdfs://10.10.1.1:8020
2015-11-29 23:44:36,280 INFO  [main]  (AbstractTFS.java:getWorkingDirectory(319)) - getWorkingDirectory: /
2015-11-29 23:44:36,281 INFO  [main]  (AbstractTFS.java:getWorkingDirectory(319)) - getWorkingDirectory: /
2015-11-29 23:44:37,238 INFO  [main] impl.TimelineClientImpl (TimelineClientImpl.java:serviceInit(285)) - Timeline service address: http://10.10.1.1:8188/ws/v1/timeline/
2015-11-29 23:44:37,367 INFO  [main] client.RMProxy (RMProxy.java:createRMProxy(98)) - Connecting to ResourceManager at /10.10.1.1:8032
2015-11-29 23:44:37,492 INFO  [main] client.AHSProxy (AHSProxy.java:createAHSProxy(42)) - Connecting to Application History server at /10.10.1.1:10200
2015-11-29 23:44:37,536 INFO  [main] mapreduce.Cluster (Cluster.java:initialize(113)) - Failed to use org.apache.hadoop.mapred.YarnClientProtocolProvider due to error: Error in instantiating YarnClient
java.io.IOException: Cannot initialize Cluster. Please check your configuration for mapreduce.framework.name and the correspond server addresses.
    at org.apache.hadoop.mapreduce.Cluster.initialize(Cluster.java:120)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:82)
    at org.apache.hadoop.mapreduce.Cluster.<init>(Cluster.java:75)
    at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1266)
    at org.apache.hadoop.mapreduce.Job$9.run(Job.java:1262)

    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)

    at org.apache.hadoop.mapreduce.Job.connect(Job.java:1261)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1290)

    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at org.apache.hadoop.examples.WordCount.main(WordCount.java:87)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
    at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
    at org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:74)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:606)
    at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

Regards,

Nishant

[Jiří Šimša]

Hi Nishant,

This looks like a configuration error outside of Tachyon and it is not clear to me what is causing it. Is that the full stack trace? Have you tried running the same command with an HDFS URI instead of Tachyon URI?

Best,

--

Jiří Šimša

[ashish kumar]

Hi All,

I started using Hive with Alluxio. And after few initial hurdles I'm able to make it work. Now, Hive with MR is working fine but the same query failing with Hive on TEZ engine.

Hive Version 1.2

Alluxio Version 1.0.1

TEZ version 0.7

Below is the error log:

Error:

org.apache.tez.dag.api.TezUncheckedException: Unable to instantiate class with 1 arguments: org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator
	at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:80)
	at org.apache.tez.common.ReflectionUtils.createClazzInstance(ReflectionUtils.java:98)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager.createInitializer(RootInputInitializerManager.java:137)
	at org.apache.tez.dag.app.dag.RootInputInitializerManager.runInputInitializers(RootInputInitializerManager.java:114)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.setupInputInitializerManager(VertexImpl.java:4392)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.access$4300(VertexImpl.java:199)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:3233)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:3183)
	at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:3164)
	at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
	at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1812)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:198)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1943)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1929)
	at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
	at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
	at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
	at org.apache.tez.common.ReflectionUtils.getNewInstance(ReflectionUtils.java:69)
	... 20 more
Caused by: java.lang.NoSuchMethodError: org.apache.commons.codec.binary.Base64.decodeBase64(Ljava/lang/String;)[B
	at org.apache.hadoop.hive.ql.exec.Utilities.getBaseWork(Utilities.java:419)
	at org.apache.hadoop.hive.ql.exec.Utilities.getMapWork(Utilities.java:308)
	at org.apache.hadoop.hive.ql.exec.tez.HiveSplitGenerator.<init>(HiveSplitGenerator.java:92)
	... 25 more
2016-08-11 08:37:57,399 ERROR [Dispatcher thread: Central] impl.VertexImpl: Can't handle Invalid event V_START on vertex Map 1 with vertexId vertex_1470806535809_0006_1_00 at current state NEW
org.apache.hadoop.yarn.state.InvalidStateTransitonException: Invalid event: V_START at NEW
	at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:305)
	at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
	at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
	at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1812)
	at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:198)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1943)
	at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1929)
	at org.apache.tez.common.AsyncDispatcher.dispatch(AsyncDispatcher.java:183)
	at org.apache.tez.common.AsyncDispatcher$1.run(AsyncDispatcher.java:114)
	at java.lang.Thread.run(Thread.java:745)

Anybody have any idea?

-Ashish

[Gene Pang]

Hi,

I am not familiar with how Tez works, but does Tez use Hive? Does Hive use Tez? In the errors you have shown, I don't see any Alluxio errors in there?

Thanks,

Gene

[ashish kumar]

Hi Gene,

Hive is having TEZ as a processing engine like MR. I was just expecting if there is any compatible version of TEZ can be used with Alluxio. Because the error shows some dependency versions mismatch.

Thanks,

Ashish

[Gene Pang]

Hi Ashish,

I see. It is possible that there are some dependency conflicts. Is there an error message that shows that?

One thing you could try is to build Alluxio with the same hadoop version that Tez is compiled with. Details can be found here: http://www.alluxio.org/docs/master/en/Building-Alluxio-Master-Branch.html#distro-support

It would also be helpful to know which package is in conflict.

Thanks,

Gene

[Gene Pang]

Hi Ashish,

Were you able to resolve your issue?

Thanks,

Gene