Cascading 3.0 WIP with Apache Tez platform support

Chris K Wensel

unread,

Sep 17, 2014, 2:28:14 PM9/17/14

to cascadi...@googlegroups.com, patter...@googlegroups.com, lingua...@googlegroups.com, us...@tez.apache.org

Hey all

Happy to let everyone know we just pushed out a new 3.0 WIP that supports Apache Tez as a runtime platform.

The official announcement is here:

http://www.cascading.org/2014/09/17/cascading-3-0-wip-now-supports-apache-tez/

Note the wip-50 download doesn't include hadoop2-tez.jar, it will show up in wip-51. But conjars has the binaries.

http://conjars.org/cascading/cascading-hadoop2-tez

And the JavaDoc can be found here:

http://docs.concurrentinc.com/cascading/3.0/javadoc/cascading-hadoop2-tez/

The current working wip branch:

https://github.com/cwensel/cascading/tree/wip-3.0

Notes and knowns issues can be tracked here:

https://github.com/cwensel/cascading/tree/wip-3.0/cascading-hadoop2-tez

Know all platform tests do pass, and the resulting binaries have been pushed to conjars.org.

That said, we expect there to be issues, so please email the cascading-user list with any questions or bugs.

http://www.cascading.org/support/

We have also updated the sample apps to use Tez, other projects will also be updated as well over time.

https://github.com/Cascading/cascading.samples/tree/wip-3.0

cheers,

chris

--

Chris K Wensel

ch...@concurrentinc.com

http://concurrentinc.com

Chris K Wensel

unread,

Sep 17, 2014, 7:01:36 PM9/17/14

to cascadi...@googlegroups.com

fyi

3.0.0-wip-51 is up with all the bits

http://www.cascading.org/wip/

and if you play with the samples below, know you need to checkout wip-3.0 branch. we try to keep the latest stable branches as the default in most projects.

ckw

Oscar Boykin

unread,

Sep 17, 2014, 8:01:39 PM9/17/14

to cascadi...@googlegroups.com, scaldi...@googlegroups.com

Those who are interested in scalding support here, please join scalding-dev.

It will be great to get some help in getting this running ASAP.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/C0821723-15C4-4DEB-A415-02282756E2C7%40wensel.net.

For more options, visit https://groups.google.com/d/optout.

--

Oscar Boykin :: @posco :: http://twitter.com/posco

Luis Casillas

unread,

Sep 18, 2014, 4:26:50 PM9/18/14

to cascadi...@googlegroups.com

Well, I gave this a quick try yesterday, and I probably did something wrong but I did manage to get one of my flows to run under Tez. Hopefully my brief notes might be of help to somebody:

I had to modify the flow explicitly set the gatherPartitions for it to run. I just copied the code from one of the example apps.
Because I was in a hurry and cheapened things out, I didn't properly install Tez and instead just included libraries as regular compile-time dependencies instead of runtime dependencies like the examples do.
The hadoop jar command hung after my flow completed. This is probably because I didn't do a proper install of Tez.

Looking good!

Luis

This message and any files or text attached to it are intended only for the recipients named above, and contain information that is confidential or privileged. If you are not an intended recipient, you must not read, copy, use or disclose this communication. Please also notify the sender by replying to this message, and then delete all copies of it from your system.

Chris K Wensel

unread,

Sep 18, 2014, 4:53:34 PM9/18/14

to cascadi...@googlegroups.com

Thanks for the feedback..

check out:

https://github.com/cwensel/cascading/tree/wip-3.0/cascading-hadoop2-tez#notes-and-known-issues

I think it covers most of your issues, which implies that your hang was a result of running in Tez local mode, so you need to add a System.exit

ckw

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/ca9b692f-5e0b-4651-8937-df42cefc727d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Chris K Wensel

unread,

Oct 2, 2014, 6:16:54 PM10/2/14

to cascadi...@googlegroups.com

Hey all

Just a quick update.

We are still actively improving our support for Apache Tez. And there is an imminent 0.5.1 release of Tez that we also plan to update too.

That said, we have some improvements in the pipeline but our CI server farm is face down in the dirt squirming around right now (has been for a week). So I can't validate changes at the level I'm comfortable with or even cut a new WIP. The tests currently run for 4.5 hours, that in and of itself adds some lag to wips (which means we need to propose a new project and release structure at some point).

For those wanting to play with Cascading 3 on Tez, I suggest you hold off till the next wip release. Hopefully by Monday/Tuesday.

When that happens, feel free to clone our vagrant tez setup

https://github.com/cwensel/vagrant-cascading-hadoop-cluster/blob/wip-3.0-tez-vmware/README-tez.md

right now its pinned to vmware, since some folk think its more stable. once some dust settles we will make it work with both virtual box and vmware.

ckw

On Sep 17, 2014, at 11:28 AM, Chris K Wensel <ch...@wensel.net> wrote:

Christopher Severs

unread,

Jan 21, 2015, 8:01:01 PM1/21/15

to scaldi...@googlegroups.com, cascadi...@googlegroups.com

Has anyone poked at this lately? I've got most stuff compiling but the DistributedCache changed significantly in Hadoop 2.x which is screwing things up. Basically DistributedCache still exists but doesn't seem to have any of the previous Hadoop 1.x methods and the new preferred way is to use the actual Hadoop Job object to add files to the cache.

Any ideas?

Chris K Wensel

unread,

Jan 21, 2015, 8:19:23 PM1/21/15

to Christopher Severs, scaldi...@googlegroups.com, cascadi...@googlegroups.com

outside of libraries and such, DistCacheTap is the preferred method for handing data to a HashJoin small side.

in Cascading 2, you can have Checkpoint taps decorated with it on the fly

http://docs.cascading.org/cascading/2.6/javadoc/cascading-core/cascading/flow/FlowConnectorProps.html

in Cascading 3 on MR, this should continue to work, so say the tests.

But on tez we haven’t ported DistCacheTap over yet since the mechanism is different as you noticed (relies on YARN apis). fwiw, Checkpoints are not honored in Tez. they currently aren’t possible via the Tez api.

if interested, happy to see a pull request for a tez flavored DistCacheTap, we hope to get to it soonish otherwise.

ckw

You received this message because you are subscribed to the Google Groups "Scalding Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalding-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net

Oscar Boykin

unread,

Feb 16, 2015, 9:02:44 PM2/16/15

to Christopher Severs, scaldi...@googlegroups.com, cascadi...@googlegroups.com

Chris, somehow I didn't notice this email from almost a month ago.

What is the latest? Have you been able to run anything (even commenting out the distributed cache API)?

--
You received this message because you are subscribed to the Google Groups "Scalding Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scalding-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Christopher Severs

unread,

Feb 17, 2015, 7:37:03 PM2/17/15

to cascadi...@googlegroups.com, chris....@gmail.com, scaldi...@googlegroups.com

I'm mostly stuck on Cascading 3 changes. I don't think they're all that related to Tez. Everything compiles nicely except maple, which I've run into a few blocks on. I showed the branch I'm working on to Chris W. and co to see if they can help out a bit since they're a lot more familiar with the Cascading 3 changes.

I did try without changing maple but in that case a lot of the Scalding tests fail and I didn't look carefully enough to see if it was due to maple but I suspect having maple using cascading-hadoop and everything else using cascading-tez is not helping.

Chris K Wensel

unread,

Feb 27, 2015, 1:44:02 PM2/27/15

to cascadi...@googlegroups.com, patter...@googlegroups.com, us...@tez.apache.org, lingua...@googlegroups.com

Hey all

Just a heads up we are up to wip-72 now.

This moves Tez to 0.6.0, and changes the default Hadoop version to 2.6. Tez intends to remain compatible with Hadoop 2.4. So from a runtime perspective, you shouldn’t see a difference.

This also provides deep counter support to the task/slice level. This in turn requires a YARN history server to be running, things will gracefully downgrade without it.

I’ve also updated the README to include help on using Tez on Amazon EMR

https://github.com/cwensel/cascading/blob/wip-3.0/cascading-hadoop2-tez/README.md

As a convenience we are hosting Tez binaries in our S3 buckets.

https://github.com/cwensel/cascading/blob/wip-3.0/cascading-hadoop2-tez/README.md#tez-binaries

Lastly, we only have a few more items to build out for Tez listed below before we are feature complete for 3.0

http://docs.cascading.org/roadmap/cascading/cascading-3.html

If you haven’t done so, please test your apps and languages on 3.0 so we can address any issues, especially API changes, before 3.0 final.

We are also publishing 3.0 Fluid artifacts as well.

http://www.cascading.org/fluid/

ckw

On Sep 17, 2014, at 11:28 AM, Chris K Wensel <ch...@wensel.net> wrote:

—

Chris K Wensel

ch...@wensel.net

Luis Casillas

unread,

Feb 27, 2015, 5:32:17 PM2/27/15

to cascadi...@googlegroups.com, patter...@googlegroups.com, us...@tez.apache.org, lingua...@googlegroups.com

On Friday, February 27, 2015 at 10:44:02 AM UTC-8, Chris K Wensel wrote:

Hey all

Just a heads up we are up to wip-72 now.

This moves Tez to 0.6.0, and changes the default Hadoop version to 2.6. Tez intends to remain compatible with Hadoop 2.4. So from a runtime perspective, you shouldn’t see a difference.

Thanks! I gave another try at running our flow in Cascading/Tez and EMR. I've gotten further ahead than last time, but still having problems.

First: it would be enormously valuable to figure out how to submit a Cascading/Tez application as a step to an EMR cluster, instead of running it by logging in to the master and issuing commands. I had a shot at this and the challenge was that EMR bootstrap actions run before HDFS is up. I think the solution here is to have both a bootstrap action (to set up environment variables, configuration files and such) and an EMR step that performs the actual copy of the jars, but I haven't had time to try this yet.

Second: the README you linked still says tez-0.5.3, I think you mean to update that to 0.6.0.

Third: I had to hardcode the -DH arguments in the README to my app, because I couldn't figure out in 5 minutes the correct way to specify options to the hadoop jar command so that my Cascading app's option processor doesn't see them. Not difficult to do, I'm sure, but the command template shown in the README for kicking off an app didn't actually work.

Fourth: my application (which sources from and sinks to S3, and uses HfsProps.setUseCombinedInput()) fails with two ClassNotFoundExceptions, one for com.amazon.ws.emr.hadoop.fs.EmrFileSystem and the other for cascading.tap.hadoop.Hfs$CombinedInputFormat.

Vertex failed, vertexName=DF233A0DD77540CCB87B54D63CD5CE02, vertexId=vertex_1425073885856_0004_1_02, diagnostics=[Vertex init failed : java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2379)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2392)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:89)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2431)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2413)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:368)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter.(FileOutputCommitter.java:91)
at org.apache.hadoop.mapred.FileOutputCommitter.getWrapped(FileOutputCommitter.java:65)
at org.apache.hadoop.mapred.FileOutputCommitter.setupJob(FileOutputCommitter.java:131)
at org.apache.hadoop.mapred.OutputCommitter.setupJob(OutputCommitter.java:213)
at org.apache.tez.mapreduce.committer.MROutputCommitter.setupOutput(MROutputCommitter.java:91)
at org.apache.tez.dag.app.dag.impl.VertexImpl$2.run(VertexImpl.java:1965)
at org.apache.tez.dag.app.dag.impl.VertexImpl$2.run(VertexImpl.java:1945)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.tez.dag.app.dag.impl.VertexImpl.initializeCommitters(VertexImpl.java:1945)
at org.apache.tez.dag.app.dag.impl.VertexImpl.initializeVertex(VertexImpl.java:1975)
at org.apache.tez.dag.app.dag.impl.VertexImpl.access$4300(VertexImpl.java:183)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.handleInitEvent(VertexImpl.java:3033)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2942)
at org.apache.tez.dag.app.dag.impl.VertexImpl$InitTransition.transition(VertexImpl.java:2923)
at org.apache.hadoop.yarn.state.StateMachineFactory$MultipleInternalArc.doTransition(StateMachineFactory.java:385)
at org.apache.hadoop.yarn.state.StateMachineFactory.doTransition(StateMachineFactory.java:302)
at org.apache.hadoop.yarn.state.StateMachineFactory.access$300(StateMachineFactory.java:46)
at org.apache.hadoop.yarn.state.StateMachineFactory$InternalStateMachine.doTransition(StateMachineFactory.java:448)
at org.apache.tez.state.StateMachineTez.doTransition(StateMachineTez.java:57)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:1587)
at org.apache.tez.dag.app.dag.impl.VertexImpl.handle(VertexImpl.java:182)
at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1768)
at org.apache.tez.dag.app.DAGAppMaster$VertexEventDispatcher.handle(DAGAppMaster.java:1754)
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:173)
at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:106)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Class com.amazon.ws.emr.hadoop.fs.EmrFileSystem not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 35 more
, Vertex vertex_1425073885856_0004_1_02 [DF233A0DD77540CCB87B54D63CD5CE02] killed/failed due to:null]
Vertex failed, vertexName=C80F022F9BC448099A57447133356E96, vertexId=vertex_1425073885856_0004_1_01, diagnostics=[Vertex vertex_1425073885856_0004_1_01 [C80F022F9BC448099A57447133356E96] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: 37D24F5D40D44FAB924984B63029AEBC initializer failed, vertex=vertex_1425073885856_0004_1_01 [C80F022F9BC448099A57447133356E96], org.apache.tez.dag.api.TezUncheckedException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:426)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:295)
at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1927)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:686)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:424)
... 13 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1919)
... 15 more
Caused by: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 16 more
]
Vertex failed, vertexName=C8082EF9E44B4C8BA8021E4CF7C07ADB, vertexId=vertex_1425073885856_0004_1_03, diagnostics=[Vertex vertex_1425073885856_0004_1_03 [C8082EF9E44B4C8BA8021E4CF7C07ADB] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: 57AB02C1620643C3974B2BC9C06A4C69 initializer failed, vertex=vertex_1425073885856_0004_1_03 [C8082EF9E44B4C8BA8021E4CF7C07ADB], org.apache.tez.dag.api.TezUncheckedException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:426)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:295)
at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1927)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:686)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:424)
... 13 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1919)
... 15 more
Caused by: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 16 more
]
Vertex failed, vertexName=DC9B873F6A454D1FB032F8252E33C9C7, vertexId=vertex_1425073885856_0004_1_00, diagnostics=[Vertex vertex_1425073885856_0004_1_00 [DC9B873F6A454D1FB032F8252E33C9C7] killed/failed due to:ROOT_INPUT_INIT_FAILURE, Vertex Input: 67A10F19A80B41B8A68522303348387A initializer failed, vertex=vertex_1425073885856_0004_1_00 [DC9B873F6A454D1FB032F8252E33C9C7], org.apache.tez.dag.api.TezUncheckedException: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:426)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateInputSplitsToMem(MRInputHelpers.java:295)
at org.apache.tez.mapreduce.common.MRInputAMSplitGenerator.initialize(MRInputAMSplitGenerator.java:122)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:245)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable$1.run(RootInputInitializerManager.java:239)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1556)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:239)
at org.apache.tez.dag.app.dag.RootInputInitializerManager$InputInitializerCallable.call(RootInputInitializerManager.java:226)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1927)
at org.apache.hadoop.mapred.JobConf.getInputFormat(JobConf.java:686)
at org.apache.tez.mapreduce.hadoop.MRInputHelpers.generateOldSplits(MRInputHelpers.java:424)
... 13 more
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1895)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1919)
... 15 more
Caused by: java.lang.ClassNotFoundException: Class cascading.tap.hadoop.Hfs$CombinedInputFormat not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1801)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:1893)
... 16 more
]
Vertex killed, vertexName=A968F45732AC42EF8E595623D056A4B9, vertexId=vertex_1425073885856_0004_1_04, diagnostics=[Vertex received Kill in NEW state., Vertex vertex_1425073885856_0004_1_04 [A968F45732AC42EF8E595623D056A4B9] killed/failed due to:null]
DAG failed due to vertex failure. failedVertices:4 killedVertices:1

Chris K Wensel

unread,

Feb 27, 2015, 6:50:58 PM2/27/15

to cascadi...@googlegroups.com

keeping this on the cascading list

First: it would be enormously valuable to figure out how to submit a Cascading/Tez application as a step to an EMR cluster, instead of running it by logging in to the master and issuing commands. I had a shot at this and the challenge was that EMR bootstrap actions run before HDFS is up. I think the solution here is to have both a bootstrap action (to set up environment variables, configuration files and such) and an EMR step that performs the actual copy of the jars, but I haven't had time to try this yet.

no good answer for that. I always shell in.

Third: I had to hardcode the -DH arguments in the README to my app, because I couldn't figure out in 5 minutes the correct way to specify options to the hadoop jar command so that my Cascading app's option processor doesn't see them. Not difficult to do, I'm sure, but the command template shown in the README for kicking off an app didn't actually work.

i see the problem, -DH are Cascading.Load specific params. I’ll clean that up. there might not be a way to pass values to "hadoop jar"

Fourth: my application (which sources from and sinks to S3, and uses HfsProps.setUseCombinedInput()) fails with two ClassNotFoundExceptions, one for com.amazon.ws.emr.hadoop.fs.EmrFileSystem and the other for cascading.tap.hadoop.Hfs$CombinedInputFormat.

let me get a fix for that, CombinedInputFormat is not supported on Tez, splitting is delegated to the app master. it would be responsible for that.

—

Chris K Wensel

ch...@wensel.net

Reply all

Reply to author

Forward