Exception Handling and identifying failed subassemblies.

[ANIKET MORE]

Hi,

I am implementing a use case wherein I need to identify the failed subassembly/pipe. Also I want to catch the exception thrown by the subassembly/pipe. For this, I am executing a cascading job which is as below-

 Input file (id, name, city) => Each(Custom Function) => Output file

To make the assembly fail and throw exception, I have implemented below logic in operate method-

TupleEntry te = functionCall.getArguments();

Object o=te.getObject(5);

The job throws below exception-

Caused by: cascading.pipe.OperatorException: [pipeName]  operator Each failed executing operation

I am unable to catch this exception since cascading doesn't provide the control to catch it. 

Is there any way to catch the above Exception(OperatorException) and identify which pipe/sub-assembly is causing the failure, if the job fails at runtime.

Can someone please provide any solution for this problem? 

Cascading version : 3.1.0

Hadoop version : 2.6 

Thanks

[Chris K Wensel]

can you share the logger output from the local console?

if you see "please see cluster logs for failure messages”, you need to go to the cluster and see the logs there.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at https://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/9d014c11-6c4a-4c4f-9a23-484d3bdbcbcf%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

415-203-5022

ch...@wensel.net

https://www.linkedin.com/in/cwensel

[ANIKET MORE]

Below is the logger output from console :

2016-10-14 17:37:28.670 ERROR assembly.Fun: invalid index: 5, length: 3

java.lang.IllegalArgumentException: invalid index: 5, length: 3

at cascading.tuple.util.NarrowTupleList.get(NarrowTupleList.java:69)

at cascading.tuple.Tuple.getObject(Tuple.java:180)

at cascading.tuple.TupleEntry.getObject(TupleEntry.java:454)

at hydrograph.engine.cascading.assembly.Fun.operate(Fun.java:33)

at cascading.flow.stream.element.FunctionEachStage.receive(FunctionEachStage.java:100)

at cascading.flow.stream.element.FunctionEachStage.receive(FunctionEachStage.java:40)

at cascading.flow.stream.element.SourceStage.map(SourceStage.java:110)

at cascading.flow.stream.element.SourceStage.run(SourceStage.java:66)

at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:142)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745)

2016-10-14 17:37:28.694 ERROR element.TrapHandler: caught Throwable, no trap available, rethrowing

cascading.pipe.OperatorException: [delimitedInputFile][com.cascading.example.Cascading_BasicEx.main(Cascading_BasicEx.java:59)] operator Each failed executing operation

at cascading.flow.stream.element.FunctionEachStage.receive(FunctionEachStage.java:108)

at cascading.flow.stream.element.FunctionEachStage.receive(FunctionEachStage.java:40)

at cascading.flow.stream.element.SourceStage.map(SourceStage.java:110)

at cascading.flow.stream.element.SourceStage.run(SourceStage.java:66)

at cascading.flow.hadoop.FlowMapper.run(FlowMapper.java:142)

at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)

at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)

at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:243)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask.run(FutureTask.java:262)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

at java.lang.Thread.run(Thread.java:745

Is there any way to catch above exception and identify failed subassembly/pipe ?

Thanks

[Chris K Wensel]

you can’t catch exceptions that exit operations because when you run on a cluster, they happen remotely in the cluster. the fact you see the exception below is because you ran in hadoop’s standalone/local mode.

what you can do is register a trap to capture the details of the failure, and allow processing to continue.

http://docs.cascading.org/cascading/3.1/userguide/ch15-advanced.html#failure-traps

ckw

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/20a95eeb-c5e1-4942-b8cf-c79a0db965f7%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.