How do I fail a flow when trap throws an exception?

[redshift-etl-user]

Hi,

I have a flow with a series of pipes that have individual traps attached to them, and I have run into an issue where the failure of a trap is causing unwanted behavior: Because of the way FunctionEachStage.receive() works, if a downstream trap fails (e.g. because the trap's output path cannot be written to), then its exception is caught by the upstream pipe and hence written to its trap. What I'd like to do is for the flow to fail in this case. Is that possible?

More generally, is it intentional that traps for upstream pipes get the exceptions of downstream pipes?

I'm using Cascading 2.5.2. 

Thanks.

[Chris K Wensel]

The intent is that a Trap captures any failure in a named branch. Branch names are inherited from upstream pipes.

if you can supply a test case, we can fix the issue, forcing a failure if writing to a trap fails. sounds like the Exception is being wrapped passed fully up the chain to cause the task to fail.

in theory, you should be using a tap that has the least brittleness, Hfs and writing as simple text files, which typically punt to #toString if the type isn’t recognized. making writes safe.

making trap writes less robust (writing to a JDBCTap or something) is going to be problematic.

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/2eb21a5c-68fe-4fa4-9749-29c63b2fa566%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net

[redshift-etl-user]

Thanks, Chris. Yes, having an exception type that is passed up the chain rather than caught seems like a potential solution.

Here's code that exhibits this behavior: https://gist.github.com/redshiftetl/9e6f821e6319cbf49185

The second pipe's trap's path can't be written to, and so the exception is handled by the first pipe's trap. In other words, the output is written to "goodPath.txt", whereas the expected behavior is that the flow fails.

Let me know if it would be better to submit a PR or something.

Thanks again.

[Chris K Wensel]

thanks. will take more time to test this out and provide a fix.

If you could issue a pull request on an existing test suite, that simplifies matters. TrapPlatformTest is a great candidate.

https://github.com/Cascading/cascading/blob/2.6/CONTRIBUTING.md

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/c64f1168-3388-47d3-9eda-eadf95dd372d%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net

[redshift-etl-user]

OK, test and proposed fix submitted, PR #29.

[Chris K Wensel]

this has been committed. thanks!

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/ac03fde6-1bb9-4ee3-8596-f49d548462b4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net