Cascade Hanging?

[Will Briggs]

I have a cascade with multiple flows, which I submit hourly via Oozie. In the past day or two, I have started to have problems with the Cascade finishing (and logging) a flow, but never starting the subsequent flow in the Cascade - it just sits there, with no additional log output, forever - killing and restarting the whole cascade typically "fixes" the problem, but that's obviously not ideal. The problem is intermittent, which makes it even more frustrating to debug. Has anyone run into this sort of thing? I'm running Cascading 2.6.1 on YARN, with CDH 5.3.0.

Thanks,

Will

[Andre Kelpe]

Is the flow not started at all or does it submit jobs, but those are never started? I have seen behaviour like this, when YARN believes it is overcommited on ressources (memory), but would never tell you that. There are some config options to ease that, but before we go there, I'd like to know which behavior you are seeing.

- André

--
You received this message because you are subscribed to the Google Groups "cascading-user" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cascading-use...@googlegroups.com.
To post to this group, send email to cascadi...@googlegroups.com.
Visit this group at http://groups.google.com/group/cascading-user.
To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/d9288abf-1105-4f5e-83c6-956abc99af24%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

André Kelpe
an...@concurrentinc.com
http://concurrentinc.com

[William Briggs]

Hi Andre, from what I can tell based on logging, the flow is never started - I can't find any jobs in the NEW, SUBMITTED or ACCEPTED state in the YARN resource manager, and the last output from the cascade is a statement telling me that the previous flow finished.

-Will

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAA2tiYJuBbjpZX5ZL8baPR2TuWCyPha_zAequ_0A9Fz5sB03-g%40mail.gmail.com.

[Chris K Wensel]

Are you running the mapreduce job history server?

its probable that job tracker shuts down before the client side knows the last job in the tracker is completed.

also, can you drop a stack/thread dump of the Cascading client side and post it? would be useful to know where the hang actually is.

ckw

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAPa6qbrio%3Dv%2BSWt47UPPE_yLnuGWXtMw26oGSpEKFu07E_jB8A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

—

Chris K Wensel

ch...@wensel.net

[William Briggs]

Thanks to both of you for the responses; we are running the MapReduce history server. I have attached a copy of the stack dump for the most recent version of the hung app.

Thanks,

Will

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/436130B8-6B8F-429B-B92B-624B8DFA9DD6%40wensel.net.

[Gera Shegalov]

Multiple job submitter threads hang in DBInputFormat split calculation. There could be too many concurrent connections to your Vertica.

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAPa6qbqbsGzkQs5Zob-0ZGTo7O0f0xAX17i3ELGUN%2BiandyKaQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

@gerashegalov

[William Briggs]

Thanks Gera; unfortunately, that doesn't seem to be the problem. As you noticed, the stack trace shows that the code is blocked when the DriverManager is instantiating JDBC drivers via reflection. This is still an intermittent issue, and it ends up causing the whole Cascade to hang, since none of the Flows that use the JDBC taps as input can run. Does anyone have some suggestions on debugging this issue further? It's driving me nuts.

Thanks,

Will

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CACiptKwFmF_%2BBy-2cNuXrv-BCNne7w%2BrbCKv-e%3DRmA1wQy4sNQ%40mail.gmail.com.

[Andre Kelpe]

Can you confirm that you are able to load the jdbc driver from a simple java app from your cluster? Are you able to run the standard cascading-jdbc tests?

- André

To view this discussion on the web visit https://groups.google.com/d/msgid/cascading-user/CAPa6qbpFFFQKpwy-_hW1R3MApM8toL39SxYK-AkyxEk40XjMTw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.