Not sure where am I going wrong. We are using CDH 5.2
We have a use case where if any of the flowstep fail we want to abort entire flow. So we used flowsteplistener and onthrowable we call flow.stop(), It is killing all hadoop jobs(rest of the flowsteps) but even after killing it is waiting for 5 mins after printing a message that
"shutting down job executor"
I checked some code and found that in BaseFlow it calls this method:
protected void handleExecutorShutdown()
{
if( spawnStrategy.isCompleted( this ) )
return;
logInfo( "shutting down job executor" );
try
{
spawnStrategy.complete( this, 5 * 60, TimeUnit.SECONDS );
}
catch( InterruptedException exception )
{
// ignore
}
logInfo( "shutdown complete" );
}
Then this spawnStrategy.compIete is timing out after 5 mins as given duration. On checking further it calls ThreadPoolExecutor.awaitTermination
Not sure why this method is still waiting for termination even when all the hadoop jobs are killed remotely by calling flow.stop method?