Shell or REST api to find long running jobs and kill them

[Amirdha Gopal]

Dear Hue,

We are using Hue 3.10 in Cloudera Enterprise 5.8.3.

In the UI, the job browser allows to view the running jobs along with their duration, as well as to kill the jobs. We need to automate the killing of any long running jobs, say more than 10 hours, after notification.

Does hue provide any REST or Shell command to fetch the running jobs filtered by duration and issue kill command to them?

Is there any alternative approach to perform this without manual intervention?

[Bob Freitas]

Hi Amirdha,

Hue is just the UI that provides access.  Hue is not the place for what you want.  

What you are looking for is the monitoring that is part of the Oozie, https://oozie.apache.org/docs/3.3.1/AG_Monitoring.html.  You can set these up with a workflow and trigger alarms or take actions.  Of course, you will need to submit your jobs as Oozie workflows to make this work.

Best Regards

Bob

--
You received this message because you are subscribed to the Google Groups "Hue-Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hue-user+unsubscribe@cloudera.org.

[Amirdha Gopal]

Thanks Bob. What you say makes sense.  

However, there is a bug in Cloudera cluster that killing oozie doesn't kill the spark job that it spawned. Also, we have huge number of workflows and occasionally few spark jobs run forever. So we are looking for an external solution that doesn't require code change for now. Is there any provision to do this in Cloudera cluster?

--

R.Amirdha Gopal
Application Developer,
Thoughtworks

Computers are incredibly fast, accurate, and stupid: humans are incredibly slow, inaccurate and brilliant; together they are powerful beyond imagination.
--Albert Einstein

[Romain Rigaux]

You could go on the Hue Job Browser page and look at the REST call made by Hue to list the jobs.

Alternatively you could also check the YARN API https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Applications_API (Hue uses this one under the cover)

[Karan Kumar]

We have solved it using the Yarn Resource Manager API. A job is scheduled in oozie which runs at 15 mins and kills all application Id's whose running time is greater than 4 hrs. 

Thanks

Karan