Monitoring streaming job using genie

[Priyanka Gugale]

Hi,

I wanted to know if genie can be used for monitoring of jobs. If yes how?

Also any idea how stream jobs can be monitored or even killed by genie.

-Priyanka

[Priyanka Gugale]

Hi All,

After further investigation I can see that I can use genie to getStatus of job and kill it. Still I couldn't figure out how to work with streaming jobs.

Example,

My genie job is to launch a streaming job on hadoop. My setup script just has command line to launch this streaming job. When I run the job the "launch" command is run and status of launch command is returned by genie. 

So say my launch command was successful but launched streaming job on yarn failed for some reason, the status in genie is still SUCCESSFUL & FINISHED.

Is there already some provision to getStatus streaming jobs in hadoop or I can plug in some scripts / code to achieve this? Can you please refer me to some handles.

-Priyanka

[Marco Primi]

Hello Priyanka,

It may be easier if you stop by or book some time to talk about this in person.

In general, the Compute group maintains command and cluster configurations on Genie.

You should not need to roll your own, you should just be able to say "run X with Hadoop”. We take care of the “with Hadoop” part of it.

A couple more inline comments below.

On Jul 6, 2018, at 2:06 AM, Priyanka Gugale <priyank...@gmail.com> wrote:

Hi All,

After further investigation I can see that I can use genie to getStatus of job and kill it. Still I couldn't figure out how to work with streaming jobs.

Example,

My genie job is to launch a streaming job on hadoop.

A streaming job is just like a regular job from Genie perspective, except it runs indefinitely.

The current Genie (v3) has a few limitations in this respect, since it was designed for ETL.

In V4 we have better support for streaming (i.e. long-running) jobs. But V4 is still in the works.

So for the moment, if you want to run your streaming jobs on Genie please come talk to us so we can figure something out.

My setup script just has command line to launch this streaming job.

Setup scripts are not meant to launch the command itself, it should just do setup (extract archives, download configurations, etc).

The command itself needs to be launched and monitored by Genie later.

If you are seeing weird things with the job final status, this might be why.

When I run the job the "launch" command is run and status of launch command is returned by genie. 

So say my launch command was successful but launched streaming job on yarn failed for some reason, the status in genie is still SUCCESSFUL & FINISHED.

Is there already some provision to getStatus streaming jobs in hadoop or I can plug in some scripts / code to achieve this? Can you please refer me to some handles.

-Priyanka

On Thursday, July 5, 2018 at 4:54:11 PM UTC+5:30, Priyanka Gugale wrote:

Hi,

I wanted to know if genie can be used for monitoring of jobs. If yes how?

Also any idea how stream jobs can be monitored or even killed by genie.

-Priyanka

--
You received this message because you are subscribed to the Google Groups "genie" group.
To unsubscribe from this group and stop receiving emails from it, send an email to genieoss+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marco Primi]

Priyanka, sorry I assumed you were an internal user, but you’re not.

So we aren’t providing Hadoop clusters and commands for you :-)

But here you can find an overview of the model we use.

In short, administrator register commands, clusters, applications.

When you submit a job you just need to provide parameters to resolve the above (for example via tag and run using a given build of Hadoop, or a specific cluster), and parameters for the job itself (it could be a query or query parameters, depending on how the command is structured).

The demo walks you through an example that uses Hadoop.

[Priyanka Gugale]

Thanks for reply Marco.

I do understand I have to just register commands, applications and clusters and genie will take care of launch part. My concern is getting status of job after launch.

When genie runs the command to launch job it instantly returns status and marks job as completed in UI. Wherein the job is actually still running in backend (as it's a streaming job) and the true status of job is not fetched by genie.

Can you please help me with this part?

-Priyanka

[Marco Primi]

I suspect somewhere in your command you are launching a children process and detaching from it. 

Genie sees the process it launched as completed, since it has no way of knowing there’s a children process still running.

So the solution is probably for your command process to stay alive wait on the children, rather than forking and detaching from it.

This way Genie sees the job as running and you can poll the API to check on its status.

If you want, zip up a Genie job folder for one of this jobs after it executed and send it over, we can take a quick look.

[Priyanka Gugale]

Thanks Marco. From your response looks like I can't monitor streaming jobs running on hadoop.

Just wondering same applies for spark streaming jobs launched with Genie?

-Priyanka

[Marco Primi]

Genie monitors a local process, set up by your ‘command’.

If your command submits a job then immediately terminates (while the actual job is still running on the cluster), then yes, looking at genie status won’t be very useful.

Monitoring streaming jobs in Hadoop and Spark is possible. 

But the process you launch just needs to have the same lifecycle as the job running on the cluster:

 - Execute as long as the job is running on the cluster

 - Exit with code 0 if the job succeeded

 - Exit with code non-zero if the job fails.

An example is the Hadoop executor.

Spark-submit may not behave this way natively (unless there is some kind of —wait option).

So you may need to adapt it by wrapping it into a script that stays running as long as the job is running.

[Priyanka Gugale]

This helps. Thanks a lot for clarifying all my confusions. Can you refer me to Hadoop Executor example? Is it part of demo?

-Priyanka