Implementing YARN based Rest endpoint.

32 views
Skip to first unread message

Vicky Kak

unread,
Apr 26, 2017, 6:06:45 AM4/26/17
to gobblin-users
I have been able to implement the plugin based service for the rest endpoint for the standalone deployment, however I was expecting similar approach working for YARN/MR however it is not going to work with YARN deployment.
Here are the list of changes that I see 
- Having a RestEndpoint in the GobblinYarnAppLauncher, this endpoint will make invocations to the Yarn ApplicationMaster.
- We need one more RestEndpoint in ApplicationMaster which will be invoked via RestEndpoint of the GobblinYarnAppLauncher.
- The RestEndpoint can be plugged in as service, this service will internally trigger the GobblinHelixJobLaucher.

I can see the similar flow in the GobblinServiceManager but have yet not taken a deeper look.

Before I make changes in the code and test it I wanted to hear more from the dev team and community members

We also used to rely on the hot deployment of jobs, we would create the new .pull file and drop in the job work directory and gobblin would detect it dynamically. We need the similar feature with the YARN/services.I don't think it is going to work with the YARN Deployment. I would like to hear on this before I raise an issue and get started with it.

Thanks,
Vicky
Reply all
Reply to author
Forward
0 new messages