Codahale metrics for Livy

[Janki Akhani]

Hi,

We have started Livy at LinkedIn. It is in poc stage right now. We are thinking of adding Metrics and Monitoring for it. It will be useful to add Codahale metrics to livy. Can you please provide your thoughts for the same? I am looking into possible solutions for the same. I will share those detail soon.

--

Thanks & Regards,

Janki Akhani
Data Analytics and Infrastructure

+91 96200 51508
jak...@linkedin.com
linkedin.com/in/jakhani

[Alex Bozarth]

Hi Janki,

I'm currently working on adding a Livy Web UI and would love to potentially include any metrics and monitoring you add in the UI. I was wondering what new metrics you were looking to add? Livy already has the MetricsAdminServet from Coda Hale mounted at / (or for 0.4, /metrics if the UI is enabled). Currently I have no plans to include those metrics in the UI though. Adding any Coda Hale metrics should be straight forward since the servlet mounts are included with Scalatra, which Livy uses.

Alex Bozarth

Spark Technology Center

[Janki Akhani]

Hi Alex,

It's good to know about Livy Web UI. My plan is to collect System level metrics as well as below mentioned user level metrics.

• Unique users using livy at any given time
• # of active sessions by each users
• Total number of concurrent connections to livy
• Queue, Driver and executor memory requested in eash session, queue usage etc

Basically at any given point in time, we would like to know who are active users, sessions, maximum number of active sessions, requested resources, # of exceptions and 400s and 500s returned in response.

As you mentioned, you are planning to add /metrics UI page. Are you planning to show collected metrics in graph or json format?

I was thinking of using Dropwizard to collect System and user level metrics in livy.

Please share your thoughts for the same. I have not yet checked UI and MetricsAdminServet. I will have a look at it.

Thanks,

Janki Akhani

[Alex Bozarth]

Thanks for the update Janki,

The current /metrics is an auto-generated html page with links to json or text formatted metrics and is very minimal. After some testing I decided to leave it alone and outside the ui. The metrics you wish to add would be useful in a Metric UI tab though. Are you planning on adding a new API that returns the metrics in json format similar to the current Livy API? The UI gets it's data from the API via js ajax calls, not direct from the code, so json formated metrics would be the easiest to use in an eventual Metrics UI tab. I haven't done too much reading into the available metrics servlets in Scalatra (Coda Hale), but there are probably extendable servlets already available.

For the Web UI code, only the first PR has been merged so far, but you can follow my progress via the umbrella JIRA here: https://issues.cloudera.org/browse/LIVY-87 I have a few open PRs currently and code ready for the next few.

Alex Bozarth Software Engineer Spark Technology Center

E-mail: ajbo...@us.ibm.com GitHub: github.com/ajbozarth	505 Howard Street San Francisco, CA 94105 United States

Janki Akhani ---06/01/2017 02:38:15 AM---Hi Alex, It's good to know about Livy Web UI. My plan is to collect System level