Monitoring storm-on-yarn

[kumar y]

Has anybody done any work on monitoring Storm-on-yarn . Since the storm workers run dynamically on different containers/nodes, its is not straight forward to monitor the storm workers , nimbus etc.

[Lorenz Fischer]

Hi

I use graphite [1] as the rendering platform in combination with the metrics collection framework [2] to do some of the realtime monitoring. For evaluation purposes I use the metrics collection framework [2] to collect the metrics before writing them into a performance log using logback with remote appenders [3,4]. I run the logback server on a computer with a known IP (laptop, submission host, openstack vm, or similar) and parse/evaluate the collected performance metrics in that log file after my evaluation runs.

To configure logback, I copy my custom logback.xml file (with the socket appender configuration) into the storm.zip before deploying it HDFS.

I hope this helps,

Lorenz

[1] http://graphite.wikidot.com/

[2] https://storm.incubator.apache.org/documentation/Metrics.html

[3] http://logback.qos.ch/manual/appenders.html

[4] http://logback.qos.ch/xref/ch/qos/logback/classic/net/SocketAppender.html