thoughts on JMX / REST API / management UI web app (and an aside on hawtio & jolokia...)

[James Strachan]

Howdy folks! I've seen various threads on management UI / JMX and so forth but figured it was maybe simpler to start a new thread; I wanted to share my recent experiences on the topic.

So I've been working on an open source web based management console, initially for Apache ActiveMQ, Camel & Fuse Fabric. Its actually a pluggable HTML5 web console to manage stuff in a Java container. Its called hawtio:

http://hawt.io/

Its a browser centric console based on a plugin model; we've plugins for JMX, logging, ActiveMQ, Camel, a dashboard, charting and so forth. One of the cool things about hawtio is that plugins can be discovered at runtime and plugins dynamically visualise themselves in the console based on whats running in the JVM. So if you ran hawtio on vertx, then you deployed a vertx module which used, say, Camel, you'd get the Camel tab appearing in the hawtio console.  So different vertx modules could have their own UIs which come and go based on the modules you deploy in vertx.

So if folks wanted, we could certainly help create some vertx plugins for hawtio which would give some nice eye candy for logging/charting/viewing the underlying metrics in vertx itself or deploying modules etc.

Whether folks use hawtio or not though I also wanted to mention a library we're using in hawtio for the back end; jolokia:

http://jolokia.org/

Jolokia is a small Java library (about 300K) which exposes JMX over REST / JSON; along with having a nice Java & JS API (and an interactive shell). Note I'm just a user of jolokia; but I've used it heavily and cannot recommend it enough :).

So one thing that kinda amazed me when we created the Apache Camel and ActiveMQ plugins for hawtio was; the only server side we have is Jolokia along with the existing MBeans for the projects. Everything else is basically static content (HTML/CSS/images/JavaScript - using AngularJS). i.e. the JavaScript uses the Jolokia JS library to access JMX on the JVM via REST/JSON APIs (which support polling & batching nicely - though websockets would be even cooler :). Its also handy that Jolokia can connect from the browser to any JVM with Jolokia inside (CORS allowing). So one vertx instance could serve up the management console but the console could then connect directly to any JVM running jolokia. (Or jolokia's server supports proxying too).

I've often found JMX a bit of a PITA to work with; though jolokia makes it really simple (even fun :) - everything's just a simple JSON REST endpoint - or a simple little Jolokia JS API call. Its also a pretty handy JVM lowest common denominator; most frameworks & libraries & the JVM itself offers lots of useful MBeans & its easy to write mbeans in any JVM language.

So having JMX MBeans for vertx sounds like a good thing & they'd then just work over jolokia. It'd be good if we created a jolokia and/or hawtio module for vertx so its easy reuse the REST API for management or even hawtio for a nice UI console. 

Thats not to say that hawtio can only work with JMX over jolokia; we could make plugins which use any kind of REST endpoint; though it seems JMX is a nice simple JVM way of exposing stats and operations in a language agnostic way.

Anyway, just thought I'd give folks a heads up if anyone wanted to hack on a management console or was looking into management REST APIs etc

[Pid]

On 31/01/2013 13:51, James Strachan wrote:
> Howdy folks! I've seen various threads on management UI / JMX and so
> forth but figured it was maybe simpler to start a new thread; I wanted
> to share my recent experiences on the topic.
>
> So I've been working on an open source web based management console,
> initially for Apache ActiveMQ, Camel & Fuse Fabric. Its actually a
> pluggable HTML5 web console to manage stuff in a Java container. Its
> called hawtio:
> http://hawt.io/
>
> Its a browser centric console based on a plugin model; we've plugins for
> JMX, logging, ActiveMQ, Camel, a dashboard, charting and so forth. One
> of the cool things about hawtio is that plugins can be discovered at
> runtime and plugins dynamically visualise themselves in the console
> based on whats running in the JVM. So if you ran hawtio on vertx, then
> you deployed a vertx module which used, say, Camel, you'd get the Camel
> tab appearing in the hawtio console. So different vertx modules could
> have their own UIs which come and go based on the modules you deploy in
> vertx.

My first response is "that's interesting and might save me writing it
from scratch".

> So if folks wanted, we could certainly help create some vertx plugins
> for hawtio which would give some nice eye candy for
> logging/charting/viewing the underlying metrics in vertx itself or
> deploying modules etc.
>
>
> Whether folks use hawtio or not though I also wanted to mention a
> library we're using in hawtio for the back end; jolokia:
> http://jolokia.org/
>
> Jolokia is a small Java library (about 300K) which exposes JMX over REST
> / JSON; along with having a nice Java & JS API (and an interactive
> shell). Note I'm just a user of jolokia; but I've used it heavily and
> cannot recommend it enough :).
>
> So one thing that kinda amazed me when we created the Apache Camel and
> ActiveMQ plugins for hawtio was; the only server side we have is Jolokia
> along with the existing MBeans for the projects. Everything else is
> basically static content (HTML/CSS/images/JavaScript - using AngularJS).
> i.e. the JavaScript uses the Jolokia JS library to access JMX on the JVM
> via REST/JSON APIs (which support polling & batching nicely - though
> websockets would be even cooler :). Its also handy that Jolokia can
> connect from the browser to any JVM with Jolokia inside (CORS allowing).
> So one vertx instance could serve up the management console but the
> console could then connect directly to any JVM running jolokia. (Or
> jolokia's server supports proxying too).

My intention was/is to publish from nodes onto the vert.x eventbus, with
a node that can aggregate & publish collated stats. A REST API would be
part of that.

> I've often found JMX a bit of a PITA to work with;

Yup.

> though jolokia makes
> it really simple (even fun :) - everything's just a simple JSON REST
> endpoint - or a simple little Jolokia JS API call. Its also a pretty
> handy JVM lowest common denominator; most frameworks & libraries & the
> JVM itself offers lots of useful MBeans & its easy to write mbeans in
> any JVM language.
>
> So having JMX MBeans for vertx sounds like a good thing & they'd then
> just work over jolokia. It'd be good if we created a jolokia and/or
> hawtio module for vertx so its easy reuse the REST API for management or
> even hawtio for a nice UI console.

Yep, plan A is to expose the thread pools & some bus related metrics,
followed by adding some (obvious) counts around the other I/O endpoints
and if possible, verticle context selection counts.

> Thats not to say that hawtio can only work with JMX over jolokia; we
> could make plugins which use any kind of REST endpoint; though it seems
> JMX is a nice simple JVM way of exposing stats and operations in a
> language agnostic way.
>
> Anyway, just thought I'd give folks a heads up if anyone wanted to hack
> on a management console or was looking into management REST APIs etc

This is good. I'm mostly offline today/tomorrow but I'll take a look
and post back on this thread...


p


> --
> You received this message because you are subscribed to the Google
> Groups "vert.x" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to vertx+un...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>


--

[key:62590808]

[James Strachan]

On 31 January 2013 13:57, Pid <p...@pidster.com> wrote:
> On 31/01/2013 13:51, James Strachan wrote:
>> Howdy folks! I've seen various threads on management UI / JMX and so
>> forth but figured it was maybe simpler to start a new thread; I wanted
>> to share my recent experiences on the topic.
>>
>> So I've been working on an open source web based management console,
>> initially for Apache ActiveMQ, Camel & Fuse Fabric. Its actually a
>> pluggable HTML5 web console to manage stuff in a Java container. Its
>> called hawtio:
>> http://hawt.io/
>>
>> Its a browser centric console based on a plugin model; we've plugins for
>> JMX, logging, ActiveMQ, Camel, a dashboard, charting and so forth. One
>> of the cool things about hawtio is that plugins can be discovered at
>> runtime and plugins dynamically visualise themselves in the console
>> based on whats running in the JVM. So if you ran hawtio on vertx, then
>> you deployed a vertx module which used, say, Camel, you'd get the Camel
>> tab appearing in the hawtio console. So different vertx modules could
>> have their own UIs which come and go based on the modules you deploy in
>> vertx.
>
> My first response is "that's interesting and might save me writing it
> from scratch".

Phew :)

Yeah; I was gonna say in my previous mail, it'd be interesting to look
at bridging Jolokia & vert.x eventbus. I raised a Jolokia issue a
while back for a more websockety API to access JMX stats in the
browser:
https://github.com/rhuss/jolokia/issues/55

Bridging on the JVM side of things Jolokia to the vert.x event bus
would be pretty straightforward I suspect - as its all JSON. At least
jolokia takes care of all the JMX stuff (which is most of the work
with all that CompositeData / TabularData ickyness ;-).

We'd just need a little bootstrap/config thing that decides when to
create a bridge & whether to poll or use the JMX notification stuff
etc. e.g. do we bridge on startup; or wait for a consumer of an event
before registering a JMX notification and then emitting events - and
which beans do we expose & how to map the mbeans/attributes to event
bus names etc.

>> though jolokia makes
>> it really simple (even fun :) - everything's just a simple JSON REST
>> endpoint - or a simple little Jolokia JS API call. Its also a pretty
>> handy JVM lowest common denominator; most frameworks & libraries & the
>> JVM itself offers lots of useful MBeans & its easy to write mbeans in
>> any JVM language.
>>
>> So having JMX MBeans for vertx sounds like a good thing & they'd then
>> just work over jolokia. It'd be good if we created a jolokia and/or
>> hawtio module for vertx so its easy reuse the REST API for management or
>> even hawtio for a nice UI console.
>
> Yep, plan A is to expose the thread pools & some bus related metrics,
> followed by adding some (obvious) counts around the other I/O endpoints
> and if possible, verticle context selection counts.

Event bus metrics would be nice and easy to consume too. As an aside
we've found loading the mbean tree into the browser is handy for
figuring out which bits of the UI to visualise & seeing the big
picture of whats in the JVM.

>> Thats not to say that hawtio can only work with JMX over jolokia; we
>> could make plugins which use any kind of REST endpoint; though it seems
>> JMX is a nice simple JVM way of exposing stats and operations in a
>> language agnostic way.
>>
>> Anyway, just thought I'd give folks a heads up if anyone wanted to hack
>> on a management console or was looking into management REST APIs etc
>
> This is good. I'm mostly offline today/tomorrow but I'll take a look
> and post back on this thread...

Cool thanks!

--
James
-------
Red Hat

Email: jstr...@redhat.com
Web: http://fusesource.com
Twitter: jstrachan, fusenews
Blog: http://macstrac.blogspot.com/

Open Source Integration

[Nate McCall]

>>
>> My intention was/is to publish from nodes onto the vert.x eventbus, with
>> a node that can aggregate & publish collated stats. A REST API would be
>> part of that.

Just so it's on the radar - I definitely would like to see this
"aggregate and publish" service be plug-able.

The goal would be to support publishing to as many external monitoring
systems as possible (eg. graphite with pickle protocol would be my
first implementation).

James - thanks for those links. I had not seen either project before.
Both look really interesting.

[James Strachan]

On 31 January 2013 16:47, Nate McCall <zzna...@gmail.com> wrote:
>>>
>>> My intention was/is to publish from nodes onto the vert.x eventbus, with
>>> a node that can aggregate & publish collated stats. A REST API would be
>>> part of that.
>
> Just so it's on the radar - I definitely would like to see this
> "aggregate and publish" service be plug-able.
>
> The goal would be to support publishing to as many external monitoring
> systems as possible (eg. graphite with pickle protocol would be my
> first implementation).

Agreed. Graphite's my favourite too BTW - shame its a bitch to install
(never managed to figure it out on OS X :)

On the aggregate and publish thing, I spotted jmxtrans a while ago
which is a simple library which has back ends for Graphite, StatsD,
Ganglia, catci/rrdtool etc
https://github.com/jmxtrans/jmxtrans#readme

you use JSON to configure it with the mbean attributes you want to
monitor & it writes to whatever servers are required. Pretty simple
really.

Here's a little module I hacked up to add jmxtrans logging of metrics
to a JVM in Fuse Fabric
https://github.com/fusesource/fuse/tree/master/insight/insight-graph

e.g. here's some JSON to log stats from all mbeans for ActiveMQ queues:
https://github.com/fusesource/fuse/blob/master/fabric/fuse-fabric/src/main/resources/distro/fabric/import/fabric/configs/versions/1.0/profiles/mq-base/org.fusesource.insight.graph.json

> James - thanks for those links. I had not seen either project before.
> Both look really interesting.

Thanks!

[Brian Lalor]

On Jan 31, 2013, at 12:06 PM, James Strachan <james.s...@gmail.com> wrote:

> Agreed. Graphite's my favourite too BTW - shame its a bitch to install
> (never managed to figure it out on OS X :)

It's not pretty, for sure. It took me a while, but I think homebrew ended up being the key. It took me ages to find a good system for packing them as RPMs for our CentOS systems.

> On the aggregate and publish thing, I spotted jmxtrans a while ago
> which is a simple library which has back ends for Graphite, StatsD,
> Ganglia, catci/rrdtool etc
> https://github.com/jmxtrans/jmxtrans#readme

I'm playing around with that. I even have some Puppet config to automate the installation and setup. Their JSON format is really awful, tho, once you start adding multiple servers with the same config.

I'm currently using jmxtrans with Vert.x so that I can monitor basic statistics (heap, permgen, threads, ehcache and hibernate stats). I'm also sending metrics to StatsD from Vert.x, but I had to write a worker module for that because there's no native support for UDP in Vert.x.

I really want to find something for the JVM that will directly publish JMX data to Graphite. That will really minimize the configuration required. If I use Jolokia, I'll have to configure collectd to poll from the servlet. If I use jmxtrans I have to coordinate ports, make sure they don't conflict if there are multiple JVMs on one host, and remember how to map hostname_port back to a particular JVM/application.

[James Strachan]

Getting each JVM to push stats to (say) Graphite is fairly easy.

e.g. It looks like the jmxtrans code just writes the String values of
managed attributes in a text line over a socket for graphite:
https://github.com/jmxtrans/jmxtrans/blob/master/src/com/googlecode/jmxtrans/model/output/GraphiteWriter.java#L105

So should be easy to do with or without jmxtrans. Jolokia would let
you remotely pull over HTTP/JSON; though its maybe easier for each JVM
to just push (asynchronously in the background) to some graphite
server(s). Most of the pain with JMX is dealing with the JMX API which
jmxtrans/jolokia hide from you (and the latter lets you access JMX in
a rich polyglot HTTP/JSON way so is ideally suited to being wired into
the vertx event bus).


The trickiest bit is figuring out how many graphite servers are
running & to discover them. In the Fuse Fabric project use the use the
ZooKeeper registry to discover at runtime all the graphite servers
which apply to a particular geographic cluster. e.g. you may want to
run 2 graphite servers in case one goes down or something. We also
have a ZeroConf plugin in case folks use that to announce Graphite
servers. Then there's no configuration per se in the JVM for graphite;
it just discovers the graphite servers at runtime.

Maybe use the shared data / event bus in vertx to announce the
presence of graphite servers & then discover them at runtime inside
each vertx container?

Having a jolokia / vertx event bus for JMX stats/events, then a vertx
plugin to write stats to graphite (with shared discovery) sounds
pretty good? Then folks could push or pull stats and do custom
aggregation/filtering before it goes to graphite?

--
James

[JC]

Hello everybody,

is there anything going on further in this direction? I mean, JMX is not only great for stats, but also for un/deploying stuff.
I develop using Eclipse and don't really like m2e. So I looked these days into what would it take to create a vert.x server extension for Eclipse WTP, you know for start/stop/choose target server and whatever else I'd need during development of vert.x modules (and vertices too), and with MBeans that'd be so much easier...

Many thanks,
JC

[James Strachan]

I'm still more than happy to hack a nice UI for vertx in hawtio if someone points me at either a REST API or some MBeans :) 

http://hawt.io/

--
You received this message because you are subscribed to a topic in the Google Groups "vert.x" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/vertx/dqeE42_A-_Q/unsubscribe.
To unsubscribe from this group and all of its topics, send an email to vertx+un...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

[JC]

...while Netty lacks JMX support as well - somebody proposed it in 2010 and got ZERO answers...
https://groups.google.com/forum/#!topic/netty/OKVX7vhLwNk

[Tim Fox]

James,

This would be awesome, unfortunately we don't have much in the way of Mbeans at the moment (it's on the list) :(

You received this message because you are subscribed to the Google Groups "vert.x" group.
To unsubscribe from this group and stop receiving emails from it, send an email to vertx+un...@googlegroups.com.

[James Strachan]

I can work with REST APIs and JSON too :) 

/me ducks

[S C]

I'd be able to fiddle together a MBean, that'd be the easy part. BUT as long the container itself offers no insight in its verticle stuff, there's not much one actually could publish today. I can get deployment IDs, but there's nothing yet one can do with that ID except undeploy it. There was a short discussion about management, not long time ago:
https://groups.google.com/forum/#!msg/vertx/VN9Jlagv8V4/2QmxyKtTUoMJ

Anyway, a good starting point would be, just like for Netty, somebody putting together the needs of this management interface, then look what can already be offered, what should be changed and what needs to be added. As a beginner with vert.x I'm totally unqualified for this task :) I'd love to hear from the heavy users what kind of metrics they missed during use...

[S C]

Wrong link sorry:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=415403

[S C]

Do I understand correctly that Bugzilla entry?

- I don't think the Bus module itself should expose JMX, rather a JMX module plugged to the bus would handle management messages

- the Container, PlatformManager and EventBus could offer lots of interesting infos about themselves and let one also modify some internals, as "management" means also interference :)

- all new modules would need to implement themselves the kind of runtime infos they publish, at least with log entries (and MetricsAPI would instrument the logger)

- clogging the bus with events is a real possibility... maybe mitigated with "trace levels" see above
- ...

[Otis Gospodnetic]

+1 for some JMX'n!  We want to support for Vert.x to SPM! (see http://sematext.com/spm )

Otis

--

Performance Monitoring -- http://sematext.com/spm

[Tim Yates]

Would something like this be of use?

https://github.com/timyates/mod-metrics

Or is that overly simplistic?

[Norman Maurer]

I think we said before that we may be able to trigger events for stats over the eventbus and so expose them over JMX then.

Not sure anyone has started with work on this yet.

[Tim Yates]

*cough* ;-)

[Abdel Dridi]

were there any news on this topic?

Is there a web console tool we can use with vertx?