Rendering Pretty Sensor Data

[Andrew Kennedy (Cloudsoft)]

Zaid Mohsin wrote:
> I always thought if it’s possible to bundle sensors in a page instead
> of having them one on each line. For example, drilling down to a
> sensor group to get a page of all the sensors related to that group.
> Maybe it’s nitpicking, but that would be a cool addition to the UI.

Hi, Zaid.

I'm not sure exactly what you're suggesting here. Do you mean
aggregating all sensors for a group of entities in a single view? A
concrete example or use case would be helpful.

On 4 Apr 2014, at 22:50, Andrew Kennedy
(Cloudsoft)<andrew....@cloudsoftcorp.com> wrote:
> - Adding a mechanism to transform sensor values for display in the Brooklyn UI

To expand, this was motivated by a user wanting to make the sensor data
for `VanillaJavaApp` more friendly, because large heap sizes in the
sensor view are easy to misread, by counting the number of zeros
incorrectly. Is '100000000' a Gigabyte or a hundred Megabytes, for
instance? Additionally, it would be good to be able to present sensor
values that are entities as links that direct the user to the page for
that entity in the console. For lists of entities, such as the
`GROUP_MEMBERS` sensor added in #1299, this would allow the set of
member groups of a `DynamicMultiGroup` to give access to *their* member
entities.

I have added a new `RendererHints` inner class implementing
`Hint<AttributeSensor<?>>` called `DisplayValue` that takes a
`Function<?, ?>` to process the sensor value before display. The display
value hints are applied to the REST API calls to get the batched current
state of all sensors on an entity ant to the single sensor value request
with type 'text/plain' that returns a String. The JSON sensor value
request is not modified, so third-party dashboard or similar apps can
still use this to perform calculations on raw sensor data.

To validate the approach, the `JavaAppUtils` class has a new `init()`
method that registers display value hints for the `UsesJavaMXBeans`
sensors that return memory information (USED_HEAP_MEMORY,
INIT_HEAP_MEMORY, ...) by transforming them using a `ByteSizeStrings`
function (1000000000L -> "1 GB" etc.) and time and duration sensors are
similarly transformed to String representations.

This seemed to be the only way to proceed without making any UI level
changes in the 'brooklyn-jsgui' project. You can see the effect by
merging pull request #1303 and starting a Brooklyn application
containing any Java based entity.

- https://github.com/brooklyncentral/brooklyn/pull/1303

The sensor data for a `Tomcat7Server` instance is shown in the attached
screenshot. Note the memory size values in 'MB' for the
'java.metrics.heap.*' sensors, the 'java.metrics.processCpuTime.*'
durations and the 'java.metrics.starttime' timestamp, from
`System.currentTimeMillis()`.

Another way to achieve this would have been to add a 'displayValue'
attribute to the returned JSON data, but I wan't clear on how best to
accomplish this with the batched sensor values call, and even the single
sensor value call would need to be modified along with the JavaScript UI
code. For example, the REST server code would need to generate something
like the following JSON data, which the UI JavaScript would then need to
parse:

{
"displayValue":"12 MiB",
"data":"12582912"
}

However, I think this would have introduced unnecessary complexity
without enough obvious gain in functionality over the current, simpler
approach. Also, note that the current mechanism for adding 'Open' links
to the sensor table using `RendererHints.NamedActionWithUrl` is
implemented oddly. The data for the link is included in the
`SensorSummary` object, which is not polled for changes in the same way
as the 'current-state' data. The 'webapp.url' summary JSON looks like this:

{
"name":"webapp.url",
"type":"java.lang.String",
"description":"URL",
"links":{

"self":"/v1/applications/F2NxAqBy/entities/zkymHnHy/sensors/webapp.url",
"application":"/v1/applications/F2NxAqBy",
"entity":"/v1/applications/F2NxAqBy/entities/zkymHnHy",

"action:json":"/v1/applications/F2NxAqBy/entities/zkymHnHy/sensors/webapp.url",
"action:open":"http://sodium:8080/"
}
}

And then polling on the 'self' or 'action:json' links will return the
current value, which is normally the same as the value given for
'action:open' here, but I am unsure what would happen if this changed?
Perhaps the summary sensor data should _always_ contain the current
value, and an additional REST API call should be added to return summary
data for a _single_ sensor? This would be the obvious place to add the
'data' and 'displayValue' entries. The existing API calls would operate
as they do in this pull request.

Note that the code has been slightly modified since the comment from
@ahgittin on the pull request, so the raw sensor JSON will be returned
un-transformed if requested. As stated above, I believe that this will
still allow clients of the REST API to function, with only the UI
requesting batched sensor data that is transformed for display.

Any suggestions about this or any other, alternative approaches, such as
a purely client-side javaScript solution, would be welcome.

Cheers,
Andrew.
--
-- andrew kennedy ? software engineer : https://github.com/grkvlt/ ;

[Zaid Naji]

For example,
we can have something similar to Activity pane, e.g. drilling down from install->ssh stdout and so on. We can have a sensor group “EC2 Sensors” that drills down to Address,Hostname,Security Group, and so on.

Is it clear now?

> <tomcat-sensor-display-values.png>

[grkvlt]

On Saturday, 5 April 2014 22:01:33 UTC+1, Zaid Naji wrote:

For example,
we can have something similar to Activity pane, e.g. drilling down from install->ssh stdout and so on. We can have a sensor group “EC2 Sensors” that drills down to Address,Hostname,Security Group, and so on.

On 5 Apr 2014, at 21:56, Andrew Kennedy (Cloudsoft) <andrew....@cloudsoftcorp.com> wrote:

> Zaid Mohsin wrote:
>> I always thought if it’s possible to bundle sensors in a page instead
>> of having them one on each line. For example,  drilling down to a
>> sensor group to get a page of all the sensors related to that group.
>> Maybe it’s nitpicking, but that would be a cool addition to the UI.
>
> Hi, Zaid.
>
> I'm not sure exactly what you're suggesting here. Do you mean aggregating all sensors for a group of entities in a single view? A concrete example or use case would be helpful.

Ok, that makes sense. To use the Java MXBean sensors as an example, we could split according to the dotted sensor name, and use that as a hierarchy. So we get the following tree structure:

 + java
   + metrics
     + heap
       - used
       - committed
       - max
     + processCpuTime
       - total
       + fraction
         - last
         - windowed
     + starttime

This would be straightforward enough to implement, but I'm not sure it's much of an improvement over the current setup. Also, there would need to be affordances that indicate that the tree entries are expandable. For instance, some entries may be both a sensor name and the parent of other sensors. I have noticed that some users do not realise that the task rows in the 'Activity' pane table are clickable to give child details.

Andrew.

[Duncan Johnston-Watt]

Andrew

I think the UI improvement you have made for Waratek is very useful generally. This makes the sensor data much easier to read.

Zaid

I'm not convinced alternate views is sensors makes a whole lot of sense in our console. In some some client engagements we are doing less direct instrumentation of entities and instead hooking into things like collectd/statsd. In other words there are better tools designed for large scale data visualisation.

I would be interested in what other people think but our focus has to be on management IMO. 

Best

--

Duncan Johnston-Watt

CEO | Cloudsoft Corporation

+44 777 190 2653 | @duncanjw

Sent from my iPhone

[Andrew Kennedy (Cloudsoft)]

Andrew Kennedy (Cloudsoft) wrote:
> This seemed to be the only way to proceed without making any UI level
> changes in the 'brooklyn-jsgui' project. You can see the effect by
> merging pull request #1303 and starting a Brooklyn application
> containing any Java based entity.
>
> - https://github.com/brooklyncentral/brooklyn/pull/1303
>
> The sensor data for a `Tomcat7Server` instance is shown in the attached
> screenshot. Note the memory size values in 'MB' for the
> 'java.metrics.heap.*' sensors, the 'java.metrics.processCpuTime.*'
> durations and the 'java.metrics.starttime' timestamp, from
> `System.currentTimeMillis()`.

[...]

There are multiple 'pretty' interpretations for a particularly typed
sensor, and the `SensorSummary` indicates only that
'java.metrics.heap.max' and 'java.metrics.starttime' are both
'java.lang.Long' values but with no semantic information beyond their
natural language description. The only components that know the semantic
intent of the sensor data are the entities, and their support classes,
which is where the rendering hints are currently applied, so the
`JavaWebAppUtls` code both sets up the sensors and also the hints for
rendering Long values as either byte sizes or timestamps.

I appreciate the concern about encapsulation, and presentation level
code being kept in a single place, rather than cutting through layers.
However, if we ever wanted to move the 'pretty' rendering to the
presentation (i.e. JavaScript UI) layer, we will need to find a way of
pushing the semantic details up there, also without unnecessary
duplication of existing rendering code from classes like
`ByteSizeStrings` which are used for logging and console output.

[Andrew Kennedy (Cloudsoft)]

Hi.

I've now marked the new `RendererHints.DisplayValue` class with the
`@Beta` annotation, and added a comment indicating that this may not be
the final mechanism used. I think we *do* need a way of rendering these
'pretty' display values somehow, so this gives us a way of achieving the
functionality in the near-term, without necessarily committing to use it
in future, and will allow users to see what is possible and experiment
with different options.

Unless there is a strong objection, I'd like to merge the #1303 pull
request before the next 0.7.0 milestone release of Brooklyn.