Request for feedback: moving mean and variance into aggregate operation

[Sergio Bossa]

Hi all,

as you may know, Nimrod currently computes mean and variance online,
every time it receives a metric value; this is very nice and useful
for short-lived metrics, but not so useful for long lived metrics, as
rolling mean and variance over a long term window makes very difficult
to identify trends.

Weighted mean could be an obvious solution, possibly averaged over
time periods, but configuring weights and actually choosing time
windows may be cumbersome and not that flexible.

In my experience, what you really need is to compute mean and variance
over an arbitrary period of time, so here's what could be the
solution: moving mean and variance computations into the aggregate
operation, together with count, percentiles and median
(http://github.com/sbtourist/nimrod/wiki/API-Reference).
This would obviously make the aggregate operation slower: but I
already had an implementation spike at it, and I get all those
computations for 1 million distinct metric points in ~30 seconds on my
poor man laptop, which is not bad for what I want.

But what's your opinion? What would be your desired performance target?

Every kind of feedback will be highly appreciated.
Thanks,

Sergio B.

--
Sergio Bossa
http://www.linkedin.com/in/sergiob