Filtering with smooth and rbe based on a rolling average by time (not count)?

[Christopher Biggs]

Hi Folks,

   I've been using node-red to monitor exception levels in an internet site, and send alerts

to pushover when levels are "unusual".  

I am looking for a better way to get "trending error" alerts.  So far I've tried:

   * alert if more than N instances of an exception in the last 15 minutes (this is finicky to tune)

   * use RBE to alert if error rate up more than 10% since last alert 

The latter could take a long time to alert on a slow monotonic increase.

What I'm looking for is an alert if, say the rolling average over configured interval (say, the last 5 mins),

exceeds the average over a longer window (say, 20 mins) so that I can send an alert of the form "Error X is trending up", 

regardless of how much up.    

The difference from using smooth and rbe alone is that those appear to give me only count based smoothing not time based,

if I'm not able to take a sample at regular intervals (impicitly recognising that N samples is M minutes), then they don't quite give what I need.

Does anyone know of a way to achieve what I've described?   If not, I'll try to add it.

What I intend to try is to add "rolling average of inputs in the last X minutes"  to the smooth node, and 

as a comparison option in RBE's deadband, in addition to existing last-input and last-output comparisons.

This would allow me to form a flow like  (measure level at irregular intervals) -> (smooth to 5min avg) -> (restrict to 1/5min) -> (rbe if input exceeds 20min avg) -> pushover

Anyone think that's nuts?

--chris

[Dean Cording]

Hi Chris,

Sounds like what you are after is a leaky bucket algorithm.  You could implement this with a counter node - errors increment the counter and a regular inject node decrements the counter.  If the counter value increases above a set number then you have a problem.  You could also sample the counter value at regular intervals using a delay node and then RBE if the value changes by more than a specified percentage.

Dean

[Dave C-J]

I'm not really clear on the changes you are proposing for the rbe node, what is the comparison you want to add ?

For the smooth node, what would you output if nothing arrives ? And how would the next node (rbe) handle maths on that ?

[Christopher Biggs]

Hi Dave,

Currently I can say "pass the input if it differs in some way from previous inputs".  The problem is I have a signal that is quite spiky, and that I can't establish a "normal" level for.

So I want to be able to use RBE to say "pass the input if the value exceeds the average of inputs in the last 20 minutes by X%", rather than "exceeds the most recent input by X%".     So, it's effectively extending the trigger level to have another choice beyond "last input" and "last output"

In the "trending up" use-case, my input would be smoothed to "average of smooth node's inputs to over the last 5 minutes" (rather than "average of last N samples" which smooth can do now).

So I'm just talking about using time-based windows instead of fixed sized windows for the smoothing function.  If no input comes in to the smoothing during the smoothing window, then it should produce no output, just as the last-N-samples smoothing node would do now.

I had an idea overnight, perhaps I should be thinking about a "trend" node that takes two inputs "trigger level" and "current value", and computing a trigger level using existing smooth.   I'm not quite sure how to do multiple inputs, probably treating topic "foo_trigger" as the trigger level, and "foo" as the value.  Has anyone made a multi-input node in another way?

--chris

[Dave C-J]

Yes, using another property is the way to distinguish between two inputs.
But would you need to if you did it all in a trend node? You could calculate the average over both time periods, saving all values in the same array internally. (The worry of course is what happens if you end up with massive buffers, both on performance, and blowing memory space)

I'm thinking having a separate node would also allow you to have options for other output modes, like if up by more than (as you want), down by more than, simple up/down indicator, etc

[Mike Bell]

Chris,

If you haven't already solved your problem, you might look at the following flow. It is loosely based on Dean's suggestion of a leaky bucket algorithm. The trend is computed from the outputs of two smoothing nodes, one of which averages over a long sample and the other over a short one. Both smoothing nodes are fed by your exception source and by a clock that generates inputs at a rate much higher than the exception rate.  The clock very nearly, but not quite, converts the averaging basis to time rather than events. The discrepancy, which is small for low exception rates, could be corrected by some additional calculation, if necessary. The switch and topic timeframe trigger modes are there to see that the trend is always computed from current data; there may be better ways to handle that.

Mike

[{"id":"7954f7fe.86ab08","type":"inject","z":"f09d051b.0f62f8","name":"Clock","topic":"","payload":"0","payloadType":"num","repeat":"1","crontab":"","once":false,"x":90,"y":40,"wires":[["ba6c6ca8.45939","2d30e60a.d2cf1a"]]},{"id":"76f6e837.890918","type":"inject","z":"f09d051b.0f62f8","name":"Input","topic":"","payload":"1","payloadType":"num","repeat":"","crontab":"","once":false,"x":90,"y":120,"wires":[["ba6c6ca8.45939","2d30e60a.d2cf1a"]]},{"id":"a2270350.5dd9","type":"smooth","z":"f09d051b.0f62f8","name":"Short","action":"mean","count":"10","round":"","x":430,"y":120,"wires":[["435b003c.bca5"]]},{"id":"ba6c6ca8.45939","type":"change","z":"f09d051b.0f62f8","name":"topic=long","rules":[{"t":"set","p":"topic","pt":"msg","to":"long","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":40,"wires":[["db4690d2.24b97"]]},{"id":"db4690d2.24b97","type":"smooth","z":"f09d051b.0f62f8","name":"Long","action":"mean","count":"100","round":"","x":430,"y":40,"wires":[["ceb46457.314b98"]]},{"id":"2d30e60a.d2cf1a","type":"change","z":"f09d051b.0f62f8","name":"topic=short","rules":[{"t":"set","p":"topic","pt":"msg","to":"short","tot":"str"}],"action":"","property":"","from":"","to":"","reg":false,"x":290,"y":120,"wires":[["a2270350.5dd9"]]},{"id":"af991dfe.5066e","type":"function","z":"f09d051b.0f62f8","name":"Trend","func":"var long = flow.get('long');\nvar short = flow.get('short');\nvar trend =  0;\nif (long !== 0) {\n    trend = (short - long) / long;\n}\nmsg.payload = trend;\nnode.status({text:trend.toFixed(1)});\nreturn msg;","outputs":1,"noerr":0,"x":830,"y":80,"wires":[["9726be65.68d94"]]},{"id":"9726be65.68d94","type":"debug","z":"f09d051b.0f62f8","name":"","active":false,"console":"false","complete":"true","x":950,"y":80,"wires":[]},{"id":"ceb46457.314b98","type":"function","z":"f09d051b.0f62f8","name":"Set long","func":"var long = flow.set('long',msg.payload);\nlong = flow.get('long');\nnode.status({text:long.toFixed(2)});\nreturn msg;","outputs":1,"noerr":0,"x":560,"y":40,"wires":[["9ec4ff9d.613b"]]},{"id":"435b003c.bca5","type":"function","z":"f09d051b.0f62f8","name":"Set short","func":"var short = flow.set('short',msg.payload);\nshort = flow.get('short');\nnode.status({text:short.toFixed(2)});\nreturn msg;","outputs":1,"noerr":0,"x":560,"y":120,"wires":[["9ec4ff9d.613b"]]},{"id":"9ec4ff9d.613b","type":"topic-timeframe-trigger","z":"f09d051b.0f62f8","name":"Trigger","topics":"2","count":"1","timeframe":"1","triggeredtopic":"triggered","triggeredpayload":"go","x":710,"y":80,"wires":[["af991dfe.5066e"]]}]

[Christopher Biggs]

Thanks, Dave,  I agree a dedicated node would avoid the need to find a way to feed in calculated thresholds as well as actual levels.   I started out with that idea, then started thinking about how RBE /nearly/ did what I had in mind.

I'm going to work on a dedicated node, then we'll see how that looks.

--chris

--
http://nodered.org
 
Join us on Slack to continue the conversation: http://nodered.org/slack
---
You received this message because you are subscribed to a topic in the Google Groups "Node-RED" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/node-red/IkdONfS7o5o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to node-red+u...@googlegroups.com.
To post to this group, send email to node...@googlegroups.com.
Visit this group at https://groups.google.com/group/node-red.
For more options, visit https://groups.google.com/d/optout.

[j...@mansey.com]

+1 vote for a trend node

[Dave C-J]

Thinking about it some more, would a proper Kalman filter be a "better" node to have ? As it is a well known technique with proven abilities to smooth and indicate trends ?

[Mike Bell]

Kalman filtering might be very useful, especially to folks interested in robotics. The down side is that it can get computationally intensive, and strictly speaking it assumes that the data comes from an underlying linear dynamic system, which may not be true in a case like the one Chris described. 

Something you might also consider, possibly even as an addition to the current smooth node, is Savitsky-Golay filtering (https://en.wikipedia.org/wiki/Savitzky–Golay_filter). This gives a least-squares polynomial fit to equi-spaced data and is very fast, since it uses pre-computed coefficients.

[Christopher Biggs]

Thanks, Mike, Dave (and also for your example flow in the other message, Mike)

I'm going to start with a the simplest possible thing, (averaging a sliding window) and see how that performs but plan for pluggable algorithms, then move on to try some more complex ones.

--chris

[Christopher Biggs]

Seems there's already a a Kalman filter node: https://www.npmjs.com/package/node-red-contrib-kalman

[Walter Kraembring]

I have a similar solution based on moving averages of the current sunlight level (one fast/short 5 min and one slow/long 45 min). When they are crossing certain thresholds, data is fed to my automated awning control in Node-RED. In my case, I have this written in Python but it could easily have been in javascript instead. I run this script as a service in a RPi and communicate the results to Node-RED via MQTT. The graph below shows how those average values are changing during the day. Awnings are controlled to go down when the fast moving average is crossing the orange 'DownLevel' from below and goes up when the slow moving average is crossing the green 'UpLevel' from above.

So in your case, if you could have a service providing a sample every minute holding the number of exceptions captured last minute, I think you could use a similar approach.

BR Walter 

[Christopher Biggs]

Nice, very much the kind of spike smoothing I'm looking for.

I've implemented the filter, just wrestling with getting my code working in a new node.  

[Juha Autioniemi]

Hi,

I am not sure if this was mentioned before, but there is also node:  node-red-contrib-aggregator, which can be used to aggregate values over configured time span. Available methods for aggregated values are for example: arithmetic mean (average), geometric mean, harmonic mean, median, maximum, minimum.

Maybe this would be useful for keeping track averages over different time periods.

Juha

[Christopher Biggs]

Thanks, Juha.

I was sure something like that existed, I remember seeing the "master message" concept, but I had not been able to find it again.

--chris

[Arman Assylkhan]

Hello, Christopher!

I am currently dealing with RSSI filtering for iBeacon detection and distance estimation with the tools of Node-RED.

Have you managed to install and implement Kalman Filter node?

I tried to install in on my Raspberry Pi 3 but it doesn't in the list of nodes

Thank you!