Software noise filters?

[Vadim Tkachenko]

Hello lurkers,

Under some circumstances, analog based sensors may produce false
readings, sometimes significantly false, sufficient to kick DZ's PID
math into unusable zone.

Anyone here has any idea about good noise filtering algorithms and,
possibly, open source implementations? Both Java and C/C++ are good.

--vt

[Jerry Scharf]

Vadim,

From my time working with NTP, I am fond of median filters. Dave Mills
is a wizard of all such things and uses a median filter to remove noise
from client time updates. You can only use this when the update rate is
significantly faster that the system response time. Dave uses this to
front his time software PLL. This is a guy who thinks that 1ms errors
are ok for the masses but not anywhere near acceptable for anyone who
cares. I've had more than one machine running his software that is
software clock traceable to <10us. I have also caught GPS time sources
having microsecond level systemic offsets.

It's simple enough that I don't use a library. You keep an odd length
queue (maybe 3 or 5 long) . When a new entry comes in, you sort the
queue into a list and take the entry in the middle of the list. This
prevents a single (or two) outliers from impacting the second stage. It
comes at the price of a 1-2 read cycle delay when a real change comes
in. Since in general with heating/cooling systems, the read time is much
faster than the system response time, this is acceptable.

This is what I use (queue length 3) for my zone temp filter. If you are
using 1-wire sensors, this assumes that you have filtered out false 85c
readings first, since it is theoretically possible to get a string of these.

enjoy,
jerry

[Vadim Tkachenko]

Hello Jerry,

> It's simple enough that I don't use a library. You keep an odd length queue
> (maybe 3 or 5 long) . When a new entry comes in, you sort the queue into a
> list and take the entry in the middle of the list. This prevents a single
> (or two) outliers from impacting the second stage.

Genius. Much lighter on the system than, say, mean filters.

> This is what I use (queue length 3) for my zone temp filter. If you are
> using 1-wire sensors, this assumes that you have filtered out false 85c
> readings first, since it is theoretically possible to get a string of these.

Actually, 85C readings are ignored (it is known that for some reason
DS18x20 developers decided to make that an exception). Otherwise,
1-Wire sensors are not known to be producing noise.

Real problem lies with XBees, and, possibly, any other analog sensors.
I'm getting up to -20C spikes on my TMP36, and TK is getting insanely
high readings (400C?) on his LM34.

> enjoy,

Thanks, will get to it.

> jerry

--vt

[Tomasz]

> Real problem lies with XBees, and, possibly, any other analog sensors.
> I'm getting up to -20C spikes on my TMP36, and TK is getting insanely
> high readings (400C?) on his LM34.

More like 199.98. Now that I think about it, it may be a just an
'empty reading' (as in a reading when sensor returns nothing, Xbee
samples it and returns some end of range value) and then sensor
convertor gets it and comes with 199.98C. Would have to go there and
check what voltage ranges LM34 work with.

Tomasz