Spike Detection

[Claire]

Hi -

I'm trying to eliminate the noise caused by data drop-outs and have
been experimenting with the wild points and spike detection
functionality of IADS. The wild point filtering appears to be working
consistently, but I'm not getting as reliable results with the spike
detection.

I've tried using the Absolute Change setting, and setting the Change
Limit to 500. Yet when the data goes from say 50 to 3,000, the spike
often remains. It was my understanding from the user's manual that
starting with a value of 50, if the next point were 551 or higher, it
would hold the previous value of 50. Any idea why this isn't working
consistently? I'm having similar issues when using the Slope Change
setting, although it's behaving as I'd expect for 98% of the data
corrections.

Thanks in advance for your help,
Claire

[Mike Burt]

Hi Claire:

If I had to speculate...I would guess that you are probably observing
side effects from attempting to perform spike correction on multiple
consecutive spikes. The algorithm was made to correct single isolated
spikes occurring in the data and not necessarily handle extended
periods of sync loss...when multiple consecutive spikes occur the
algorithm is probably attempting to establish a new 'baseline' based
on multiple spike values.

This differs from the wild point editing method which filters data
outside of a specified range for all data values...the limitation with
this method is that you need to have previous knowledge on the
possible range of the data being edited to be able to establish
logical limits.

Another option you could attempt is if you have decom status words
available with your setup you could create some derived equations that
utilize the _IadsDecomStatusN_ parameters to sense when sync loss
conditions occur where you could then output something like Last Good
Value (LGV).

Please let me know if the above theory seems to apply in your case.

[James Bretz]

Hi Claire,

Sounds like it's working, but just not picking up 100% of the spikes. I'm
really not sure why it would be skipping those points.

Are you seeing this issue while scrolling back in time, or is this happening
while the AW is "streaming" data?
Jim

[Claire]

It's doing it when I'm replaying the data. It seems to eliminate the
majority of spikes, but there are instances where there are spikes
every few seconds and it isn't dropping them - the sample rate is
about 40Hz, so I wouldn't think this would be an issue. I've found it
interesting that the threshhold alarms do not necessarily go off with
these false spikes, although they're still showing up on the graphs.
It seems to be a toss-up as to if the alarms sound or not. I assume
this has something to do with the Test Group's configuration of IADS,
although I'm not familiar with their end of things.

Thanks for your help Mike and James!

[James Bretz]

Claire,

>It's doing it when I'm replaying the data. It seems to eliminate the
>majority of spikes, but there are instances where there are spikes
>every few seconds and it isn't dropping them - the sample rate is
>about 40Hz, so I wouldn't think this would be an issue. I've found it
>interesting that the threshhold alarms do not necessarily go off with
>these false spikes, although they're still showing up on the graphs.
>It seems to be a toss-up as to if the alarms sound or not. I assume
>this has something to do with the Test Group's configuration of IADS,
>although I'm not familiar with their end of things.

Hmmm.... Are you saying that you see a spike exceed the threshold line but
it doesn't register? If that's the case then you must have "Threshold
Protection" active. In this case, you supply it an equation based on "decom
status" so it can determine if the data is good or bad (i.e. if telemetry
system is experiencing communication failure or not). If it registers as bad
data according to the equation, it doesn't trip the threshold alarms. You
can confirm this by right clicking on the threshold panel on the dashboard
(bottom left hand corner of screen), and see if the "use threshold
protection" option is checked.

If you want a discussion as to why this equation/protection system will
probably never protect against data spikes 100% of the time then let me
know. There are inherent flaws in the error detection capability of the
existing telemetry systems. Also, sounds like Mike already explained why our
spike detection system has trouble getting 100% of the errors as well.... so
I'm not sure if we can completely eliminate all problems at this time.

Having said that, there is hope on the horizon. We are working with the
telemetry vendors in order to ensure a mechanism that bad data can be caught
100% and properly corrected.

>Thanks for your help Mike and James!

No problem ;)
Jim

[Claire]

Hi again,

Unexpected issue - The test guy just added in the wild point and spike
detection parameters to the master config file. They were added
directly to the (TPP) gages, or the data being streamed off the A/C
rather than to a derived parameter. My strip plots have all turned to
reading 0 (or a constant value <5). I created a test derived
parameter, and set it equal to the TPP streaming data - copied and
pasted all of the wild point and spike detection settings. It appears
to be working as expected - that is, it is filtering out unexpectedly
high data, but otherwise returning reasonable looking (non-constant)
data rather than flat-lining. Do I need to to create yet another
derived parameter in order to apply the drop out protection? I would
think I could do it directly to the data coming in (I realize the data
itself isn't modified, but is stored on the server and the filters are
applied as needed to the display data - thus why the derived parameter
isn't also 0 for my test case above). If I do need another set of DP,
could you please explain why?

Thanks,
Claire

[Claire]

OK I've figured it out... When I sent the test group our file, I had
it nicely formatted in excel complete with commas (eg 5,950). The wild
point settings appear not to accept commas as a valid number imput..
Will have to resend non-formatted numbers to test group.

And yes, Threshold protection has been activated.

Thanks again!!
Claire

> > Jim- Hide quoted text -
>
> - Show quoted text -

[Jim Bretz]

>OK I've figured it out... When I sent the test group our file, I had
>it nicely formatted in excel complete with commas (eg 5,950). The wild
>point settings appear not to accept commas as a valid number imput..
>Will have to resend non-formatted numbers to test group.
>And yes, Threshold protection has been activated.

Nice! Yes, the embedded commas probably hosed up the import process.

>Thanks again!!

No problem and I'm glad you got things working ;)
Jim

[Jim Bretz]

Hi Claire,

>Unexpected issue - The test guy just added in the wild point and spike
>detection parameters to the master config file. They were added
>directly to the (TPP) gages, or the data being streamed off the A/C
>rather than to a derived parameter.

Ok, understand.

>My strip plots have all turned to reading 0 (or a constant value <5).

Hmmm... Well if you are using these Tpp parameter directly in your display
then I'm a little puzzled as to why it's flat lining. Are there also filters
applied to these Tpp parameters?

>I created a test derived parameter, and set it equal to the TPP streaming
>data - copied and
>pasted all of the wild point and spike detection settings. It appears
>to be working as expected - that is, it is filtering out unexpectedly
>high data, but otherwise returning reasonable looking (non-constant)
>data rather than flat-lining.

Ok, this *might* be because the original parameters had filtering set on
them and your new derived parameters not... or there might be some other
settting that I'm missing. Try to compare every column value with the
original parameter and look for differences.

>Do I need to to create yet another derived parameter in order to apply the
>drop out protection?
>I would think I could do it directly to the data coming in (I realize the
>data
>itself isn't modified, but is stored on the server and the filters are
>applied as needed to the display data - thus why the derived parameter
>isn't also 0 for my test case above). If I do need another set of DP,
>could you please explain why?

No, you shouldn't need to... I'm not quite sure what's going on. Anyone else
have an idea other than filtering?
Jim

[Michael Jones]

Claire, is it possible you have multipole Parameter Default Entries
for the same Parameter? Right-Click Sort Ascending on the Parameter
name column, then navigate to the Param entry.

[Mike Burt]

Claire...just to clarify...was the issue you figured out with the
embedded commas in reference to the 'Unexpected issue'?

Addressing one of the data replay issues mentioned earlier...when
there were spikes 'every few seconds' were there instances of multiple
consecutive data samples containing 'spike' values (specifically in
the pre-edited data)? If so that would explain why some values were
not being edited when applying the spike correction algorithm...

[Claire]

Delayed responce: I've been out of office.

On Sep 2, 2:31 pm, Mike Burt <mb...@iads-soft.com> wrote:
> Claire...just to clarify...was the issue you figured out with the
> embedded commas in reference to the 'Unexpected issue'?
>

You are correct - removing the commas fixed the "Unexpected issue".
Sorry for any confusion!

> Addressing one of the data replay issues mentioned earlier...when
> there were spikes 'every few seconds' were there instances of multiple
> consecutive data samples containing 'spike' values (specifically in
> the pre-edited data)? If so that would explain why some values were
> not being edited when applying the spike correction algorithm...
>

Correct again!

On to new filtering questions. For the spike detection using the slope
change method the manual says it compares the average slope values of
up to five data points - this is actual slope, not slope magnitude,
correct? Also, if we have a true data increase (where it has a sudden
huge increases and holds approximately constant at the new reading)
and are sampling at 40hz, the screen should update to the true value
within 1/8th of a second worst case, correct?

Thanks for all your help!
Claire

[Mike Burt]

Hi Claire:

> On to new filtering questions. For the spike detection using the slope
> change method the manual says it compares the average slope values of
> up to five data points - this is actual slope, not slope magnitude,
> correct?

The average slope value is based on the average of the last 5 sample
differences...i.e.

avg = ((s6-s5)+(s5-s4)+(s4-s3)+(s3-s2)+(s2-s1)) / 5

This average is used to calculate a ratio against the difference of
the last 2 sample values (i.e. (s1-s0)/avg). The resultant ratio is
then compared to the SpikeChangeLimit value specified in the
ParameterDefaults table to determine whether spike correction should
be performed.

> Also, if we have a true data increase (where it has a sudden
> huge increases and holds approximately constant at the new reading)
> and are sampling at 40hz, the screen should update to the true value
> within 1/8th of a second worst case, correct?

I would have said correct when using the Slope Change detection method
but in looking at the code seems as though the container holding the
last 5 value differences is not being updated with the current
difference when spike correction is applied. Will need to investigate
further but looks like there is a potential bug there which may
prevent data transitions to the new sequence. So I am guessing you are
asking this question because you are observing problems...correct?

[Claire]

> >  Also, if we have a true data increase (where it has a sudden
> > huge increases and holds approximately constant at the new reading)
> > and are sampling at 40hz, the screen should update to the true value
> > within 1/8th of a second worst case, correct?
>
> I would have said correct when using the Slope Change detection method
> but in looking at the code seems as though the container holding the
> last 5 value differences is not being updated with the current
> difference when spike correction is applied. Will need to investigate
> further but looks like there is a potential bug there which may
> prevent data transitions to the new sequence. So I am guessing you are
> asking this question because you are observing problems...correct?

Yes - I've seen a few instances where it takes several seconds for the
data to correct itself. What is the worst case length of time for it
to update as the code stands now?

[Mike Burt]

> Yes - I've seen a few instances where it takes several seconds for the
> data to correct itself. What is the worst case length of time for it
> to update as the code stands now?

If my intrepretation of the code is correct I would say the length of
time to update would be indeterminate...but I will need to verify...

[Claire]

Please let me know what you find out, and if there is any sort of work-
around. As it stands now I think we're going to have to ditch the
slope filtering unless you have a fix.

Basically some of the drop-outs we're seeing are less than the wild
points, but are obviously data spikes and are caught with the slope
filtering - we've been using a combination of spike and wild points to
minimize these drop-outs. The current concern is that during a hard
landing or similar event a gage may correctly spike to less than the
wildpoint, but with a greater slope than the spike allows, hold steady
at that state for a second or two before unloading. If the spike
detection were able to pick up on this at worst case every 1/8th
second this wouldn't be a big deal since we aren't dealing with
frequencies.. but we aren't willing to sign up to an inderterminate
delay..

Any suggestions would be greatly appreciated!

[Claire]

One other thing I've been noticing with the wild point filtering.. I
have it set up to use the last-good-value. However in some instances,
it appears that IADS comes up with it's own value. For example, the
wild point was set to -40, and as it decreased from -30 to -50 it
tracked really well until -40, at which point the value for the
filtered data jumped to +35. I've also seen this data waver if the
gage is outside of the wildpoint range for too long (so 10 seconds at
+35, then it drops down to +30 for the next 10 seconds etc). Any idea
what's causing this? My concern here is that it won't be triggering an
alarm as the magic IADS value may or may not be greater than the
threshholds (even though the wildpoint is larger).

Thanks for your help sorting out these filtering issues!

[Mike Burt]

Hi Claire:

> Please let me know what you find out, and if there is any sort of work-
> around. As it stands now I think we're going to have to ditch the
> slope filtering unless you have a fix.

Ok...will let you know what I come up with...

> Basically some of the drop-outs we're seeing are less than the wild
> points, but are obviously data spikes and are caught with the slope
> filtering - we've been using a combination of spike and wild points to
> minimize these drop-outs. The current concern is that during a hard
> landing or similar event a gage may correctly spike to less than the
> wildpoint, but with a greater slope than the spike allows, hold steady
> at that state for a second or two before unloading. If the spike
> detection were able to pick up on this at worst case every 1/8th
> second this wouldn't be a big deal since we aren't dealing with
> frequencies.. but we aren't willing to sign up to an inderterminate
> delay..

Understand...thanks for the feedback...as another possible option...if
the drop-outs are due to sync loss conditions (which may also be
triggering the spikes) you may want to consider looking at the
_IadsDecomStatusN_ parameters as a mechanism to sense periods of valid/
invalid data conditions. If you have decom status words defined in the
data stream sent to the CDS you could create derived parameters that
utilize the _IadsDecomstatusN_ parameter to determine when to apply
data filtering (e.g. during sync loss conditions output LGV).

[Mike Burt]

Not sure...will try to reproduce on my end...do you have any other
data editing active besides wild point editing (e.g. sign change,
Butterworth or Elliptic filters, Null Correction, etc.)?

[Claire]

> Not sure...will try to reproduce on my end...do you have any other
> data editing active besides wild point editing (e.g. sign change,
> Butterworth or Elliptic filters, Null Correction, etc.)?

Only using Wild Point and Spike Detection filtering. I can scan in
screen printouts and send them your way it that would help.

[Claire]

> Understand...thanks for the feedback...as another possible option...if
> the drop-outs are due to sync loss conditions (which may also be
> triggering the spikes) you may want to consider looking at the
> _IadsDecomStatusN_ parameters as a mechanism to sense periods of valid/
> invalid data conditions. If you have decom status words defined in the
> data stream sent to the CDS you could create derived parameters that
> utilize the _IadsDecomstatusN_ parameter to determine when to apply
> data filtering (e.g. during sync loss conditions output LGV).

Thanks for the suggestion - I'll go talk to our test group and see if
they can show me how to impliment this.

[Claire]

Here's an absolute value spike detection question with two examples.

Absolute value spike detection – set to 10
Set 1:
Point 1: 1.2
Point 2: 10.3
Point 3: 20.4
Point 4: 30.5
Point 6: 35
Set 2:
Point 1: 1.2
Point 2: 10.3
Point 3: 12.4
Point 4: 13.5
Resulting edited data:
Set 1:
Point 1: 1.2
Point 2: 1.2
Point 3: 1.2
Point 4: 1.2
Point 5: 35
Set 2:
Point 1: 1.2
Point 2: 1.2
Point 3: 12.4
Point 4: 13.5
Is the above a correct interpretation/implementation of the absolute
value spike detection data editing?

Also, how long will IADS hold Last Good Value?

Thanks!
Claire

[Mike Burt]

Hi Claire:

To answer your latter question...in the case of using the Absolute
Value method of spike detection/correction IADS will hold LGV for one
sample...in the case of using the Slope method of spike detection/
correction the LGV may be held for up to 5 samples due to the nature
of the algorithm (see above post for further detail).

I setup a test using the data sequence in your example (produced a csv
file and created an IADS data file using the PostTestDataServer)
applying the Absolute Value spike detection method, LastValue
correction method and a Spike Limit of 10 with the following results:

Set 1:
Point 1: 1.2
Point 2: 10.3

Point 3: 10.3
Point 4: 30.5
Point 5: 35.0

Set 2:
Point 1: 1.2
Point 2: 10.3
Point 3: 12.4
Point 4: 13.5

These results are what I would expect...in Set 1 after the correction
was applied the detection state is reset and the next value in the
sequence is applied...in Set 2 there is no correction because the
limit is never exceeded. I would speculate on a couple reasons why you
may be observing the results you specified...either there is some
other data editing/filtering active with that parameter or you are
using separate IADS generated derived parameters (e.g. applying the
sinewave function) to compare the edited and non-edited values and
their data flows are disjoint during this sequence (maybe from jumping
around in nonconsective time frames)...