Ranking random variables that show step change
Gene
(statistics). I'd like to find the ones that show a large and abrupt
change at time T. This may be a simple type of data mining, but as I
can't find a data mining newsgroup, I thought I'd post to some related
groups. There is a fair amount of noise in the data, and in addition
to the step change I am looking for, the variables tend to contain
other slower moving changes (so far I have been avoiding the last
issue by looking at a relatively short time windows).
What I have done so far is this:
- Correlate each varaible with a step function at time T. This does a
good job of finding variables with the abrupt change, and I can use
the correlation coefficient (R^2) to rank them.
- Calculate an average value before/after time T, and rank based on
the percent change.
My questions are:
- Is this a known problem that has some clever, well thought out
solution?
- Any ideas on other ways to do my ranking?
- Any ideas on good ways to combine various rankings to produce a
composite ranking? The motivation is I'm most interested in varaibles
that show an abrupt AND significant change in value.
Gene
Sebastian Nowozin
Dear Gene,
On Dec 26, 9:54 pm, Gene <genecolgr...@gmail.com> wrote:
> My questions are:
> - Is this a known problem that has some clever, well thought out
> solution?
> - Any ideas on other ways to do my ranking?
The problem is a special case of "change point detection" for time
series, for which statistical tests and (simple) algorithms exist.
Sebastian
Gene
> The problem is a special case of "change point detection" for time
> series, for which statistical tests and (simple) algorithms exist.
Thanks for the pointer. I assumed this problem must be well-studied,
but I didn't know the terminology, or even what field would study it.
Now I have lots of reading to do!
IRISHSTAT
As Gene the responder wrote, there are procedures to dectec time point
changes. AFS offers a 30 day free trial demo of it's awar winning
package AUTOBOX. Simply download
http://www.autobox.com/30day.exe and load your data in and go. The
procedures used allow autoregressive memory to be seemlessly
integrated into the level shift detection procedures. For more on
Intervention Detection and textbook references, go to the autobox
website and google "INTERVENTION DETECTION".
If I can help , please call me.
Dave Reilly
AFS
http://www.autobox.com
215-675-0652