anomaly detection via Markov switching models - example data ?

[josef...@gmail.com]

I'm still thinking about some practical application of time series analysis, but I don't have any use case myself. The question is whether anyone has or knows of datasets that could be used as example and test cases.

Chad and Valera are working on adding Markov switching models to statsmodels during this GSOC.

my random thought:

A long time ago when I was trying out Markov switching models (in GAUSS), then one of the observations was that, if I had too many states specified, then the extra states just captured a few outliers or outlying observations. In that case, this was an annoyance and overparameterized for my purpose (or maybe it was an unlucky local optimum).

However, for outlier and anomaly detection especially if anomalies lasts for several periods this would be a desired feature. So, it might be possible to exploit this for anomaly detection.

Two more observations:

Given the background of our contributors (Skipper, Chad and Wes) the time series models in statsmodels have more of a macro econometrics, and possibly finance, background. In contrast, Hyndman in R is all into forecasting.

However, it's not clear to me what the real difference is. For example, statsmodels still doesn't have exponential smoothing/Holt-Winters, but Chad added `UnobservedComponents` based on the statespace representation. However, in terms of use cases for forecasting those two types of models look very close to me.

(I was just briefly visiting time series analysis and forecasting before going back to ... )

Josef

[Leoysen Liu]

It's my domain with anomaly detection but I dont know how to use Markov Switching detecting outlier?

Any related paper or form?

I have many datasets to detect anomaly but unlabeled....I will do something if you need help.

在 2016年6月4日星期六 UTC+8上午11:43:37，josefpktd写道：

[josef...@gmail.com]

Markov switching models are essentially the univariate time series
analog to Gaussian Mixture Modellling (the machine learning GMM).
So some of the intuition and application should be similar to it, but
with persistence, serial correlation in the underlying state and in
the within state autoregression. Outliers would be short lived regimes
or episodes that show up every once in a while, like recessions in the
economy or financial crises.

I haven't read anything in a while, and my references were old
econometrics text books, like Hamilton.

https://github.com/statsmodels/statsmodels/pull/2921
Since Valera and Chad do all the heavy stuff, I think it should be
possible just to run it semi-automatically over various datasets and
check usability for outlier detection.
Similar to automatic forecasting, there would be a need for trying out
different specifications, e.g. choosing the number of states.
Additionally ,mixture models often have multiple local minima that
need to be worked around for applications.


Josef