out of sample non-dynamic prediction with SARIMAX model

[Leo]

Hi,

I have divided my time series to training data and test data. After fitting a SARIMAX model, I'd like to use the model to predict values with the test data (non-dynamic) and evaluate the model.

From the doc, http://www.statsmodels.org/dev/generated/statsmodels.tsa.statespace.sarimax.SARIMAXResults.get_prediction.html#statsmodels.tsa.statespace.sarimax.SARIMAXResults.get_prediction, it seems possible by using kwargs, but it doesn't show the name of the parameter. I cannot find any clue from the source code either. Could someone please help me out?

Thanks in advance!

Leo

[Chad Fulton]

Hello,

I think you're looking for something like:

https://github.com/statsmodels/statsmodels/issues/3623#issuecomment-300969821

Best,

Chad

[Leo]

Hi Chad,

Thanks a lot for your advice. So, with filter/smooth, modelB will be able to fit modelA's params and make prediction with modelA's params and the new test data? I guess I can also get resid (.resid()) similarly with the test data? Are there any difference between filter and smooth? There is no documentation on either of them.

Thanks again,

Leo

[Chad Fulton]

On Wed, Oct 18, 2017 at 9:06 AM, 'Leo' via pystatsmodels <pystat...@googlegroups.com> wrote:

Hi Chad,

Thanks a lot for your advice. So, with filter/smooth, modelB will be able to fit modelA's params and make prediction with modelA's params and the new test data? I guess I can also get resid (.resid()) similarly with the test data?

Yes, that is correct. Unless it takes a long time, you may as well use `smooth` in case you need any smoothed values.

 

Are there any difference between filter and smooth? There is no documentation on either of them.

`filter` performs filtering, which is a forward pass through the data, at the given `params`. This yields most of the things you're likely to want (e.g. resid, fittedvalues, predict, forecasting, etc).

`smooth` performs an additional backwards pass through the data, yielding some additional results like estimates of the unobserved state elements based on the full sample.

There is no documentation on either of them.

Thanks for pointing that out! 

[Leo]

Thanks again for your helpful information, Chad. I really appreciate it. So, to confirm, filter only uses the existing params to make forecasting based on the new data and won't change the params, whereas smooth will change the params based on the new data. Is it correct? If so, is there any issue when there are gap between the old data (say 2017-01-01 to 2017-06-07) and the new data (say 2017-07-02 to 2017-10-03)?  I guess filter is fine as it only uses the new data, but not sure about smooth.

Another related question, for k-fold cross-validation, is it possible to train the model with one set of data, say 2017-07-01 to 2017-09-30, and then train the model further with another set of data, say 2017-01-01 to 2017-03-31? If so, how? Can smooth do the job? Will the inverse order of time cause any issue?

One last question regarding the predict function, , it tries to predict values before the p slot. Does it make sense as value depends on all p observation before it? Does it only use available observations, where n_obs<p?

Thanks,

Leo

[Chad Fulton]

Thanks again for your helpful information, Chad. I really appreciate it. So, to confirm, filter only uses the existing params to make forecasting based on the new data and won't change the params, whereas smooth will change the params based on the new data. Is it correct? If so, is there any issue when there are gap between the old data (say 2017-01-01 to 2017-06-07) and the new data (say 2017-07-02 to 2017-10-03)?  I guess filter is fine as it only uses the new data, but not sure about smooth.

No problem. Actually, both `filter` and `smooth` use the given parameters and will not change the parameters. Only the `fit` method actually finds new parameters.

The distinction is the output that the methods produce. `smooth` produces everything that `filter` produces, but also computes some additional output. This distinction is important for the underlying class of models (state space models), but is less relevant for SARIMAX.

If you have a given set of parameters (e.g. from a training sample), you can use either method. Prediction, resid, fittedvalues, etc. will be available (and will be the same regardless of which you use).

 

Another related question, for k-fold cross-validation, is it possible to train the model with one set of data, say 2017-07-01 to 2017-09-30, and then train the model further with another set of data, say 2017-01-01 to 2017-03-31? If so, how? Can smooth do the job? Will the inverse order of time cause any issue?

What you call "training the model", we describe as finding the parameters that maximize the likelihood function associated with a given sample. So if you use `fit` to "train" the model on 2017-07-01 to 2017-09-30, that will find the parameters to maximize the likelihood associated with that sample. If you use `fit` to "train" the model on 2017-01-01 to 2017-03-31, you will be finding parameters to maximize the likelihood associated with that specific sample. There's not really a concept of "training the model further".

What might get closest to what you want would be to use `fit` to "train" the model on the dataset including *both* samples. Since SARIMAX allows missing observations, you can simply fit the model on 2017-01-01 to 2017-09-30 with missing values (np.nan) for all of the elements in 2017-04-01 to 2017-06-30.

 

One last question regarding the predict function, , it tries to predict values before the p slot. Does it make sense as value depends on all p observation before it? Does it only use available observations, where n_obs<p?

Our SARIMAX model does not condition on the first p observations. Instead, it is "initialized" using the unconditional distribution of the process (if the model is stationary) or a combination of the unconditional distribution with diffuse elements (if the model is not stationary).

[Leo]

Thanks again for your prompt and informative reply, Chad. It has cleared up my confusions a lot ;). One question about "fit the model on 2017-01-01 to 2017-09-30 with missing values (np.nan) for all of the elements in 2017-04-01 to 2017-06-30", will the missing data affect the fitted model somehow? Does the model try to fill the gap? In my case, I actually have all the data in 2017-04-01 to 2017-06-30, but I would like to leave them for testing/cross-validation. If the missing data/gap makes it produce an inferior model, it's probably not good for my purpose ;).

Many thanks,

Leo