Bootstrapped conf_int for AR(I)MA out of sample prediction

[Niels Wouda]

In its current state, out-of-sample prediction from an ARMA/ARIMA model returns confidence intervals based on normal theory. I recently wrote something (quite quickly) to base these confidence intervals on a bootstrapped approach instead.

Specifically, I considered the residuals of the in-sample fit, resampled from those to create many new paths (starting from the last observation in the series) of len(steps). At each individual point, I take the lower (alpha/2) and upper (1 - alpha/2) quantiles as the boodstrapped confidence interval for that step.

This approach works quite well for me.

My whole point with this is pretty much as follows: would this be a useful addition to statsmodels? I could clean up my code a bit, write a unit test and create a pull request for it.

[josef...@gmail.com]

Sorry for the delay, I'm still a bit slow right now.

I think it would be a good enhancement and should fit well, especially
given that the current focus is based mostly on normality assumptions.
We have bootstrap confidence interval in some parts, e.g. for impulse
response function in VAR, IIRC.

I'm not familiar with bootstrap approaches for prediction intervals
(and only ever used it for Poisson). Chad, Kevin or Skipper should
know more. I started to browse a few abstracts and can do some
readings if necessary.

Do you have a reference for your bootstrap? Do you have your code
already public?
Is this intended to be a standalone function or integrated into the
model and results classes?

Josef

[Chad Fulton]

This does sound interesting, and I'd be happy to help out with integration particularly if there was a reference I could look at (I'm too busy in the next few months to do a lot of research myself).

(Although I know you're focused on the ARIMA models, my ulterior motive here is moving towards bootstrapped confidence intervals in the state space models, particularly for impulse response functions - not that you'd have to do this, but doing this will get us down that path).

Chad

[Niels Wouda]

I'll definitely look into this more and intend to pick this up in early February, as that's

the earliest I can devote some time to this. I'll look for some more formal references

 before then, and share as I find them.

Depending on the approach, this could be either a standalone function or be more

model dependent - I am undecided about any direction, but I take it from Chad's response

that he would like to move towards state space models in general.

Shall I open a WIP pull request for this sometime soon?

Op maandag 11 januari 2016 23:35:50 UTC+1 schreef Chad Fulton:

[Niels Wouda]

I found these,

• http://comisef.eu/files/wps024.pdf 
  
• http://www.sciencedirect.com/science/article/pii/S0167947306005007 
  

And I quickly scanned the stata manual (not sure if useful): http://www.stata.com/manuals13/tsfcastcompute.pdf

There is the forecast package in R, and I do recall that it is already doing something similar to this. I will take a look sometime soon.

Op zondag 17 januari 2016 00:13:12 UTC+1 schreef Niels Wouda:

[josef...@gmail.com]

On Sun, Jan 17, 2016 at 9:17 AM, Niels Wouda <niels...@gmail.com> wrote:

I found these,

• http://comisef.eu/files/wps024.pdf 
  
• http://www.sciencedirect.com/science/article/pii/S0167947306005007 
  

And I quickly scanned the stata manual (not sure if useful): http://www.stata.com/manuals13/tsfcastcompute.pdf

There is the forecast package in R, and I do recall that it is already doing something similar to this. I will take a look sometime soon.

VAR/SVAR already has bootstrap confidence intervals for some things, impulse response functions ?

I think it's important to add a `method` argument to the bootstrap functions, so it is possible to implement and use simple methods first, and then add more expensive methods. 

After browsing a few abstracts last week, it looks like there are many different versions, the simplest ignore all model uncertainty, the full confidence intervals would have to do the entire model (lag) search and estimation for each bootstrap replication, the latter sounds slow to me.

 

Op zondag 17 januari 2016 00:13:12 UTC+1 schreef Niels Wouda:

I'll definitely look into this more and intend to pick this up in early February, as that's

the earliest I can devote some time to this. I'll look for some more formal references

 before then, and share as I find them.

Depending on the approach, this could be either a standalone function or be more

model dependent - I am undecided about any direction, but I take it from Chad's response

that he would like to move towards state space models in general.

Shall I open a WIP pull request for this sometime soon?

Opening an early WIP PR has two advantages.:

It allows commenting on the PR and makes early review more likely which is helpful if something is unclear or needs design/integration decisions.

It triggers running the test suite on TravisCI. If there are no unit tests, then those are redundant. This is often the reason for me to add unit tests early and then open a PR to get the unit test results for the different versions.

Josef