Conceptual question about time-varying parameter models

[James Savage]

All - 

I'm trying to reconcile the performance difference of two models. The first model is estimated on some (extending) window of data, at the end of which it produces a 1-year ahead forecast. For example, it estimates up to the end of 1975 then produces a forecast for 1976; the window then extends to include 1976, the model is re-fitted, and I generate a forecast for 1977. As the window size is increased, I get a series of true out-of-sample forecasts. 

The second model is a time-varying parameter model estimated over the full data set. In each period within the sample, I generate a 1-year ahead forecast using that period's parameters. So the parameters for 1975 are used to forecast for 1976, and those for 1976 are used to forecast 1977. 

The second model performs much better than the first, which is not surprising. My question is whether this is a valid comparison; in a time-varying-parameter model with a parameter theta[t] ~ normal(theta[t-1], sigma), are the values of theta[t-1] influenced at all by the data that come after? Another way of putting it is if i were to estimate the model to the end of 1975, would I have the same 1975 parameters in expectation as the case of estimating the model to today? 

Cheers, 

Jim

[Michael Betancourt]

It depends on the details of the model.  For example, if the parameters you’re

estimating are stationary then it’s valid to use the fit from the full data to inform

the 1-year forecast simulations.  This would be true for the parameters in 

AR, MA, ARMA, etc processes, but not the states themselves.

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[James Savage]

I'm modelling the parameters as adjusting due to a random walk (although ex-post they look stationary). So to my mind that would invalidate the comparison? 

[Michael Betancourt]

I’m not sure what you mean by that.  Either you have parameters

that are constant in time or parameters that vary in time.  If a

parameter varies in time (really you have a different parameter

for each time point) then future values can certainly improve the 

inference of past values (think smoothing in a Kalman filter).

Now you can use those improved inferences to simulate forecasts

but you have to be careful with the interpretation — it’s a valid

posterior predictive check _on the complete time series that was

measured_ but not a valid measure of predictive performance on

entirely new data.   

[James Savage]

Thanks a thousand Michael, that clears things up. 

I misunderstood "stationary", which has a special meaning in time-series. 

[Michael Betancourt]

Yeah, sorry, I was trying to refer to parameters that generate stationary.