Linear Regression

[Dominic Steinitz]

I have attempted to cast linear regression as a state space model. The bit I am unsure about is how to specify the observations which are both the independent and the dependent variables. I am also unsure about how to set up the data using the R package rbi but I’d like to get the model right first. Why am I doing this? Because I want to create a more sophisticated model with a change point and I wanted to reproduce results I already have for simple linear regression.

> model LR {
> param sigma;
> state m, c;
> obs x, y;
>
> sub parameter {
> sigma ~ uniform(0.0, 1000.0)
> }
>
> sub proposal_parameter {
> sigma ~ truncated_gaussian(sigma, 0.2, 0.0, 1.0)
> }
>
> // We know the slope and intercept and their approximate standard
> // deviations from other methods.
> sub initial {
> m ~ normal(0.17, 0.01)
> c ~ normal(-1700.0, 200.0)
> }
>
> // Allow the slope and intercept to vary slightly. Probably folklore
> // but may avoid particle collapse.
> sub transition {
> eta ~ normal(0.0, 1.0e-6)
> m <- m + eta
> epsilon ~ normal(0.0, 1.0e-6)
> c <- c + epsilon
> }
>
> sub observation {
> y ~ normal(m * x + c, sigma)
> }
> }

Dominic Steinitz
dom...@steinitz.org
http://idontgetoutmuch.wordpress.com

[Lawrence Murray]

Hi Dominic,

1. I'd specify x as an input, not an obs.

2. Indeed this slowly varying parameter business is folklore. What I
would do is make m and c parameters, put priors on them in the parameter
block, a proposal on them in the proposal_parameter block, and leave
your transition block empty. Your "state-space model" is then
deterministic, and just computes the likelihood one x-y pair at a time.
You'll just be doing Metropolis-Hastings with a random walk proposal.
Just use --nparticles 1, there's no point doing otherwise.

3. This might work quite well with SMC^2. It would introduce one data
point at a time and adapt the proposal distribution over your parameters
as it goes, so would avoid you having to tune the proposal distribution
too much. Of course, because you're using --nparticles 1, it would just
be SMC, not SMC^2, but the implementation reduces gracefully.

Cheers,

Lawrence