predicted values

[Paul Lantos]

I'm trying to figure out how to obtain residuals from an INLA model. Is there a way to pull predicted values or to compute observed - predicted?

My suspicion is I'd need to create a prediction stack with the same coordinates and covariates as the input data but with y = NA, right? I could then take the fitted values from the estimation model and subtract the ones from the prediction model?

[Håvard Rue]

On Thu, 2017-02-16 at 09:29 -0800, Paul Lantos wrote:
> I'm trying to figure out how to obtain residuals from an INLA model.
> Is there a way to pull predicted values or to compute observed -
> predicted?

you have access to the posterior for the linear predictor (or the
fitted values) and from this you could compute the posterior for the
residual, if that is what you want. or as you say, you might for
simplicity do the (obs-pred)/stdev

H


--
Håvard Rue
Department of Mathematical Sciences
Norwegian University of Science and Technology
N-7491 Trondheim, Norway
Voice: +47-7359-3533 URL : http://www.math.ntnu.no/~hrue
Mobile: +47-9260-0021 Email: havar...@math.ntnu.no

R-INLA: www.r-inla.org

[Paul Lantos]

Thanks,
So for the observed in a binary model I could just use the observed 0 or 1?

[Håvard Rue]

On Fri, 2017-02-17 at 07:42 -0500, Paul Lantos wrote:
> Thanks,
> So for the observed in a binary model I could just use the observed 0
> or 1?
>

yes. but in this case the 'concept of residuls' make less sense.

[Paul Lantos]

The overall question is whether there is spatial clustering of residual deviance, and whether this improves with different model parameters. I guess plotting CPO is one way to address the question.

But I also want to be able to compare models done in INLA to alternative approaches. Residuals are easy to export from say a GAM, for instance. If there's a way to do a cross validation for GAMs that's comparable to CPO perhaps that would be the best - I'm not sure how to do that, though.

[Finn Lindgren]

Perhaps find out precisely what the residuals are as defined by you other software?

That might reveal what the equivalent thing from INLA would be. The benefit of the CPO calculations is that they already do the leave-one-out part. Does your other GAM software do that as well? (Note the models in inla are also GAMs, so it doesn't quite make sense to say one compares GAMs to INLA; the comparison is likely more between a specific frequentistic estimation method and the specific Bayesian approximation method provided by INLA.)

Finn

Finn

--
You received this message because you are subscribed to the Google Groups "R-inla discussion group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to r-inla-discussion...@googlegroups.com.
To post to this group, send email to r-inla-disc...@googlegroups.com.
Visit this group at https://groups.google.com/group/r-inla-discussion-group.
For more options, visit https://groups.google.com/d/optout.

[Paul Lantos]

Yes, comparing different approaches to GAMs would be more accurate way of describing it -- both LOESS-based ones in the GAM package, spline-based ones in the MGCV package, and Bayesian ones in INLA. But adjacency and distance are specified very differently in INLA (with the mesh) vs frequentist GAMs where long and lat are treated more simply as vectors.

MGCV uses an internal generalized cross validation, and there may be some way to extract pointwise measurements of prediction accuracy. It is not as far as I know a leave one out CV; and as you know not being the world's most experienced R coder I'm not sure how to code a LOOCV from scratch. Been looking for some help with that.

In the GAM package we have typically picked a smoothing bandwidth just based on AIC, but again I worry that this may ignore heterogeneity in errors, and I'd prefer to use a cross-validation.

In the end these are the important research questions:

- If different approaches to modeling produce different results, how do we compare their accuracy relative to one another?

- How sensitive are these different techniques to edge effects and data density?

Paul

On Fri, Feb 17, 2017 at 10:07 AM, Finn Lindgren <finn.l...@gmail.com> wrote:

Perhaps find out precisely what the residuals are as defined by you other software?

That might reveal what the equivalent thing from INLA would be. The benefit of the CPO calculations is that they already do the leave-one-out part. Does your other GAM software do that as well? (Note the models in inla are also GAMs, so it doesn't quite make sense to say one compares GAMs to INLA; the comparison is likely more between a specific frequentistic estimation method and the specific Bayesian approximation method provided by INLA.)

Finn

Finn

On 17 Feb 2017, at 14:57, Paul Lantos <paul....@gmail.com> wrote:

The overall question is whether there is spatial clustering of residual deviance, and whether this improves with different model parameters. I guess plotting CPO is one way to address the question.

But I also want to be able to compare models done in INLA to alternative approaches. Residuals are easy to export from say a GAM, for instance. If there's a way to do a cross validation for GAMs that's comparable to CPO perhaps that would be the best - I'm not sure how to do that, though.

On Feb 17, 2017 8:46 AM, "Håvard Rue" <hr...@r-inla.org> wrote:

On Fri, 2017-02-17 at 07:42 -0500, Paul Lantos wrote:
> Thanks,
> So for the observed in a binary model I could just use the observed 0
> or 1?
>

yes. but in this case the 'concept of residuls' make less sense.


--
Håvard Rue
Department of Mathematical Sciences
Norwegian University of Science and Technology
N-7491 Trondheim, Norway
Voice:  +47-7359-3533    URL  : http://www.math.ntnu.no/~hrue
Mobile: +47-9260-0021    Email: havar...@math.ntnu.no

R-INLA: www.r-inla.org

--
You received this message because you are subscribed to the Google Groups "R-inla discussion group" group.

To unsubscribe from this group and stop receiving emails from it, send an email to r-inla-discussion-group+unsub...@googlegroups.com.
To post to this group, send email to r-inla-discussion-group@googlegroups.com.

[Paul Lantos]

Ok, make sense. I do see in the output how different methods converge upon similar results. However I am definitely seeing differences between mgcv, gam, and inla, including sensitivity to edge effects, extremes of model predictions (i.e. the odds range), smoothness, and significance estimates. The differences may or may not be important, but I think it's important to justify why one model vs another was used.

As for heterogeneity being built into the model, at least for LOESS-based gams it is in a global rather than local sense. If I'm picking a span size based on AIC, then this choice is based on a bias-variance tradeoff over the entirety of the dataset, right?

On Fri, Feb 17, 2017 at 11:45 AM, Finn Lindgren <finn.l...@gmail.com> wrote:

On 17 Feb 2017, at 15:21, Paul Lantos <paul....@gmail.com> wrote:

Yes, comparing different approaches to GAMs would be more accurate way of describing it -- both LOESS-based ones in the GAM package, spline-based ones in the MGCV package, and Bayesian ones in INLA. But adjacency and distance are specified very differently in INLA (with the mesh) vs frequentist GAMs where long and lat are treated more simply as vectors.

No, INLA just exposes more of the details; conceptually and often also in the code (e.g. in mgcv) very similar things are used.

MGCV uses an internal generalized cross validation, and there may be some way to extract pointwise measurements of prediction accuracy. It is not as far as I know a leave one out CV; and as you know not being the world's most experienced R coder I'm not sure how to code a LOOCV from scratch. Been looking for some help with that.

You could do a more straightforward data split into estimation data and validation data.

In the GAM package we have typically picked a smoothing bandwidth just based on AIC, but again I worry that this may ignore heterogeneity in errors, and I'd prefer to use a cross-validation.

That's a detail of the estimation method; in inla the smoothing bandwidth is determined by the spde parameters and their posterior distributions.  Heterogeneity should ideally be built in to the model, regardless of frequentist or Bayesian approach.

In the end these are the important research questions:

- If different approaches to modeling produce different results, how do we compare their accuracy relative to one another?

Holdout validation/comparison data.  Decide on what aspect you want to test, and pick a scoring method that tests that.  If all you care about is the point estimate of parameters, you need a very different assessment than if you want to check probabilistic spatial predictions.  In the latter case, you need at point estimates and kriging std.dev.s in the freq. case and posterior mean and std.dev. in the Bayesian case. Then you need to figure out a useful score for binary data.  Take a look at the Gneiting and Raftery 2007 JASA paper "Strictly Proper Scoring Rules, Prediction, and Estimation" (doi 10.1198/016214506000001337)

Finn

To unsubscribe from this group and stop receiving emails from it, send an email to r-inla-discussion-group+unsubscr...@googlegroups.com.

[Finn Lindgren]

On 17 Feb 2017, at 20:42, Paul Lantos <paul....@gmail.com> wrote:

Ok, make sense. I do see in the output how different methods converge upon similar results. However I am definitely seeing differences between mgcv, gam, and inla, including sensitivity to edge effects, extremes of model predictions (i.e. the odds range), smoothness, and significance estimates. The differences may or may not be important, but I think it's important to justify why one model vs another was used.

The edge effects in mgcv will be different than in the current inla spde models, yes. But neither have very strong motivations (and the spde edge modeling will be expanded to something closer to mgcv in the hopefully not too distant future); in reality both are wrong (all models are wrong but some are useful...)

As for heterogeneity being built into the model, at least for LOESS-based gams it is in a global rather than local sense. If I'm picking a span size based on AIC, then this choice is based on a bias-variance tradeoff over the entirety of the dataset, right?

Yes, just as when using a stationary spde model, or nonlinear effects via rw2 models, etc.  No fundamental difference between mgcv and INLA apart from which specific subclasses of models are easily available in the interfaces.

Finn

[Paul Lantos]

That all models are wrong in a sense is why I want a good way to compare them :) 

The adaptive smoothing in mgcv s(x, bs="ad") if I understand it allow the smoothing parameters change over x. The AIC values don't change much, making it questionable whether the computational costs are worth it (but I haven't tried a cross validation comparing the two). Is there any comparable approach with spde?

[Finn Lindgren]

I see, yes, you can do nonstationary 1-dimensional spdes for the same effect.

Finn

[Finn Lindgren]

On 17 Feb 2017, at 15:21, Paul Lantos <paul....@gmail.com> wrote:

Yes, comparing different approaches to GAMs would be more accurate way of describing it -- both LOESS-based ones in the GAM package, spline-based ones in the MGCV package, and Bayesian ones in INLA. But adjacency and distance are specified very differently in INLA (with the mesh) vs frequentist GAMs where long and lat are treated more simply as vectors.

No, INLA just exposes more of the details; conceptually and often also in the code (e.g. in mgcv) very similar things are used.

MGCV uses an internal generalized cross validation, and there may be some way to extract pointwise measurements of prediction accuracy. It is not as far as I know a leave one out CV; and as you know not being the world's most experienced R coder I'm not sure how to code a LOOCV from scratch. Been looking for some help with that.

You could do a more straightforward data split into estimation data and validation data.

In the GAM package we have typically picked a smoothing bandwidth just based on AIC, but again I worry that this may ignore heterogeneity in errors, and I'd prefer to use a cross-validation.

That's a detail of the estimation method; in inla the smoothing bandwidth is determined by the spde parameters and their posterior distributions.  Heterogeneity should ideally be built in to the model, regardless of frequentist or Bayesian approach.

In the end these are the important research questions:

- If different approaches to modeling produce different results, how do we compare their accuracy relative to one another?

Holdout validation/comparison data.  Decide on what aspect you want to test, and pick a scoring method that tests that.  If all you care about is the point estimate of parameters, you need a very different assessment than if you want to check probabilistic spatial predictions.  In the latter case, you need at point estimates and kriging std.dev.s in the freq. case and posterior mean and std.dev. in the Bayesian case. Then you need to figure out a useful score for binary data.  Take a look at the Gneiting and Raftery 2007 JASA paper "Strictly Proper Scoring Rules, Prediction, and Estimation" (doi 10.1198/016214506000001337)

Finn