I am using INLA to derive predictions + uncertainty estimates.
I find that the prediction interval coverage probability (PCIP, ie the percentage of actual observations falling within the prediction interval) is generally poor.
Here is an example I have used in the past on an unrelated note with Finn Lindgren (thanks to him for the development of this example):
# Load example data
data(meatspec, package = "faraway")
# Scaling from the original example; numerically unstable without it,
# at least with the default prior
scaley <- 100
# To be able to control scopes, encapsulate the data and avoid global variables.
the_data <- list(
X = matrix(1, nrow = nrow(meatspec), ncol = 1),
# Z is the matrix of predictors (in this case, a matrix of reflectance values)
Z = as.matrix(meatspec[,which(colnames(meatspec) != "fat")]),
# The outcome is the meat fat content
orig_outcome = meatspec$fat / scaley,
outcome = meatspec$fat / scaley
# Setting some to NA to check performance on held out samples
the_data$outcome[173:215] <- NA
# INLA model, based on https://julianfaraway.github.io/brinla/examples/ridge.html#ridge-regression-using-bayes
# with modifications from Finn
mod_inla <- inla(
formula = outcome ~ -1 + X + f(psi, model = "z", Z = Z, hyper = list()),
data = c(the_data,
list(psi = seq_len(nrow(meatspec)))),
control.predictor = list(compute = TRUE)
The PICP is then computed as follows:
# PICP on held-out samples for the 95% PI
bind_cols(obs = the_data$orig_outcome[173:215]) %>%
mutate(obs_in_pi = obs >= `0.025quant` & obs <= `0.975quant`) %>%
summarise(picp = sum(obs_in_pi) / n())
I am getting pcip = 0.51, while the expected value for the prediction interval used in this example is 0.95.
Any reason for this systematic underestimation of the prediction interval?