Do probabilities obtained with GAM-INLA need to be calibrated?
Lola Riesgo Torres
I am developing a species distribution model for a rare, data-scarce species. To explore the effect of sample prevalence, I fitted several models with different presence–absence ratios by randomly sampling pseudo-absences while keeping presences fixed. After choosing priors for parameters and hyperparameters and fitting the models with INLA, I evaluated their performance. To check whether predicted probabilities are ecologically realistic, I plotted a reliability (calibration) diagram.
The plot suggests miscalibration. I have not found studies that use INLA and then explicitly recalibrate predicted probabilities, as is commonly done for some machine-learning models. In your experience, is post-hoc calibration (e.g., logistic recalibration/Platt scaling, isotonic or beta calibration) advisable in this context, or would it add bias? Alternatively, is adjusting only the intercept to account for the discrepancy between sample and true prevalence sufficient?
Thank you in advance,
Lola R
Helpdesk (Haavard Rue)
there is no guarantee that predictive probabilities are well calibrated, unless
you're using the true model. for a 'rare & data-scarce' species, it might be
harder to get well calibrated predictions, due to little data, hence more
depends on model assumptions.
You can try to simulate data under the model you're using, so its a perfect
case, to see how well you can doit under perfect conditions.
I do not know enough about post-hoc calibration to say anything about that
best
H
> You received this message because you are subscribed to the Google Groups "R-
> inla discussion group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to r-inla-discussion...@googlegroups.com.
> To view this discussion, visit
> https://groups.google.com/d/msgid/r-inla-discussion-group/1a70e392-fa88-42db-bb55-bc7bd6e51f15n%40googlegroups.com
> .
--
Håvard Rue
he...@r-inla.org