validate / calibrate in rms

[Daniel Smith]

Dear Frank,

From my understanding validate doesn’t adjust our original model parameter estimates, though it does correct for biased estimates of how good our model is. So the index corrected r2 is a better representation of the amount of variance explained by our predictors than the r2 from our original fit? Is it a case of if we have poor validation, we then go back and change the model? I don’t really understand the difference between validate and calibrate. Do we need to do both? Would calibrate tune the original parameter estimates for us and re-fit the model? Or just tell us how much calibration was performed in the original fit?

Thanks!

[Frank Harrell]

Neither of these are for tuning parameter estimates.  They are for telling you how badly the model is likely to perform on samples from the same stream of subjects.

Both validate and calibrate are generally needed.  validate is more aimed at predictive discrimination and calibrate is aimed solely at absolute prediction accuracy.