minimum detectable negligible difference (MDND) model validation

[Andrew Nelson]

Hi All,

I have been thinking of alternative methods of model validation than
the typical BIAS and RMSE. Aaron W. and I were thinking of looking
towards equivalence testing. One of the methods which stood out was
calculating the minimum detectable negligible difference (MDND)
developed by Parkhurst (2001) and Radtke and Robinson (2006) which
essentially is a minimum cutoff where the predicted values are not
longer related to the observed values. The function below calculate
the MDND and then gives a estimation of the deviation of the
predicted:observed ratio to the MDND to hopefully tell us how far away
the models are from the cutoff value for model selection. I am hoping
to use this technique to assist in selecting among regional biomass
models for small trees here in Maine.

Any comments or suggestions would be great.

Regards,
Andrew

##Minimum Detectable Negligible Difference Function
##Based on Parkhurst (2001) and Radtke & Robinson (2006)
##Function using equivalence testing for model validation
##Ho: model is not valid
##Ha: model is valid
##This function outputs:
#mean of observed values (mean.obs)
#mean of predicted values (mean.pred)
#1-(predicted/observed) (PredtoObs) (%) (%PO)
#MDND/observed (MDNDtoObs) (%) ##Smallest negligible
difference to conclude predictions are similar to observed
#Deviation of predicted from MDND (Dev) (%) ##Determines how far
the predicted values are from the cutoff, smaller values = predictions
closer to observed

##if %PO < (-MDNDtoObs) or if %PO >(+MDNDtoObs), then cannot reject
the null hypothesis that the model is not valid
##(i.e. predicted values do not accurately represent the observed
values)
##if %PO > (-MDNDtoObs) or if %PO < (+MDNDtoObs), then reject the
null and not reject the alternatitve hypothesis that model is valid
##(i.e. predicted values accurately represent the observed values)


MDND<-function(x,y,alpha){ #x = predicted values, y = observed
values, alpha is significance level (i.e. 0.05)
mean.obs<-mean(y) #calculates the mean of the observed values
mean.pred<-mean(x) #calculates the mean of the predicted values
mean.diff<-mean.obs-mean.pred #calculates the difference between the
observed and predicted means
se.diff<-sd(y-x)/sqrt(length(x)) #calculates the standard error of
the observed minus predicted values
aaa.lm<-lm(y~x)
t<-qt(1-alpha,df=aaa.lm$df.residual) #calculates the t value for
the lm model
mdnd.est<-(-1*((t*se.diff)-mean.diff)) #calculates the minimum
detectable negligible difference (MDND) MDNDtoObs<-
abs((mdnd.est/mean.obs)*100) #smallest negligible difference to
conclude predictions are similar to observed

PredtoObs<-abs(1-(mean.pred/mean.obs))*100 #calculates the percent
of predicted values to observed values
Dev<-PredtoObs-MDNDtoObs ##Deviation of the Pred:Obs ratio from
MDND:Obs ratio;
##negative values: Pred:Obs is within the
MDND cutoff range
##Positive value: Pred:Obs is not within
MDND cutoff range

return(if(PredtoObs<(-1*(MDNDtoObs)) | PredtoObs>MDNDtoObs){
writeLines("H0 not rejected. predicted values not similar to
observed")

return(list(mean.obs=mean.obs,mean.pred=mean.pred,PredtoObs=PredtoObs,MDNDtoObs=MDNDtoObs,Dev=Dev))}
else if (PredtoObs>(-1*(MDNDtoObs)) | PredtoObs<MDNDtoObs){
writeLines("H0 rejected. predicted values are similar to
observed")

return(list(mean.obs=mean.obs,mean.pred=mean.pred,PredtoObs=PredtoObs,MDNDtoObs=MDNDtoObs,Dev=Dev))})
}