Strategy for presenting model evaluations

[Ken]

Hi all,

I have produced 3 seasonal models relating species occurrence (binomial) to a mix of continuous and categorical environmental variables. I am presenting the AUC values from the evaluation output (reviewers/readers will be on familiar ground here) and I initially thought that the pseudo xR2 values would be another metric that reviewers/readers could intuitively grasp.

But I'm not so sure about that after doing some reading on pseudo R2. As I understand it there are more than a few different pseudo R2 metrics out there and they can produce quite different values on the same data set (~0.3-0.5 for one example). And it appears that the pseudo R2 in HN is unique to HN?

It seems to me that the xR2 values are fine for comparing fit among my models, but may actually mislead readers who expect the pseudo xR2 to have values similar to a traditional R2.

So I have two questions: for the pseudo xR2 in HN, what method is used (who should I cite for the method), and how might pseudo xR2 values compare to traditional R2 other than that they are correlated? 

Secondly, how can I best present the model evaluation output from the models so that reviewers/readers can judge if the models are garbage or not without resorting to hand waving (supporting evidence from other analyses that are consistent with model results & general consistency among models with different neighborhood sizes)? I especially ask this in light of deserved push back/growing focus on the too frequent lack of including transparent model evaluations in papers using an AIC approach. I expect reviewers will not be familiar with NPMR and they will really drill down on the model evaluations (as they should).

My xR2 values range from 0.19 to 0.3 (and result from using Random samples vs. Present samples rather than present/absent samples with relatively high errors of Commission = 22% vs Omission = 0.02%).

Thank you,

Ken

[Bruce McCune]

1. On the pseudo-R2 in HyperNiche -- it's pretty standard, the only thing that is different about it is that it is cross-validated during model fitting, which means it is penalized for overfitting (if any).

One possible citation for it is: Agresti, A. 1990. Categorical data analysis. Wiley, New
York, New York, USA.

(probably doesn't mention crossvalidation)

The pseudoR2 and R2 are really measuring different things, even though the math is essentially the same. The pseudoR2 pretends that the goal with binary data is to predict either 1 or 0 -- which is not the case. You're really trying to estimate the proportion of 1s and 0s at any given point. This leads into the use of likelihood ratio statistics, which are more appropriate given that goal.

2. The likelihood ratio statistics in HyperNiche are pretty standard too. LogB is the log of the likelihood ratio. It depends on sample size (i.e. gets bigger with bigger sample size, even with same degree of fit to the data), so some people prefer AUC as a bounded measure.

I like AveB, the average increment to the likelihood ratio with the addition of one sample unit, but it's not commonly used.

If you want a measure with a fixed endpoint, I suggest reporting the AUC. That isn't in the model list, but you can get it from Evaluate Selected Model.

Hope this helps.

Bruce McCune

--
You received this message because you are subscribed to the Google Groups "HyperNiche and NPMR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hyperniche+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ken]

Thank you Bruce, your reply does help indeed. I had come to the conclusion that AUC would be the best evaluation metric to report, but I wanted to see what your opinion was: I suppose that if I had only continuous variables, xR2 would be helpful too. But, I don't and that's the way it is :)

FWIW: I really like using HyperNiche. It is a smooth running package and it is not a black box. I've run analyses on about 6 data sets now and results/graphics offer much more biological insight than CART approaches I have been using (though the big picture results/conclusions have always been consistent between the methods too). 

Thanks!

Ken

To unsubscribe from this group and stop receiving emails from it, send an email to hyperniche+...@googlegroups.com.

[Bruce McCune]

Great, thanks for your comments. Yes, if you have continuous response variables, then xR2 is the way to go.

Bruce

To unsubscribe from this group and stop receiving emails from it, send an email to hyperniche+unsubscribe@googlegroups.com.