xR2 differences/model selection

[Autumn]

Hello, 

I'm using NPMR to evaluate how an experimental (categorical) treatment affects the (continuous) distribution of species across two (continuous) environmental variables.  To do this, I have run two sets of models, a set with the continuous response and two continuous predictors, and another set with those same variables as well as the categorical treatment variable.  I then compared the xR2 between the best fit models with and without the categorical variable for each species and noted that there was a treatment effect whenever there was at least a 3% change in the xR2, saying in my manuscript "When the increase in xR² resulting from addition of the categorical treatment to the model was greater than the 3% convention used in Arkle et al. (2012) , we considered the difference meaningful."  Reviewers noted that 3% was a small increase in xR2 and asked for more information about our model selection procedures.  Is the following an accurate statement to include in my manuscript?  "Since NPMR fitted Gaussian weighted models rather than forced linear models, a small improvement in xR² is thought to be more meaningful than comparable changes in traditional R²."  Any suggestions for other explanations or citations?

Thanks much!

[Bruce McCune]

Autumn, this all sounds reasonable, and I like your innovation for evaluating the contribution of the categorical variables, but I have a couple of comments. First, whether 3% is big or not really depends on the situation -- I wouldn't like to have to use one number mechanically for every case. I think 3% is reasonable if you want to be cautious about throwing out subtle but real effects. On the other hand, if you had two predictors that combined to 80% xR2, then I would tend to ignore an additional 3%.

Second, your final quote could be strengthened by pointing out more clearly that this is an increase in a cross-validated R2, which is harder to improve than regular R2, particularly is the sample size is smallish. (You do cite xR2, but I think you could be more explicit about this).

Bruce

--
You received this message because you are subscribed to the Google Groups "HyperNiche and NPMR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hyperniche+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Autumn Sabo]

I really appreciate your quick and helpful feedback, Bruce!

--
You received this message because you are subscribed to a topic in the Google Groups "HyperNiche and NPMR" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hyperniche/FIGTpEF1oEM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hyperniche+unsubscribe@googlegroups.com.