Discrepencies in Free Search vs Evaluation xR2

[nick warren]

Hello,

I'm hoping someone can help me understand why a model xR2 following a free search would differ dramatically from the xR2 within the evaluate selected model feature.

Some context: First, this isn't true of all our data sets. Trying another produces xR2's in agreement with each other. The models in question have SU's in empty / small neighborhoods. The free search and model evaluation setup use the same settings, however I have also tried manually adjusting the minimum and maximum neighborhood size for an acceptable model - just to see if it would effect the evaluation. 

Provided that these differences are not an indication of a bigger issue, I'm OK reporting the more conservative xR2 (lower) coming from our model evaluation. But part of my motivation to try and understand this better is that the rank order of the models will change. For example, the two best single predictor models are as follows:

Free Search Model 1, Predictor A, xR2 = 0.40, and Model 2, Predictor B, xR2 = 0.39

Evaluation  Model 1, Predictor A, xR2 = 0.19, and Model 2, Predictor B, xR2 = 0.20

Any insight would be greatly appreciated!

Nick

[Bruce McCune]

Nick, you're right, this is a big difference that needs an explanation. Given your description of the data and what you've done, I'd suggest comparing settings for the minimum neighborhood size needed to make an estimate. Sometimes that can have a surprisingly large effect.

Bruce McCune 

--
You received this message because you are subscribed to the Google Groups "HyperNiche and NPMR" group.
To unsubscribe from this group and stop receiving emails from it, send an email to hyperniche+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hyperniche/fbbf7a9b-2995-477d-ad5c-8d53466d8390n%40googlegroups.com.

[nick warren]

Thank you so much for getting back to me, and for hosting this forum - it's a huge help.

I've done iterations of settings and datasets to try and identify where this is coming from and haven't had any luck so far, other than to narrow it down to the one particular data set. I'll list some details below only if you're interested. Given that the initial xR2 in so many other cases should equal the evaluation xR2, I'm curious if you would feel comfortable trusting the initial free search value, or if this sounds like an indication that there could be a deeper issue with the model? For what it's worth, the story that the models are telling us is consistent with what simple linear regression suggests, so I don't feel that it is misleading by suggesting non-important predictors etc. 

Thanks again,

Nick

Here's a summary of what I've tried so far:

With a new data set, I have evaluated single and multi-factor models, varying the following settings (starting with defaults, then using neighborhood size minimum and maximum, with over fitting controls aggressive and conservative). All cases yield free search xR2 = evaluation xR2.

I have tried all of the above with data set in question, and all cases the evaluation xR2 drops to a consistent/stable lower number. I used both an untransformed and transformed response which shifts the absolute values but does not change the relationship between search and evaluation.

Out of curiosity I doubled the data set in question, and this too changes the the xR2 values but does not change the relationship (the decrease) between free search and evaluation.

[Bruce McCune]

You may have done all this, but please double check:

Evaluate Selected Model dialog....

Model Options tab, overfitting control same as what created the model

Model Options tab, Min Ave Neighborhood Size, choose "manual" and type in setting from your Free Search

Output Options tab, Min Neigh Size for estimate, manual, then enter value from free search.

Bootstrap tab, make sure Bootstrap is not checked.

If you are still seeing a difference, please send the response and predictor files and model file to me, and I will try to reproduce the problem.  But don't send your data in reply to this message; instead,  send it outside the Group to the address listed for Help | Technical Support.

Thanks,

Bruce

To view this discussion on the web visit https://groups.google.com/d/msgid/hyperniche/e1fcab2b-e3b9-4de5-948f-97084aa7d4b6n%40googlegroups.com.

[nick warren]

Good morning Bruce,

Following the advice in your last post I found all settings to be the same except for one:

Evaluate Selected Model Setup > Output Options > Minimum Neighborhood Size for Estimate

By default this number was lower than the value that was in the Free Search.

Raising it to match free search value, decreased xR2 dramatically.

Lowering it to an artificial minimum (0.001) , matched the xR2 between Free Search and Evaluation.

I typically use the automatic setting for Output > Min Size for Est within the Free Search Setup, however it doesn't seem like changing this setting affects the underlying models. Or, I haven't found any evidence that it does yet. 

As you mentioned in the forum and elsewhere - usually this issue doesn't occur. Hopefully this thread will be helpful for anyone in a similar situation.

Thank you again for your help!

Nick