Maximum sensitivity plus specificity - test or training?

[Benjamin Bleyhl]

Hi everybody,
I am using Maxent to map habitat suitability of European bison. To derive areas that could serve as potential "core areas", I need to use a threshold. A range of authors suggest maximum sensitivity plus specificity as a good threshold to derive binary maps, yet I found different suggestions in terms of if this value should be based on training or test data. Maxent output gives both values. Any suggestions?
Thanks in advance,
Benjamin

[brevi...@gmail.com]

Have you figured out which one to use? I would also like to know, do you use test or training omission to calculate sensitivity? I'm using three different thresholds (minTSS, MaxTss, EquTSS) and need to calculate sensitivity and specificity for each threshold. Please advise, do I use the 1-(fractional predicted area) as specificity and the test  omission rate as sensitivity?

Please someone advise 

[Adam Smith]

Honestly, I don't think there's a "best" answer anyone can give you without looking at the maps that would be created by applying the threshold from test or training data.  Ergo, do both and see which seems more reasonable!

(This is, I am assuming, a different question from "How do I evaluate my model, using test or training data?") in which case the answer would be to use the threshold applied to test data to determine sensitivity/specificity.

Best,
Adam

[Jamie M. Kass]

I second Adam’s suggestion — do a couple of reasonable thresholds, then put on your ecologist hat and determine if any make more sense than the others. Thresholding is not an exact science, and appropriate thresholds differ between species, model settings, and a range of other inputs. Because of the lack of a consensus, and because often thresholds are applied blindly without much thought, some researchers detest thresholding so much that they advocate against it completely! In your case, I can see why thresholding would be useful though,

I would add, however, that at least equally important to choosing which threshold to use is tuning your model (exploring different model settings to determine which are optimal for your data). If you have an overfit model, all your maps will be wrong regardless of which threshold you pick.

Jamie Kass
PhD Candidate
City College of NY