Testing RM values. A graph of AUC results - Advice?

[Steve Research]

I ran a series of Maxent test runs using my model details with all the options set for my final runs at this stage. I changed the RM from 1 to 3 in steps to look at the effects of this on AUC, to help me think about over-fitting etc (the difference is just plottedx40 to emphasize the shape of the curve).

The graph above shows the two AUC values (Training and Test). They are pretty close all the way through but there is a definite, detectable change in the gradient of a graph of the difference between them. This has a clear break-of-slope at RM=2. So my questions are:

•  The TraningAUC and TestAUC are pretty close anyway. Should I just conclude that over-fitting is not much of an issue in this model and run at RM=1?

• Is that change in he gradient at RM=2 too good to be true? Is there some mathematical reason why it is likely to do that, other than it being a significant improvement of the model which just happens to peak at that point?

• I am also looking at the binomial values and TSS for this data. What should dominate my choices here? Should I take this RM=2 value and go ahead so long as the binomial and TSS values are OK, even if it brings them down a little?
  

Cheers,

Steve

PhD (University of Wales)

Currently working on plants of the SWAFR

[Lawrence, Kevin]

Should compare the omission rates at a couple of cut points as well.

Honestly I would select the model based on the AICc using the ENMeval package and almost forget about the AUC.

Kevin

--
You received this message because you are subscribed to the Google Groups "Maxent" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.
To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.