High AUC, Decent TSS, not super great Kappa

[evan teal]

Hello all, 

I've ran some data using the Banta lab's "Maxent using R" script and have gotten back some pretty encouraging results. However, the more I look through the data the more I am unsure about how valid they are. 

As an example

AUC Test:  0.8303

AUC Train:  0.8241

AUC Dif:  0.0062

COR:  0.2670

Kappa:  0.1190

TSS:  0.6715

I know that in general people will take a AUC >.7 and just run with it, but I'm applying this to medical entomology, and just want to be super sure what I have is moderately viable as it is. The map outputs have a strong biological basis when I look it over with my wasp specialist colleague but I wanted some further confirmation! 

This is my first post and I really appreciate how much help this site has provided, its been invaluable!

[Gebreyohannes Zenebe]

Hello,

It’s encouraging that your outputs align with biological expectations.

One point to clarify: you didn’t mention whether you screened your predictors for multicollinearity. Ensuring that non-collinear variables are selected using correlation thresholds or VIF is important, as this can strongly influence model stability and interpretability.

To strengthen the analysis, consider also going beyond AUC, Kappa, and TSS. Adding the Continuous Boyce Index and partial ROC would be particularly valuable for a presence-only MaxEnt model, along with reporting omission rates and evaluating sensitivity and specificity at appropriate thresholds.

It would also be beneficial to use spatial cross-validation instead of random splits to obtain more realistic performance estimates. Additionally, applying AICc for model selection and assessing sensitivity to regularization settings could further improve robustness.

Good luck!

With kind regards,

Gebreyohannes Zenebe
MSc. Geo-Information and Earth Observation Science for Natural Resources Management

Alternative email: gebreyohan...@mu.edu.et / g.ze...@seefconsult.com

Cell Phone: +251930527887 / +251993155395

ORCID id: https://orcid.org/0000-0002-6100-4908

--
You received this message because you are subscribed to the Google Groups "Maxent" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/maxent/7b39b9b3-295f-4b68-9bd7-3a8862bb66een%40googlegroups.com.

[evan teal]

Yes, I did screen for collinearity between the worldclim (after clipping and masking) using VIF and i used the categorical NLCD for land cover data. After screening, I only used 6 bioclim variables that had -.7<x<.7 collinearity between them. Ok thank you for the other metrics to look at, I believe I have ROC in my data somewhere. 

I appreciate the leads on what to do, thank you very much

Evan

To view this discussion visit https://groups.google.com/d/msgid/maxent/CA%2BPW5sbTWALDSqzqDssRDEkiY7Q%2BnBwMvtzPnfDMDHg_TRKPog%40mail.gmail.com.

[Gourav Devtalla]

I want to extract bioclimatic variables from future climatic variables but all it has are bands. Please help

To view this discussion visit https://groups.google.com/d/msgid/maxent/CAGu85OECnbv8-t4skdE6XH8vsCPnzWLkm9TsfP73MgucVM9QAA%40mail.gmail.com.

[Gebreyohannes Zenebe]

Are you familiar with R programming language?

If so, you can use the below code to unstack the layers

library(terra)

# 1. Define directories

input_file <- "E:/data/bioclim/future_bioclim.tif"

output_dir <- "E:/data/output/"

# Create output folder if it doesn't exist

if (!dir.exists(output_dir)) {

  dir.create(output_dir, recursive = TRUE)

}

# 2. Load stacked raster

r <- rast(input_file)

# 3. Assign layer names

names(r) <- paste0("BIO", 1:nlyr(r))

# Check names

print(names(r))

# 4. Unstack and save layers

for (i in 1:nlyr(r)) {

  writeRaster(

    r[[i]],

    filename = paste0(output_dir, names(r)[i], ".tif"),

    overwrite = TRUE

  )

}

To view this discussion visit https://groups.google.com/d/msgid/maxent/CALFY_0Yr0DJzVPwyg7zYQL1Pv%2Bmp%2BpZq9hjtv7qgnt2fg3G4uQ%40mail.gmail.com.