Choosing features and regularization multiplier manually?

[Lukas]

Dear all,

I'm testing my models with different numbers of samples for different species of one genus. I’m using different combinations of features (L, H, LQ, LHQ) and regularization multiplier (from 1 to 6 - seq. by 0,5). Do you think it's a good way how to choose the best combination for each species?

My results untill now:

For example I have two species with the same number of samples (13), but the best combination for the first one is LQ1 and for the second one L1.

Then:

23 sampl. - LQ1,5,

40 sampl. - H1

26 sampl. - H2

14 sampl. - H2,5

6 sampl. -H1

I'm using Pearson et al.' Jackknife method for species < 20 samples and as the best combination = lowest OR + highest AUC, and 10 replicates average AUC for species > 20 samples and as the best combination = highest AUC.

I will be glad if someone could help.

Thank you,

Lukas

[Mariah]

I would also look into an R package called ENMeval. There are many different ways to automatically run evaluations of  combos of features and regularization multipliers, as well as AICc in addition to AUC scores used to select the best combination.

ENMeval: An R package for conducting spatially independent evaluations and estimating optimal model complexity for Maxent ecological niche models. Muscarella et al 2014

Mariah

[Jamie M. Kass]

Your methods look pretty sound. For low sample sizes, also check out Shcheglovitova & Anderson 2013. My group developed an R package called "ENMeval" that iterates multiple model production over varying feature class and reg multiplier combinations and reports their evaluation statistics. You should check it out -- it will not only make your life easier, but make your work reproducible.

Jamie Kass
PhD Candidate
City College, NYC

[Jamie M. Kass]

Oops, looks like Mariah beat me to the punch. Thanks!

-J