ENMeval - single final model and prediction or average over folds?

[makani...@gmail.com]

I am generating some ENMs for bats of North America. I've run ENMevaluate from the ENMeval package with spatial block folds and partitioned background settings. I am selecting the optimal models using the or.10p.avg and auc.val.avg. My question is on what to do next. It appears that selecting the model through ENMeval gives a model and prediction that is then from the whole dataset run with the optimal fc and rm and not an average over the folds. The variable importances, lambdas, and other bits are also from this model that includes all the occurrences and background points with no folds. In the standalone MaxEnt java software it provides an average prediction over the various folds. I'm just seeing what would be the best way forward, or if there is one. Do I take the average prediction and variable importances from the folds? That would just mean 4 separately trained models with distinct lambdas went into one prediction. Or do I just take the single whole model and count the tuning cross-validation step as the validation for that model? That would mean one final model with just one set of lambdas. Any thoughts, help, guidance is appreciated.

[Jamie M. Kass]

This response is late, so my apologies. The default value of the "replicates" argument in the Java implementation of Maxent is 1, meaning no cross-validation is done on models and only training evaluation is done. This means the whole dataset is tested on itself, which should show higher values than cross-validation. Setting "replicates" to values higher than 1 defaults to cross-validation, which will be random. Looking at the documentation, it does seem that the model returned in the Java software is an average of these folds. 

In ENMeval, all the evaluation is handled by the package and not by the Maxent software, including random cross-validation. ENMeval does not return an average of folds, but rather the model constructed on the full dataset. The point here is that we want to use all the data for the final prediction and only use the cross-validation results to <choose> model settings (i.e., feature classes and regularization multipliers). Averages of random cross-validation folds might make sense to produce a "stable" prediction, but you are always using an incomplete dataset to build each model, so the competing philosophy is to use the full dataset to make the final model and predictions. However, for block cross-validation that uses spatial or temporal (or other) blocks, averaging the folds makes even less sense. This is because each model is expected to be incomplete on purpose so that we get better estimates of how well the model can transfer to new conditions. We should not expect that an average of a set of purposefully incomplete models should produce a more accurate prediction than a model built with the full dataset. This is why we should use the block cross-validation results to <choose> settings only. 

Hope this helps.

-Jamie