Permutation importance dominated by one or a few variables

[Ahmed El-Gabbas]

Hello,

Is there a reason why the permutation importance of a series of cross-validated models of a species are dominated by a few or only one variable (attached figures)?

Prediction maps, response curves, and model performance seem to be fine.

The only reason I find is that species observations are recorded from one side of these important variables (biased). Thus, shuffling of values of these variables lead to highest reduction in training AUC.

Is it advisable in this case to use Percent contribution instead?

Also, is it possible to re-estimate the permutation importance of established model using R? Can we estimate permutation importance using for example, testing AUC or other metric?

Cheers,

Ahmed

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Dr. Ahmed El-Gabbas,

Ocean Acoustics Lab, Alfred-Wegener-Institut

My Website:  https://elgabbas.netlify.com/

[Amir Sohail Choudhury]

Hello

Please Check your final maxent result (excel file), Contribution of variables are measured by permutation importance.

Also check your jackknife result. 

Regards

--
You received this message because you are subscribed to the Google Groups "Maxent" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/maxent/29bc14df-3f18-404b-84c6-ff9221369f61%40googlegroups.com.

[Ahmed El-Gabbas]

Thanks Amir for your response. The same results are shown in the excel file.
I turned jackknife test off, as the data is huge and each cross-validated model needs more than 2 days to be performed on HPC cluster without jackknifing. With Jackknifing, I found it impossible to run the models in due time.

Ahmed

On Monday, 4 May 2020 23:04:47 UTC+2, Amir Sohail Choudhury wrote:

Hello

Please Check your final maxent result (excel file), Contribution of variables are measured by permutation importance.

Also check your jackknife result. 

Regards

On May 5, 2020 2:27 AM, "Ahmed El-Gabbas" <elga...@gmail.com> wrote:

Hello,

Is there a reason why the permutation importance of a series of cross-validated models of a species are dominated by a few or only one variable (attached figures)?

Prediction maps, response curves, and model performance seem to be fine.

The only reason I find is that species observations are recorded from one side of these important variables (biased). Thus, shuffling of values of these variables lead to highest reduction in training AUC.

Is it advisable in this case to use Percent contribution instead?

Also, is it possible to re-estimate the permutation importance of established model using R? Can we estimate permutation importance using for example, testing AUC or other metric?

Cheers,

Ahmed

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Dr. Ahmed El-Gabbas,

Ocean Acoustics Lab, Alfred-Wegener-Institut

My Website:  https://elgabbas.netlify.com/

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To unsubscribe from this group and stop receiving emails from it, send an email to max...@googlegroups.com.

[Jamie M. Kass]

Ahmed,

The permutation importance is deterministic, and the percent contribution is not. Thus, with every model run, percent contribution will change. Permutation importance should reflect the variables with non-zero coefficients in the lambdas file. Can you confirm that's true?

Jamie

[Adam Smith]

@Jamie: Just curious, is permutation importance really deterministic given that it relies on permuting the data?  Wouldn't it need to use the same random seed (probably in the java code, not R) to be exactly the same?

[Martin Lowry]

The code was released as Open Source some time ago. You can see that the permutation does indeed use a hardcoded random seed.

PermutationImportance.java, line 122: Random generator = new Random(11111);

Cheers!