Jackknife (leave-one-out) procedure for model evaluation

[Arta]

Dear Maxenter

Which parameter in Maxent output exactly refers to Jackknife (leave-one-out) procedure for model evaluation?

How can interpret it?

Many thanks

-- Arta

[Lauren Yee]

Hi Arta,

If you're running the maxent in the standalone program, there is a check box to turn on jack-knife. It will then be output in the bottom of your html file.

If you're running this using dismo, then you would specify "jackknife=TRUE" in your arguments.

[Arta]

Thank you, Lauren

The last diagram shows jakknife of AUC for test data, it means that with and without a varible, model building with test data has how many AUC. If this is true, so where is the jackknife test result (withholding a location not variable and running the model with remained) to check the model validation?

Is this included in the html file?

[Lauren Yee]

I'm not sure if I understand your question,  but the html output usually shows three graphs for jacknife: training gain, testing gain and auc.
The two colours on the graph signify whether the model is run with just the one variable (dark blue) vs with all the variables (light blue).
Does that answer your question?

[Forough Goudarzi]

Sure, but this is not my concern.

Is there any jackknife analysis for model validation included in maxent output?

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/X0KnJm34Mi8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.
To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Paulo]

Hi

I copied the answer from old posts of the group:

Bill Peterman

"Maxent can be used to get the data for the Pearson et al. approach. 
You simply need to set the cross validation number equal to the number 
of samples in your data set (i.e., only one sample is used for testing 
while the others are used for model training). You then need to decide 
what threshold level to use (e.g. minimum training presence) and then 
go through each iteration to see if the Maxent probability value of 
each test point for each model iteration was equal to or greater than 
the threshold level for that particular model."

and this is also useful when using pearson's pValueCompute.exe

Lukas:

"you have to open the maxent.csv file and go to the column "Minimum
training presence test omission" for the numbers of the first row. But
you have to change the 0 to 1 and the 1 to 0 because Maxent is showing
the test omission and the p-value software using the errors. So just
the other way around.
Then use the values of the column "Minimum training presence area" for
the predicted area”

I hope it help you

Best luck, Paulo

[Forough Goudarzi]

Many thanks Paulo

It's really helpful.

I will also try pValueCompute.exe and let me come back if question derived!

Best wishes

--Arta

[Jamie M. Kass]

You can also do jackknife cross-validation (k=n, where k is number of partitions and n is number of points), and many others, using ENMeval in R (http://onlinelibrary.wiley.com/doi/10.1111/2041-210X.12261/abstract), or SDMToolbox (http://onlinelibrary.wiley.com/doi/10.1111/2041-210X.12200/abstract; http://sdmtoolbox.org) in ArcGIS.

Jamie Kass
PhD Candidate
City College, NYC

> To unsubscribe from this group and all its topics, send an email to maxent+un...@googlegroups.com.

>
> To post to this group, send email to max...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/maxent.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/X0KnJm34Mi8/unsubscribe.
>

> To unsubscribe from this group and all its topics, send an email to maxent+un...@googlegroups.com.

[Forough Goudarzi]

Many thanks All

> To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.

>
> To post to this group, send email to max...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/maxent.
>
> For more options, visit https://groups.google.com/d/optout.
>
>
>
>
>
>
>
>
>
> --
>
> You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
>
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/X0KnJm34Mi8/unsubscribe.
>

> To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.

>
> To post to this group, send email to max...@googlegroups.com.
>
> Visit this group at https://groups.google.com/group/maxent.
>
> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/X0KnJm34Mi8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.

[Forough Goudarzi]

Dear Paulo

I want to perform jackknife base on your comment but: 

does "minimum training presence"  be Minimum presence training area in maxentresults.csv file? if yes, for every replicate we must use any value in front of any replicate as the threshold, is this true?

where can i find "Maxent probability value of each test point for each model iteration" for comparison with that threshold? is this equal to Minimum training presence area?

Many thanks

--Arta

[Paulo Vinicius Ferraz Corrêa]

Hello Arta,

 

I will try to make the process clearer for you.

First, you must select which threshold you are going to use: for what I read, people usually use "Minimum training presence" threshold for this test.

After that, you just need to make a txt file to use in the pValueCompute.exe (the software comes with a template). That file must have two columns: the first one is whether the replicate was successful (value 1) or not (value 0) to predict the location of the sample not used. To do this, go to the column "[name of threshold] test omission" in the maxentresults.csv file. If the value is 0, that means the model successful predict the sample not used, so you should put 1 in the txt file. If the value is 1, that means the sample was omitted in the model, so you should put 0 in the txt file.

Now for the second column, just copy the values of the "[name of threshold] area" column.

 

Feel free to ask if you have any further doubts

Best regards, 

Paulo 

[M Mf]

Dear colleagues,

I downloaded the program pValueCompute.exe but this did not work on my computer (with windows 10). Has anyone had the same problem? I already have the table with the data of the models by jackknife (live-one-out) ready, however I am not able to do the analysis because the program does not work. Does anyone know how to do this p-value evaluation analysis in R? 

Hugs.

 |

V

The problem in the program:

[Paulo Vinicius Ferraz Corrêa]

Hi, 

I had the same problem with the pValueCompute.exe. One workaround is to manually download and register the missing file. Just Google the name of the file to see how to do that. The problem is that there is one or two more missing files after you correct the first one (at least it is what happened to me). 

There is a script in python to calculate the p value.

http://journals.plos.org/plospathogens/article?id=10.1371/journal.ppat.1005780

https://github.com/AndrewJudson/jackknife

 

As far as I know, there is not a script or package to calculate these values in R.

Cheers,

Paulo

[M Mf]

Dear Paulo,

I got it! 

I downloaded the missing file and pasted it into all system folders on my computer ... Either I would get a super virus or run this analyses, but in the end everything worked out! Thank you for your help!

Hugs.

Marília

Em sexta-feira, 5 de maio de 2017 16:02:28 UTC-3, Paulo escreveu:

Hi, 

I had the same problem with the pValueCompute.exe. One workaround is to manually download and register the missing file. Just Google the name of the file to see how to do that. The problem is that there is one or two more missing files after you correct the first one (at least it is what happened to me). 

There is a script in python to calculate the p value.

http://journals.plos.org/plospathogens/article?id=10.1371/journal.ppat.1005780

https://github.com/AndrewJudson/jackknife

 

As far as I know, there is not a script or package to calculate these values in R.

Cheers,

Paulo

On May 4, 2017 16:07, "M Mf" <marilia...@gmail.com> wrote:

Dear colleagues,

I downloaded the program pValueCompute.exe but this did not work on my computer (with windows 10). Has anyone had the same problem? I already have the table with the data of the models by jackknife (live-one-out) ready, however I am not able to do the analysis because the program does not work. Does anyone know how to do this p-value evaluation analysis in R? 

Hugs.

 |

V

The problem in the program:

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/X0KnJm34Mi8/unsubscribe.

To unsubscribe from this group and all its topics, send an email to maxent+un...@googlegroups.com.

[ndimhypervol]

For anyone else interested, you can perform jackknife cross-validation using ENMeval in R very easily. Just specify method="jackknife".

Jamie Kass

PhD Candidate, CCNY

cell: 917-602-5787

[Милош Поповић]

Does ENMeval calculate the P value?

Best regards,
Miloš

У пет, 05. 05 2017. у 21:04 -0400, ndimhypervol пише:

[ndimhypervol]

If this is in reference to Pearson et al 2007, the reason they derived the P value approach was because they decided AUC was not appropriate for poorly sampled areas. This may be true for some studies with small sample sizes.

ENMeval calculates the average and variance test AUC, AUCdiff (sensu Warren and Seifert 2011) , and average and variance of omission rates for MTP and 10% across all groups (for jackknife this would be the model built after leaving out each individual record). It also calculates full model AUC and AICc.

If you require P values and knew some R, you could tweak the code to make it do that without much hardship. If there is a huge need for this, users can request new additions to ENMeval through its Github repository.

Jamie Kass
PhD Candidate, CCNY

[M Mf]

Dear Miloš,

The ENMeval doesn't make the p-value of Pearson's (2007) work, but Pearson make an R script and made this avaible in his own site: <http://www.ucl.ac.uk/cber/pearson>, in "Data and Software". I hope I've helped!

Hugs,

Marília

[ndimhypervol]

Interesting, thanks Marília. I wasn't aware that an R script existed. If there is demand, it wouldn't be too hard to make a script that formats ENMeval output into the input format for this script.

Jamie Kass

PhD Candidate, CCNY

cell: 917-602-5787

--

[M Mf]

I've also thought of that! It would be too interesting! I still do not understand why this technique is not available in the R packages for ecological niche modeling :\ But I think it's time! 

Hugs.

Marília

[Paulo Vinicius Ferraz Corrêa]

Thank you Marília for sharing this information. I am sure this will help a lot of people. 

Cheers, 

Paulo 

To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.

[M Mf]

You're welcome, Paulo (: And I hope it helps!

Hugs,

Marília

[Милош Поповић]

Thank you all for the info! I am glad that we can now compute P value within R.
Last time I have to go to my neighbour to calculate P value under Windows :)
Also, thanks for the effort on ENMeval, since I am currently using it and I
think AUC procedure for selecting regularisation and model complexity works
great.

All the best,
Miloš

[Arne Adam]

Hi Marília,

I tried to go to the like were Pearson put his Rscript to calculate the p value from the ENMeval output but the link is not working anymore. I was wondering if you could provide me with the R script as it would be very useful for my research. I hope you can help me with this. Thank you in advance.

Arne Adam

PhD student

Curtin University

Op maandag 8 mei 2017 21:05:21 UTC+8 schreef M Mf:

[Areej Jaradat]

Dear All

Thank you for the valuable input. I have two questions. 1. Does this test only made when crossvalidation method is used? can it work if I used bootstrap? 

If yes then, I noticed in all of the examples I came cross that the values of the first column (obtained from Minimum training presence test omission) are always 0 or 1 while in my file I have some with decimal value .. so could I say that since the 0 is converted to 1, so the 0.2 in my file will be converted to o.8 ?

Also do I have to apply the MTP threshold in maxent settings before running the model? Or it doesn’t matter?