Re: Model evaluation - AUC and ROC, too easy?

[Francisco Rodriguez Sanchez]

Hi,

Unfortunately AUC is not enough in many cases, e.g. have a look at
Phillips & Elith Ecology 2010, or Warren & Seifert Ecol. Appl 2011.
These papers suggest complementary measures (AIC, POC plots) that can be
obtained in R or ENMTools (enmtools.blogspot.com).

But if you don't need to compare different models, your response curves
and maps look sensible and the AUC is also good, you might skip the
above for your master project. Up to you!

Cheers

Paco



El 21/09/2012 09:09, MyHanh Tu escribi�:
> Hey guys!
> I've made a maxent-model to predict the distribution of Spanish sparrows
> in Spain (I have around 900 locationpoints) with both landcover and
> climate data. I'm now at the stage where I have to evaluate my model.
> I'm pretty new at both statistics and maxent, but I've done some
> research and I hope you can help me verify my method. I've read in three
> articles where they have used the ROC and AUC to evaluate the model's
> accuracy and prediction power, respectively. Is this enough? And is the
> ROC and AUC i find in the output file all I need (which I feel sounds
> too easy to be true, but I would LOVE it if it was this easy), or do I
> need to use a statistical program to further test it?
>
> I'm a masterstudent and kind of running out of time, so I would be very
> very very happy and appreciative of your help. I'll be so happy, I'll
> include you in my "acknowledgements". Seriously.
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google
> Groups "Maxent" group.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/maxent/-/ow9Lnq7RvhEJ.
> To post to this group, send email to max...@googlegroups.com.
> To unsubscribe from this group, send email to
> maxent+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/maxent?hl=en.

--
Dr Francisco Rodriguez-Sanchez
Forest Ecology and Conservation Group
Department of Plant Sciences
University of Cambridge
Downing Street
Cambridge CB2 3EA
United Kingdom
http://sites.google.com/site/rodriguezsanchezf

[MyHanh Tu]

Thanks for the reply!. I'm now looking at Cohen's kappa coefficient in addition to the AUC, as many studies have done. However, I'm getting very low kappa values.. Around 0,08!! I can't make sense of it, as my AUC values are quite high. Any idea?

[Alaa Eldeen]

hi,
I would suggest you to use True Skill Statistic ,TSS, instead Kapa, where Kapa usually give very low value with presence data only.
I have used it in my research and it gave me a reliable output.
Kindly go through this publication it will help you.
Allouche, O., Tsoar, A. & Kadmon, R. (2006) Assessing the accuracy of species distribution
models: prevalence, kappa and the true skill statistic (TSS). J. Appl. Ecol, 43, 1223–1232.

In case you stuck in it, I have developed an excel sheet for TSS I can forward it to you, all what you need is to input your data and it will give you the TSS value.

Cheers

--

You received this message because you are subscribed to the Google Groups "Maxent" group.

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/Q4jmENhDQFUJ.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

--

Alaa Eldin Soultan,  B.Sc, MSc.
Wildlife Conservationist,
St Katherine Protectorate,
National parks of Egypt,
Nature Conservation Sector,
Egyptian Environmental Affairs Agency, EEAA
Tel-Fax: + 2069- 3470033
Mobil:+201225773922

[MyHanh Tu]

Thanks!! I just read through it, and it looks like I can very well skip Kappa and go for TSS. However, I have no idea how to find the TSS, could you please help me?

[Luiz Menini]

Dear Alaa,

I am following this discussion and appreciate if you could send this excel sheet to me (menin...@gmail.com) or to the group. And

Thank you,

Luiz.

2012/9/26 Alaa Eldeen <alaael...@gmail.com>

--
Dr. Luiz Menini Neto
Centro de Ensino Superior de Juiz de Fora
Juiz de Fora - MG - Brasil

[Alaa Eldeen]

Dear All,
Kindly find the attached file it contains TSS excel sheet.

Just to give some info.  to use  TSS file,  first before your running the maxent model  you need to tell maxent to write backgroundPredictions file by selecting this option from the setting, by that, maxent will create backgroundPredictions file in the output directory. All what you need to do is just to copy the last column (logistic) from the backgroundPredictions file and paste it in the first column in TSS file, then go to the output director and open file named samplePredictions and select only cells from the last column named (Logistic prediction) that are adopted to be test not train(you can do this by checking the 3rd column which named Test or train) copy it and paste it in the 3rd column (test) in the TSS file.

Now close all files you have except TSS file, select the 1th column (background) and doing ascending sorting for the current selection only, repeat this step with the 3rd column (test). Then you need to identify you threshold value based on the adopted threshold criterion that you want to use. all what you need to do is just count how many cells have values lower than the value of your threshold, and how many one higher than it. in the TSS file you need is to write these numbers in the highlighted cells with blue and red colors. where,

F6	Number of cells that have value above the threshold value in test column
F7	Number of cells that have value below the threshold value in test column
G6	Number of cells that have value above the threshold value in bachground column
G7	Number of cells that have value below the threshold value in bachground column
H8	Is the sum of F6,F7,G6,G7
I3	Tottal No. of the background points selected from the maxent setting

 TSS value will appear in the yellow cell (I15).

By the way you will find an example in the same file for demonstration.
And also you will find the Kappa Value in cell (I17) and Kappa max in cell (K17) that I was trying to compare between the value of Kappa and TSS for presence data only.
I hope it helps you.
Please let me know if you need more clarification or if you faced any problem with.
I'm  happy to receive you comments
Best of luck

[Angela Cuervo]

Wow!

Thank you very much Alaa!!!

2012/9/29 Alaa Eldeen <alaael...@gmail.com>

--

M. en C. Angela P. Cuervo-Robayo
Estudiante de Doctorado
Laboratorio de biología evolutiva

Universidad Autónoma del Estado de México

México

[Alaa Eldeen]

you welcome.

Just let me know if you encounter any problem.

Cheers

[MyHanh Tu]

Thank you so much! I very much appreciate it. I'll try it out soon.

....You know, you might just be my Maxent-hero!

[Alaa Eldeen]

Thanks MyHanh.
I loved it Maxent-hero, Maxent-hero. LOL
Best of luck

On Sat, Sep 29, 2012 at 10:34 PM, MyHanh Tu <myh...@gmail.com> wrote:

Thank you so much! I very much appreciate it. I'll try it out soon.

....You know, you might just be my Maxent-hero!

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/Dg8ixuueBLMJ.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

[MyHanh Tu]

So, question.. How do I find/define my threshold value? Is there a "standard"-value?

[Alaa Eldeen]

Hi Dear,
If you wanna to identify your threshold value, I would suggest you to use 10 percentile threshold value as it reported to be is the best.
In order to to get it, you need to open the html file located in the output directory of the maxnet file you will find a table, see the attached file, and the select the value in the second column (logistic threshold) in the 5th cell, this value is the threshold value that you will use in TSS.

Back to me if you stuck in it.

Cheers

On Tue, Oct 2, 2012 at 9:45 AM, MyHanh Tu <myh...@gmail.com> wrote:

So, question.. How do I find/define my threshold value? Is there a "standard"-value?

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/-62dFQ2ZpV4J.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

[MyHanh Tu]

Hey. So I got the TSS value of 0.566, and a AUC value of 0.822. How does that sound to you? I guess the TSS value is a lot better than the kappa value I got, but isn't a 0.566 still a quite low value? I'm not sure what conclusions I should draw from this. Hope you can give me a hand on this one. Thank you so much for everything else:))

[Alaa Eldeen]

Dear MyHanh,

First of all, 0.5 value for TSS is not bad, as you might know that the values of TSS lays between 0 and 1, where 
0 - 0.4 = poor
0.4 - 0.5 = fair
0.5 - 0.7 = good
0.7 -0.85 = V. good
0.85 - 0.9 = excellent
0.9 -1 = perfect

According to that I see that you have a good model for predicting your sp.
Regrading the value  of AUC, as I told you before that the AUC value is correlated to the size of the study area and the prevalence of the occurrence points, for instance, if you use a small study region or if the occurrence points are localized in small area and have small prevalence, you will get a high AUC value. unlike AUC, TSS value are not affected by the prevalence of the occurrence point or the size of the study region.
Although I don't know about your occurrence points prevalence and study region, but the first thought of mine is that you are using occurrence points that may cover a small parts of your study area.
Please kindly find the attached paper it might give you more information about AUC, and why I don't recommend it.
Please back to me to know if my thought was right or not.
Cheers 

On Fri, Oct 5, 2012 at 12:56 PM, MyHanh Tu <myh...@gmail.com> wrote:

Hey. So I got the TSS value of 0.566, and a AUC value of 0.822. How does that sound to you? I guess the TSS value is a lot better than the kappa value I got, but isn't a 0.566 still a quite low value? I'm not sure what conclusions I should draw from this. Hope you can give me a hand on this one. Thank you so much for everything else:))

--

You received this message because you are subscribed to the Google Groups "Maxent" group.

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/K7htJBA-M0YJ.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

[MyHanh Tu]

Ok, I agree as that paper felt very strongly about not using AUC. But I kind of felt strongly convinced to use it as all the studies I've read have used it! Do you think I should just push it out of the way and only use TSS, or use both and then later discuss it? Is it then enough just to use TSS alone? May I ask where you found the scale for poor-very good values for TSS? I just assumed that the scale was somewhat the same for AUC as TSS, that's why I thought it was bad. Sorry, my bad!

Other information: I have around 900 presence points for the whole Iberian peninsula. The points covers about 40-50% of the land.

[Alaa Eldeen]

If I were you, I will adopt TSS to be the model evaluation method, and in my discussion I will discuss AUC and TSS.
Again as I mentioned in the previous email since the AUC score is greater than 0.7 this indicate that your model is doing well and better than random one.
If you convinced with AUC, I would suggest you to use A null-model for significance testing (see the attached zipped file).
Regarding the TSS score, I have attached the source and you will find it in the table in the page 11.

I think the presence points and the percentage of cover is the reason for the AUC score.

Let me know if it works with you. and keep me updated

Cheers

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/if5INtVydz8J.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

[MyHanh Tu]

I'm sorry, this might be a stupid question, but I want to make sure I know this. Why is the TSS score scale the same as Kappa score scale?

I guess I'll just follow your advice and only use TSS, but discuss them both in my discussion. Thank you.

[Alaa Eldeen]

No at all, it is a good question.
Since bath Kappa and TSS are based on confusion matrix,  and TSS is just a modification for Kappa where there is no absence points, so it is logic to use one scale for all of them.
So, using Kappa's scale in case of TSS is acceptable.
I wish it reveal your confusion.

I so interested to see study when you finish.

Cheers

[Jenni]

Thankyou Alaa and everyone else - this conversation and the attachments have been extremely useful for me too!

Cheers
Jenni

[epiphyte]

Hi Alaa,

Thanks for you guys discussion and contributions to this group. :)

However, I am a bit confused. In your reference, it says Kappa has been criticized for being dependent on prevalence (Allouche et. al. 2006). Since TSS is a modification from Kappa, how could it independent from prevalence? On the contrary, in Swets (1988)'s report says that AUC is "a measure independent of event frequency". I think that AUC is not very reliable is because "the fraction of the total study area predicted present" is used instead of "commission rate" in Maxent (Maxent manual). Since the absence data is not available, I wonder the TSS would use similar way for calculation?

Best,

Rebecca  

[Alaa Eldeen]

Hi Rebecca,

Based on my little knowledge, both Kappa and AUC are  weighting omission and commission errors equally, and this reliable if we use presence/absence model and realized model. but in case of presence/pseudo absence  and potential model where the omission errors weight more strongly than commission errors, Kappa and AUC are not reliable. 
I would suggest you to go through these publications:

Anderson, R.P., Lew, D. & Peterson, A.T. (2003) Evaluating predictive models of species’ distributions: criteria for selecting optimal models. Ecological Modelling , 162 , 211–232.

Lobo, J.M., A. Jiménez-Valverde, and R. Real, (2008), AUC: a misleading measure of the performance of predictive distribution models. Global Ecology and Biogeography, 17, 145-151.

Also see the attached file it contains a paper that might add to your knowledge.

I wish it reveals your confusion.

I would call all modeler in this group to response if they have another interpretation.

Cheers

--

You received this message because you are subscribed to the Google Groups "Maxent" group.

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/BuOrXUjosqQJ.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

[Alaa Eldeen]

Dear Phaius,
Sorry to being late in reply to your message I have limited access to internet.
Unoriginality you can't use TSS with replicated models, exactly like AIC.
So I suggest you to do is just to do different models without replication and use TSS to evaluate them, and then re-run the best fit model with replication.
I would be grateful if any one in this group can provide use with another way to solve this dilemma.

All the best

On Fri, Jan 25, 2013 at 8:57 AM, <pha...@gmail.com> wrote:

Dear Alaa,

Thanks for your TSS excel sheet. It is very useful to me.

When I run MaxEnt by setting 10 replicates, I got 10 backgroundPredictions files and 10 samplePredictions files.

How do I select the cells of “logistic” and “Logistic prediction” columns from 10 backgroundPredictions files and 10 samplePredictions files to paste in TSS excel sheet.

Thanks!

 

Best regards,

Phaius

alaaeldeen80於 2012年9月30日星期日UTC+8上午2時32分11秒寫道：

Visit this group at http://groups.google.com/group/maxent?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

[pha...@gmail.com]

Dear Alaa,

Thanks for your reply.

I'll think about your suggestion and to try that.

Many thanks.

 Phaius

alaaeldeen80於 2013年1月27日星期日UTC+8上午2時49分03秒寫道：

[Alfred Chitiki]

Dear Alaa,

Although this discussion is quite old, it is still valuable to some of us.  I would like to get some clarification on the use of TSS and ‘leave-one-out’ procedure as outline below. I have already posted it to maxent group but I would like to specifically share with you.

On using TSS

Is TSS appropriate for smaller sample sizes (<25) or only for large samples (>25)?

While setting Maxent to get the 10 percentile threshold value to be used in TSS, do I need to set the Crossvalidate, Bootstrap or Subsample function as a Replicated run type? Which one is more appropriate and with how many Replicates? Do I need to specify the Random test percentage?

On using ‘leave-one-out’procedure

For smaller sample sizes (<25) it is advocated to use the method by Pearson et al. (2007) called (‘leave-one-out’), in which “model performance is assessed based on its ability to predict the single locality that is excluded from the ‘training’ dataset”. I am a bit confused about this statement. If we have 11 observations, and so we run the model 11 times: does it mean that we run the model with 11, 10, 9, 8, 7, 6, 5, 4, 3, 2 and 1 samples? How does it work?

 

What about the final prediction map of a particular species (How do I use the results of all 11 different runs to a final map)?

 

Do I set values for the Random test percentage and Replicates? How much?

 

On using 10 percentile training presence logistic threshold to classify suitable and unsuitable habitats we use the value obtained from maxentResults.csv file to set a threshold between suitable and unsuitable habitats in Arcmap.  How do I use the ‘lowest presence threshold’ (LPT) and a fixed threshold?

 

Your responses are highly appreciated

 

Alfred

[Alaa Eldeen]

Hello Alfred,

Sorry for being late in reply to your message, I have limited access to internet nowadays.

Up to my knowledge, TSS test is working well with small sample sized models.

Maxent is using Crossvalidation option as a default you don't need to set it.

Regarding the No. of replications, it is recommended to run 10 replicates for your model, but you have to bear in mind that once you are using more than one replicates you will not be able to assess your model based on TSS, Kappa, AIC, or null model. The only way you can use is AUC curve. Otherwise you can firstly run your model without replication and then assess it using TSS, Kappa, AIC, or null model and then rerun the best fit model with replication.

Bear in mind that if you use model with replication you don't need to set the replication run type and leave it.

Regarding the Random test percentage, I would say yes you need to specify it at 25%

Unfortunately, I didn't work with leave-one-out’procedure before so I don't have any idea about it.

What about the final prediction map of a particular species (How do I use the results of all 11 different runs to a final map)?

If you mean here 11 replications, in this case you will find in maxent output directory a raster map named "species_average" this map that you will use, where it is an average of all 11 replications you ran for your model.

How do I use the ‘lowest presence threshold’ (LPT) and a fixed threshold?

In Maxent LPT is called Minimum training presence, and you will find its value in the cell above 10 percentile training presence's cell in the same table in Maxent HTML file.

Regarding fixed threshold I don't recommend it based on some publications

Liu C, Berry, PM, Dawson TP, Pearson RG. 2005. Selecting thresholds of occurrence in the prediction of species distributions. Ecography 28: 385 -393.

At the end, I would suggest you to use AIC threshold dependent evaluation, and you can use ENMtool to get AIC score. a lot of studies pointed out that AIC is being a good criterion for selecting the best fitted model and you can use AICc if your sample size less than 30 points.

Best of luck

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at http://groups.google.com/group/maxent.

For more options, visit https://groups.google.com/groups/opt_out.

[antonio rodriguez]

WOW, This conversation was long time ago but.... There are some thing that I dont undersand:


1) Can you use TSS on Maxent models using replicates? (I ask this because while I was reading this text sometimes I undestood yes and sometimes I understood no.

2) And more important: In this excel, you use specificity as in Maxent´s AUC, without absence data for testing the model. Are there any paper which use this aproach (Background values instead of true absences)? I didnt find it.

Thanks

[kaky]

Dear alaaeldeen

I am Kaky

please when I run maxent with different feature for types of threshold in maxent every time one environmental variable affect the model more other one (for example BIO3) but when I run maxent with Hinge feature for each threshold there are one variable most affected than other some time BIO 3 some time altitude some time BIO4, can you help me to find method to find correlation between them after that delete some of these variable because i used about 11 variable.

regards 

On Wednesday, 26 September 2012 14:42:42 UTC+1, alaaeldeen80 wrote:

hi,
I would suggest you to use True Skill Statistic ,TSS, instead Kapa, where Kapa usually give very low value with presence data only.
I have used it in my research and it gave me a reliable output.
Kindly go through this publication it will help you.
Allouche, O., Tsoar, A. & Kadmon, R. (2006) Assessing the accuracy of species distribution
models: prevalence, kappa and the true skill statistic (TSS). J. Appl. Ecol, 43, 1223–1232.

In case you stuck in it, I have developed an excel sheet for TSS I can forward it to you, all what you need is to input your data and it will give you the TSS value.

Cheers

On Tue, Sep 25, 2012 at 8:21 PM, MyHanh Tu <myh...@gmail.com> wrote:

Thanks for the reply!. I'm now looking at Cohen's kappa coefficient in addition to the AUC, as many studies have done. However, I'm getting very low kappa values.. Around 0,08!! I can't make sense of it, as my AUC values are quite high. Any idea?

kl. 10:09:04 UTC+2 fredag 21. september 2012 skrev MyHanh Tu følgende:

Hey guys!
I've made a maxent-model to predict the distribution of Spanish sparrows in Spain (I have around 900 locationpoints) with both landcover and climate data. I'm now at the stage where I have to evaluate my model. I'm pretty new at both statistics and maxent, but I've done some research and I hope you can help me verify my method. I've read in three articles where they have used the ROC and AUC to evaluate the model's accuracy and prediction power, respectively. Is this enough? And is the ROC and AUC i find in the output file all I need (which I feel sounds too easy to be true, but I would LOVE it if it was this easy), or do I need to use a statistical program to further test it?

I'm a masterstudent and kind of running out of time, so I would be very very very happy and appreciative of your help. I'll be so happy, I'll include you in my "acknowledgements". Seriously.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/Q4jmENhDQFUJ.

To post to this group, send email to max...@googlegroups.com.

To unsubscribe from this group, send email to maxent+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

[Hira Fatima]

Dear Alaa..
Thankyou for TSS sheet, it is very useful and all other attachments. you suggested, to run model checking TSS for individual models and then rerun the best fit model with replicates, and you said TSS is not for models with replicates, what i did is i calculated TSS values for each replicate based on their individual threshold values and then took the average of all the 10 TSS values.. Is it a wrong way.. i just came out with this suggestion on my own, i dont know if i am right. please help.!!
Cheers

[Nega Tassie Abate]

Dear Alaaa..

Thank you so much for your nice explanations about MaxEnt.I am using maxent to do species distribution modelling. Would you please send the excel sheet to (negat...@yahoo.com)  to calculate true skill statistics?

With best regards

Nega

[许仲林]

Dear Nega

You can find the TSS sheet in the message of alaaeldeen80 send to maxent group,  here is the URL

https://groups.google.com/forum/#!topic/maxent/eCgJ_0vdOb0

--

You received this message because you are subscribed to the Google Groups "Maxent" group.

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at http://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--

Best regards!

Zhonglin Xu

College of Resource & Environmental Science

Xinjiang University

Shengli Road 14,Urumqi, 830046

China

E-mail:gal...@gmail.com

Cell:+8618099112497

[Alaa Eldeen]

Hi antonio,

I'm very sorry to being late in reply, it is out of my hand.

No you can not use TSS with replications.

It is much better to calculate TSS for different models without replications and then choose the best model and rerun it again with replication.

Please accept my excuse for delaying

[Alaa Eldeen]

Hi Kaky,
I'm very sorry to being late in reply, i had technical problem in my Gmail.
I would suggest you to use VIF through Regression analysis, or you can use ENMtool and request correlation function form it.

Hope it helps

Please accept my excuse for delaying

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at http://groups.google.com/group/maxent.

For more options, visit https://groups.google.com/d/optout.

[Alaa Eldeen]

Hi Hira Fatima,

I'm sorry for late reply but I had technical problem with my email.

No you can not get average value for TSS it is not statistical correct.

If you need to use the average, I suggest you to use a null model but not AIC or TSS.

Best of luck

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at http://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Vinicius Tonetti]

Hi Alaa, 

First, I would like to thank you a lot for post the sheet and teach us how to calculate TSS for maxent models. Thanks for all other explanations too.

I would like to ask one more thing. Do you have some references for the choice of this value of threshold (10 percentile) ? I'm gonna use this value, but would be nice if I quoting any reference to justify it.

Thanks again,
vinicius

[Alaa Eldeen]

Dear Vinicius,

Sorry to being late in reply to you.

here is some of the papers
http://link.springer.com/article/10.1007%2Fs10530-010-9798-4

http://www.plosone.org/article/info%3Adoi%2F10.1371%2Fjournal.pone.0010360

http://journals.cambridge.org/action/displayAbstract?fromPage=online&aid=4867232&fileId=S0959270908007703

Hopefully it help

Cheers

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at http://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[morgann...@gmail.com]

Hi Alaa,

 

I would like to use the TSS spreadsheet to test model performance but I cannot get any of the links to work to download it. Anychance you could email it to me? morg...@hotmail.com

 

Thank you for all the great information!!!

Morganne

[Vinicius Tonetti]

Wow! I will read the papers carefully! 

Many thanks for the help, Alaa.

Best regards!

[Angélica]

Hello Alaa,

Could you please facilitate me the paper of Liu et al 2005.

Thanks a lot for your help through this conversation!!

[Alaa Eldeen]

Hi Angélica,

You will find the Liu paper in the attached file.

Is it what you looking for or I misunderstood you?

Cheers

For more options, visit https://groups.google.com/d/optout.

--

Alaa Eldin Soultan,  B.Sc, MSc.
Wildlife Conservationist,
St Katherine Protectorate,
National parks of Egypt,
Nature Conservation Sector,
Egyptian Environmental Affairs Agency, EEAA
Tel-Fax: + 2069- 3470033
Mobil:+201225773922

Ps:  It takes 24 trees to produce 1 ton of office paper! Think… is it really necessary to print this email?

[Sony Rk]

Dear All,

Thanks for this wonderful discussion thread. Ala, thanks for sharing the tss calculator. But I am wondering if anybody know how to do it this in R. I am trying with a script provided by Gustav. But it is not giving any values other than the csv files with columns

[Francisco Rodriguez Sanchez]

Hi,

There is a script here https://github.com/PNHP/SDM_maxent/blob/master/TSS.R. But I haven't tried it

Good luck

Paco

-- 
Dr Francisco Rodriguez-Sanchez
Integrative Ecology Group
Estacion Biologica de Doñana - CSIC
Avda. Americo Vespucio s/n
41092 Sevilla (Spain)
http://bit.ly/frod_san

[Alaa Eldeen]

Hi Sony,

What other values you need?

Alaaeldin Soultan
PhD researcher
Max Planck Institute for Ornithology
Am Obstberg 1
78315 Radolfzell, Germany
Tel: +49 (0)7732 1501 911
Mob.+4915757474779

[Poonam Tripathi]

Dear Allaa Eldin,

I am following this forum and I saw your excel file for TSS and Kappa. I am modeling my species separately with the GLM and GAM models where I want to evaluate model performance other than AIC. I am working on the count data means grid based cumulative number of species data (not binary data) therefore is it possible to use TSS for that??   As I am confused for the threshold limit how to judge then?. kindly guide.

2012/9/26 Alaa Eldeen <alaael...@gmail.com>

--

You received this message because you are subscribed to the Google Groups "Maxent" group.

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/Q4jmENhDQFUJ.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

--

Alaa Eldin Soultan,  B.Sc, MSc.
Wildlife Conservationist,
St Katherine Protectorate,
National parks of Egypt,
Nature Conservation Sector,
Egyptian Environmental Affairs Agency, EEAA
Tel-Fax: + 2069- 3470033
Mobil:+201225773922

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

--

Dr. Luiz Menini Neto
Centro de Ensino Superior de Juiz de Fora
Juiz de Fora - MG - Brasil

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

[Alaa Eldeen]

Hi Poonam,

If I understood you well, you mean that your output is a continuous scale. So you can normally use the TSS but yes it depend on the confusion matrix so you need a threshold to convert your output to a binary.

Or you can use a logistic function to convert it to binary based on the probability not on the threshold.

Have a look to these papers
Meynard C.N. & Kaplan D.M. 2013. Using virtual species to study species distributions and model performance. Journal of Biogeography 40:1-8

Meynard C.N. & Kaplan D.M. 2011. The effect of a gradual response to the environment on species distribution model performance. Ecography 35:499-509

Cheers

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at https://groups.google.com/group/maxent.

For more options, visit https://groups.google.com/d/optout.

--

Alaaeldin Soultan
PhD researcher
MaxPlanck Institute for Ornithology

Am Obstberg 1
78315 Radolfzell, Germany
Tel: +49 (0)7732 1501 911
Mob.+4915757474779

[Ignacio M.]

Hello everyone.

Excellent and very useful discussion. Has anyone tried the script R "https://github.com/PNHP/SDM_maxent/blob/master/TSS.R"?

I have tested this script and also the Alaaeldin's excel spreadsheet. I selected one of my replicates from the R output, but I did not get the same results for the same replicate in excel.

Someone has compared the results of TSS of these two possible ways (R script and Alaaeldin's excel) to perform this analysis?

Cheers

Ignacio

El martes, 25 de septiembre de 2012, 11:36:25 (UTC-3), Francisco Rodriguez-Sanchez escribió:

Hi,

Unfortunately AUC is not enough in many cases, e.g. have a look at
Phillips & Elith Ecology 2010, or Warren & Seifert Ecol. Appl 2011.
These papers suggest complementary measures (AIC, POC plots) that can be
obtained in R or ENMTools (enmtools.blogspot.com).

But if you don't need to compare different models, your response curves
and maps look sensible and the AUC is also good, you might skip the
above for your master project. Up to you!

Cheers

Paco



El 21/09/2012 09:09, MyHanh Tu escribi�:

> Hey guys!
> I've made a maxent-model to predict the distribution of Spanish sparrows
> in Spain (I have around 900 locationpoints) with both landcover and
> climate data. I'm now at the stage where I have to evaluate my model.
> I'm pretty new at both statistics and maxent, but I've done some
> research and I hope you can help me verify my method. I've read in three
> articles where they have used the ROC and AUC to evaluate the model's
> accuracy and prediction power, respectively. Is this enough? And is the
> ROC and AUC i find in the output file all I need (which I feel sounds
> too easy to be true, but I would LOVE it if it was this easy), or do I
> need to use a statistical program to further test it?
>
> I'm a masterstudent and kind of running out of time, so I would be very
> very very happy and appreciative of your help. I'll be so happy, I'll
> include you in my "acknowledgements". Seriously.
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google
> Groups "Maxent" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/maxent/-/ow9Lnq7RvhEJ.

> To post to this group, send email to max...@googlegroups.com.
> To unsubscribe from this group, send email to
> maxent+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/maxent?hl=en.

--

Dr Francisco Rodriguez-Sanchez
Forest Ecology and Conservation Group
Department of Plant Sciences
University of Cambridge
Downing Street
Cambridge CB2 3EA
United Kingdom
http://sites.google.com/site/rodriguezsanchezf

[Alaa Eldeen]

HI Ignacio,

Thanks for sharing your experience with TSS.

I don't know exactly from where we you get this discordance.

I did review for the R code and It is the same as the excel file.

By the way I have also created R code based on my excel file and I guess it is the same for what is on github.

I have attached it and you can test my code with excel and with the one on github and keep me updated.

I hope it helps

 

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Ignacio M.]

Hi Alaa,

Thank you very much for your help! I tested the code and I found my problem. In this function "lapply(filesname, function(name){....", I did not change the threshold which I previously selected.

This column change in each run and species. So, to ensure that this don't happen again, I added this lines before the function:

max_res <- read.csv("maxentResults.csv",sep=",") # Read the summary results
names(max_res) # Set which threshold will be chosen and change this inside the function

After this, I run successfully the TSS code on my species' replicates.

Maybe this could help other stuck people!!

PD. Alaa, I saw a previous answer of yours. Where did you get this:

Dear MyHanh,

First of all, 0.5 value for TSS is not bad, as you might know that the values of TSS lays between 0 and 1, where 
0 - 0.4 = poor
0.4 - 0.5 = fair
0.5 - 0.7 = good
0.7 -0.85 = V. good
0.85 - 0.9 = excellent
0.9 -1 = perfect

Which paper stated this? Maybe I can cite a published paper which established this rank.

Thanks again.

Cheers fom Patagonia

[Alaa Eldeen]

HI Ignacio,

I'm sure that will make it easier for all of us.
Regarding the scale of TSS ,since bath Kappa and TSS are based on confusion matrix,  and TSS is just a modification for Kappa where there is no absence points, so it is logic to use one scale for all of them.
So, using Kappa's scale in case of TSS is acceptable.

Have a look to the table in page 11 in this papre
http://www.sciencedirect.com/science/article/pii/030438009290003W

If you can't access it please let me know

Cheers

[Seshadri k.s]

Dear Alaa, 

Thank you for this excel sheet!

I have a rather basic question. I am running maxent on a species of frog in India and I have run 15 replicates. If I want to calculate TSS, do I use the samplePredictions and backgroundPredictions for each of the runs? I do not see a samplePredictions and backgroundPredictions file for the cumulative runs. 

Thank you.

Seshadri

2012/9/26 Alaa Eldeen <alaael...@gmail.com>

Hey guys!
I've made a maxent-model to predict the distribution of Spanish sparrows in Spain (I have around 900 locationpoints) with both landcover and climate data. I'm now at the stage where I have to evaluate my model. I'm pretty new at both statistics and maxent, but I've done some research and I hope you can help me verify my method. I've read in three articles where they have used the ROC and AUC to evaluate the model's accuracy and prediction power, respectively. Is this enough? And is the ROC and AUC i find in the output file all I need (which I feel sounds too easy to be true, but I would LOVE it if it was this easy), or do I need to use a statistical program to further test it?

I'm a masterstudent and kind of running out of time, so I would be very very very happy and appreciative of your help. I'll be so happy, I'll include you in my "acknowledgements". Seriously.

Thanks!

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To view this discussion on the web visit https://groups.google.com/d/msg/maxent/-/Q4jmENhDQFUJ.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

--

Alaa Eldin Soultan,  B.Sc, MSc.
Wildlife Conservationist,
St Katherine Protectorate,
National parks of Egypt,
Nature Conservation Sector,
Egyptian Environmental Affairs Agency, EEAA
Tel-Fax: + 2069- 3470033
Mobil:+201225773922

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

--

Dr. Luiz Menini Neto
Centro de Ensino Superior de Juiz de Fora
Juiz de Fora - MG - Brasil

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To post to this group, send email to max...@googlegroups.com.
To unsubscribe from this group, send email to maxent+un...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/maxent?hl=en.

--

[Alaaeldin Soultan]

Hi Seshadri,

You need to use the samplePredictions and backgroundPredictions for each of the runs and calculate the mean for it.

Happy new year

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--

Alaaeldin Soultan
PhD researcher
MaxPlanck Institute for Ornithology

Am Obstberg 1
78315 Radolfzell, Germany

Tel: +49 (0)7732 1501 35
Mob.+4915757474779

[Jeronimo Silva]

Hi all,

This discussion is incredible! Alaa, special thanks to you for providing the excel sheet, the R code, and lots of resources to back everything up. I used your R code to analyze my model and it all worked perfectly and matched the excel sheet (excel results were off by 0.02 sometimes but I assume small discrepancies are ok?). I know someone has asked this before, but I wanted to double check that we CAN use TSS to evaluate models with replicates. I used MaxEnt to create a model for the hellbender salamander in Tennessee, USA. I ran about 9 models and picked the best one based on the AUC value. I am using pseudo-absences, so during analysis I became aware of the issues with using only AUC values to evaluate model performance. I decided to use TSS instead, but i ran my model with 10 replicates. So I calculated the TSS for each replicate and then took the average which is the value I plan on using on my manuscript. Is that correct? Has anyone here done something similar? If it is NOT correct, why not and what would be a good alternative?

I appreciate any input/suggestions!

Best,

Jeronimo
------------------------------------------------------------------

Jeronimo Gomes Da Silva Neto
M.S. candidate
Dept. of Agricultural and Environmental Sciences
Tennessee State University
Nashville, TN

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Alaaeldin Soultan]

Hi Jeronimo,

Glad to hear that the R code and excel file are helpful.

Yes you are right, you can get the average value for many replica and using it.

Yes, earlier I thought that TSS can't be average, but I was wrong.

I hope it helps.

Good luck in your MS.

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[BALIMA Larba Hubert]

Hello guys
Happy to join you in SDM
I am working on the modelling for my PhD and would appreciate to doing it in R instead of Maxent and Arc_GIS.
Does somebody have any idea about that?

Regards

Le mardi 25 septembre 2012 14:36:25 UTC, Francisco Rodriguez-Sanchez a écrit :

Hi,

Unfortunately AUC is not enough in many cases, e.g. have a look at
Phillips & Elith Ecology 2010, or Warren & Seifert Ecol. Appl 2011.
These papers suggest complementary measures (AIC, POC plots) that can be
obtained in R or ENMTools (enmtools.blogspot.com).

But if you don't need to compare different models, your response curves
and maps look sensible and the AUC is also good, you might skip the
above for your master project. Up to you!

Cheers

Paco



El 21/09/2012 09:09, MyHanh Tu escribi�:

> Hey guys!
> I've made a maxent-model to predict the distribution of Spanish sparrows
> in Spain (I have around 900 locationpoints) with both landcover and
> climate data. I'm now at the stage where I have to evaluate my model.
> I'm pretty new at both statistics and maxent, but I've done some
> research and I hope you can help me verify my method. I've read in three
> articles where they have used the ROC and AUC to evaluate the model's
> accuracy and prediction power, respectively. Is this enough? And is the
> ROC and AUC i find in the output file all I need (which I feel sounds
> too easy to be true, but I would LOVE it if it was this easy), or do I
> need to use a statistical program to further test it?
>
> I'm a masterstudent and kind of running out of time, so I would be very
> very very happy and appreciative of your help. I'll be so happy, I'll
> include you in my "acknowledgements". Seriously.
>
> Thanks!
>
> --
> You received this message because you are subscribed to the Google
> Groups "Maxent" group.
> To view this discussion on the web visit

> https://groups.google.com/d/msg/maxent/-/ow9Lnq7RvhEJ.

> To post to this group, send email to max...@googlegroups.com.
> To unsubscribe from this group, send email to
> maxent+un...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/maxent?hl=en.

--

[Alaaeldin Soultan]

U can use Biomod2  or Dismo, and mony other R packages are available

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--

Alaaeldin Soultan
PhD candidate

[BALIMA Larba Hubert]

Did you already try it?

Thanks

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/eCgJ_0vdOb0/unsubscribe.
To unsubscribe from this group and all its topics, send an email to maxent+unsubscribe@googlegroups.com.

[Alaaeldin Soultan]

Yes i use Biomod2

[Areej Jaradat]

Thank you all for this beneficial discussion, special thanks to AlaaEldeen. 

 

This method actually provoked a question in my head. If my aim is to present the original continuous model produced by maxent (0-1) along another commonly used habitat suitability map of 4 classes (highly suitable, moderate, least, unsuitable) for example - that would be developed in any suitable program like ArcGIS depending on the data from the original maxent ascii file of course -. So the question here, does it make since to use TSS to test the model performance knowing that it requires a threshold, which means the TSS is evaluating a binary model (above threshold suitable, below threshold not suitable) from maxent and I'm presenting the original continuous model (0-1) that I didn't apply that threshold to.  Even the habitat suitability map of 4 classes is developed using the original data file of the continuous model

So Is this a valid way to test the performance of a continuous model? 

Yes i use Biomod2

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.

Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--

Alaaeldin Soultan
PhD candidate
MaxPlanck Institute for Ornithology
Am Obstberg 1
78315 Radolfzell, Germany
Tel: +49 (0)7732 1501 35
Mob.+4915757474779

--
You received this message because you are subscribed to a topic in the Google Groups "Maxent" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/maxent/eCgJ_0vdOb0/unsubscribe.

To unsubscribe from this group and all its topics, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Sk Mujibar Rahaman]

Thank you all for this beneficial discussion, 

Thanks to AlaaEldeen

I have a question regarding the TSS. I have run a maxent model with 10 number of replicates. I use the average output as final result and I have to calculate TSS for the average output result. After that I want to evaluate the output of maxent using TSS. On the above mention method and providing excel file I need background Predictions and sample Predictions for calculation TSS. But for the 10 replicates result have only this two (background Predictions and sample Predictions ) CSV file. For average output result have not any  background Predictions and sample Predictions file. Then how to I calculate the TSS for average output result?