TSS - What is considered a "good" score?

[Anon]

Hello,

A couple questions regarding the True Skill Statistic.  

What would a "good" score be considered when using the TSS to calculate statistics in MaxEnt? Higher is better, obviously, and anything lower than 0 is worse than random.  My sensitivity scores are coming out decently high, but my specificity scores are basically in the trenches, hovering not much higher than 0 or .1 in many cases.  After calculating TSS, I am left with values at .125 to .4. 

I recognize there may be guidelines for good Kappa scores (I think I've seen one listed on this forum, but couldn't find any literature to support it), but have yet to find anything for TSS.  Would they be the same guidelines and, if so, do these guidelines change when using presence/absence data vs. presence only data?  I would think presence/absence data should receive higher Kappa/TSS scores to be considered "good".

Thanks for any thoughts you may have.

Anon

[Anon]

I have one more question.  I have seen a number of studies refer to the their True Skill Statistic scores as a percent, rather than a unitless value ranging from -1 to 1.  Can someone explain how they might be getting percentages when calculating the TSS or where this convention arose?

Thanks again.

[Alaa Eldeen]

Hi Anon,

regarding what is consider good for TSS, I normally use the same scale applied for KAPPA since they are rely on the same approach.

So I would say the score is >= 0.4 is what you consider.

And I do cite this paper for that

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.

--
You received this message because you are subscribed to the Google Groups "Maxent" group.
To unsubscribe from this group and stop receiving emails from it, send an email to maxent+un...@googlegroups.com.
To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

--

Alaaeldin Soultan
PhD researcher
MaxPlanck Institute for Ornithology
Am Obstberg 1
78315 Radolfzell, Germany
Tel: +49 (0)7732 1501 911
Mob.+4915757474779


Ps:  It takes 24 trees to produce 1 ton of office paper! Think… is it really necessary to print this email?

[Alaaeldin Soultan]

Hi Arta,

You need to calculate the TSS for each run separately and use the average.

Cheers

On Tue, Jan 3, 2017 at 10:07 PM, Arta <fghgo...@gmail.com> wrote:

Dear Alaadin

I have run Maxent models with 10 sub-sample replicates. Every replicate produces a specific background file and sample file. which one must be used in TSS (excel file that you upload in another post)? How can I have an average of them?

Many thanks

--Arta

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Arta]

Dear Alaaedin

I have another simple question: is it difference to run 10 models separately or set replicate as 10? 

In my case, when i run 10 model separately the AUCs are high (>0.9) but when i run a model with 10 replicate the result is different!

Many thanks for your help.

[Alaaeldin Soultan]

HI Arta,

I would say both procedures should give similar AUC but not exact.

In your case how much is the differences between the AUC in these two procedures?

Cheers

To unsubscribe from this group and stop receiving emails from it, send an email to maxent+unsubscribe@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Sean Isidore]

Dear Arta,

Not sure if this is still relevant, but I hope it may help you for further studies. The reason we want to run replicates within a single MaxEnt run is due to the way MaxEnt sets aside training and testing data. By running the models separately, the training and test data is exactly the same for all 10 models and only the pseudo-absence points are whats changing. For setting replicates as 10 however, depending on if its bootstrap or cross-validation, the training and test data is cycled, actually giving you 10 "proper" replicates. Which is why your AUCs for the 10 model separately is more consistent with fewer variations, simply because the same training and test points are being used. This means that the average AUC for the 10 model separately is not a proper representation of your actual dataset.

Hope this helps.

Cheers!

[Carlos Eduardo Carvalho]

hello i have a question, how to improve the models then?

Em quinta-feira, 28 de janeiro de 2016 13:43:56 UTC-3, alaaeldeen80 escreveu:

Hi Anon,

regarding what is consider good for TSS, I normally use the same scale applied for KAPPA since they are rely on the same approach.

So I would say the score is >= 0.4 is what you consider.

And I do cite this paper for that

Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics 1977;33:159-74.

On Fri, Jan 8, 2016 at 6:07 PM, Anon <kristoph...@gmail.com> wrote:

I have one more question.  I have seen a number of studies refer to the their True Skill Statistic scores as a percent, rather than a unitless value ranging from -1 to 1.  Can someone explain how they might be getting percentages when calculating the TSS or where this convention arose?

Thanks again.

On Friday, January 8, 2016 at 11:01:35 AM UTC-6, Anon wrote:

Hello,

A couple questions regarding the True Skill Statistic.  

What would a "good" score be considered when using the TSS to calculate statistics in MaxEnt? Higher is better, obviously, and anything lower than 0 is worse than random.  My sensitivity scores are coming out decently high, but my specificity scores are basically in the trenches, hovering not much higher than 0 or .1 in many cases.  After calculating TSS, I am left with values at .125 to .4. 

I recognize there may be guidelines for good Kappa scores (I think I've seen one listed on this forum, but couldn't find any literature to support it), but have yet to find anything for TSS.  Would they be the same guidelines and, if so, do these guidelines change when using presence/absence data vs. presence only data?  I would think presence/absence data should receive higher Kappa/TSS scores to be considered "good".

Thanks for any thoughts you may have.

Anon

--
You received this message because you are subscribed to the Google Groups "Maxent" group.

To unsubscribe from this group and stop receiving emails from it, send an email to max...@googlegroups.com.

To post to this group, send email to max...@googlegroups.com.
Visit this group at https://groups.google.com/group/maxent.
For more options, visit https://groups.google.com/d/optout.

[Jamie M. Kass]

Models can be improved by:
1) processing the occurrence data to remove clustering or biases (i.e., via spatial thinning)
2) changing the environmental predictor variables (i.e., using ones you as the ecologist hypothesize relates most to the species' range)
3) using an appropriate study extent (i.e., limit the area sampled for background points by the area accessible to the species)
4) tuning the model settings (i.e., explore different levels of complexity and select an optimal model)

There are probably more, but I'd start there.

I also wanted to ask this group if they've read the paper Leroy et al. 2016 or others criticizing the use of TSS (and by extension kappa) for presence-background models:
https://doi.org/10.1111/jbi.13402

Jamie Kass
Postdoctoral Scholar
Okinawa Institute of Science and Technology

[Simon Dedman]

See also https://natureconservation.pensoft.net/article/33918/ for proposed alternatives to TSS.

Cheers

Simon