Test AUC standard deviation

[Bothrops]

Dear colleagues
Which is the difference between a model with a AUC Test value of 0.99
with standard deviation of -1.000 and a model with a AUC Test value of
0.99 but with standard deviation of 0.001 (calculated as in DeLong,
DeLong & Clarke-Pearson 1988, equation 2)?. What this difference in
standard deviation means to the model?. What is the biology meaning of
this standard deviation differences?
Thanks
Nicolas

[Paco Rodríguez]

Hi Nicolas

 

I would say that your standard deviation of -1.000 must be wrong. To my knowledge, standard deviations must always be positive, and given that AUC ranges between 0 and 1, much lower values than 1 should be obtained for the standard deviation. Perhaps try repeating the analysis to find out what could have failed.

 

Standard deviation is a measure of the uncertainty of the parameter. If you obtain a low standard deviation you can have higher confidence on your estimate of AUC. In your case, a standard deviation of 0.001 means that the actual test AUC value must be very close to 0.99.

 

Hope this helps,

 

Paco

 

 

2008/5/5 Bothrops <nurb...@gmail.com>:

[Bothrops]

Dear Paco
Thanks for your response. I keep having some SD values of -1. To date
I have run MaxEnt models for 158 reptiles and 285 amphibians in Mexico
and of those 443 species I have 76 models with a SD of -1 but without
an apparent relation with AUC values (those 76 models ranged from a
Test AUC of 1 to 0.6 with a mean of 0.98) or number of registers
(those 76 models ranged from a number of registers of 6 to 20, with a
mean of 7 registers)
It could be due to some problems with MaxEnt?.
I have run those 443 models with 25% for test and 75% for training. Do
you think if I use just 10% of registers to test the model the Test
AUC SD could be fixed?
Thanks
Best
Nicolas

On May 7, 3:26 am, "Paco Rodríguez" <frodrigu...@gmail.com> wrote:
> Hi Nicolas
>
> I would say that your standard deviation of -1.000 must be wrong. To my
> knowledge, standard deviations must always be positive, and given that AUC
> ranges between 0 and 1, much lower values than 1 should be obtained for the
> standard deviation. Perhaps try repeating the analysis to find out what
> could have failed.
>
> Standard deviation is a measure of the uncertainty of the parameter. If
> you obtain a low standard deviation you can have higher confidence on your
> estimate of AUC. In your case, a standard deviation of 0.001 means that the
> actual test AUC value must be very close to 0.99.
>
> Hope this helps,
>
> Paco
>

> 2008/5/5 Bothrops <nurbin...@gmail.com>:

[Steven Phillips]

Dear Nicolas,

The value of -1 for the standard deviation of the AUC indicates that
the standard deviation cannot be calculated. I expect that this is
happening because you have only 6 test presence records for some
species, and using 25% of the records for testing gives you only a
single test presence record to use in the AUC calculation. The
standard deviation cannot be determined from a single test presence
record.

I've made a change to the code so that in the next release an
explanation of the -1 value will be given in the .html output if
there's only 1 test presence.

-- Steven

Date: Wed, 7 May 2008 09:37:03 -0700 (PDT)
From: Bothrops <nurb...@gmail.com>

[Emma Marotte]

I am also having this issue of getting -1 SD value in my maxentResults output csv file. I am running R to model habitat suitability for blue whales using 222 sighting records. I am doing 100 replicates for 8 different sampling scenarios, and for the first 50-60 replicates or so I have regular AUC and AUC SD values. But the rest have an SD of -1... Wondering what is wrong and how I can fix this. Any additional help on this topic would be really appreciated, I am very new to MaxEnt and still trying to figure things out as I go!