Biased and unbiased std dev
Vinay
I was wondering if someone can explain in simple terms why std dev s=
sqrt(sum_(x_i-\bar{x})/n) is biased. I understand that 's' should be
invalid when n=1 and hence they divide by (n-1). But is it just to
avoid this case? Or is there more intersting fact for that.
Why does dividing by n always give biased estimates? Is this because
there are n-1 intervals between x_1 to x_n?
Thanks,
Vinay
Reef Fish
The notion of "unbiasedness" is based on the definition:
A statistic T is an "unbiased" estimate for a parameter Theta, if
E( T ) = Theta.
Where E is the mathematical expectation of the statistic T.
That's ALL.
E( s^2) = sigma^2, hence an unbiased estimate for sigma^2.
E( s ) is NOT equal to sigma, hence a biased estimate of sigma,
whether you use (n-1), n, (n+1) as the denominator of s^2.
-- Bob.
Eric Bohlman
@g43g2000cwa.googlegroups.com:
As Bob has pointed out, it's only the variance that you can get an
unbiased estimate of by dividing by (n-1); taking the square root biases
the result.
The reason for the bias is that when you take the average sum of squares
around the *sample* mean rather than the population mean, you end up
subtracting out the variance of the sample mean itself (which will be
sigma^2/N, where sigma^2 is the population variance) and thus wind up
with a value that's too small by a factor of (n-1)/n. So you multiply
the result by n/(n-1) and come up with the standard formula.
A more intuitive explanation is that the mean of a set of data is, by
definition, the value that minimizes the sum of squared deviations around
it. For a sample, the sample mean will have a smaller sum of squared
deviations around it than any other value. But if you're trying to
estimate the *population* variance, you're really interested in the
deviations around the *population* mean. Since the population mean will
not in general be *exactly* equal to the sample mean, the sum of squared
deviations around it will be greater than the sum of squared deviations
around the sample mean, and therefore the latter will be biased low.
Reef Fish
Eric Bohlman wrote:
> "Vinay" <vink...@gmail.com> wrote in news:1121288120.240622.52330
> @g43g2000cwa.googlegroups.com:
>
> > I was wondering if someone can explain in simple terms why std dev s=
> > sqrt(sum_(x_i-\bar{x})/n) is biased. I understand that 's' should be
> > invalid when n=1 and hence they divide by (n-1). But is it just to
> > avoid this case? Or is there more intersting fact for that.
> >
> > Why does dividing by n always give biased estimates? Is this because
> > there are n-1 intervals between x_1 to x_n?
>
> As Bob has pointed out, it's only the variance that you can get an
> unbiased estimate of by dividing by (n-1); taking the square root biases
> the result.
That's only ONE consequence of the definition of an unbiased estimate.
What you said below is neither necessary nor correct:
>
> The reason for the bias is that when you take the average sum of squares
> around the *sample* mean rather than the population mean, you end up
> subtracting out the variance of the sample mean itself (which will be
> sigma^2/N, where sigma^2 is the population variance) and thus wind up
> with a value that's too small by a factor of (n-1)/n. So you multiply
> the result by n/(n-1) and come up with the standard formula.
>
> A more intuitive explanation is that the mean of a set of data is, by
> definition, the value that minimizes the sum of squared deviations around
> it. For a sample, the sample mean will have a smaller sum of squared
> deviations around it than any other value. But if you're trying to
> estimate the *population* variance, you're really interested in the
> deviations around the *population* mean. Since the population mean will
> not in general be *exactly* equal to the sample mean, the sum of squared
> deviations around it will be greater than the sum of squared deviations
> around the sample mean, and therefore the latter will be biased low.
The same subject came up when I FIRST read this group, in February
2005.
I already noticed Richard Ulrich making his characteristic errors and
misinformation, and prompted me to say to him (Ulrich), after he had
made FOUR posts in that thread, of his pointless MISINFORMATION:
RF> You missed my original point -- that the unbiasedness criterion
RF> is plain SILLY, especially when one needs and uses the sample
RF> standard deviation s, and NOT the sample variance s-squared.
The unbiased estimate of the standard deviation was given by
Jack Tomsky, whom I said was the ONLY person in the world I knew
who had used an unbiased estimate of sigma:
JT> I actually do know someone who uses an unbiased estimate of sigma
on
JT> occasion,namely myself. When I need an unbiased estimate of the
p-th
JT> quantile of a normal distribution, particularly for small sample
JT> sizes, I use
JT> xbar + (z_p)*(s*a_N)
JT> where a_N = Gamma(N/2)/[sqrt((N-1)/2)*Gamma((N-1)/2)].
I could only recall an article in an ancient copy of the American
Statistician giving that result.
-- Bob.
Herman Rubin
Since the estimate of s^2 is unbiased, that of s MUST be biased
unless s^2 is evaluated without error.
The estimate of s^2, when divided by n-1, is unbiased because
the mean (the regression on the variable all 1) is subtracted.
More generally, if the residuals of the x's from their regression
on k linearly independed vectors is subtracted, the sum of squares
of the residuals divided by n-k is unbiased, if the regression
model is correct. In the case of normality of the true residuals,
it has the exact same distribution as the sum of squares of
normal random variables with the same variance, the number being
the number of degrees of freedom.
--
This address is for information only. I do not claim that these views
are those of the Statistics Department or of Purdue University.
Herman Rubin, Department of Statistics, Purdue University
hru...@stat.purdue.edu Phone: (765)494-6054 FAX: (765)494-0558
Richard Ulrich
On 13 Jul 2005 16:50:58 -0700, "Reef Fish"
<Large_Nass...@Yahoo.com> wrote:
[snip, about bias and variance estimates, etc.]
>
> The same subject came up when I FIRST read this group, in February
> 2005.
>
> I already noticed Richard Ulrich making his characteristic errors and
> misinformation, and prompted me to say to him (Ulrich), after he had
> made FOUR posts in that thread, of his pointless MISINFORMATION:
[ ... ]
I few weeks ago, I pointed out that I had screwed up a post
in that February thread, inadvertently stating my point
confusingly, and backwards; and did not notice at the time.
I apologized to Bob Ling a few weeks ago for that error,
and for my shrugging defense of my *intended* statement
at the time. (My defense made sense if one read the
statement as intended, not as written.)
If my "characteristic error" is an occasional fumbling of words
that I'm willing to admit and apologize for, then I guess
I'm in pretty good shape. But I don't fumble as much as
Bob mis-reads.
I would say that Bob Ling's first characteristic error is that he
reads badly. He mis-reads posts -- splitting paragraphs, missing
sentences. That has been the origin of several of his
complaints against me, and he's screwed up with other people,
too. Also, he mis-reads literature to find support for his
nearly-unique take on regression with observational data.
He second characteristic error, which took me longer to
winkle out, is that he assumes he knows a lot of stuff
where he is truly clueless -- there have now been multiple
examples in social science content and practice. A couple:
Social science "power analysis" is not identical to
computing a power function (whence, his complaint about
my "alpha" and "beta" errors). Flynn's observations about
IQ are neither simplistic nor obscure, and Bob made an
amazingly naive response on that topic.
Bob has made only a couple of possible admissions of error
to me -- framed (speciously) as accusations of me nitpicking.
Never an apology.
--
Rich Ulrich, wpi...@pitt.edu
http://www.pitt.edu/~wpilib/index.html
Reef Fish
Richard Ulrich wrote:
> Off-thread, comment to Bob's digression.
What digression? I explained the meaning of an unbiased estimator,
and pointed out errors and omissions by others.
>
> On 13 Jul 2005 16:50:58 -0700, "Reef Fish"
> <Large_Nass...@Yahoo.com> wrote:
>
> [snip, about bias and variance estimates, etc.]
That's the statistical SUBSTANCE of my post. Not digression.
You ALWAYS snip all thes statistical substance of what I
posted and then make excuses for your own ERRORS.
> > The same subject came up when I FIRST read this group, in February
> > 2005.
> >
> > I already noticed Richard Ulrich making his characteristic errors and
> > misinformation, and prompted me to say to him (Ulrich), after he had
> > made FOUR posts in that thread, of his pointless MISINFORMATION:
> [ ... ]
and Richard snipped MY correct explanation to the OP, after Richard
had bungled in FOUR posts getting nowhere!
> I would say that Bob Ling's first characteristic error is that he
> reads badly.
This is getting to be Ulrich's stuck record, every time an ERROR
or his was pointed out.
Richard, let the READERS decide, whether yours were ERRORS or NOT.
We don't need your self-serving and pointless excuses for your
blatent errors!
> Social science "power analysis" is not identical to
> computing a power function (whence, his complaint about
> my "alpha" and "beta" errors).
Let readers READ the garbage you wrote, instead of TWISTING
what you posted. This was Richard Ulrich's gibberish when the
OP asks about Type I and Type II errors:
RU> Clinical research designs typically compute the type 2 power
RU> before attempting the research.
Statistics is statistics! Ulrich typically precedes whatever
nonsense he was about to spew with "In epidemiology" or
"Clinical research", ... which are entirely irrelevant to
the question of what Type I and Type II errors are or what
alpha and beta signififies.
RU> This shows the "power" (1 minus the type-2 error)
Type-2 error is not even a NUMBER. Richard badly mangles
the PROBABILITY of Type II error with the definition of
"Type II error", and made a statement about "power" that
is complete nonsense.
Richard's excuse was:
RU> I would say that Bob Ling's first characteristic error
RU> is that he reads badly.
As I had said to you numerous times already:
Learn some STATISTICS before you post your MISINFORMATION and
ERRORS. What I cited above of what you posted is a very
BLATENT example of your Quackery:
RU> Clinical research designs typically compute the type 2 power
RU> before attempting the research. This shows the "power"
RU> (1 minus the type-2 error) for achieving a stated alpha,
This is complete gibberish in responding to someone who asks
about Type-1 and Type-2 errors!
-- Bob.
Richard Ulrich
This is in response to -
On 13 Jul 2005 21:52:31 -0700, "Reef Fish"
<Large_Nass...@Yahoo.com> wrote:
in which I show the grace of my "characteristic error";
and illustrate the apparent ignorance of Bob Ling on
power analysis.
I'm going to back up and give the overview of what
I wrote and why I wrote it, before commenting directly
on Bob's reply - which is included in its entirety.
Bob had made one more swipe at me in his post on
the topic. My intro line stated that I was going to
respond to the digression, which I then quoted.
He cited a previous error of mine as "characteristic".
I replied, more or less, that it was flattering to have him
consider my "characteristic" error to be a flubbing of words
that I had previously pointed out and apologized for, without
prompting, a few weeks ago. That puts me in undeservedly
good shape in the "characteristic error" race.
Then I took the occasion to point out what I consider
Bob Ling's *2* characteristic errors. He misreads, in a
couple of different ways (and I gave examples).
And sometimes he writes with confidence when he
is clueless (plus examples). The new thing here is that
I introduced "cluelessness" as an explanatory principle.
I think readers of the groups will find that categorization
to be useful, without having to wonder how he achieved
such cluelessness.
=== Here is Bob's post, with notes.
>
>
> Richard Ulrich wrote:
> > Off-thread, comment to Bob's digression.
>
> What digression? I explained the meaning of an unbiased estimator,
> and pointed out errors and omissions by others.
>
> >
> > On 13 Jul 2005 16:50:58 -0700, "Reef Fish"
> > <Large_Nass...@Yahoo.com> wrote:
> >
> > [snip, about bias and variance estimates, etc.]
>
> That's the statistical SUBSTANCE of my post. Not digression.
>
> You ALWAYS snip all thes statistical substance of what I
> posted and then make excuses for your own ERRORS.
>
I suppose I might call it ... Good posting habits?
I snip what I'm not going to talk about, especially when
it is totally irrelevant to what I *am* going to talk about.
> > > The same subject came up when I FIRST read this group, in February
> > > 2005.
> > >
> > > I already noticed Richard Ulrich making his characteristic errors and
> > > misinformation, and prompted me to say to him (Ulrich), after he had
> > > made FOUR posts in that thread, of his pointless MISINFORMATION:
> > [ ... ]
>
> and Richard snipped MY correct explanation to the OP, after Richard
> had bungled in FOUR posts getting nowhere!
Here is the crucial post. I quote Bob's comment that
quotes my error, and I give a very short reply -- in which
I obviously *assume* that I posted something opposite
in the first one.
===from earlier thread - RU/RF/RU attributions added.
On 8 Feb 2005 07:27:59 -0800, Large_Nass...@Yahoo.com wrote:
[...]
RU > >
> > Did you know ANYONE who does not choose to combine
> > sigmas, instead of sigma-squareds?
RF >
> As in a pooled standard deivation? In the commonly used form,
> the pooling is done on the VARIANCES, and the pooled standard
> deviation is still biased.
>
> So what's your point about the good sense of your post?
RU >
Oh. Sorry, I assumed you would immediately recognize
that the variances were unbiased. And that unbiasedness
was a main reason to combine variances, instead of
combining the deviations.
===
The negative valence and inverted order "got" me.
I wrote it wrong, and later quickly read it wrong.
Without detailing it, I pointed this out a few weeks.
[We all make some of these posting and reading mistakes
if we post a lot. Some of us apologize for our errors.]
As the reader may see: I replied, "unbiasedness was
a main reason to combine variances, instead of combining
the deviations." Of course, this is contradictory to my first
post; and I was patently unaware of that fact, or I wouldn't
have said that. I surely was not defending the error of
adding standard deviations. Can Bob parse that?
Now, when Bob raises this *against* me, is he flattering
me, or is he still mis-reading the whole exchange?
>
>
> > I would say that Bob Ling's first characteristic error is that he
> > reads badly.
>
> This is getting to be Ulrich's stuck record, every time an ERROR
> or his was pointed out.
>
> Richard, let the READERS decide, whether yours were ERRORS or NOT.
> We don't need your self-serving and pointless excuses for your
> blatent errors!
Oh, I'm happy to let the readers decide. Except when I am
making the generic point, as in that note, I use the term
when I wish to direct the reader's attention to how they
might resolve the contradiction; "Waste less time in trying
to make sense of Bob's critique, more time reviewing the
original."
"Reading badly" might encompass bad memory, and so on.
I've lately concluded that it can't explain some other
problems, which I call "cluelessness."
Coming just before the next lines, here is the intro that
Bob snipped -
"His second characteristic error, which took me longer to
winkle out, is that he assumes he knows a lot of stuff
where he is truly clueless -- there have now been multiple
examples in social science content and practice. A couple:"
> > Social science "power analysis" is not identical to
> > computing a power function (whence, his complaint about
> > my "alpha" and "beta" errors).
>
> Let readers READ the garbage you wrote, instead of TWISTING
> what you posted. This was Richard Ulrich's gibberish when the
> OP asks about Type I and Type II errors:
Well, in my reading, the OP seemed confused. I figured that
he knew the definitions of Type 1 and Type 2 errors; he knew
how Type 1 was used for tests; and he asked about a "test"
for Type 2. My response was to tell him how Type 2 error
was incorporated in clinical research, which I am well familiar
with -- Power analysis. I've been doing power analyses for
research projects for at least 25 years. They have been
required for NIMH projects in clinical research for 10 or
15 years, IIRC. I have several basic posts on power analysis
in my stats-FAQ from 1995, at
http://www.pitt.edu/~wpilib/statfaq/95power.html
Google reports 39,900 hits for < power-analysis Cohen >
and all the first 20 are relevant to Jacob Cohen's book
on the subject, "Statistical power analysis for the
behavioral sciences."
Now, if I muffed my description, I would be happy to see a
better one. However, nowhere in several posts by Bob Ling
on the subject do I detect any indication that he has ever
*heard* of power analysis. This is what inspired me to
conclude that "clueless" had to be another explanation for
some of his posts, in addition to "reading badly" (taken
however loosely one wants to take the term).
>
> RU> Clinical research designs typically compute the type 2 power
> RU> before attempting the research.
>
> Statistics is statistics! Ulrich typically precedes whatever
> nonsense he was about to spew with "In epidemiology" or
> "Clinical research", ... which are entirely irrelevant to
> the question of what Type I and Type II errors are or what
> alpha and beta signififies.
Saying "in clinical research", etc.?
Mainly from posting to these groups, I've learned that
statisticians in other areas don't always know the same
terms or procedures, or use them in the same way.
So, I give the reader some warning.
Apparently "power analysis" is a case of unfamiliarity
for Bob Ling.
>
>
> RU> This shows the "power" (1 minus the type-2 error)
>
> Type-2 error is not even a NUMBER. Richard badly mangles
> the PROBABILITY of Type II error with the definition of
> "Type II error", and made a statement about "power" that
> is complete nonsense.
- Eliding the word "rate" when referring to error seems no more
serious an offense than when Bob Ling this week called a
PP-plot a "test." Just to offer a sense of proportion.
>
> Richard's excuse was:
>
> RU> I would say that Bob Ling's first characteristic error
> RU> is that he reads badly.
>
I don't know where I said that about Bob on Power analysis.
It might apply to a limited extent, since I wrote about Power
analysis and Bob replied about Power functions.
The part on Type 1 and 2 errors was illuminating his *second*
characteristic error, "cluelessness." -- Posting with arrogance
when he knows nothing.
>
> As I had said to you numerous times already:
>
> Learn some STATISTICS before you post your MISINFORMATION and
> ERRORS. What I cited above of what you posted is a very
> BLATENT example of your Quackery:
>
Will you show me a proper definition of a power analysis,
at variance with mine, or do I conclude you are clueless/
that I have rebutted this "quackery"?
Will this example be repeated? I can reflect in reply that
Bob's proud ignorance represents the cluelessness of a
net.kook -- I previously posted the Jargon file definition of that.
>
> RU> Clinical research designs typically compute the type 2 power
> RU> before attempting the research. This shows the "power"
> RU> (1 minus the type-2 error) for achieving a stated alpha,
>
> This is complete gibberish in responding to someone who asks
> about Type-1 and Type-2 errors!
>
> -- Bob.
Google groups reports me posting in 127 threads on
power-analysis in the last 10 years. I've given something
like the above definition a number of times. Bob Ling is
the first reader who thought the definition must be "gibberish."
That word, along with fraud and Quack, says to me he thinks I
made the whole thing up; not that I might have botched it
somehow. What an imagination I must have! And those
posters asking about power analyses, where did they come
from?
Reef Fish
Alpha = P(Type I Error); Beta = P (Type II Error).
Richard Ulrich wrote at length, AGAIN, to excuse himself of his errors:
>
> === Here is Bob's post, with notes.
> >
> > You ALWAYS snip all thes statistical substance of what I
> > posted and then make excuses for your own ERRORS.
> >
>
> I suppose I might call it ... Good posting habits?
What else can anyone expect Richard Ulrich to say but that?
> [...]
>
> The negative valence and inverted order "got" me.
> I wrote it wrong, and later quickly read it wrong.
> Without detailing it, I pointed this out a few weeks.
> [We all make some of these posting and reading mistakes
> if we post a lot. Some of us apologize for our errors.]
Then what's your POINT in the PRESENT post?
>
> As the reader may see: I replied, "unbiasedness was
> a main reason to combine variances, instead of combining
> the deviations." Of course, this is contradictory to my first
> post; and I was patently unaware of that fact, or I wouldn't
> have said that. I surely was not defending the error of
> adding standard deviations. Can Bob parse that?
That you repeatedly made errors and repeatedly excused yourself?
> > Richard, let the READERS decide, whether yours were ERRORS or NOT.
> > We don't need your self-serving and pointless excuses for your
> > blatent errors!
>
> Oh, I'm happy to let the readers decide.
Then why DIDN'T you? Why this post, in which you've added NOTHING
of substance other than twisting everything to excuse yourself,
once more?
> > > Social science "power analysis" is not identical to
> > > computing a power function (whence, his complaint about
> > > my "alpha" and "beta" errors).
> >
> > Let readers READ the garbage you wrote, instead of TWISTING
> > what you posted. This was Richard Ulrich's gibberish when the
> > OP asks about Type I and Type II errors:
Why did you snipped what you wrote???
RU> Clinical research designs typically compute the type 2 power
RU> before attempting the research. This shows the "power"
RU> (1 minus the type-2 error) for achieving a stated alpha,
RF> This is complete gibberish in responding to someone who asks
RF> about Type-1 and Type-2 errors!
type-2 error is NOT a number. "1 minus the type-2 error" is
gibberish. The reference to alpha was gibberish. The
statement about "type 2 power" was gibberish. The statememnt
about "power" was gibberish.
EVERYTHING Richard Ulrich wrote in those three lines are gibberish.
Yet Richard is returning ... for the nth time, to excuse himself!
> Well, in my reading, the OP seemed confused.
As if Richard Ulrich wasn't.
>
> Now, if I muffed my description, I would be happy to see a
> better one.
How about this one:
http://tinyurl.com/b7u84
You should read my post(s) on the thread of Type I and Type II
errors, instead of THIS thread which was about your errors
about biased and unbiased estimates! Or read THIS one:
It's all deja vu about Richard Ulrich's ERRORS on MISINFORMING
a reader about Type I and Type II errors.
> > As I had said to you numerous times already:
> >
> > Learn some STATISTICS before you post your MISINFORMATION and
> > ERRORS. What I cited above of what you posted is a very
> > BLATENT example of your Quackery:
> Google groups reports me posting in 127 threads on
> power-analysis in the last 10 years.
It doesn't matter how many times you MIS-DIRECT, they do not excuse:
RU> Clinical research designs typically compute the type 2 power
RU> before attempting the research. This shows the "power"
RU> (1 minus the type-2 error) for achieving a stated alpha,
RF> This is complete gibberish in responding to someone who asks
RF> about Type-1 and Type-2 errors!
> I've given something
> like the above definition a number of times.
Well, we now have a confession of REPEATING the same errors!
Read this AGAIN until you understand what those definitions are:
> Bob Ling is
> the first reader who thought the definition must be "gibberish."
type-2 error is NOT a number. "1 minus the type-2 error" is
gibberish. The reference to alpha was gibberish. The
statement about "type 2 power" was gibberish. The statememnt
about "power" was gibberish.
EVERYTHING Richard Ulrich wrote in those three lines are gibberish.
> That word, along with fraud and Quack, says to me he thinks I
> made the whole thing up;
The fraud and Quack characterization referred to your ignorance in
making a statement that is complete gibberish in response to a
question by someone about Type I and Type II errors!!
RU> Clinical research designs typically compute the type 2 power
RU> before attempting the research. This shows the "power"
RU> (1 minus the type-2 error) for achieving a stated alpha,
RF> Type 1 and Type II errors are NOT alpha or beta.
RF> See: http://www.intuitor.com/statistics/T1T2Errors.html
RF> alpha = PROBABILITY of Type I error.
RF> beta = PROBABILITY of Type II error.
RF> The POWER function is the PROBABILITY of REJECTING a Null
Hypothesis
RF> for any given value of the parameter in question.
RF> It is NOT "1 - minus type-2 error".
RF> See, e.g., http://www.amstat.org/publications/jse/v11n1/pinto.html
Richard, the more you try to excuse yourself of your blunders, the
clearer it made that you ARE hopelessly ignorant in matters of
STATISTICS and BASIC statistical definitions!!!
Re-read these posts, and let READERS read them for themselves!
-- Bob.
glenb...@geocities.com
If you were computing the squared deviations from the population mean,
the "divide by n" version is right (in the sense that on average it has
the right value - i.e. it's unbiased).
The sample mean is closer to the data (in the least squares sense) than
the population mean. There are several ways to see this, but I won't
show it now.
As a result, if you divide the sum of squared deviations from the
sample mean by n, the estimate of the variance is too small (though
it's still maximum likelihood). The n-1 divisor exactly adjusts for
that, again making it unbiased.
However, it's not unbiased for the standard deviation, only the
variance.
Glen