Lesson 2.3 Question 5: does standard deviation estimation depend on number of samples?

[Jeehoon Kang]

Hi, this is Jeehoon.

Lesson 2.3 Question 5 asks:

"If you did the above experiment with 10 different random seeds rather than 5, how would you expect this to affect the mean and standard deviation?"

The stated correct answer is:

"The mean would be about the same and the standard deviation would be a little smaller."

However, as far as I know, the number of samples has no correlation on the estimation of standard deviation. This is why we divided the sum of squares of differences by (n-1), not n.

Would you please point out where I was wrong?

Thank you

Jeehoon

[s9r...@gmail.com]

Intuitively, think of this as the case of Law of Large Numbers. As you add more samples, the observed sample mean converges to the true mean of the distribution.

In that sense, the variance and therefore the standard deviation of the distribution decreases.

Best,

Sirisha

[Jeehoon Kang]

As the mean of samples converges to the true mean, the estimated standard deviation of the samples also converges to the true standard deviation.

See:

http://en.wikipedia.org/wiki/Standard_deviation#Estimation

http://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation

2013년 9월 19일 목요일 오전 5시 12분 47초 UTC+9, s9r...@gmail.com 님의 말:

[s9r...@gmail.com]

Yes. 

Variance and Standard Deviations are a measure of uncertainty in the distribution of the data. If your variance went up, you would be that much more uncertain in estimating the true mean.

As you add more data points/samples, the uncertainty decreases. But this is true only for "unbiased" estimators, as in this case.

If this helps, you can also think of this as: the more samples you have, then likely that "extreme" cases cancel out.

Sirisha

[Manav Brar]

for my results, the correct answer should be "Both the mean and standard deviation would be a bit smaller." if I look at the Weka data 

I am comparing the data with the 5 different random seeds

[Karen B]

I agree there is a problem in one of the early chapters that asserts that as the sample size increases we should see something *systematic* with the standard deviation of S.  More appropriate would be to say that “the sample standard deviation will become more precise as sample size increases”, and not something like “the standard deviation will decrease, on average, as sample size increases”.

 

Properties of Estimate of Variance, S^2

------------------------------------------------------------

Expected value of S^2

When population mean is known, estimate of variance = S^2 = (sum(Xi-mu))^2/N is unbiased. The expected value of the variance computed this way is equal to the true population variance, E(S^2) = Sigma^2, Where Sigma^2 is the true unknown population variance. Note that as N increases, there is no change to the expectation for S^2, E(S^2).

 

When population mean is known, variance = S^2 = (sum(Xi-mu))^2/N is *biased*. The expected value of the variance computed this way is smaller than the true population variance, E(S^2) = Sigma^2*(N-1)/N). Note that (N-1)/N is lower than 1 but approaches 1 as N increases, so this estimator is a little too small on average but rises to match true Sigma^2 as N increases.

 

When population mean is unknown, variance = S^2 = (sum(Xi-mu))^2/(N-1) is unbiased. The expected value of the variance computed this way is equal to the true population variance, E(S^2) = Sigma^2, so we say it is unbiased. Note that as N increases, there is no change to the expectation for S^2.  That is why we apply N-1 to the denominator of the S^2 calculation in practice.

 

Precision of S^2

How variable is our estimate of S^2 as a function of N? The variance of the sampling distribution of S^2 is given by  Variance(S^2) = 2*Sigma^4/(N-1). So as N increases, 1/(N-1) decreases, and the sample estimates of the variance have decreasing variability; they are increasingly more precise. Hence we can say that “the sample standard deviation will become more precise, or less variable, as sample size increases”

 

 

Footnote: There is another fine point, meaning of smaller consequence, and not the main point here: By taking the square root to find S = sqrt(var),  or a = sqrt(b) where b is an unbiased estimator of a true population quantity, there is a very small downward bias introduced in S as an estimate of Sigma, E(S) < Sigma, by Jensen's inequality, due to the square root being a concave function. The bias in the variance (S^2) is easily corrected (by applying the N-1 as above), but the bias from the square root function to solve for S is more difficult to correct, and depends on the distribution in question. Luckily, it is very small.  The error in this approximation decays quadratically (as 1/N²), and so S as an estimate of true population sigma is suited for all but the smallest samples or highest precision: for n = 3 the bias is equal to 1.3%, and for n = 9 the bias is already less than 0.1%.