Bootstrap confidence intervals by sorting -- not recommended?

[John Uebersax]

Hello Group,

In the past I've been rather cavalier when it comes to bootstrap
confidence intervals. For convenience, I've just sorted bootstrap
estimates and selected the values closest to, say, the 2.5% and 97.5%
percentiles (with maybe an adjustment for ties).

One might call this a fully non-parametric confidence interval,
because it makes no distributional assumptions at all.

This is admittedly primitive, but I didn't realize how much so until
noticing that it's scarcely mentioned (if at all!) in Efron and
Tibshirani's Introduction to the Bootstrap (1994).

Rather, they suggest that bootstrap estimates generally approach a
normal (or t) distribution, and so base confidence intervals on
parametric assumptions (e.g., in the simplest case, taking, say, +/-
1.96 * bootstrap std err.)

Is that the general view among statisticians -- to avoid "fully
nonparametric" confidence intervals (i.e., based on specified
quantiles of sorted estimates) whenever possible?

Is there a good reference that discusses this issue?

Thanks in advance.

John Uebersax
http://www.john-uebersax.com

[Jeff Allard]

John-

I think what you are describing is the percentile method. I would say if the distribution of bootstrap estimates appears normal, use that the norm or t distribution as you say. I think the Bca method is one of the best (http://www.lexjansen.com/phuse/2005/pk/pk02.pdf) CI methods. This reference touches on a few and shows the derivation within SAS.

Curious what others have to say.

HTH. 

2009/10/15 John Uebersax <jsueb...@gmail.com>

[John Whittington]

Maybe I'm missing something, but I find it hard to see why there
could/should be a view that one should "avoid 'fully non-parametric'
confidence intervals whenever possible". If there were such a view, what
would be the reason?.

Provided only that one has a large enough number of bootstrap estimates
(and a very large number is generally possible/practical with available
technology), it would seem to me that the method which avoids any
distributional assumptions (which, like John, is what I normally do) is the
'purest' approach.

Kind Regards,
John

At 14:19 15/10/2009 -0700, John Uebersax wrote:
>In the past I've been rather cavalier when it comes to bootstrap
>confidence intervals. For convenience, I've just sorted bootstrap
>estimates and selected the values closest to, say, the 2.5% and 97.5%
>percentiles (with maybe an adjustment for ties).
>
>One might call this a fully non-parametric confidence interval,
>because it makes no distributional assumptions at all.
>
>This is admittedly primitive, but I didn't realize how much so until
>noticing that it's scarcely mentioned (if at all!) in Efron and
>Tibshirani's Introduction to the Bootstrap (1994).
>
>Rather, they suggest that bootstrap estimates generally approach a
>normal (or t) distribution, and so base confidence intervals on
>parametric assumptions (e.g., in the simplest case, taking, say, +/-
>1.96 * bootstrap std err.)
>
>Is that the general view among statisticians -- to avoid "fully
>nonparametric" confidence intervals (i.e., based on specified
>quantiles of sorted estimates) whenever possible?

John

----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225
Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
Buckingham MK18 4EL, UK
----------------------------------------------------------------

[Adrian Sayers]

Bootstrap confidence intervals: when, which, what? A practical guide
for medical statisticians.
by: J. Carpenter, J. Bithell
Stat Med, Vol. 19, No. 9. (15 May 2000), pp. 1141-1164.

might be of interest!

Adrian

2009/10/15 John Uebersax <jsueb...@gmail.com>:

[John Uebersax]

Thanks for the replies.

Jeff,

Yes, I think you're right. Efron and Tibshirani is unusually well
written, but in the case of the percentile method their notation was a
little more complex than I was expecting for a simple nonparametric
approach and I misinterpreted it.

Second, the Barker paper you suggested helped me identify the
problem. I had used quantiles produced by SAS proc univariate to
identify the 2.5% and 97.% limits. The advantage with this over
sorting is that one can request quantiles for several variables
at once.

However this produced inaccurate CIs, prompting my original post.
When I simply sorted bootstrap estimates and selected
the ones at the 2.5% and 97.% percentiles, the CIs looked much closer
to what is expected. This attempts no kind of adjustment for ties
(and perhaps that's the issue with proc univariate).

John W.,

As noted above, when I re-ran my nonparametric bootstrap estimates
without proc univariate they were *much* better. That
effectively removes my concerns about the method, and I am no longer
reluctant to use it.

--
John Uebersax
http://www.john-uebersax.com

[John Uebersax]

Some followup:

After further study, I'm not so skeptical about the SAS proc
univariate quartiles after all.

SAS supplies both unadjusted ('observed') and adjusted ('estimated')
quartiles, and this probably confused me earlier.

The former seem to be exactly what one gets by simple sorting and
selection of values at the target percentiles.

Based on a couple of examples, it seems that the estimated quartiles
might be slightly closer to expected results.

So, to set straight any misinformation my earlier post may have
presented, my conclusions (so far):

- nonparametric bootstrap is fine
- SAS proc univariate quartiles are fine
- *possibly* the estimated quartiles are a little better

There are several options in SAS for handling quartiles and I've by no
means looked at them closely.

Whether to adjust quartiles for possible ties (or other things?), then
remains an open question for me.