Re: {MEDSTATS} Re: Bootstrap confidence intervals by sorting -- not recommended?

[John Sorkin]

I may have missed a part of this thread, so please forgive me if I cover territory already well trod.
If the observed centiles at the tail of a distribution do not agree with the estimated centiles (for something that is supposed to be normally distributed) it is probably because you have not run enough simulations. Remember that if you want a 95% confidence interval, even with 1,000 simulations, each tail has only 25 observations (50/1,000 divided in two tails). With samples of this small size, the observed centiles may not agree with the predicted values. If, on the other hand you run 10,000 simulations (250 observations in each tail) your observed and predicted centiles should agree very closely. In an earlier posting I noted that you needed a sufficiently large number of simulations. When trying to estimate tails of a distribution you need to remember that more simulations will be needed for a given level of precision to estimate tails than would be needed to estimate a measure of central tendency like the mean or median.
John





John Sorkin M.D., Ph.D.
Chief, Biostatistics and Informatics
Baltimore VA Medical Center GRECC,
University of Maryland School of Medicine Claude D. Pepper OAIC,
University of Maryland Clinical Nutrition Research Unit, and
Baltimore VA Center Stroke of Excellence

University of Maryland School of Medicine
Division of Gerontology
Baltimore VA Medical Center
10 North Greene Street
GRECC (BT/18/GR)
Baltimore, MD 21201-1524

(Phone) 410-605-7119
(Fax) 410-605-7913 (Please call phone number above prior to faxing)
jso...@grecc.umaryland.edu
>>> John Uebersax <jsueb...@gmail.com> 10/16/09 3:37 PM >>>

Some followup:

After further study, I'm not so skeptical about the SAS proc
univariate quartiles after all.

SAS supplies both unadjusted ('observed') and adjusted ('estimated')
quartiles, and this probably confused me earlier.

The former seem to be exactly what one gets by simple sorting and
selection of values at the target percentiles.

Based on a couple of examples, it seems that the estimated quartiles
might be slightly closer to expected results.

So, to set straight any misinformation my earlier post may have
presented, my conclusions (so far):

- nonparametric bootstrap is fine
- SAS proc univariate quartiles are fine
- *possibly* the estimated quartiles are a little better

There are several options in SAS for handling quartiles and I've by no
means looked at them closely.

Whether to adjust quartiles for possible ties (or other things?), then
remains an open question for me.

--
John Uebersax
http://www.john-uebersax.com



Confidentiality Statement:
This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

[BXC (Bendix Carstensen)]

As far as I can see it is only the "raw" quantiels that are transformtion
invariant. Suppose you bootstrap to get some parameter, theta.
If you instead wanted log(theta), wouldn't it be reasonable to require that
the quantiles for log(theta) were the log of the quantiles for theta?

This property does not hold for the predicted quantiles???

Or have I completely misunderstood the previously given definition of the
"predicted" quantiles?

Best regards,
Bendix

[John Uebersax]

I'm definitely using a large number of simulations (e.g,., 100,000).
My concern is more what to do with possible ties in the rank ordering
of estimates near the target percentiles. I think that in the present
case this might have more to do with the original sample N (i.e., only
moderate) than with the number of bootstrap replications.

John Uebersax

> > faxing) jsor...@grecc.umaryland.edu
> > >>> John Uebersax <jsueber...@gmail.com> 10/16/09 3:37 PM >>>

[John Uebersax]

Checking, I find lots of ties at the 2.5% and 97.5% levels. That is,
the bootstrap estimate at the, e.g., 2.5% percentile also occupies
adjacent higher and lower percentiles.

So couldn't a case be made for interpolation here?

Suppose 1000 bootstrap (yes, too few, but this is an example) with
these results:

rank estimate

22 .563
23 .572
24 .572
25 .572
26 .581

Then should the bootstrap lower bound of the 2-sided 95% CI be taken
as .572? Or is it somewhere between .572 and .563
(or between .572 and .581)?

John Uebersax
http://www.john-uebersax.com

> > > reply email and destroy all copies of the original message.- Hide quoted text -
>
> - Show quoted text -

[BXC (Bendix Carstensen)]

If you have a lot of ties, it might be an indication that your initial sample is too small. Specifically, the bootstrap will only ever give you one of a finite number of values, because there is only a finite number of possible bootstrap samples (and some of these may even give the same estimate). Usually this finite number is very large, but occasionally it might just be a few thousand, and then you get ties. And have the opportunity to very precisely evaluate the empirical distribution of you estimator.

My point was that it is an odd idea to assume that the empricical distribution of some
parameter has a particular form. Maybe this idea was a product of times where you were effectively restricted to "small" bootstrap samples like 348.

Best regards,
Bendix Carstensen