CIs for positive/negative predictive value via likeihood ratios?
John Uebersax
I ran across this paragraph in an FDA Draft Guidance for Industry and
am puzzled:
http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidance/GuidanceDocuments/ucm181509.htm
"For the 95% confidence intervals for sensitivity and specificity, a
score method is recommended…"
Okay, that's clear; it's recommending use of Wilson (score) confidence
intervals for binomial proportions. But then it continues:
"The confidence intervals for the predictive values [i.e., positive
and negative predictive value, or PPV and NPV] can be calculated (when
prevalence is constant) based on the confidence intervals of the
corresponding likelihood ratios (an estimate of the likelihood ratio
is a ratio of two independent proportions; therefore, the exact
confidence intervals for ratio of two independent proportions can be
used)."
This seems to suggest estimating the confidence intervals of PPV and
NPV based on
likelihood ratios. So two questions:
1. Why can't one use Wilson score intervals to estimate the CIs of PPV
and NPV? Isn't this just a transpose of the problem of estimating
sensitivity and specificity? For example, with data in this form:
Standard
+ -
+ a b
Test
- c d
if PPV = a/(a+b), why can't one estimate a standard CI for a binomial
proportion. Does the issue relate to the stated stipulation "when
prevalence is constant"?
2. Can anyone suggest references or examples of the method the
Guidance is describing?
Thanks in advance.
John Uebersax PhD
http://www.john-uebersax.com
Andrew Hayen
Here's one reference that discusses the confidence intervals for
predictive values:
Nathaniel DM, Kit FL, Xiao HZ. Confidence intervals for predictive
values with an emphasis to case-control studies. Stat Med.
2007;26(10):2170-83.
I haven't read the guidelines you mention, but I part of the issue is
that with many diagnostic accuracy studies, the prevalence cannot be
estimated from the study itself because cases may be oversampled.
Andrew
thomas
the relationship predictive values - DLR might be addressed in the
Pepe book "Statistical Methods in Diagnostic Medicine" on page 45
(last paragraph) ff.
In the chapter, she refers to
Sox, Jr. et al. 1989 "Assessment of diagnostic technology in health
care..., National academic press
and
Diamond and Forester, Analysis of Probability as an aid ... N.
Engl.J. Med. 300. 1350 (1979)
By the way, DLRs are currently "en vogue" in this FDA department.
However, I would interpret the proposal in FDA guideline as a cohort
study of ASC-US patients allowing direct estimation of predictive
values (?).
CLSI guideline EP12 (which the FDA guideline refers to in some extend)
proposes exact as well as score CI.
Regards Thomas
________________
Dr. Thomas Keller
ACOMED statistik
Leipzig
Germany
On 14 Okt., 01:23, John Uebersax <jsueber...@gmail.com> wrote:
> Hello Group,
>
> I ran across this paragraph in an FDA Draft Guidance for Industry and
> am puzzled:
>
thomas
the relationship predictive values - DLR might be addressed in the
(last paragraph) ff.
Th. Keller
John Uebersax
> CLSI guideline EP12 (which the FDA guideline refers to in some extend)
> proposes exact as well as score CI.
Indeed -- causing much mischief and the need to explain to scientists
that "exact" (i.e., Clopper-Pearson) intervals are by no means exact!
John
Steve Simon, P.Mean Consulting
> "The confidence intervals for the predictive values [i.e., positive
> and negative predictive value, or PPV and NPV] can be calculated (when
> prevalence is constant) based on the confidence intervals of the
> corresponding likelihood ratios (an estimate of the likelihood ratio
> is a ratio of two independent proportions; therefore, the exact
> confidence intervals for ratio of two independent proportions can be
> used)."
>
> This seems to suggest estimating the confidence intervals of PPV and
> NPV based on likelihood ratios. So two questions:
>
> 1. Why can't one use Wilson score intervals to estimate the CIs of PPV
> and NPV? Isn't this just a transpose of the problem of estimating
> sensitivity and specificity? For example, with data in this form:
>
> Standard
> + -
> + a b
> Test
> - c d
>
> if PPV = a/(a+b), why can't one estimate a standard CI for a binomial
> proportion. Does the issue relate to the stated stipulation "when
> prevalence is constant"?
As another reader has mentioned, the prevalence in a study may not be
the prevalence in a population. In particular, in a case-control study,
the prevalence is often artificially controlled at 50%. In this case,
you would pick a more relevant prevalence (Pv) and use the following
formula:
PPV odds = Pv odds * LR+
NPV odds = Pv odds * LR-
If the prevalence (Pv) is known to sufficient precision that it can be
considered a constant, then confidence limits for PPV and NPV would be
computed by replacing the likelihood ratios by their confidence
intervals. So if LR+ has CI of 16,24 and the Pv=0.125, then Pv odds =
1/8 and the interval for PPV odds would be 2,3 corresponding to
probabilities of 0.67, 0.75.
> 2. Can anyone suggest references or examples of the method the
> Guidance is describing?
Sorry, but I'm unaware of a good reference for this.
--
Steve Simon, Standard Disclaimer
Sign up for The Monthly Mean at www.pmean.com/news
John Uebersax
A late reply to your helpful message. Thanks - the citation you gave
turns out to be very relevant to my question.
The article seems specifically concerned with estimating confidence
intervals of PPV and NPV for case-control studies (i.e., where the
sample prevalence is different than population prevalence). However
for a regular population study or clinical trial, I can't see any
obvious reason not to estimate the confidence intervals of PPV and NPV
based on treating them as simple binomial proportions; although a
couple of statements in the article strike me as ambiguous, there
isn't anything there that would specifically argue against this.
John Uebersax
http://www.john-uebersax.com
On Oct 13, 3:36 pm, Andrew Hayen <aha...@health.usyd.edu.au> wrote:
> hi
>
> Here's one reference that discusses the confidence intervals for
> predictive values:
>
> Nathaniel DM, Kit FL, Xiao HZ. Confidence intervals for predictive
> values with an emphasis to case-control studies. Stat Med.
> 2007;26(10):2170-83.
>
> I haven't read the guidelines you mention, but I part of the issue is
> that with many diagnostic accuracy studies, the prevalence cannot be
> estimated from the study itself because cases may be oversampled.
>
> Andrew
>
> > Hello Group,
>
> > I ran across this paragraph in an FDA Draft Guidance for Industry and
> > am puzzled:
>
John Uebersax
Nathaniel D. Mercaldo
Kit F. Lau
Xiao H. Zhou
so that, if I'm not mistaken, the citation would be:
Mercaldo ND, Lau KF, Zhou XH.
Confidence intervals for predictive values with an emphasis to case–
control studies.
Statist Med 2007; 26:2170–2183.
Link: http://www3.interscience.wiley.com/journal/112768102/abstract
--
John Uebersax
http://www.john-uebersax.com
mcap
think you have all covered this better than I could. As I read it in
plain language:
-normally for a diagnostic test, you are interested in the probability
of the disease given a positive test. This is the PPV
-As was stated previously, the PPV is only useful if the sample
prevalence is similar to your target population prevalence.
Validation studies are often configured to include high numbers of
individuals with the disease in question.
-A likelihood ratio can be converted to a post-test probability of
diease. This post test probability is essentially a PPV with a more
realistic estimate of your target population prevalence. As to why
you couldn't use a CI for a binomial proportion to caclulate a CI for
the post test probability.........no idea!
Marc