Q use of multiple metrics

[Cosine]

Hi:

We would use more than one metrics to test the significance of a study. The most often used ones are sensitivity (SE) and specificity (SP). However, this pair would be affected by the disease prevalence. In contrast, the positive/negative predictive values (P/NPVs) are not affected by the prevalence.

The first question now is: if we choose to use the P/NPVs for testing the significance, do we also test the significance for the SE and SP? Or do we only use one of the two pairs?

The second question is that the value of the area under the receiver operating characteristic (AU-ROC) curve is a summary measure of the sensitivity and specificity. Does it mean that if we use the value of the AU-ROC for testing the significance, we do not test for the SE and SP?

The third question is that is there a replacement of the value of the AU-ROC if we are concerned about the influence of the prevalence?

Thank you,

[Rich Ulrich]

On Thu, 25 Nov 2021 12:36:29 -0800 (PST), Cosine <ase...@gmail.com>
wrote:

>Hi:
>
> We would use more than one metrics to test the significance of a
> study. The most often used ones are sensitivity (SE) and specificity
> (SP). However, this pair would be affected by the disease prevalence.
> In contrast, the positive/negative predictive values (P/NPVs) are not
> affected by the prevalence.

I would not say it that way. The PPV is /based on/ the prevalence.
It assumes a single value for the prevalence. The rarer the condition,
the more likely the Positive is False.

If you know the prevalence pretty well, you should use it for
your description. "Testing the significance" uses the same set
of numbers, same 2x2 table (I presume) for SE and SP, so you
don't expect more power from one than for the other. It should
be the same test.

If you are interested in tests across the whole ROC curve, you
test the curve. If you are interested in some specific prevalence,
you test at that value.

>
> The first question now is: if we choose to use the P/NPVs for testing the significance, do we also test the significance for the SE and SP? Or do we only use one of the two pairs?
>
> The second question is that the value of the area under the receiver operating characteristic (AU-ROC) curve is a summary measure of the sensitivity and specificity. Does it mean that if we use the value of the AU-ROC for testing the significance, we do not test for the SE and SP?
>
> The third question is that is there a replacement of the value of the AU-ROC if we are concerned about the influence of the prevalence?
>

--
Rich Ulrich

[Bruce Weaver]

On Friday, November 26, 2021 at 1:56:03 PM UTC-5, Rich Ulrich wrote:
> On Thu, 25 Nov 2021 12:36:29 -0800 (PST), Cosine <ase...@gmail.com>
> wrote:
> >Hi:
> >
> > We would use more than one metrics to test the significance of a
> > study. The most often used ones are sensitivity (SE) and specificity
> > (SP). However, this pair would be affected by the disease prevalence.
> > In contrast, the positive/negative predictive values (P/NPVs) are not
> > affected by the prevalence.
> I would not say it that way. The PPV is /based on/ the prevalence.
> It assumes a single value for the prevalence. The rarer the condition,
> the more likely the Positive is False.
>
> If you know the prevalence pretty well, you should use it for
> your description. "Testing the significance" uses the same set
> of numbers, same 2x2 table (I presume) for SE and SP, so you
> don't expect more power from one than for the other. It should
> be the same test.
>
> If you are interested in tests across the whole ROC curve, you
> test the curve. If you are interested in some specific prevalence,
> you test at that value.

The equations in this BMJ Stats Note (Altman & Bland, 1994) show how prevalence is related to PPV and NPV:

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2540558/pdf/bmj00448-0038a.pdf

Note as well that nowadays, some authors are using the following terms for the predictive values:

Predictive value of a positive test (PV+)
Predictive value of a negative test (PV-)

I like this terminology and notation better than PPV/NPV, because it makes it clear that it is the *test result*, not the predictive value, that is either positive or negative.

HTH.