Sample size in diagnostic studies

[Thomas Keller]

Dear all, 

in recent published paper "Zapf et al:  Zapf et al: Adaptive trial designs in diagnostic accuracy research, Stat in Med. 39 (2020) pp591 (DOI: 10.1002/sim.8430)", the following sample size calculation is described:

"To illustrate, the following scenario is considered. For the approval of an assay, a confirmatory study is needed to demon-strate that sensitivity fulfills a predefined acceptance criterion, ie, the lower limit of a two-sided 95% confidence interval (LLCI) is at least 90%, for a point estimate of 96%. "

"Conventional fixed design. For a single cohort fixed design with an α of 5% (two-sided), a minimum of 100 true positive cases is required to allow for a maximum of four false negatives so that the LLCI is at least 90% with a point estimate of 96%."

Note: "true positives" means here disease positives D+.

I think, the sample size consideration refers to an estimation of a proportion (exploratory trial), and not to a confirmatory study (where I get appr. 196 D+ for 90% power). 

I have noticed that diagnostic studies often use sample sizes corresponding to an estimate because they are much smaller. But typically then it is known and underlined, that the estimation is the objective of the trial. 

The connection of a confirmatory study and sample size consideration for an estimation, (without considering power), however, is new for me. 

I would be happy if you would share your thoughts about this.

Kind regards Thomas

ACOMED statistik Leipzig, Germany

[John Whittington]

Thomas, I have been trying to get my head around
your question (well, an answer to it!) and was
hoping that others might respond - but, since
no-one yet has, and at risk of making a fool of
myself, I thought I would do a little 'thinking aloud'.

One aspect which initially attracted my attention was your statement:

>The connection of a confirmatory study and
>sample size consideration for an estimation,
>(without considering power), however, is new for me.

Having given some thought, is it not the case
that 'power' is a concept which only really
exists in relation to inference, and not to
estimation? Power refers to the probability of a
voiding a Type II error but, in the cases of
estimation, in the absence of a dichotomised
result ('significant' vs. 'non-significant') I
can't really see that there can be any equivalent
of a Type II error (hence power to avoid it).

Consider first the situation if criteria for
approval of the assay were written in terms of
inference - i.e. if, using the figures you
mention, the requirement was that there should be
95% confidence (i.e. alpha=0.05) that the true
result was at least 90% if/when the point
estimate were 96%. In that situation, one could
calculate the sample size required to provide any
given power to achieve that - and the 'power' one
has used would be the probability of avoiding a
Type II Error, such an error arising if the
(statistical) test failed to produce a
positive/'significant' result when it 'should
have done'. On that basis, I seem to get
required sample sizes around 180 for 90% power
for two-sided and about 155 for (probably more
appropriate) one-sided (as compared with your 196)

On the other hand, if the stated requirement is
(as you say) merely that the LLCI of an estimate
of 96% should be at least 90%, I cannot see what
'power' could mean. Am I missing something?

This is where I get a bit confused, and perhaps
display ignorance and/or faulty thinking! If the
estimate is 96% then, for any assumed
distribution (say, binomial), the CI for any
sample size is a fixed and calculable entity,
whose lower limit can be made as close as one
likes to 96% by choosing a large enough sample size.

I suspect that this is probably where their
"minimum sample size of 100" comes from, since
the (Wilson) binomial confidence interval for a
success rate of 96% with n=100 is 90.16% to
98.43% (and the Clopper-Perason Exact CI is
90.07% to 98.90%), with a LLCI just fractionally
above 90%. What I'm struggling to decide is how
reasonable an approach I think this is - because
my first inclination was to feel that it wasn't (a 'reasonable approach').

The above may be nonsense, but I'd be interested to hear any thoughts!

Kind Regards,
John

At 09:31 30/04/2021, Thomas Keller wrote:
>Dear all,
>
>in recent published paper "Zapf et al: Zapf et
>al: Adaptive trial designs in diagnostic
>accuracy research, Stat in Med. 39 (2020) pp591
>(DOI: 10.1002/sim.8430)", the following sample size calculation is described:
>
>"To illustrate, the following scenario is
>considered. For the approval of an assay, a
>confirmatory study is needed to demon-strate
>that sensitivity fulfills a predefined
>acceptance criterion, ie, the lower limit of a
>two-sided 95% confidence interval (LLCI) is at
>least 90%, for a point estimate of 96%. "
>"Conventional fixed design. For a single cohort

>fixed design with an α of 5% (two-sided), a

>minimum of 100 true positive cases is required
>to allow for a maximum of four false negatives
>so that the LLCI is at least 90% with a point estimate of 96%."
>
>Note: "true positives" means here disease positives D+.
>
>I think, the sample size consideration refers to
>an estimation of a proportion (exploratory
>trial), and not to a confirmatory study (where I
>get appr. 196 D+ for 90% power).
>
>I have noticed that diagnostic studies often use
>sample sizes corresponding to an estimate
>because they are much smaller. But typically
>then it is known and underlined, that the
>estimation is the objective of the trial.
>The connection of a confirmatory study and
>sample size consideration for an estimation,
>(without considering power), however, is new for me.
>
>I would be happy if you would share your thoughts about this.
>
>Kind regards Thomas
>ACOMED statistik Leipzig, Germany

John

----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225
Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
Buckingham MK18 4EL, UK
----------------------------------------------------------------

[John Whittington]

Further to what I wrote yesterday (below), I've
done some more thinking and have come to realise
that one of the odd things here is ..

> ".... For the approval of an assay, a

> confirmatory study is needed to demon-strate
> that sensitivity fulfills a predefined
> acceptance criterion, ie, the lower limit of a
> two-sided 95% confidence interval (LLCI) is at
> least 90%, for a point estimate of 96%. "

That appears to be a quote from the paper which
Thomas mentioned but, unless it is a paraphrase
or 'example', it seems to be very odd that a
'requirement for approval' should be stated in
that fashion. It would make sense (to me) if it
were not for the last few words ("... for a point
estimate of 96% "), but with those words seems
very odd. In other words, I would expect it to
simply say that "the LLCI must be at least 90%".

However, even if it did say 'just that', it would
surely effectively only be saying that the
estimate (hence, by implication, the 'true
value') had to be above 90% - since, provided
only that the 'true value' were above 90%, one
could always achieve the requirement that the
LLCI should be above 90% by invoking a large enough sample size, couldn't one?

Does that make sense? Any other thoughts?

Kind Regards,
John

>--
>--
>To post a new thread to MedStats, send email to MedS...@googlegroups.com .
>MedStats' home page is http://groups.google.com/group/MedStats .
>Rules: http://groups.google.com/group/MedStats/web/medstats-rules
>
>--- You received this message because you are
>subscribed to the Google Groups "MedStats" group.
>To unsubscribe from this group and stop
>receiving emails from it, send an email to
>medstats+u...@googlegroups.com.
>To view this discussion on the web, visit
>https://groups.google.com/d/msgid/medstats/202105021855.142ItS5g011707%40mail33c50.megamailservers.eu.