RE: MEDSTATS: Implications of "early stopping" of randomized trials
Val Gebski
I think this and excellent example of when NOT to stop a study. Some points:
a) was the decision to stop the study taken BEFORE the results were known? If the driving question was the cost of the study then the decision to close the study should have been made before any results were known. Otherwise they have compromised the sicence
b) the group would not have been so excited if this informal analysis yielded a p-value of say 0.08!
c) What was the process of having a look & why? eg was the call to stop the stduy thru a Data Safety and Monitoring Committee or the steering comittee
d) If one looks at the ISIS 1 data see Sallm Yusuf: "Challenges in the conduct and interpretation of Phase II (pilot) randomized trials" Am Heart J 2000; 139:s 136s 142 the results showed a significant detrimet p < 0.05 after 120 deaths were observed. The trial wen on to show a significant benefit. This illustrates the danger of casual looks at the data
e)Having stopped the study early, the study has now failed to scientifically demonstrate whether the intervention is better and another trial needs to be done - what a waste!
f) The original trial was presumably designed to detect a certain effect size on a particular population based on known information about the intervention, disease history and standard care. How did all this change in this trial & if things were the same all the more reason to continue
g)Based on this trial alone, I would guess the regulatory authorities would be skeptical.
Val Gebski
NHMRC Clinical Trials Centre University of Sydney,
Camperdown NSW 1450 Australia
-----Original Message-----
From: MedS...@googlegroups.com on behalf of jlm
Sent: Wed 01/18/06 2:48 PM
To: MedStats
Subject: MEDSTATS: Implications of "early stopping" of randomized trials
Dear Medstats Group
I am seeking opinions regarding the implications of "early stopping",
not when certain pre-defined boundaries have been crossed, rather
because, quite simply, "the money ran out".
The randomized trial I have been asked to comment upon (multicentre and
trans-national) declared a favourable difference (binary outcome) of
-16% (-27%, -4%) at a P =0.009 with only 55% of projected recruitment
(no interim analyses appear to have been performed; at least, none were
reported in the methodology statements).
Two questions if I may: (i) is this just "good luck / management" and
(ii) if this was a drug trial (which, in this case, it was not) could
said drug have been licensed with regulatory authorities
####################################################################################################################
IMPORTANT NOTICE: This e-mail and any attachment to it are intended only to be read or used by the named addressee.
It is confidential and may contain legally privileged information. No confidentiality or privilege is waived or lost
by any mistaken transmission to you. The CTC is not responsible for any unauthorised alterations to this e-mail or
attachment to it. Views expressed in this message are those of the individual sender, and are not necessarily the
views of the CTC. If you receive this e-mail in error, please immediately delete it and notify the sender. You must
not disclose, copy or use any part of this e-mail if you are not the intended recipient.
#####################################################################################################################
Ted Harding
> Dear jlm?
>
> I think this and excellent example of when NOT to stop a study.
> Some points:
>
> a) was the decision to stop the study taken BEFORE the results
> were known? If the driving question was the cost of the study
> then the decision to close the study should have been made
> before any results were known. Otherwise they have compromised
> the sicence
> b) the group would not have been so excited if this informal
> analysis yielded a p-value of say 0.08!
> c) What was the process of having a look & why? eg was the call
> to stop the stduy thru a Data Safety and Monitoring Committee
> or the steering comittee
> d) If one looks at the ISIS 1 data see Sallm Yusuf: "Challenges
> in the conduct and interpretation of Phase II (pilot) randomized
> trials" Am Heart J 2000; 139:s 136s 142 the results showed a
> significant detrimet p < 0.05 after 120 deaths were observed.
> the danger of casual looks at the data
> e)Having stopped the study early, the study has now failed to
> scientifically demonstrate whether the intervention is better
> and another trial needs to be done - what a waste!
> f) The original trial was presumably designed to detect a certain
> effect size on a particular population based on known information
> about the intervention, disease history and standard care. How did
> all this change in this trial & if things were the same all the
> more reason to continue
> g)Based on this trial alone, I would guess the regulatory authorities
> would be skeptical.
Val makes some good points, and it is certainly clear that
interpretation must depend on what really really happend when
the decision to terminate was taken. However, I am not so
absolutely negative as Cal appears to be.
While I'm in no position to commment from the point of view
of a regulatory bureaucrat, I'd like to make the following
point:
-- If the trial was properly randomised (over sequence in time
as well as over subjects), and
-- if the reason for stopping was the occurrence of circumstances
which are not informative about the treatment effect, and
-- if the stopping took place "blind" (i.e. in ignorance of
outcomes)
then the data which have been obtained up to the stopping time
are unbiased, and randomised, and any conclusions which may be
drawn by analysing them on that basis are valid.
Admittedly the trial has no doubt fallen short of possessing
the properties it was originally designed to have (e.g. adequate
power for given effect size), but as far as it went it does
have (conditional on the numbers treated) a certain power, etc.
Of course if any of those "if"s fails, or is doubtful, then
so also will such a conclusion fail or be doubtful; but judging
this depends on what you know about what really really happened.
If you don't really know about this, then you don't really
know how valid the trial was, as far as it went; and that's
where you stand -- namely, you don't know!
Best wishes to all,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 18-Jan-06 Time: 10:08:51
------------------------------ XFMail ------------------------------
Ted Harding
the result of Val's "TNEF" attachment crashing my mailer
when I started to edit my reply, before I sussed out
that this was the source of the problem and deleted the
attachment from the mail before opening it for replying!
Best wishes to all,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 18-Jan-06 Time: 10:23:45
------------------------------ XFMail ------------------------------
Bland, M.
money ran out, then the treatment comparison is OK.
In my experience many trials have great difficulty achieving their
sample size. Indeed, my solution to the MRSA problem is to set up a
national trial of MRSA treatment, where all patients will be enrolled in
the trial. They will disappear like snow in August, end of problem. If
we said that a trial had to achieve its planned sample size, very few
public sector trials would ever be reported. I don't know whether the
Pharma sector does any better.
Martin
(Ted Harding) wrote:
--
***************************************************
J. Martin Bland
Prof. of Health Statistics
Dept. of Health Sciences
Seebohm Rowntree Building Area 2
University of York
Heslington
York YO10 5DD
Email: mb...@york.ac.uk
Phone: 01904 321334
Fax: 01904 321382
Web site: http://www-users.york.ac.uk/~mb55/
***************************************************
Moran, John (NWAHS)
into this question.
I think that the cautions raised by Val Gebski are to the point in the
particular case in question, because, although a very influential study
and published in a "premier" journal, the said trial, in the _published_
methodology statements:
(i) gave _no_ statement regarding sample size or minimally clinically
important difference. The 55% figure that I quoted was a ratio of actual
enrolment (in the published study) to what was quoted in a small review
article published 1 year after the start of the trial (and not
referenced in the final published trial report).
(ii) the _only_ information regarding "stopping" was a terse statement
in the published article that enrolment was low and funding had ended.
There is mention in the published article (as an Appendix) of a
"Monitoring Committee"
(iii) the trial enrolled patients over a 4.5 year period.
Why I raised the question in this forum were the obvious concerns above
(i-iii) and the question of (prolonged) enrolment and stopping in a
multicentre trial. The best quoted caution to this process appears to be
"The Coronary Drug Project Research Group. Practical aspects of decision
making in clinical trials: the coronary drug project as a case study.
Contol.Clin.Trials. 1 (4):363-376, 1981", the section "Nontermination of
the clofibrate group", Figure 7, where it was noted that 3 times in the
first 30 months of the study the z value exceeded the -1.96 boundary. At
conclusion there was no difference (the opposite to the article by
Yusuf, which Val quoted)
Employing Ted Harding's series of "if" qualifiers, in this case we
obviously do "not know", based upon i-iii above.
This raises the obvious question of adoption, in clinical practice, of
non-drug therapies based upon trials which may not have scaled
regulatory heights, although I take Martin Bland's comments regarding
Pharma. The best way of cautioning clinicians, the majority of whom
cannot stand by and do nothing, appears to ask them whether they think
that their favourite non-drug therapy would have passed regulatory
strictures.
Thanks again to Val, Ted and Martin for their comments
John Moran
Department of Intensive Care Medicine
The Queen Elizabeth Hospital
28 Woodville Road
Woodville SA 5011
Australia
Tel: 61 08 8222 6463
Fax 61 08 8222 6045
E-mail: john....@nwahs.sa.gov.au
John Whittington
>I think Ted is correct, that if the only reason for stopping is that the
>money ran out, then the treatment comparison is OK.
That seems to make total sense, although it might often be impossible to be
very confident that money WAS the only reason for stopping.
People often seem to forget that, in some situations, no amount of
blindness can prevent observers gaining a pretty good idea of how well the
test treatment is performing - e.g. in those situations in which the
comparator treatment is (necessarily or by choice) essentially
ineffective. One therefore would have to look very carefully before even
starting to believe that a trial had been stopped 'only because the money
ran out'.
>In my experience many trials have great difficulty achieving their sample
>size. ... If we said that a trial had to achieve its planned sample size,
>very few public sector trials would ever be reported. I don't know
>whether the Pharma sector does any better.
I think you can rest assured that pharmaceutical industry experience is
much the same as yours, particularly in relation to larger trials;
recruitment is usually the greatest management problem in any trial. As
you have said, it's often tempting to suggest that the way to make a
disease disappear is to set up a clinical trial to study it!!
Of course, this can be intimately tied up with the 'running out of money'
issue, since recruitment problems can generate considerable additional
costs. Even worse for a pharmaceutical company than direct costs is that
if prolongation of a trial results in delay of a product reaching the
market, that can have massive financial implications, not only in terms of
cashflow but also in absolute terms (since the delay eats into remaining
patent life).
Kind Regards
John
----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225
Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
Buckingham MK18 4EL, UK medis...@compuserve.com
----------------------------------------------------------------
Moran, John (NWAHS)
As detailed in my previous reply (Thu 19/01/2006) to comments, the
problem, for the trial under scrutiny, was precisely that it was
impossible to know from the publication.
Thus the paradox: in the pharmaceutical industry there are attendant
penalties (with regard to regulatory bodies) for falling off ones' horse
before the finishing line, as it were; yet in non-Pharma trials one can
apparently remount with a lesser penalty, and clinicians being
clinicians, be awarded the prize regardless.
Or am I overstating the case?
John Moran
Department of Intensive Care Medicine
The Queen Elizabeth Hospital
28 Woodville Road
Woodville SA 5011
Australia
Tel: 61 08 8222 6463
Fax 61 08 8222 6045
E-mail: john....@nwahs.sa.gov.au
-----Original Message-----
From: MedS...@googlegroups.com [mailto:MedS...@googlegroups.com] On
Behalf Of John Whittington
Sent: Saturday, 21 January 2006 4:32 AM
To: MedS...@googlegroups.com
Subject: MEDSTATS: Re: Implications of "early stopping" of randomized
Ted Harding
>
> Thanks to John Whittington for those comments.
>
> As detailed in my previous reply (Thu 19/01/2006) to comments, the
> problem, for the trial under scrutiny, was precisely that it was
> impossible to know from the publication.
>
> Thus the paradox: in the pharmaceutical industry there are attendant
> penalties (with regard to regulatory bodies) for falling off ones'
> horse before the finishing line, as it were; yet in non-Pharma trials
> one can apparently remount with a lesser penalty, and clinicians being
> clinicians, be awarded the prize regardless.
>
> Or am I overstating the case?
A nice image! I think one might say that, if the objective is
scientific rather than regulatory, then what matters is getting
to the finishing line -- you could fall off, catch the horse,
and walk with it to the line leading it by the bridle, so long
as you get there! (Staying within the confines of the racetrack,
however -- it's unscientific to take a short cut across the
inner area; though sometimes one suspects that this can happen
in regulatory work, so long as one is seen to stay in the saddle).
With regard to your case, though, the fact that you cannot tell
from the publication means that the publication does not clear
up the legitimate questions relevant to whether the results
can be validly interpreted at face value, and therefore the
publication is scientifically inadequate.
Best wishes,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 21-Jan-06 Time: 08:15:33
------------------------------ XFMail ------------------------------
John Whittington
>With regard to your case, though, the fact that you cannot tell
>from the publication means that the publication does not clear
>up the legitimate questions relevant to whether the results
>can be validly interpreted at face value, and therefore the
>publication is scientifically inadequate.
True, but as I have suggested, even if the publication DID include
statements which appeared to indicate that the results could be 'validly
interpreted at face value', it would be extraordinarily difficult (I guess
a statistician should not say 'impossible'!) to be sure that those
statements were true, even if they were written in good faith.
In those situations in which blinding cannot fully conceal aspects of the
emerging outcome, those managing trials may be influenced 'subconsciously',
even if not 'consciously and deliberately'. Even 'Running out of money' is
not as straightforward as it may first sound. Trials very often go 'over
budget', but that does not necessarily cause them to be prematurely
terminated. There is scope for those deciding whether or not to make
additional funds available (to allow a trial to continue to its planned
end) to be (consciously or unconsciously) influenced by having some idea of
how the results are developing.
Kind Regards,
Ted Harding
> At 08:15 21/01/06 +0000, Ted Harding wrote (in part):
>>With regard to your case, though, the fact that you cannot tell
>>from the publication means that the publication does not clear
>>up the legitimate questions relevant to whether the results
>>can be validly interpreted at face value, and therefore the
>>publication is scientifically inadequate.
>
> True, but as I have suggested, even if the publication DID
> include statements which appeared to indicate that the results
> could be 'validly interpreted at face value', it would be
> extraordinarily difficult (I guess a statistician should not
> say 'impossible'!) to be sure that those statements were true,
> even if they were written in good faith.
>
> In those situations in which blinding cannot fully conceal
> aspects of the emerging outcome, those managing trials may be
> influenced 'subconsciously', even if not 'consciously and
> deliberately'. Even 'Running out of money' is not as straightforward
> as it may first sound. Trials very often go 'over budget', but
> that does not necessarily cause them to be prematurely terminated.
> There is scope for those deciding whether or not to make additional
> funds available (to allow a trial to continue to its planned end)
> to be (consciously or unconsciously) influenced by having some
> idea of how the results are developing.
I don't disagree with any of this! In practice the issue of
influence of "how the results are developing" on decisions
about whether the bring the trial to a halt is likely to be
a "grey area". I was originally only spelling out conditions
necessary for treating the outcomes up to that stage as giving
a valid basis. To approach "sufficient condition" status they
would need to be supported by substantial convincing detail
on the circumstances, and even then, as you say, it would
be a matter of judgement as to how convincing one found them
to be.
One cannot avoid a suspicion that, when a decision is taken
that no further funding is available, it may have seemed
that it might be opportune to stop there since further
work might take the edge off the results obtained. And,
I think, it was mentioned whether the decision was taken
(or conmfirmed) before or after seeing the results!
But apparently the publication does not even refer to these
issues, saying only, in effect, "the money ran out". Whatever
the truth may be about what really went on, the absence of
any discussion seriously devalues the publication.
Best wishes,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 23-Jan-06 Time: 14:34:26
------------------------------ XFMail ------------------------------
John Whittington
>Thanks to John Whittington for those comments.
>As detailed in my previous reply (Thu 19/01/2006) to comments, the
>problem, for the trial under scrutiny, was precisely that it was
>impossible to know from the publication.
Indeed, and as I've just stressed in my response to Ted's message, even if
the publication DID appear to provide the information one required, it
would be extremely difficulty to know to what extent one could trust that
information.
>Thus the paradox: in the pharmaceutical industry there are attendant
>penalties (with regard to regulatory bodies) for falling off ones' horse
>before the finishing line, as it were; yet in non-Pharma trials one can
>apparently remount with a lesser penalty, and clinicians being
>clinicians, be awarded the prize regardless.
>Or am I overstating the case?
I'm not sure whether I would call it 'overstating the case', but you may
possibly be over-estimating those 'penalties'. In the present context, if
one can convince a regulatory body that a premature termination of a trial
had occurred for reasons that did not introduce bias, then they would
usually be happy to consider the available data. Indeed (and this is a
whole different can of worms to discuss!), it's far from uncommon these
days for Data Monitoring Committees (or whatever they happen to be called)
to terminate trials prematurely for reasons which are very much 'outcome
related' and regulatory authorities will, again, usually be prepared to
consider the available data.
John Whittington
>I was originally only spelling out conditions
>necessary for treating the outcomes up to that stage as giving
>a valid basis. To approach "sufficient condition" status they
>would need to be supported by substantial convincing detail
>on the circumstances, and even then, as you say, it would
>be a matter of judgement as to how convincing one found them
>to be.
Exactly, and when one realises that some of the 'influences' may not have
'subconscious', there may not always be any rational basis for that judgement.
>One cannot avoid a suspicion that, when a decision is taken
>that no further funding is available, it may have seemed
>that it might be opportune to stop there since further
>work might take the edge off the results obtained.
Indeed so - or, perhaps even more commonly, because it appears that 'good'
results will not be obtained even if the trial continues to
completion. However, at least that will not result in 'false positive'
findings.
>And, I think, it was mentioned whether the decision was taken
>(or conmfirmed) before or after seeing the results!
Sure, but as I keep saying, unavoidable 'indicators of outcome' may be
present even before one 'looks at the results'. The fact that a decision
was taken 'before seeing the results' (in the normal literal sense)
therefore is not enough.
>But apparently the publication does not even refer to these
>issues, saying only, in effect, "the money ran out". Whatever
>the truth may be about what really went on, the absence of
>any discussion seriously devalues the publication.
Agreed. I was merely warning that, even if the publication appeared to
contain 'a full discussion' about these matters, one would STILL have to be
cautious.
Moran, John (NWAHS)
discussion on this vexed question
What I sense from this discussion is that the "reasons" for "early
stopping" in cases such as these may be tantamount to "looking" at the
data.
Is it appropriate then to mandate formal statistical penalties (in terms
of P values and / or estimates) as occur in up-front interim analyses?
Best wishes
John moran
John Moran
Department of Intensive Care Medicine
The Queen Elizabeth Hospital
28 Woodville Road
Woodville SA 5011
Australia
Tel: 61 08 8222 6463
Fax 61 08 8222 6045
E-mail: john....@nwahs.sa.gov.au
-----Original Message-----
From: MedS...@googlegroups.com [mailto:MedS...@googlegroups.com] On
Behalf Of John Whittington
Sent: Tuesday, 24 January 2006 1:18 AM
To: MedS...@googlegroups.com
Subject: MEDSTATS: Re: Implications of "early stopping" of randomized
Ted Harding
>
> Thanks again to John Whittington and Ted Harding for the
> continued discussion on this vexed question
>
> What I sense from this discussion is that the "reasons"
> for "early stopping" in cases such as these may be tantamount
> to "looking" at the data.
May be, but also may not be! I think the tenor of what both
John and I have said is that the situation in this respect
is undetermined. If the data were "looked at", this does not
appear from the available information. If they were not
looked at, this still does not appear from the available
information and, even it (as John pointed out) it had been
formally stated there would still be some question about
how securely one could trust such a statement (even if made
in good faith).
However, it seems that no statement either way has been
made, so we simply do not know.
> Is it appropriate then to mandate formal statistical penalties
> (in terms of P values and / or estimates) as occur in up-front
> interim analyses?
That would be tantamount to acting as if the data had been
"looked at". This could possibly be justified on a "precautionary
principle": in the case of danger, protect yourself against the
worst that could happen-- the Conservative attitude. An opposite
attitude could be to trust what is said "at face value" -- if you
like, the Principle of Non-belief in Conspiracy Theories -- the
Liberal or Optimistic attitude.
However, both of these fall into what I classify as "political
grounds": it is Type I error vs Type II error in a different guise.
Evaluate (on grounds which include non-scientific considerations)
what the detriments of different consequences may be, what
protection you want against these respective detriments, and decide
accordingly.
So: (A) you may adopt the Conservative attitude, and discount the
evidence in favour of the treatment as reported in the publication;
then you risk overlooking a beneficial treatment and also the
time and cost of the research have been wasted, if your conservatism
was misplaced. Or: (B) you may adopt the Liberal attitude, and accept
the study at face value; then you risk being misled (either by intention
or inadvertence on the part of the people who did the study) into
accepting something which may be worthless or even detrimental,
with the consequences that this may lead to, if your optimism
was misplaced.
As to the likelihood that either conservatism or optimism might
be misplaced, it seems from the available information that you
simply do not know.
So perhaps there is a third alternative: retain an open mind,
on the grounds that the publication is inadequate. Perhaps (again
for "political" reasons) this may not be an option!
However, you can also take into account any information (which
would be additional to the reported information in the study)
which may give you prior reasons to suspect that the truth
goes one way or the other.
Have the group who made the study been meticulous and manifestly
honest in the past? (I.e. do they have a good reputation)?
Do the group in question have slightly, or somewhat, or even
definitely, a "dodgy" reputation?
Is there internal evidence, in the way the study had been
reported, that either (i) the group are not careful and
meticulous, or even (ii) that something may have been
deliberately swept under the carpet?
For what it is worth (and it's only my $0.02 at most), I would
expect a competent and honest team writing up the study to have
done so with some care, and to have thought of the doubts that
can arise in such circumstances, and therefore to have explicitly
presented the considerations which would help to resolve the
doubts. It seems that this was not done, which suggests either
that they did not act with full competence, or that they have
not been fully open. But note "suggests" -- i.e. tends to tip
the balance in that direction.
There is of course also the possibility that whoever wrote up
the publication was not in possession of the full facts as
available to the researchers running the trial, but was in
the position (again a "political" factor) of being obliged to
write it up as best they could, without the help and information
they should normally expect from the trial coordinators.
Sorry to leave things apparently hanging in the air, but that
does seem to be their rightful place!
Best wishes,
Ted.
--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@nessie.mcc.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 24-Jan-06 Time: 10:31:59
------------------------------ XFMail ------------------------------
John Whittington
>What I sense from this discussion is that the "reasons" for "early
>stopping" in cases such as these may be tantamount to "looking" at the
>data.
Yes, but with the emphasis on 'may'. As Ted has said, I imagine that most
researchers are very decent about such things, and will probably even
consider and discuss the possibility that they have been subconsciously
influenced by indicators of 'outcome'.
>Is it appropriate then to mandate formal statistical penalties (in terms
>of P values and / or estimates) as occur in up-front interim analyses?
I really have nothing much to add to what Ted has said about this. As he
has said, that would be the approach (which Ted calls 'conservative') which
gives added protection against Type I errors at the cost of an increased
risk of Type II Errors. Whether that is necessarily 'better', or even
appropriate, will depend upon circumstances and viewpoint. It is not at
all difficult to think of situations in which the consequences of a Type II
Error are potentially more detrimental to mankind than a Type I Error -
particularly if that Type II error results in failure to further
investigate/develop a treatment which could be very valuable.
Moran, John (NWAHS)
On balance, I think that the Conservative option has merit on the basis
that there are many (professional) enthusiasts out there whose dictum
seems to be "this must work".
I thus prefer a stiff Type I error and like the recent advice given in a
Well-Read-Journal: "Don't just do something, stand there"
Much obliged again for your thoughts
Doug Altman
recommendations of the CONSORT Statement (www.consort-statement.org).
The CONSORT checklist includes:
"How sample size was determined and, when applicable, explanation of any
interim analyses and stopping rules."
In other words, if interim analyses were done this should be reported.
It also says:
"Describe protocol deviations from study as planned, together with reasons."
Authors thus ought to include reasons why the sample size was well sort of
that planned (that number being required in the methods section).
There is of course no way of knowing that interim analyses were done if a
paper does not report this. It is disturbing that some articles describing
RCTs mention having a data monitoring committee but say nothing about
interim analyses, yet these usually coexist.
(A review of trials stopping early for benefit makes fascinating reading:
Montori VM, et al. Randomized trials stopped early for benefit: a
systematic review. JAMA 2005;294:2203-9.)
Also of possible interest to this group, an extension of CONSORT to
noninferiority and equivalence trials is in press in JAMA and will appear
within the next 2 months.
Doug
_____________________________________________________
Doug Altman
Professor of Statistics in Medicine
Centre for Statistics in Medicine
Wolfson College Annexe
Linton Road
Oxford OX2 6UD
email: doug....@cancer.org.uk
Tel: 01865 284400 (direct line 01865 284401)
Fax: 01865 284424
Web: http://www.csm-oxford.org.uk/
John Whittington
>On balance, I think that the Conservative option has merit on the basis
>that there are many (professional) enthusiasts out there whose dictum
>seems to be "this must work".
That's certainly one viewpoint, but I personally think that it may be
unnecessarily harsh in what may well be the great majority of situations,
in which nothing sinister or inappropriate has taken place. When
researchers with a good reputation 'say all the right things' about a
prematurely-terminated study, I am probably inclined to give that study
'the benefit of the doubt', even though I have argued myself that there
still remains some degree of 'risk'. To impose a 'statistical penalty' on
a study whose power has already been reduced by a failure to achieve the
target sample size is obviously increasing further the risk of a Type II error.
Whilst I would not deny that there are undoubtedly both individual and
corporate instances of the "this must work" phenomenon, I also think that
one has to bear in mind how serious can be the consequences of Type II
errors, with the potential to cause more human suffering, and even death,
than Type I errors. If, as a result of a Type I error, an ineffective
treatment comes into use, that ineffectiveness will usually become apparent
pretty quickly. On the other hand, if a Type II error results in
abandonment of development of a valuable treatment, then mankind might be
denied that treatment 'for evermore'.
John Whittington
>Many journals expect authors to report trials according to the
>recommendations of the CONSORT Statement (www.consort-statement.org).
>... "How sample size was determined and, when applicable, explanation of
>any interim analyses and stopping rules."
>"Describe protocol deviations from study as planned, together with reasons."
>Authors thus ought to include reasons why the sample size was well sort of
>that planned (that number being required in the methods section).
Indeed so - but, as we have been discussing, the question remains as to how
one should regard and handle data from an 'incomplete' trial, even when the
circumstances surrounding the 'incompleteness' are fully documented and
discussed in a report or publication.
I know I am somewhat repeating myself, but I think the importance of the
issue is a justification. I am probably close to being unique in terms of
the membership of this group, in that I have been on the investigator's
side of the fence countless times in the past, and I have a suspicion that
the statistical community is fairly unaware of how relatively common it is
for investigators (and others close to a RCT) to have at least some degree
of awareness/ suspicion about emerging outcomes, even when 'double blind'
conditions are rigorously applied, in the ABSENCE of any 'interim analysis'.
Moran, John (NWAHS)
Although I take the point re 'the benefit of the doubt', as applied to
researchers, and the detriments of a Type II error, I see little
evidence (in clinical practice) of: " If, as a result of a Type I error,
an ineffective treatment comes into use, that ineffectiveness will
usually become apparent pretty quickly. "
That clinicians continue to use therapies which are "ineffective" (ie,
don't work) has been attested too all too frequently. Hence the
preference for heavy Type I!
Best wishes
John Moran
Department of Intensive Care Medicine
The Queen Elizabeth Hospital
28 Woodville Road
Woodville SA 5011
Australia
Tel: 61 08 8222 6463
Fax 61 08 8222 6045
E-mail: john....@nwahs.sa.gov.au
-----Original Message-----
From: MedS...@googlegroups.com [mailto:MedS...@googlegroups.com] On
Behalf Of John Whittington
Sent: Wednesday, 25 January 2006 10:59 PM
To: MedS...@googlegroups.com
Subject: MEDSTATS: Re: Implications of "early stopping" of randomized
denied that treatment 'for evermore'.
John Whittington
>Although I take the point re 'the benefit of the doubt', as applied to
>researchers, and the detriments of a Type II error, I see little
>evidence (in clinical practice) of: " If, as a result of a Type I error,
>an ineffective treatment comes into use, that ineffectiveness will
>usually become apparent pretty quickly. "
>That clinicians continue to use therapies which are "ineffective" (ie,
>don't work) has been attested too all too frequently. Hence the
>preference for heavy Type I!
'Non-evidenced-based' behaviour of clinicians is a whole subject of its
own, but I get the impression that we are probably talking about drug
treatments, in which respect clinicians are very largely constrained by
what drugs are developed and brought to market by pharmaceutical
companies. In relation to those 'treatments', and perhaps contrary to
common perceptions, I frankly doubt that you could find many drugs licensed
during, say, the past 30 years that have been truly 'ineffective' (i.e.
don't work).