"Interim analyses and statistical penalties"

[John Whittington]

Hi folks,

If an interim analysis of a trial is undertaken with the explicit
understanding that the trial will continue to its planned end, regardless
of the findings of the interim analysis, is there any reason why this
should incur a "statistical penalty" in relation to the final analysis? On
the face of it, I can't see why it should. One might also ensure that
trial-related personnel are not exposed to results of in interim analysis,
lest that might in some way bias the conduct of the ongoing trial.

Taking that question one step further, what if the understanding is that
the trial will continue to its planned end if results are "looking
promising" but that the trial might be curtailed if it looks as if it is
"going nowhere" ('futility')? Whilst I can understand that premature
termination of the trial would increase the chances of a Type II error
(i.e. the result might possibly have 'become positive' had the trial
continued), again I can see no obvious reason why such an interim analysis
would affect the Type I error in the final analysis, such that none of the
alpha is 'spent' by the interim analysis.

I strongly suspect that there is something wrong with my reasoning above -
but I can't so far work out what!

Kind Regards,

John

----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225
Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
Buckingham MK18 4EL, UK
----------------------------------------------------------------

[Swank, Paul R]

If you test the data at interim and at the end, you have done two tests, which increases the probability of a type I error unless the two tests are perfectly correlated.

Paul Swank

Sent from my iPad

> --
> --
> To post a new thread to MedStats, send email to MedS...@googlegroups.com .
> MedStats' home page is https://urldefense.proofpoint.com/v2/url?u=http-3A__groups.google.com_group_MedStats&d=BQIBaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=TJ-J8OqnBAHE_LBDjl6ChV75FrcVWN12JgpJ_BxOW3A&s=70doMWuhDLQ5Zanei6kHOy62IMh817Nj2vIS7sZTfIM&e= .
> Rules: https://urldefense.proofpoint.com/v2/url?u=http-3A__groups.google.com_group_MedStats_web_medstats-2Drules&d=BQIBaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=TJ-J8OqnBAHE_LBDjl6ChV75FrcVWN12JgpJ_BxOW3A&s=NBqJVYW5iVhUggJwSXLL_LuH_nIL5b9vb5AgB0MWhOw&e=
> --- You received this message because you are subscribed to the Google Groups "MedStats" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to medstats+u...@googlegroups.com.
> For more options, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=BQIBaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=TJ-J8OqnBAHE_LBDjl6ChV75FrcVWN12JgpJ_BxOW3A&s=eqrv7h4sszQVwLUi4JAPq_7qRko-2MvpxKW_Zzgp9hw&e= .

[John Whittington]

At 17:10 05/07/2015 +0000, Swank, Paul R wrote:
>If you test the data at interim and at the end, you have done two tests,
>which increases the probability of a type I error unless the two tests are
>perfectly correlated.

Yes, I realise that, but it only seems to have meaning/relevance if one is
going to consider the result of the first analysis once the final analysis
is undertaken (or take some action as a result of the interim analysis). If
one is simply going to 'forget about' the interim analysis, continue to the
end of the trial and then conduct the planned analysis, I don't see that
the probability of a Type I error (in that final analysis) can be any
different from what it would have been had the interim analysis not been
undertaken.

[Swank, Paul R]

If you do two significance tests (not perfectly correlated), each at the .05 level, the probability of a type I error is greater than .05 regardless of whether or not you take action. Why would the probability of a false hypothesis being rejected depend on what you do with the information?

Paul R. Swank, Ph.D., Professor
Health Promotions and Behavioral Sciences
School of Public Health
University of Texas Health Science Center Houston
________________________________________
From: meds...@googlegroups.com [meds...@googlegroups.com] on behalf of John Whittington [Joh...@mediscience.co.uk]
Sent: Sunday, July 05, 2015 12:36 PM
To: meds...@googlegroups.com
Subject: Re: {MEDSTATS} "Interim analyses and statistical penalties"

--
--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .

MedStats' home page is https://urldefense.proofpoint.com/v2/url?u=http-3A__groups.google.com_group_MedStats&d=BQIBaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=-Ri5F4zaT1h24PWdUYH25edpqDfnz5yTMcZc8i-X8ug&s=NlCh4lL29V7fVSdWXlQGYWfFV5R232KDK4kpMc3myVM&e= .
Rules: https://urldefense.proofpoint.com/v2/url?u=http-3A__groups.google.com_group_MedStats_web_medstats-2Drules&d=BQIBaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=-Ri5F4zaT1h24PWdUYH25edpqDfnz5yTMcZc8i-X8ug&s=W4PTpYPQFtIfXNwSj39wZ8eEfu79ykwNLGpvbKPODWA&e=

---
You received this message because you are subscribed to the Google Groups "MedStats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medstats+u...@googlegroups.com.

For more options, visit https://urldefense.proofpoint.com/v2/url?u=https-3A__groups.google.com_d_optout&d=BQIBaQ&c=6vgNTiRn9_pqCD9hKx9JgXN1VapJQ8JVoF8oWH1AgfQ&r=8frmz39BMbPfozSCry7R2XF1zD3P8iT3dTcbzh5VWc8&m=-Ri5F4zaT1h24PWdUYH25edpqDfnz5yTMcZc8i-X8ug&s=oJ4YjfYiWKTU7WZFYAYZE-fUnt5DIC4iGu3HtdqNcpk&e= .

[John Sorkin]

John,

I feel some sympathy for your plight. If, for example the interim test were done by a third party, who did not notify anyone of the results of the interim analysis would it be necessary to penalize the "official" statistician's analyses? 

John

John David Sorkin M.D., Ph.D.
Professor of Medicine
Chief, Biostatistics and Informatics
University of Maryland School of Medicine Division of Gerontology and Geriatric Medicine
Baltimore VA Medical Center
10 North Greene Street
GRECC (BT/18/GR)
Baltimore, MD 21201-1524
(Phone) 410-605-7119
(Fax) 410-605-7913 (Please call phone number above prior to faxing)

>>> John Whittington <Joh...@mediscience.co.uk> 07/05/15 12:59 PM >>>

Hi folks,

If an interim analysis of a trial is undertaken with the explicit
understanding that the trial will continue to its planned end, regardless
of the findings of the interim analysis, is there any reason why this
should incur a "statistical penalty" in relation to the final analysis? On
the face of it, I can't see why it should. One might also ensure that
trial-related personnel are not exposed to results of in interim analysis,
lest that might in some way bias the conduct of the ongoing trial.

Taking that question one step further, what if the understanding is that
the trial will continue to its planned end if results are "looking
promising" but that the trial might be curtailed if it looks as if it is
"going nowhere" ('futility')? Whilst I can understand that premature
termination of the trial would increase the chances of a Type II error
(i.e. the result might possibly have 'become positive' had the trial
continued), again I can see no obvious reason why such an interim analysis
would affect the Type I error in the final analysis, such that none of the
alpha is 'spent' by the interim analysis.

I strongly suspect that there is something wrong with my reasoning above -
but I can't so far work out what!

Kind Regards,

John

----------------------------------------------------------------

Dr John Whittington, Voice: +44 (0) 1296 730225+44 (0) 1296 730225

Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
Buckingham MK18 4EL, UK
----------------------------------------------------------------

--
--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .

MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules

---
You received this message because you are subscribed to the Google Groups "MedStats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medstats+u...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

Call

Send SMS

Call from mobile

Add to Skype

You'll need Skype CreditFree via Skype

Confidentiality Statement:

This email message, including any attachments, is for the sole use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message.

[Munyaradzi Dimairo]

On 5 July 2015 at 17:59, John Whittington <Joh...@mediscience.co.uk> wrote:

Hi folks,

If an interim analysis of a trial is undertaken with the explicit understanding that the trial will continue to its planned end, regardless of the findings of the interim analysis, is there any reason why this should incur a "statistical penalty" in relation to the final analysis? 

If the interim analysis is meant to modify some future aspects of the design then you will incur a statistical penalty unless the test statistics before and after the interim analysis are weighted accordingly. This is not the case if you don't change any aspects of the design such as sample size or treatment effects or number of treatment arms.

 

On the face of it, I can't see why it should.  One might also ensure that trial-related personnel are not exposed to results of in interim analysis, lest that might in some way bias the conduct of the ongoing trial.

Taking that question one step further, what if the understanding is that the trial will continue to its planned end if results are "looking promising" but that the trial might be curtailed if it looks as if it is "going nowhere" ('futility')?  Whilst I can understand that premature termination of the trial would increase the chances of a Type II error (i.e. the result might possibly have 'become positive' had the trial continued), again I can see no obvious reason why such an interim analysis would affect the Type I error in the final analysis, such that none of the alpha is 'spent' by the interim analysis.

This approach is based on defining various zones of decision making criteria with an option to change the effect size at interim. For instance, you may design a trial with assumed treatment effect of 5, but you also believe there is uncertainty around that for some reasons. At interim, a sponsor may be willing to continue the trial even when the treatment effect is 3 and increase the sample size to show statistical significance under this effect based on conditional power. There are some cases in pharmaceutical drug development where it can be useful. Competitors may have a drug with effectiveness of 2.8 say, so a company may just want to surpass that. It obvious in this case that the design has been changed. The test statistics before and after the modification are different (both information fraction and alternative treatment effects). The only way to control type I and II errors here is to weight the data and use p-value combination tests. See VALOR phase III Trial by Cytel (http://onbiostatistics.blogspot.co.uk/2014/11/valor-trial-successful-and-failed-phase.html)

Mehta, C. R. & Pocock, S. J. (2011) Adaptive increase in sample size when interim results are promising: a practical guide with examples. Statistics in medicine. 30 (28), 3267–3284

I strongly suspect that there is something wrong with my reasoning above - but I can't so far work out what!

Kind Regards,

John

----------------------------------------------------------------
Dr John Whittington,       Voice:    +44 (0) 1296 730225
Mediscience Services       Fax:      +44 (0) 1296 738893
Twyford Manor, Twyford,    E-mail:   Joh...@mediscience.co.uk
Buckingham  MK18 4EL, UK
----------------------------------------------------------------

--
--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules

--- You received this message because you are subscribed to the Google Groups "MedStats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medstats+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

"Learn from yesterday, live for today, hope for tomorrow. The important thing is not to stop questioning" ~ Albert Einstein

***********************************
Munyaradzi Dimairo
NIHR Research Fellow in Medical Statistics
University of Sheffield
School of Health and Related Research (ScHARR)
Clinical Trials Research Unit
Regent Court, 30 Regent Street
Sheffield
S1 4DA

Physical address:
Room 3.10, Innovation Centre

email: m.di...@sheffield.ac.uk 
          mdim...@gmail.com
Twitter: @mdimairo [views are my own & retweeting is not endorsement]

Tel: +44 (0) 114 222 25204

Mobile: 07531421509
Link to my publications
***********************************

[John Whittington]

At 19:12 05/07/2015 +0000, Swank, Paul R wrote:
>If you do two significance tests (not perfectly correlated), each at the
>.05 level, the probability of a type I error is greater than .05
>regardless of whether or not you take action.

I think I must be being dim. I would say that is only true if one's
decision as to whether to reject the null hypothesis and/or a decision not
to complete the trial in some way involved the result of the first test.

> Why would the probability of a false hypothesis being rejected depend on
> what you do with the information?

If the result of the first analysis played no part in the decision as to
whether or not to reject the hypothesis, and did not in any way alter the
conduct of the trial or the 'final analysis', then I really can't see how
it can effect the rejection, or otherwise, of the null
hypothesis. Consider a silly extreme example of what I'm talking about ...
an individual, or small group of individuals, undertakes an interim
analysis of an ongoing trial. All records relating to that analysis are
then destroyed, and all individuals who saw any of the results are
shot. The trial continues to its planned end and the planned analysis is
undertaken. How can the probability of a Type I error in that planned
analysis be in any way affected by what did or did not emerge from the
interim analysis??

[John Whittington]

At 15:34 05/07/2015 -0400, John Sorkin wrote:
>John, I feel some sympathy for your plight. If, for example the interim
>test were done by a third party, who did not notify anyone of the results
>of the interim analysis would it be necessary to penalize the "official"
>statistician's analyses?

Exactly. Slightly less dramatic/draconian than my suggestion of shooting
anyone who had seen results of the interim analysis, but the same concept!!

Kind Regards,


John

----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225

[John Whittington]

At 20:47 05/07/2015 +0100, Munyaradzi Dimairo wrote:

If the interim analysis is meant to modify some future aspects of the design then you will incur a statistical penalty unless the test statistics before and after the interim analysis are weighted accordingly. This is not the case if you don't change any aspects of the design such as sample size or treatment effects or number of treatment arms.

That's precisely what I thought - it is the latter of those scenarios I am talking about.

This approach is based on defining various zones of decision making criteria with an option to change the effect size at interim.

This situation is obviously a bit more complicated, but I'm not talking about a situation in which anything, even 'expectations' of treatment effect would be changed - but a situation in which a sponsor might decide to curtail a trial because it was fairly obvious (from interim analysis) that the 'expected/desired' treatment effect was not going to be seen.  If the trial had been continued to its planned end, despite the poor outcome, it's almost inevitable that the final analysis would fail to reject the null hypothesis, so the question of the probability of a Type I error would not really arise.  On the other hand, if the results were such as to not cause the trial to be curtailed, I can't really see how that could increase the probability of a Type I error in the final analysis ... so what am I missing?

[Greg Snow]

John,

I have a little different take on this from everyone else that has responded. Yes, making any type of decision based on an interim analysis will modify the type I error, but in the case that you specify it will actually reduce (not increase) the type I error rate. Your thoughts are correct that it cannot increase the type I error rate since it is impossible for the interim analysis to reject the null (or cause the null to be rejected later when it would not have been without the extra analysis). But what could happen is that a situation could occur where the full data would reject the null hypothesis, but the interim analysis would cause you to stop for futility, so the null would never be rejected. If the null is false then this increases the type II error rate as you state, but this can occasionally occur when the null is true and it would therefore reduce the type I error rate.

The result is that you would either not need a "statistical penalty" since it would become conservative and you are happy with the lower than stated type I error rate. Or you could impose a statistical anti-penalty and actually run the final test at a slightly higher alpha level to get the original type I error rate.

Since I (we) seem to be in the minority of responders, here is some evidence, a simulation in R code that checks at 50 observations (final n planned at 100) and stops for futility if the p-value is greater than 0.5:

out <- replicate(1000000, {
x <- rnorm(100)
res1 <- t.test(x[1:50])$p.value
if(res1 > 0.5) return(1)
t.test(x)$p.value
})

mean(out <= 0.05)
binom.test(sum(out<=0.05), length(out))

My run gave a type I error rate (when compared to the traditional alpha of 0.05) of about 0.0456 (with a confidence interval clearly below 0.05). The simulation can be easily modified to stop at different places and to use a different cut-off for futility. But anyone arguing for the need of a statistical penalty will need to find a combination that results in a type I error rate greater than 0.05 (or whatever the choice of alpha level is), a task that I believe to be impossible.

So, to summarize: Yes it will affect your type I error rate, but not in any way that will require a statistical penalty.


Greg Snow
Statistician
Statistical Data Center | Intermountain Healthcare
Eight Avenue and C Street
Salt Lake City, UT 84143

Office: 801.408.8111

Fax: 801.408.2120 | greg...@imail.org

[John Whittington]

At 21:52 05/07/2015 +0000, Greg Snow wrote:
>John, I have a little different take on this from everyone else that has
>responded. Yes, making any type of decision based on an interim analysis
>will modify the type I error, but in the case that you specify it will
>actually reduce (not increase) the type I error rate. Your thoughts are

>correct that it cannot increase the type I error rate since .... If the

>null is false then this increases the type II error rate as you state, but
>this can occasionally occur when the null is true and it would therefore

>reduce the type I error rate. ... So, to summarize: Yes it will affect

>your type I error rate, but not in any way that will require a statistical
>penalty.

I'm glad that at least someone agrees with me - and I confess I hadn't
thought of the possibility that it might actually _reduce_ the risk of Type
I error.

However, what worries me is how widespread is the strong assertion of the
opposite viewpoint - which, as you say, we've seen a small taste of in this
discussion. In particular, I wonder, in the light of so much apparent
'expert assertion' there is to the contrary, how easily I would be able to
convince a drug regulatory authority that neither of the scenarios I have
described would/should attract any statistical penalty (or 'worse' - like
'invalidation' of the entire study/analysis!).

[Sreenivas.V]

Deal all Medstatters

I am glad John has raised a question that has been lingering in my mind for so long.  

In my work, I have been following what John has suggested - no statistical penalty for any interim analysis, if the outcome of interim analysis has no bearing in continuation of the trial.

I also feel that any statistician involved in the trial coordination has to do 'interim analysis' in some sort or other to monitor the trial, to prepare commands to execute the analysis plan and to test such program as to its execution etc. Practically, there will be no trial where the data will be looked at only after the trial is over.

It will be interesting the views of our other friends.

Best wishes

Sreenivas

--
--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules

--- You received this message because you are subscribed to the Google Groups "MedStats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medstats+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Professor
Department of Biostatistics
All India Institute of Medical Sciences
New Delhi 110029

[Munyaradzi Dimairo]

Hi John

There is very good statistical literature on this based on partitioning the error rates ... ref page 2756 - first reference below. The penalty is there - you take a small hit on power at the end of the study bounded by (beta/(1-futility threshold)) assuming `sequential' looks, but with reduced type I error rate.  Of course you could evaluate these exact rates numerically.

Lachin, J. M. (2005) A review of methods for futility stopping based on conditional power. Statistics in medicine. 24 (18), 2747–2764.

Lachin, J. M. (2009) Futility interim monitoring with control of type I and II error probabilities using the interim Z-value or confidence limit. Clinical trials (London, England). 6 (6), 565–573.
Lan, K. G. et al. (1982) Stochastically curtailed tests in long–term clinical trials. Sequential Analysis. (May 2013), 37–41.

bw

Munya

--
--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules

--- You received this message because you are subscribed to the Google Groups "MedStats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medstats+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Martin Holt]

In the past, in this sort of scenario, Martin Bland has suggested that the interim look see be carried out by a third, independent person. If I understood him correctly, that could be done without spending alpha.

Best, Martin

 

Martin P. Holt

Freelance Medical Statistician and Quality Expert

martin...@yahoo.com

Persistence and Determination Alone are Omnipotent !

If you can't explain it simply, you don't understand it well enough.....Einstein



Linked In: https://www.linkedin.com/profile/edit?trk=nav_responsive_sub_nav_edit_profile

---

From: John Sorkin <JSo...@grecc.umaryland.edu>
To: meds...@googlegroups.com; Joh...@mediscience.co.uk
Sent: Sunday, 5 July 2015, 20:34
Subject: Re: {MEDSTATS} "Interim analyses and statistical penalties"

[Munyaradzi Dimairo]

That is more to do with minimising operational bias when an unblinded interim analysis is conducted! 

[Marc Schwartz]

Hi all,

Just to throw my $0.10 in…

Given the scenario that John originally described, if the only decision that *might* be made early, is one of futility, you can have an independent DSMB-like committee be exposed to those results in executive session, without affecting other aspects of trial operations and without having to shoot anyone… :-)

Having faced that decision for a study where I am on the DSMB, based upon discussions and limited references, we determined that a futility decision would only be made if the conditional power of the study relative to the primary outcome, declined to below 0.1 (10%). We never got there, but we did get “close”.

References that might be helpful:

Data Monitoring in Clinical Trials
A Case Studies Approach
DeMets et al
2006
http://www.springer.com/us/book/9780387203300


Data Monitoring Committees in Clinical Trials: A Practical Perspective
Ellenberg et al
2002
http://www.wiley.com/WileyCDA/WileyTitle/productCd-0471489867.html


Regards,

Marc Schwartz

> On Jul 7, 2015, at 10:09 AM, 'Martin Holt' via MedStats <meds...@googlegroups.com> wrote:
>
> In the past, in this sort of scenario, Martin Bland has suggested that the interim look see be carried out by a third, independent person. If I understood him correctly, that could be done without spending alpha.
>
> Best, Martin
>
> Martin P. Holt
>
> Freelance Medical Statistician and Quality Expert
>
> martin...@yahoo.com
>
> Persistence and Determination Alone are Omnipotent !
>
> If you can't explain it simply, you don't understand it well enough.....Einstein
>
>
>
> Linked In: https://www.linkedin.com/profile/edit?trk=nav_responsive_sub_nav_edit_profile
>

[Francisco Ramirez]

Hi

Agree with Marc up to a point. If the investigators know that an
interim analysis is planned and the possibility exist to stop for
futility, the mere fact that the trial does not stop is a sign that
the trial is going to have very likely positive results. This thought
may affect how investigators and their personnel behave in ways very
difficult to plan. I once worked in one trial with a futility interim
analysis. Just after the interim analysis the treatment effect started
to drop. Our suspicious was that everybody was optimistic and became
more lenient about withdrawing patients as soon as there was a
complaint and even a change in how efficacy was assessed as there was
an element of subjectivity.This is difficult to prove but the data
certainly looked different before and after the interim analysis. So
my advice is only include interim analysis when you really need them.
Everybody is susceptible to good and bad news.

Regards

Francisco

[Martin Bland]

My view is that, if I am on a DMEC, I can and should do analyses if I think they will help us to safeguard patients.  If patients were being harmed, I would be prepared to stop a trial and I suppose we would then disclose the results to the investigators.  I would not do this otherwise, so I do not think that me doing a DMEC analysis would affect their analysis at all.  The investigators do not know what I am doing and should not even know what data I am looking at, though usually someone has to break the blind.  That might be me, it might be an independent statistician, it might even be the trial statistician.  The last is best avoided, though I have done it.  I did not tell anybody outside the DMEC what the analysis showed.

I think that most statisticians are more interested in the the results being correct than in the results, so I think most of us would find it easy to keep quiet.

Martin

--

***************************************************
J. Martin Bland
Prof. of Health Statistics
Dept. of Health Sciences
Seebohm Rowntree Building
University of York
Heslington
York YO10 5DD

Email: martin...@york.ac.uk
Phone: 01904 321334     Fax: 01904 321382
Web site: http://martinbland.co.uk/

Statement by the University of York:
This email and its attachments may be confidential and are intended solely for the use of the intended recipient. If you are not the intended recipient of this email and its attachments, you must take no action based upon them, nor must you copy or show them to anyone. Please contact the sender if you believe you have received this email in error. Any views or opinions expressed are solely those of the author and do not necessarily represent those of The University of York.
***************************************************

[Francisco Ramirez]

HI

I do not think you need to talk about the result to affect the results. Everything is specified in the protocol and you have to say that at some point the study is going to have a futility analysis. The investigators know that a futility analysis is going to occur and if after the interim analysis the study does not stop just by logic the investigators conclude that the results are positive. And this occur even if everybody keep quiet. 

If there is a DMC and no futility analysis is included in the protocol then I cannot see a problem either. The statistician can do as many analyses he thinks are necessary without nobody knowing what he is doing unless it is relevant to safeguard patients safety.

Everybody want the result being correct. There are certain assessment that have a subjective component (e.g. depression) and if the assessor and patient are influenced in one direction or the other it is difficult to know whether you can trust your results.

Regards

Francisco

[John Whittington]

At 15:21 07/07/2015 +0100, Munyaradzi Dimairo wrote:
>There is very good statistical literature on this based on partitioning
>the error rates ... ref page 2756 - first reference below. The penalty is
>there - you take a small hit on power at the end of the study bounded by
>(beta/(1-futility threshold)) assuming `sequential' looks, but with

>reduced type I error rate. Of course you could evaluate these exact
>rates numerically.

Indeed - I acknowledged that there would be a hit on power but, as I said,
could not see how Type I error rather could possibly be _increased_. Of
course, if one ends up rejecting the null, then the issue of power becomes
irrelevant.

[John Whittington]

At 18:03 07/07/2015 +0200, Francisco Ramirez wrote:
>I once worked in one trial with a futility interim analysis. Just after
>the interim analysis the treatment effect started to drop. Our suspicious
>was that everybody was optimistic and became more lenient about
>withdrawing patients as soon as there was a complaint and even a change in
>how efficacy was assessed as there was an element of subjectivity.

Yes, that can happen, and although it can have undesirable effects (e.g.
reducing power), if the trial is a blind RCT (and the blinding actually
'works'!), it's difficult to see how that can in any way affect/bias the
comparison between two things (treatments or whatever) which are being
compared in a blind fashion.

Kind Regards,


John

----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225

[Francisco Ramirez]

Hi

Maybe I did not explain well myself. By the way this is not something
that I suddenly thought. Many years ago I did a course about interim
analyses and the speaker, someone very experienced in DMC, warned
about this effect. I tried unsuccessfully to find references to
support my comments.

What I described may only happen in clinical trials (not agricultural
or chemical experiments) where you have human assessors and human
patients. To run a clinical trial you are required to have a protocol.
If you plan to run a futility analysis you have to include it in the
protocol. The investigators must read the protocol, therefore they
know that an interim analysis is going to be run and they also know
when. So if the date arrives and you run the futility analysis and the
study is not stopped it because the results were positive, otherwise
the study would have stopped, If the RCT is blind and everybody keeps
quiet as they should, the investigators still guess quite rightly that
the interim results were positive. I insist that they guess even if
nobody tell them anything and everybody is blind. They just need to
know that the study continues from some date, and they obviously know
that the study continues as they are involved on the study. If the
investigators believe that the drug is working, and maybe they even
talk about it with the patients, you have a potential situation that
bias can occur. If you have endpoints that are more subjective to
assess or psychological influenced (for example depression, joint
counts, ...) then you have a potential bias in the assessor and maybe
also in the patient. Even conduct can change after the interim
analysis, for example in the beginning investigators may tend to
convince patients to remain in the study if they have some minor
complaints but afterwards, as the drug is working well, they may feel
that there is no reason to bother those patients with minor
complaints.

The bias does not occur if nobody knows that an interim analysis is
being run. This only occur if you are the statistician in the DMC and
you have access to the data (as Martin Bland commented yesterday) and
you decide to run an analysis. This analysis was not included in the
protocol so investigators are not aware that an interim analysis is
being run.

In the study I thought this phenomenon occurred it was quite obvious
that the placebo effect increased quite sharply from the date of the
interim analysis. Patient withdrawal also increased from the date of
the interim analysis. So we ended up with a difference in favour of
active treatment but not enough power to show a significant result.

I hope this clarifies my comment.

Regards

Francisco

[John Whittington]

At 21:51 08/07/2015 +0200, Francisco Ramirez wrote:
>The investigators must read the protocol, therefore they know that an
>interim analysis is going to be run and they also know when. So if the
>date arrives and you run the futility analysis and the study is not
>stopped it because the results were positive, otherwise the study would
>have stopped, If the RCT is blind and everybody keeps quiet as they
>should, the investigators still guess quite rightly that the interim

>results were positive. ...

Yes, I understood what you were saying, and agree with all of the above,
but ....

>... If the investigators believe that the drug is working, and maybe they

>even talk about it with the patients, you have a potential situation that
>bias can occur.

That's the bit I find more difficult to understand, if it is a blind and
comparative (e.g. placebo-controlled) RCT. The fact, that investigators
etc., and maybe even patients, know that the interim results were
'positive' (in terms of an emerging active-placebo difference) may well
affect assessments, particularly subjective or semi-subjective
ones. However, it's much more difficult (for me) to see how that could
_differentially_ effect patients according to which treatment they were
having (but no-one knew which). You appear to be suggesting that such
effects might have more effect in a placebo-treated patients than
active-treated ones (which {or the converse} is about the only way I can
think of that bias of the active-placebo difference could arise) but I'm
not at all sure about that - don't forget that active-treated patients are
as susceptible to "placebo effect" as are placebo-treated ones.

Perhaps I am missing something?

[Francisco Ramirez]

the placebo effect increases for example; in our case the active
treatment did not change much after the interim analysis but the
placebo increased clearly

[John Sorkin]

It is important to remember that interim analyses do not necessarily demonstrate that a result is positive, but rather the study is not futile and not clearly harmful, i.e. that there is hope that the null hypothesis will be rejected at some pre-specified probability.

John 

John David Sorkin M.D., Ph.D.
Professor of Medicine
Chief, Biostatistics and Informatics
University of Maryland School of Medicine Division of Gerontology and Geriatric Medicine
Baltimore VA Medical Center
10 North Greene Street
GRECC (BT/18/GR)
Baltimore, MD 21201-1524
(Phone) 410-605-7119
(Fax) 410-605-7913 (Please call phone number above prior to faxing)

>>> Francisco Ramirez <pacoram...@gmail.com> 07/08/15 3:51 PM >>>

> Dr John Whittington, Voice: +44 (0) 1296 730225+44 (0) 1296 730225

> Mediscience Services Fax: +44 (0) 1296 738893
> Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
> Buckingham MK18 4EL, UK
> ----------------------------------------------------------------
>

--
--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules

---
You received this message because you are subscribed to the Google Groups "MedStats" group.
To unsubscribe from this group and stop receiving emails from it, send an email to medstats+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Whittington]

At 22:26 08/07/2015 +0200, Francisco Ramirez wrote:
>the placebo effect increases for example; in our case the active treatment
>did not change much after the interim analysis but the placebo increased
>clearly

As I implied, that rather surprises me, if the blinding was really working
since, as I said, "placebo effect" occurs with active treatment as well as
placebo treatment.

Of course, if there are ways in which the investigator and/or patients can
get a handle on what treatment they are probably on (by virtue of efficacy
and/or unwanted/incidental effects), then that's different. However, I
would almost expect it to work the other way around. If investigators or
patients knew that the interim results were showing a 'positive'
active-placebo difference, then those who did not appear to be getting (or
did not feel they were getting) much benefit from trial treatment could
well assume they were getting placebo, which would tend to reduce the
magnitude of placebo effect in them.

[John Whittington]

At 16:35 08/07/2015 -0400, John Sorkin wrote:

It is important to remember that interim analyses do not necessarily demonstrate that a result is positive, but rather the study is not futile and not clearly harmful, i.e. that there is hope that the null hypothesis will be rejected at some pre-specified probability.

.... or that the indication is that the null hypothesis may well get rejected, but on the basis of a magnitude of effect that would be too small to be of any clinical interest - e.g. if there was a very small magnitude of effect emerging, but with very small variability.

Kind Regards,

John

----------------------------------------------------------------

Dr John Whittington,       Voice:    +44 (0) 1296 730225

[Francisco Ramirez]

I agree with J. Sorkin, we should always be aware that the interim
analysis is only a probabilistic assessment and not certainty. The
case I discussed is difficult to explain by randomness. The change in
trend in various unrelated variables after the interim analysis was
too brisk to consider it random.

Regarding J. Whittington comments, you need a big sample size because
the changes are not dramatic and because placebo patients show some
effect anyway, so you have a big number of patients in different
stages of disease, and not all patients respond. So it is not clear
cut to guess who is in placebo or active. Why the active treatment did
not change much I do not know but I guess it is easier to observe a
minor change than a major change.

[John Whittington]

At 22:58 08/07/2015 +0200, Francisco Ramirez wrote:
So it is not clear cut to guess who is in placebo or active.

Indeed, but if they believe that an interim analysis is suggesting a
significant active-placebo difference, those who perceive themselves
(because of some apparent efficacy) as probably being on active therapy
might be encouraged by the interim analysis and therefore 'improve
further', whereas those who perceive themselves as probably being on
placebo may deteriorate.

[Francisco Ramirez]

yes, i do not have an answer for that but my guess is that is it
easier to see a small change that a large change; apparently in
depression it is quite hard to get a major improvement in this time
frame

[Martin Holt]

This thread is now very long...it would be good to have a summary of where we are right now.

John?

Francisco?

Best Wishes, 

Martin

 

Martin P. Holt

Freelance Medical Statistician and Quality Expert

martin...@yahoo.com

Persistence and Determination Alone are Omnipotent !

If you can't explain it simply, you don't understand it well enough.....Einstein



Linked In: https://www.linkedin.com/profile/edit?trk=nav_responsive_sub_nav_edit_profile

---

From: Francisco Ramirez <pacoram...@gmail.com>
To: John Whittington <Joh...@mediscience.co.uk>
Cc: meds...@googlegroups.com
Sent: Wednesday, 8 July 2015, 22:12

Subject: Re: {MEDSTATS} "Interim analyses and statistical penalties"

--
--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules

---
You received this message because you are subscribed to the Google Groups "MedStats" group.

To unsubscribe from this group and stop receiving emails from it, send an email to medstats+unsub...@googlegroups.com.

[John Whittington]

At 12:38 09/07/2015 +0000, 'Martin Holt' via MedStats wrote:
>This thread is now very long...it would be good to have a summary of where
>we are right now. John? Francisco?

As I see it, the summary is very simple/brief, but I would be grateful for
any corrections/disagreements from others:

1...In terms of my primary question, despite what is so commonly asserted,
I remain of the view that if the results of an interim analysis are not
made known to anyone involved with trial conduct and if the trial is going
to be continue to its planned conclusion, regardless of results of the
interim analysis, then there is no problem, and no 'statistical penalty'.

2...As for my supplementary question, if the results of the interim
analysis are not going to be made known to anyone involved with trial
conduct and the ONLY effect that the interim analysis may have on the trial
is to cause it be curtailed by reason of 'futility', then this does not
result in an increase in the risk of Type I error, but does produce a small
increase in Type II error.

3...It has been pointed out that if it is known that a 'futility' analysis
has been undertaken and the trial not been stopped as a result, this could
have an influence on patients and/or investigators for the remainder of the
trial, particularly in relation to subjective assessments.

As I said, observations, corrections or disagreements will be very welcome!

[Martin Holt]

Thanks, John.

Kind Regards,

Martin

 

Martin P. Holt

Freelance Medical Statistician and Quality Expert

martin...@yahoo.com

Persistence and Determination Alone are Omnipotent !

If you can't explain it simply, you don't understand it well enough.....Einstein



Linked In: https://www.linkedin.com/profile/edit?trk=nav_responsive_sub_nav_edit_profile

---

From: John Whittington <Joh...@mediscience.co.uk>
To: meds...@googlegroups.com
Sent: Thursday, 9 July 2015, 13:55

Subject: Re: {MEDSTATS} "Interim analyses and statistical penalties"