Re: {MEDSTATS} Re: p-value < 0.1 for important outcomes!

[Peter Flom]

mcap <mca...@yahoo.com> wrote
>>  
>> Recently, a medical doctor colleague sent me the following query:
>>  
>> “Do you have any stats or epidemiology papers that discuss justification of alpha and beta (p values and their interpretation), especially conditions that merit altering them from what is done conventionally? My recollection is that interventions with important primary outcomes (e.g. mortality) and minimal associated adverse effects may mean that one can change p<0.05 to p<0.1 so as not to conclude there are no differences between the arms”.
>

Look at the work of Jacob Cohen - he had one article called "The Earth is round! p < .05!".

Also, there is a book called "The Cult of Statistical Significance" - by McCloskey and Ziliak, that should provide some useful citations.

As others have said, focus on effect sizes and give exact p values.

Peter

Peter L. Flom, PhD
Statistical Consultant
Website: www DOT peterflomconsulting DOT com
Writing; http://www.associatedcontent.com/user/582880/peter_flom.html
Twitter: @peterflom

[John Whittington]

Whoops - please see correction below. Apologies. John

At 09:48 11/12/2009 -0600, Marc Schwartz wrote:
>It seems to me that if one actually gets to the end of the study, only
>then to realize that there was an insufficient number of subjects
>because the underlying assumptions of the study proved false, then it
>can be argued that to a large extent, there was a failure in the
>safety oversight and review process.
>
>There is an ethical obligation by the PI's and the independent IRBs/
>DMCs/DSMBs to periodically review the study and to reasonably ensure
>that the study has a chance to achieve it's primary objective. If that
>is unlikely, then where possible, they are to recommend changes that
>do not compromise the scientific validity of the study, or if not
>possible, to recommend that the study be stopped for futility.

I agree totally, but (a) we are talking about situations in which that has
not been done and (b) although I am (for these very reasons) a great
believer in sample size estimation
THAT SHOULD READ 'SAMPLE SIZE RE-ESTIMATION' !
and other forms of adaptive designs,
such designs remain (at least in my experience) rarities at
present. Goodness knows how many trials I have been involved with, or
aware of, over the past 30 years, but the number of them in which there was
any attempt to look in real time for errors in underlying assumptions
(particularly the variability of emerging results) could undoubtedly be
counted on the fingers of one hand.

Kind Regards,


John

----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225
Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
Buckingham MK18 4EL, UK
----------------------------------------------------------------

--
To post a new thread to MedStats, send email to MedS...@googlegroups.com .
MedStats' home page is http://groups.google.com/group/MedStats .
Rules: http://groups.google.com/group/MedStats/web/medstats-rules
John

----------------------------------------------------------------
Dr John Whittington, Voice: +44 (0) 1296 730225
Mediscience Services Fax: +44 (0) 1296 738893
Twyford Manor, Twyford, E-mail: Joh...@mediscience.co.uk
Buckingham MK18 4EL, UK
----------------------------------------------------------------

[Martin Holt]

On 11/12/2009, 15:49 Marc Scwartz wrote in part:

"Other the other hand, there is likely to be some threshold for sample
size increase....."

With the proviso's you stipulate 11/12/2009 16:00:

"Returning to trials, to re-iterate, my personal feeling is that the only
situation in which a post-hoc calculation makes sense is if the a priori
sample size had been undertaken using a variance estimate that was
appreciably lower than that observed in the trial AND the trial had
produced a 'non-significant' result. In that situation, I believe that it
can be valuable in helping one to decide 'what to do next'."

re-estimation of sample-size might not be valid if there is something out of
control in the trial, and a higher observed variance estimate was also
(maybe partly) responsible for the non-significant result. But then, when
re-estimation shows that something is wrong, I guess that is when you would
find it useful in deciding "what to do next".

Best Wishes,

Martin

[Doug Altman]

Here is a real example:

Cause of the non-conclusive results. The only truly uncertain
factor of the assumptions in the power calculation
was the SD of the duration of mechanical ventilation in
days. This was originally assumed to be 2 days, based on
the findings in a subgroup in an earlier, single-centre trial.
However, when analysing the data, this was found to be
an underestimate, since the empirical SD in the study population
in this multi-centre trial turned out to be 5.3 days.
Application of this value for the SD in the power calculation
would have resulted in a sample size of 196 instead of 30
per intervention group.

Source:
van der Lee JH, Tanck MW, Wesseling J, Offringa M.
Pitfalls in the design and analysis of paediatric clinical trials: a case of a 'failed' multi-centre study, and potential solutions.
Acta Paediatr 2009 Feb;98(2):385-91.


As noted, in this case the estimate came from one centre but the trial was conducted in multiple centres.

Note that this is not the only way in which sample size calculation can go wrong - similar issues apply with a binary endpoint if the event rate is much less than anticipated.

BW
Doug

_____________________________________________________

Doug Altman
Professor of Statistics in Medicine
Centre for Statistics in Medicine
University of Oxford
Wolfson College Annexe
Linton Road
Oxford OX2 6UD

email:  doug....@csm.ox.ac.uk
Tel:    01865 284400 (direct line 01865 284401)
Fax:    01865 284424
www:     http://www.csm-oxford.org.uk/

EQUATOR Network - resources for reporting research
www: http://www.equator-network.org/

[John Whittington]

Thanks for the example, Doug. Sticking on my band wagon, I would have
thought it was useful to know (as this paper appears to have pointed out)
that a sample size estimation based on a more appropriate SD would have
resulted in a 6-7-fold increase in the required number of subjects to
provide a study of adequate power. Do you disagree?

Kind Regards,
John

>www: <http://www.equator-network.org/>http://www.equator-network.org/

>
>
>
>--
>To post a new thread to MedStats, send email to MedS...@googlegroups.com .
>MedStats' home page is

><http://groups.google.com/group/MedStats>http://groups.google.com/group/MedStats
>.
>Rules:
><http://groups.google.com/group/MedStats/web/medstats-rules>http://groups.google.com/group/MedStats/web/medstats-rules

[Peter Flom]

John Whittington <Joh...@mediscience.co.uk> wrote
>In the real world, a post-hoc power calculation or sample size estimation
>might well be a major factor in determining whether ANY future studies were
>undertaken following a 'negative' result, or whether the treatment in
>question would be 'abandoned'. Such decisions are often largely in the
>hands of people who have no 'technical' (statistical, clinical etc.)
>expertise. If they are told that a study designed to have adequate power
>to detect an effect has failed to detect an effect, they may abandon the
>project. However, if they can be shown that, with the benefit of
>hindsight, it is apparent that the trial was not adequately powered for
>purpose, then they might be more inclined to move forward into more
>adequate trials.
>

Doesn't this get into the power analysis for the NEXT grant?

That is, suppose you do a small study, and find a certain effect size, variance and so on;
it is large enough to be of practical importance, but not stat sig.

Then this should form the basis of the next application - that is, GIVEN a similar effect size,
we would need N people.

[Marc Schwartz]

If I am correctly understanding your context Peter, this is the reason
that small pilot studies are performed. They are not intended to
achieve statistical significance, but only to enable a reasonable
estimate of those factors that are relevant to designing the
subsequent powered study, where there is no a priori data available
otherwise.

However, it is made clear at the outset that the study is a pilot and
there is not a powered hypothesis expressed in the protocol. That is a
different context than using a powered study, having it fail,
performing your Fisher's post mortem and using that data to design the
next "improved" study.

Regards,

Marc Schwartz

[John Whittington]

At 10:55 12/12/2009 -0500, Peter Flom wrote:
>Doesn't this get into the power analysis for the NEXT grant?
>That is, suppose you do a small study, and find a certain effect size,
>variance and so on;
>it is large enough to be of practical importance, but not stat sig.

I suspect that our areas of experience may be somewhat different.....

Your reference to grants and applications presumably implies that you are
thinking about 'academic' studies.

I need not tell you that a high proportion of trials, particularly drug
trials, are commercially sponsored. As I said, if you just tell a
businessperson that a trial designed to have adequate power to detect a
useful effect has failed to detect such an effect, there is a distinct
possibility that there will be no 'next' of any description. On the other
hand, if you can demonstrate that (despite prior beliefs) it transpires
that the trial did NOT actually have adequate power for the intended
purpose, (s)he may well wish to investigate further.

>Then this should form the basis of the next application - that is, GIVEN a
>similar effect size,
>we would need N people.

To be clear (and assuming I understand you correctly), that is NOT what I
have been talking about, and is an approach which is surely fraught with
dangers. There should be an a priori decision as to what size of effect
(absolute, not 'standardised') would be the 'minumum clinically
important/useful' and (unless there is an intervening change in clinical
thinking) that should never be reduced throughout a programme of
trials. To estimate sample size for a future study to detect a size of
effect that was seen in an earlier study is surely very wrong if that
effect is smaller than what one has previously decided was the smallest
effect of clinical interest/importance - since it sets the scene for
detecting a 'statistically significant' effect which is too small in
magnitude to be of clinical interest (and exposing lots of subjects to the
treatment under test to achieve that). I have been talking only of
utilising a different ('updated') estimate of variance/SD.

Of course, if a trial demonstrates that the magnitude of effect is much
GREATER than the 'minimum effect of clinical importance', it is then
probably appropriate (both practically and ethically) to estimate sample
size on the basis of the ability to detect a magnitude of effect
considerably greater than that 'minimum effect of intererest', so as to
reduce the unnecessary exposure of subjects to the treatment in question.

[Peter Flom]

Marc Schwartz <marc_s...@me.com> wrote

>If I am correctly understanding your context Peter, this is the reason
>that small pilot studies are performed. They are not intended to
>achieve statistical significance, but only to enable a reasonable
>estimate of those factors that are relevant to designing the
>subsequent powered study, where there is no a priori data available
>otherwise.
>
>However, it is made clear at the outset that the study is a pilot and
>there is not a powered hypothesis expressed in the protocol. That is a
>different context than using a powered study, having it fail,
>performing your Fisher's post mortem and using that data to design the
>next "improved" study.
>

You're right, but, based on my experience, I don't think you can divide studies
quite that neatly into "pilot" and "powered". Some agencies are requiring
that *all* grants have a power analysis .... this is a bit strange, but true. I had
one colleague who submitted a grant for a meta-analysis; one of the "revise and resubmit"
requirements was that he do a power analysis (even though his total N was in the tens of thousands).

But, on 'pilot' vs. 'powered' - I think there are some studies that are in-between.

[Peter Flom]

Hi John

I think our areas of experience are so different that they don't even overlap!

I agree that one shouldn't change one's notion of what is an important effect size.

But here are two examples of what I am talking about:

1) Some doctors have a theory about a rare disease; there is not good data about the theory they have. They gather retrospective information on a small number of people at their hospital who have the disease. They come to me for a "power analysis" and I tell them that with N = 12 (or whatever) the effect would have to be HUGE for it to be sig. BUT, I say, you can get an estimate of effect size. This is news to them. They believe that if an effect is not sig, it doesn't exist.

(Based on my experience at a large hosptial/med school, doctors get VERY little training about statistics)

2) Studying a rare and stigmatized population, a grant is prepared to gather some information through a survey.
NIH or NIDA or whomever demands a power analysis.

I've never worked on a commercially funded study.

Peter

>--
>To post a new thread to MedStats, send email to MedS...@googlegroups.com .
>MedStats' home page is http://groups.google.com/group/MedStats .
>Rules: http://groups.google.com/group/MedStats/web/medstats-rules

[Ted Harding]

A few comments (some passages snipped)

On 12-Dec-09 18:13:34, Peter Flom wrote:
> Hi John
> [...]

> I agree that one shouldn't change one's notion of what is an important
> effect size.
>
> But here are two examples of what I am talking about:
> 1) Some doctors have a theory about a rare disease; there is not good
> data about the theory they have. They gather retrospective information
> on a small number of people at their hospital who have the disease.
> They come to me for a "power analysis" and I tell them that with N = 12
> (or whatever) the effect would have to be HUGE for it to be sig. BUT,
> I say, you can get an estimate of effect size. This is news to them.
> They believe that if an effect is not sig, it doesn't exist.
>
> (Based on my experience at a large hosptial/med school, doctors get
> VERY little training about statistics)

The above reflects on the common diversity of opinion about the
meaning of "significance". In a lot of everyday speech (and in much
Press reporting) "significant" means "an effect big enough to take
notice of". In statistician-speak, it means that the study yielded
enough evidence to establish (at the token significance level)
that there was indeed an effect (never mind how big it is), and
"not significant" means that there wasn't enough evidence. So you
can find statements like "the mortality rate amongst users of X
did not differ significantly from non-users" presented as though
the difference doesn't matter, preactically speaking.

The statistical meaning then triggers more language: A "significant"
result can be expressed as "The Null Hypothesis was rejected
[at the stated significance level, i.e. there was that amount of
evidence against the NH]". In researcher-speak, the "[...]" is
quite often left out, forgotten about, or not known in the first place.
And then a non-significant result is intepreted as "The NH was not
rejected", so "we tested the NH and it passed the test", i.e. there
was no effect!

The problem is that there are many different linguistic communities
out there, and varying degrees of adherence to logical thought
(in the Formal Logic, i.e. Propositional Logic, sense).

Across-the-desk education (such as Peter Flom does with his doctors)
can help to bring these things to the surface and spread understanding;
but one would expect that journal editors, sponsors, Ethics Committees
and so forth, should also be able to spell it out when they see people
in this kind of muddle (always presuming that they can perceive it).
So: can they? Do they?

#############################################################

In a preceding mail, Marc Schwarz wrote:

> If I am correctly understanding your context Peter, this is the
> reason that small pilot studies are performed. They are not
> intended to achieve statistical significance, but only to enable
> a reasonable estimate of those factors that are relevant to designing
> the subsequent powered study, where there is no a priori data
> available otherwise.
>
> However, it is made clear at the outset that the study is a pilot
> and there is not a powered hypothesis expressed in the protocol.
> That is a different context than using a powered study, having
> it fail, performing your Fisher's post mortem and using that data
> to design the next "improved" study.

This seems to imply that, officially, the pilot study has no
evidential status with respect to the objective of the trial,
being carried out in order to establish the "ball-park" of the
parameters of the situation. But then: suppose the pilot study
happened to result in highly significant evidence for a large
effect? It could happen! And the study may well (indeed should)
have been carried out with the same rigorous control of procedure
(selection of subjects, randomisation and so forth) as a "real"
("powered") study, except that it didn't go through the phase of
a power calculation (which is one of the stages officially expected
in a "real" study). So: is it valid, or invalid, as "proof" of
the existence of an effect?

Just a few thoughts ...
Ted.

--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@manchester.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 12-Dec-09 Time: 20:19:32
------------------------------ XFMail ------------------------------

[Peter Flom]

Ted.H...@manchester.ac.uk wrote

>The above reflects on the common diversity of opinion about the
>meaning of "significance". In a lot of everyday speech (and in much
>Press reporting) "significant" means "an effect big enough to take
>notice of". In statistician-speak, it means that the study yielded
>enough evidence to establish (at the token significance level)
>that there was indeed an effect (never mind how big it is), and
>"not significant" means that there wasn't enough evidence. So you
>can find statements like "the mortality rate amongst users of X
>did not differ significantly from non-users" presented as though
>the difference doesn't matter, preactically speaking.
>
>The statistical meaning then triggers more language: A "significant"
>result can be expressed as "The Null Hypothesis was rejected
>[at the stated significance level, i.e. there was that amount of
>evidence against the NH]". In researcher-speak, the "[...]" is
>quite often left out, forgotten about, or not known in the first place.
>And then a non-significant result is intepreted as "The NH was not
>rejected", so "we tested the NH and it passed the test", i.e. there
>was no effect!
>
>The problem is that there are many different linguistic communities
>out there, and varying degrees of adherence to logical thought
>(in the Formal Logic, i.e. Propositional Logic, sense).
>

Indeed.

It was, I think, a mistake for statisticians to use words that already have meaning in ways that are subtly different in meaning. As Ted points out "statistically significant" doesn't exactly match other uses of "significant". Similarly for "power". And without getting into the LOGIC of null hypothesis testing, the LANGUAGE is confusing. "Fail to reject the null hypothesis" hmmm? But most people are familiar with trials, and with "guilty" and "not guilty" - that is, at least in the USA, we find a defendant in a criminal trial "guilty" or "not guilty" never "innocent". Scientists, then, play the part of prosecutors, wanting to prove the null hypothesis guilty "beyond a reasonable doubt" and we define "reasonable" to mean "5%". This doesn't clear up that other mess of what a p-value is, but maybe it helps a little

Peter

[Martin Holt]

Thanks for those, Ted. I feel there is more to be said about the "winning of
the lottery" example. But to answer your final question, in the absence of a
sample size calculation it is quite possible that the Type II error (failure
to reject the null hypothesis when the null hypothesis is false) could be
large and yet undetected. (I've actually been there, where a pilot cancer
trial behaved just in this fashion. No-one could believe it).

I want to work out what it is that I feel has not been drawn out in the
discussion. Starting with quoting Neil Shephard (10/12/2009, 17:04):

"A simple way of characterising the point that post-hoc power tests are
pretty pointless would be the scenario of trying to tell someone who's
just one the main prize in a lottery that they shouldn't bother
playing the lottery because they have a negligible chance of winning.

They won't care because they've already got their result!

Although unfortunately I doubt saying the same thing to someone who
hasn't won would stop a habitual player from playing again!"

Winning the main prize in the lottery is taken as being a significant result
(in a trial where the sample size is such that the chances of getting a
significant result are negligible). So someone who's just won isn't going to
be interested in back-calculating the conditions for winning. The part that
I feel has been missing to-date is that isn't there a large number....any
number...of conditions that could be back-calculated ? and isn't that why
there is no point in doing so ?

Best Regards,

Martin Holt

----- Original Message -----
From: "Ted Harding" <Ted.H...@manchester.ac.uk>
To: <meds...@googlegroups.com>
Sent: Saturday, December 12, 2009 8:19 PM
Subject: RE: {MEDSTATS} Re: p-value < 0.1 for important outcomes!

[John Whittington]

At 16:16 12/12/2009 -0500, Peter Flom wrote:
>...... the LANGUAGE is confusing. "Fail to reject the null hypothesis"

>hmmm? But most people are familiar with trials, and with "guilty" and "not
>guilty" - that is, at least in the USA, we find a defendant in a criminal
>trial "guilty" or "not guilty" never "innocent". Scientists, then, play
>the part of prosecutors, wanting to prove the null hypothesis guilty
>"beyond a reasonable doubt" and we define "reasonable" to mean "5%". This
>doesn't clear up that other mess of what a p-value is, but maybe it helps
>a little

Peter, I agree, and have often used the judicial analogy in an attempt to
help non-statisticians understand what we get up to, but I fear that your
wording (above) may be at risk of actually adding to the confusion - in
particular, your phrase "....wanting to prove the null hypothesis guilty
...."?

The null hypothesis is the default, which has to be assumed unless we feel
able to reject it "beyond whatever criteria of doubt" we choose to apply.
The value of the analogy is obviously that (a) in both our countries, 'Not
Guilty' is the default ('innocent unless/until proved guilty'), hence the
null hypothesis, and only if the prosecutors can reject that null
hypothesis beyond their judicial system's definition of 'reasonable doubt'
is Guilt considered to have been established, and (b) that, like
statisticians, courts do not seek to 'prove inoccence' (or 'prove guilt') -
but only to reject (or not reject) the null hypothesis of 'Not
Guilty'. The same analogy can, in fact, be extended to try to help
explaining the concepts of Type I and Type II errors (and the trade-off
between them) to non-statisticians - since most people understand the
trade-off between allowing guilty people to remain unconvicted and risking
innocent people being convicted (and that our judicial system is biased
towards the former).

Anyway, my point vis that the phrase "...wanting to prove the null
hypothesis guilty..." is perhaps best avoided when one attempts to use this
analogy - for fairly obvious reasons.

[As an aside, when I speak of 'my country', I refer to England and Wales,
since the UK is not unified in its judicial system. As you may be aware,
in Scotland, there are three possible verdicts - 'Not Guilty', 'Guilty' and
'Not Proven'. As I understand it, the latter is used when a court believes
that the accused is probably guilty, but is accepting that a Type II error
is preferable to a Type I error (in terms of human/civil rights) if the
evidence is not compelling enough to satisfy the "beyond a reasonable
doubt" criteria for rejecting the 'Not Guilty' null. It would be
interesting if statisticians tried to create a 'third outcome' of a
hypothesis test!]

[Ted Harding]

On 12-Dec-09 23:52:10, John Whittington wrote:
> [...]

> [As an aside, when I speak of 'my country', I refer to England
> and Wales, since the UK is not unified in its judicial system.
> As you may be aware, in Scotland, there are three possible
> verdicts - 'Not Guilty', 'Guilty' and 'Not Proven'. As I understand
> it, the latter is used when a court believes that the accused is
> probably guilty, but is accepting that a Type II error is preferable
> to a Type I error (in terms of human/civil rights) if the evidence
> is not compelling enough to satisfy the "beyond a reasonable
> doubt" criteria for rejecting the 'Not Guilty' null. It would be
> interesting if statisticians tried to create a 'third outcome' of a
> hypothesis test!]
>
> Kind Regards,
> John

Statisticians already do -- in the context of sequential tests.
There is a "middle range" at any stage, equivalent in its way
to "Not Proven".

So -- in the event of "Not Proven", send the accused back to
the Police cells for further interrogation ... You never know,
this might even establish "innocent beyond reasonable doubt"!

Ted.

--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@manchester.ac.uk>
Fax-to-email: +44 (0)870 094 0861

Date: 13-Dec-09 Time: 08:49:09
------------------------------ XFMail ------------------------------