I’ve been asked to look over a statistical analysis. At the moment,
Mann-Whitney-U-Test and Wilcoxon signed-rank test were used.
It’s a tiny RCT (n1=7, n2=8 participants) in which both the comparison
of before and after values for each group and the comparison of both
groups at the end of the study are of interest. Ideally, both analyses
should be based on ‘change from baseline’.
RCTs in this field (with a reasonable sample size) are generally
analysed by ANCOVA (using before values as covariate , based on
‘change from baseline’).
Any suggestion?
Regards,
Christian
Use regression with baseline as covariate and treatment as factor to estimate the effect, but use permutations to assess the p-value --- you can acually enumerate all of the possible treatment allocations: 15 choose 7 is only 6435. Best regards, Bendix
> -----Original Message----- > From: MedStats@googlegroups.com > [mailto:MedStats@googlegroups.com] On Behalf Of Christian Lerch > Sent: 29. juni 2009 19:32 > To: MedStats > Subject: {MEDSTATS} parametric/non-parametric, change from > baseline, covariates
> I've been asked to look over a statistical analysis. At the > moment, Mann-Whitney-U-Test and Wilcoxon signed-rank test were used. > It's a tiny RCT (n1=7, n2=8 participants) in which both the > comparison of before and after values for each group and the > comparison of both groups at the end of the study are of > interest. Ideally, both analyses should be based on 'change > from baseline'. > RCTs in this field (with a reasonable sample size) are > generally analysed by ANCOVA (using before values as > covariate , based on 'change from baseline'). > Any suggestion? > Regards, > Christian
I've never heard about using "permutations to assess the p-value."
Would you mind elaborating on this a bit with a very simple, concrete
example? If you don't have the time, would you mind sharing a
reference or two on the rationale and how to do this?
Thanks!
Ryan
On Jun 29, 11:19 am, "BXC (Bendix Carstensen)" <b...@steno.dk> wrote:
> Use regression with baseline as covariate and treatment as factor to estimate the effect, but use permutations to assess the p-value --- you can acually enumerate all of the possible treatment allocations: 15 choose 7 is only 6435.
> Best regards,
> Bendix
> > -----Original Message-----
> > From: MedStats@googlegroups.com
> > [mailto:MedStats@googlegroups.com] On Behalf Of Christian Lerch
> > Sent: 29. juni 2009 19:32
> > To: MedStats
> > Subject: {MEDSTATS} parametric/non-parametric, change from
> > baseline, covariates
> > I've been asked to look over a statistical analysis. At the
> > moment, Mann-Whitney-U-Test and Wilcoxon signed-rank test were used.
> > It's a tiny RCT (n1=7, n2=8 participants) in which both the
> > comparison of before and after values for each group and the
> > comparison of both groups at the end of the study are of
> > interest. Ideally, both analyses should be based on 'change
> > from baseline'.
> > RCTs in this field (with a reasonable sample size) are
> > generally analysed by ANCOVA (using before values as
> > covariate , based on 'change from baseline').
> > Any suggestion?
> > Regards,
> > Christian- Hide quoted text -
1) Randomly assign 7 of persons to one group and 8 to the other, and pretend that this was the treatment allocation. 2) Compute the test statistic. 3) Do this for 1000 random allocations (or in your case for all 6435 possible allocations). 4) The permuation p-value is the proportion of the obtained test-statistics from random assignments that exceed the test statistic from the actual treatment assignement.
> -----Original Message----- > From: MedStats@googlegroups.com > [mailto:MedStats@googlegroups.com] On Behalf Of Ryan > Sent: 29. juni 2009 20:57 > To: MedStats > Subject: {MEDSTATS} Re: parametric/non-parametric, change > from baseline, covariates
> Dear Bendix,
> I've never heard about using "permutations to assess the p-value." > Would you mind elaborating on this a bit with a very simple, > concrete example? If you don't have the time, would you mind > sharing a reference or two on the rationale and how to do this?
> Thanks!
> Ryan
> On Jun 29, 11:19 am, "BXC (Bendix Carstensen)" <b...@steno.dk> wrote: > > Use regression with baseline as covariate and treatment as > factor to estimate the effect, but use permutations to assess > the p-value --- you can acually enumerate all of the possible > treatment allocations: 15 choose 7 is only 6435. > > Best regards, > > Bendix
> > > -----Original Message----- > > > From: MedStats@googlegroups.com > > > [mailto:MedStats@googlegroups.com] On Behalf Of Christian Lerch > > > Sent: 29. juni 2009 19:32 > > > To: MedStats > > > Subject: {MEDSTATS} parametric/non-parametric, change > from baseline, > > > covariates
> > > I've been asked to look over a statistical analysis. At > the moment, > > > Mann-Whitney-U-Test and Wilcoxon signed-rank test were used. > > > It's a tiny RCT (n1=7, n2=8 participants) in which both the > > > comparison of before and after values for each group and the > > > comparison of both groups at the end of the study are of > interest. > > > Ideally, both analyses should be based on 'change from baseline'. > > > RCTs in this field (with a reasonable sample size) are generally > > > analysed by ANCOVA (using before values as covariate , based on > > > 'change from baseline'). > > > Any suggestion? > > > Regards, > > > Christian- Hide quoted text -
On Jun 29, 2:56 pm, Ryan <Ryan.Andrew.Bl...@gmail.com> wrote:
> Dear Bendix,
> I've never heard about using "permutations to assess the p-value."
> Would you mind elaborating on this a bit with a very simple, concrete
> example? If you don't have the time, would you mind sharing a
> reference or two on the rationale and how to do this?
> Thanks!
> Ryan
Hi Ryan. In addition to the reply from Bendix, here are some links
you may find useful:
It is all in the general area of Permutation Tests or Randomisation Tests. Googling on either should throw up a lot of stuff.
The basic principle (illustrated with Christian Lerch's case of an RCT where there were 15 subjects, randomised 7:8 to Arm 1 and Arm 2, is that the Null Hypothesis is that there is no difference between Arm 1 and Arm 2. So, if Subject i were allocated to Arm 2, he would produce exactly the same result if allocated to Arm 1 (if the Null Hypothesis is true).
The precise details of how to proceed depend on the details of how the randomisation was carried out -- it is essential to respect this in doing the Randomisation Test.
But suppose that, in this case, the ones who went on Arm 1 were a random subset of size 7 out of the 15. Let T be any statistic which compares Arm 1 with Arm 2, calculated from the values returned by those who went into Arm 1, and the values returned by those who went into Arm 2.
The value of T, call it T0, obtained from the allocation that was actually used in the trial, is one out of the 6435 possible results that would have been obtained if the allocation had been different. Since the 7 are a random 7 out of the 15, all 6435 allocations are equally likely. So work out T for each of the 6435 allocations.
If the Null Hypothesis (H0) were true, then these are the possible values that really would have been obtained if the allcoation had been different, since by hypothesis it would make no difference to an individual's returned value, whichever Arm he was allocated to.
Now suppose that large values of T0 would be considered as evidence against H0. How large is significant (e.g. at P=0.05)?
Well, any value within the top 5% of the 6435 valoues of T would be significant. Is T0 one of these values? If so, then P < 0.05. A numerical P-value would be the proportion of the 6435 T-values which are as large as, or exceed, T0.
A similar approach leads to a P-value for "Is T0 significantly different (either way) from 0?" -- just use both the top and bottom ends of the 6435 T-values.
One thing which is not straightforwardly available with the use of Randomisation Tests is an approach to evaluating the Power of the Trial, since Power depends on hypothesising a particular numerical deviation from the Null Hypothesis; and when the test is as described it is far from obvious how this should be expressed so that it can be incorporated in a similar calculation. Nevertheless, in practice various ad-hoc approaches can be adopted with reasonable plausibility.
Hoping this helps, Ted (One of the top 99% of statisticians)
> I've never heard about using "permutations to assess the p-value." > Would you mind elaborating on this a bit with a very simple, concrete > example? If you don't have the time, would you mind sharing a > reference or two on the rationale and how to do this?
> Thanks! > Ryan
> On Jun 29, 11:19_am, "BXC (Bendix Carstensen)" <b...@steno.dk> wrote: >> Use regression with baseline as covariate and treatment as factor to >> estimate the effect, but use permutations to assess the p-value --- >> you can acually enumerate all of the possible treatment allocations: >> 15 choose 7 is only 6435. >> Best regards, >> Bendix
>> > -----Original Message----- >> > From: MedStats@googlegroups.com >> > [mailto:MedStats@googlegroups.com] On Behalf Of Christian Lerch >> > Sent: 29. juni 2009 19:32 >> > To: MedStats >> > Subject: {MEDSTATS} parametric/non-parametric, change from >> > baseline, covariates
>> > I've been asked to look over a statistical analysis. At the >> > moment, Mann-Whitney-U-Test and Wilcoxon signed-rank test were used. >> > It's a tiny RCT (n1=7, n2=8 participants) in which both the >> > comparison of before and after values for each group and the >> > comparison of both groups at the end of the study are of >> > interest. Ideally, both analyses should be based on 'change >> > from baseline'. >> > RCTs in this field (with a reasonable sample size) are >> > generally analysed by ANCOVA (using before values as >> > covariate , based on 'change from baseline'). >> > Any suggestion? >> > Regards, >> > Christian- Hide quoted text -
Thank you Ted, Bendix and Bruce. Each of your responses were uniquely
helpful. I grasp the simple example of group 1 versus group 2, using
either a test statistic (i.e. t statistic) or, for instance, in one of
Bruce's links, the use of the sum of group 1.
I have two follow-up questions, if anyone has the time:
(1) Aside from demonstrating that I randomly selected participants
from the population of interest and randomly assigned (perhaps with
balancing) each participant to either condition A or B, what else
would I have to demonstrate for it to be considered an acceptable
approach to compute the p-value based on permutations?
(2) If I wanted to conduct an independent samples t test, what would
be the factors that would lead you to consider computing a p-value
using permutations. Certainly sample size is critical, but is that the
only reason? What would be the rules of thumb specific to this
independent samples t test example?
Any thoughts on these would be most appreciated.
Thanks again,
Ryan
On Jun 29, 1:28 pm, (Ted Harding) <Ted.Hard...@manchester.ac.uk>
wrote:
> It is all in the general area of Permutation Tests or Randomisation
> Tests. Googling on either should throw up a lot of stuff.
> The basic principle (illustrated with Christian Lerch's case of an
> RCT where there were 15 subjects, randomised 7:8 to Arm 1 and Arm 2,
> is that the Null Hypothesis is that there is no difference between
> Arm 1 and Arm 2. So, if Subject i were allocated to Arm 2, he would
> produce exactly the same result if allocated to Arm 1 (if the Null
> Hypothesis is true).
> The precise details of how to proceed depend on the details of how
> the randomisation was carried out -- it is essential to respect
> this in doing the Randomisation Test.
> But suppose that, in this case, the ones who went on Arm 1 were a
> random subset of size 7 out of the 15. Let T be any statistic which
> compares Arm 1 with Arm 2, calculated from the values returned by
> those who went into Arm 1, and the values returned by those who
> went into Arm 2.
> The value of T, call it T0, obtained from the allocation that was
> actually used in the trial, is one out of the 6435 possible results
> that would have been obtained if the allocation had been different.
> Since the 7 are a random 7 out of the 15, all 6435 allocations are
> equally likely. So work out T for each of the 6435 allocations.
> If the Null Hypothesis (H0) were true, then these are the possible
> values that really would have been obtained if the allcoation had
> been different, since by hypothesis it would make no difference to
> an individual's returned value, whichever Arm he was allocated to.
> Now suppose that large values of T0 would be considered as evidence
> against H0. How large is significant (e.g. at P=0.05)?
> Well, any value within the top 5% of the 6435 valoues of T would be
> significant. Is T0 one of these values? If so, then P < 0.05.
> A numerical P-value would be the proportion of the 6435 T-values
> which are as large as, or exceed, T0.
> A similar approach leads to a P-value for "Is T0 significantly
> different (either way) from 0?" -- just use both the top and
> bottom ends of the 6435 T-values.
> One thing which is not straightforwardly available with the use
> of Randomisation Tests is an approach to evaluating the Power of
> the Trial, since Power depends on hypothesising a particular
> numerical deviation from the Null Hypothesis; and when the test
> is as described it is far from obvious how this should be expressed
> so that it can be incorporated in a similar calculation. Nevertheless,
> in practice various ad-hoc approaches can be adopted with reasonable
> plausibility.
> Hoping this helps,
> Ted (One of the top 99% of statisticians)
> On 29-Jun-09 18:56:56, Ryan wrote:
> > Dear Bendix,
> > I've never heard about using "permutations to assess the p-value."
> > Would you mind elaborating on this a bit with a very simple, concrete
> > example? If you don't have the time, would you mind sharing a
> > reference or two on the rationale and how to do this?
> > Thanks!
> > Ryan
> > On Jun 29, 11:19_am, "BXC (Bendix Carstensen)" <b...@steno.dk> wrote:
> >> Use regression with baseline as covariate and treatment as factor to
> >> estimate the effect, but use permutations to assess the p-value ---
> >> you can acually enumerate all of the possible treatment allocations:
> >> 15 choose 7 is only 6435.
> >> Best regards,
> >> Bendix
> >> > -----Original Message-----
> >> > From: MedStats@googlegroups.com
> >> > [mailto:MedStats@googlegroups.com] On Behalf Of Christian Lerch
> >> > Sent: 29. juni 2009 19:32
> >> > To: MedStats
> >> > Subject: {MEDSTATS} parametric/non-parametric, change from
> >> > baseline, covariates
> >> > I've been asked to look over a statistical analysis. At the
> >> > moment, Mann-Whitney-U-Test and Wilcoxon signed-rank test were used.
> >> > It's a tiny RCT (n1=7, n2=8 participants) in which both the
> >> > comparison of before and after values for each group and the
> >> > comparison of both groups at the end of the study are of
> >> > interest. Ideally, both analyses should be based on 'change
> >> > from baseline'.
> >> > RCTs in this field (with a reasonable sample size) are
> >> > generally analysed by ANCOVA (using before values as
> >> > covariate , based on 'change from baseline').
> >> > Any suggestion?
> >> > Regards,
> >> > Christian- Hide quoted text -
Many thanks to Bendix, Bruce and Ted (the top 99% statistician...).
In addition to Ryan's question: Is there any possibility to use the proposed procedure if randomisation was done after participants where matched 1:1? (I don't think so).
Thanks to all contributors to this informative thread. Best regards, Christian
> Thank you Ted, Bendix and Bruce. Each of your responses were uniquely > helpful. I grasp the simple example of group 1 versus group 2, using > either a test statistic (i.e. t statistic) or, for instance, in one of > Bruce's links, the use of the sum of group 1.
> I have two follow-up questions, if anyone has the time:
> (1) Aside from demonstrating that I randomly selected participants > from the population of interest and randomly assigned (perhaps with > balancing) each participant to either condition A or B, what else > would I have to demonstrate for it to be considered an acceptable > approach to compute the p-value based on permutations?
> (2) If I wanted to conduct an independent samples t test, what would > be the factors that would lead you to consider computing a p-value > using permutations. Certainly sample size is critical, but is that the > only reason? What would be the rules of thumb specific to this > independent samples t test example?
> Any thoughts on these would be most appreciated.
> Thanks again,
> Ryan
> On Jun 29, 1:28 pm, (Ted Harding) <Ted.Hard...@manchester.ac.uk> > wrote: >> It is all in the general area of Permutation Tests or Randomisation >> Tests. Googling on either should throw up a lot of stuff.
>> The basic principle (illustrated with Christian Lerch's case of an >> RCT where there were 15 subjects, randomised 7:8 to Arm 1 and Arm 2, >> is that the Null Hypothesis is that there is no difference between >> Arm 1 and Arm 2. So, if Subject i were allocated to Arm 2, he would >> produce exactly the same result if allocated to Arm 1 (if the Null >> Hypothesis is true).
>> The precise details of how to proceed depend on the details of how >> the randomisation was carried out -- it is essential to respect >> this in doing the Randomisation Test.
>> But suppose that, in this case, the ones who went on Arm 1 were a >> random subset of size 7 out of the 15. Let T be any statistic which >> compares Arm 1 with Arm 2, calculated from the values returned by >> those who went into Arm 1, and the values returned by those who >> went into Arm 2.
>> The value of T, call it T0, obtained from the allocation that was >> actually used in the trial, is one out of the 6435 possible results >> that would have been obtained if the allocation had been different. >> Since the 7 are a random 7 out of the 15, all 6435 allocations are >> equally likely. So work out T for each of the 6435 allocations.
>> If the Null Hypothesis (H0) were true, then these are the possible >> values that really would have been obtained if the allcoation had >> been different, since by hypothesis it would make no difference to >> an individual's returned value, whichever Arm he was allocated to.
>> Now suppose that large values of T0 would be considered as evidence >> against H0. How large is significant (e.g. at P=0.05)?
>> Well, any value within the top 5% of the 6435 valoues of T would be >> significant. Is T0 one of these values? If so, then P < 0.05. >> A numerical P-value would be the proportion of the 6435 T-values >> which are as large as, or exceed, T0.
>> A similar approach leads to a P-value for "Is T0 significantly >> different (either way) from 0?" -- just use both the top and >> bottom ends of the 6435 T-values.
>> One thing which is not straightforwardly available with the use >> of Randomisation Tests is an approach to evaluating the Power of >> the Trial, since Power depends on hypothesising a particular >> numerical deviation from the Null Hypothesis; and when the test >> is as described it is far from obvious how this should be expressed >> so that it can be incorporated in a similar calculation. Nevertheless, >> in practice various ad-hoc approaches can be adopted with reasonable >> plausibility.
>> Hoping this helps, >> Ted (One of the top 99% of statisticians)
>> On 29-Jun-09 18:56:56, Ryan wrote:
>>> Dear Bendix, >>> I've never heard about using "permutations to assess the p-value." >>> Would you mind elaborating on this a bit with a very simple, concrete >>> example? If you don't have the time, would you mind sharing a >>> reference or two on the rationale and how to do this? >>> Thanks! >>> Ryan >>> On Jun 29, 11:19_am, "BXC (Bendix Carstensen)" <b...@steno.dk> wrote: >>>> Use regression with baseline as covariate and treatment as factor to >>>> estimate the effect, but use permutations to assess the p-value --- >>>> you can acually enumerate all of the possible treatment allocations: >>>> 15 choose 7 is only 6435. >>>> Best regards, >>>> Bendix >>>>> -----Original Message----- >>>>> From: MedStats@googlegroups.com >>>>> [mailto:MedStats@googlegroups.com] On Behalf Of Christian Lerch >>>>> Sent: 29. juni 2009 19:32 >>>>> To: MedStats >>>>> Subject: {MEDSTATS} parametric/non-parametric, change from >>>>> baseline, covariates >>>>> I've been asked to look over a statistical analysis. At the >>>>> moment, Mann-Whitney-U-Test and Wilcoxon signed-rank test were used. >>>>> It's a tiny RCT (n1=7, n2=8 participants) in which both the >>>>> comparison of before and after values for each group and the >>>>> comparison of both groups at the end of the study are of >>>>> interest. Ideally, both analyses should be based on 'change >>>>> from baseline'. >>>>> RCTs in this field (with a reasonable sample size) are >>>>> generally analysed by ANCOVA (using before values as >>>>> covariate , based on 'change from baseline'). >>>>> Any suggestion? >>>>> Regards, >>>>> Christian- Hide quoted text - >>>> - Show quoted text - >> -------------------------------------------------------------------- >> E-Mail: (Ted Harding) <Ted.Hard...@manchester.ac.uk> >> Fax-to-email: +44 (0)870 094 0861 >> Date: 29-Jun-09 Time: 21:28:50 >> ------------------------------ XFMail ------------------------------- Hide quoted text -
I also have a small list of other resampling/randoization books, not dedicated to medicine (I'm an ecologist) but still with a biological bias, if anyone is interested.
|
