parametric/non-parametric, change from baseline, covariates

[Christian Lerch]

I’ve been asked to look over a statistical analysis. At the moment,
Mann-Whitney-U-Test and Wilcoxon signed-rank test were used.
It’s a tiny RCT (n1=7, n2=8 participants) in which both the comparison
of before and after values for each group and the comparison of both
groups at the end of the study are of interest. Ideally, both analyses
should be based on ‘change from baseline’.
RCTs in this field (with a reasonable sample size) are generally
analysed by ANCOVA (using before values as covariate , based on
‘change from baseline’).
Any suggestion?
Regards,
Christian

[BXC (Bendix Carstensen)]

Use regression with baseline as covariate and treatment as factor to estimate the effect, but use permutations to assess the p-value --- you can acually enumerate all of the possible treatment allocations: 15 choose 7 is only 6435.
Best regards,
Bendix

[Ryan]

Dear Bendix,

I've never heard about using "permutations to assess the p-value."
Would you mind elaborating on this a bit with a very simple, concrete
example? If you don't have the time, would you mind sharing a
reference or two on the rationale and how to do this?

Thanks!

Ryan

> > Christian- Hide quoted text -
>
> - Show quoted text -

[BXC (Bendix Carstensen)]

1) Randomly assign 7 of persons to one group and 8 to the other, and pretend that this
was the treatment allocation.
2) Compute the test statistic.
3) Do this for 1000 random allocations (or in your case for all 6435 possible
allocations).
4) The permuation p-value is the proportion of the obtained test-statistics from random
assignments that exceed the test statistic from the actual treatment assignement.

[Bruce Weaver]

On Jun 29, 2:56 pm, Ryan <Ryan.Andrew.Bl...@gmail.com> wrote:
> Dear Bendix,
>
> I've never heard about using "permutations to assess the p-value."
> Would you mind elaborating on this a bit with a very simple, concrete
> example? If you don't have the time, would you mind sharing a
> reference or two on the rationale and how to do this?
>
> Thanks!
>
> Ryan

Hi Ryan. In addition to the reply from Bendix, here are some links
you may find useful:

http://en.wikipedia.org/wiki/Permutation_test
http://www.gseis.ucla.edu/courses/ed230a2/notes/permute.html
http://www.statistics101.net/

IIRC, you have Stata. The 2nd link above gives an example using the
"permute" command in Stata.

Cheers,
Bruce
--
Bruce Weaver
bwe...@lakeheadu.ca
http://sites.google.com/a/lakeheadu.ca/bweaver/
"When all else fails, RTFM."

[Ted Harding]

It is all in the general area of Permutation Tests or Randomisation
Tests. Googling on either should throw up a lot of stuff.

The basic principle (illustrated with Christian Lerch's case of an
RCT where there were 15 subjects, randomised 7:8 to Arm 1 and Arm 2,
is that the Null Hypothesis is that there is no difference between
Arm 1 and Arm 2. So, if Subject i were allocated to Arm 2, he would
produce exactly the same result if allocated to Arm 1 (if the Null
Hypothesis is true).

The precise details of how to proceed depend on the details of how
the randomisation was carried out -- it is essential to respect
this in doing the Randomisation Test.

But suppose that, in this case, the ones who went on Arm 1 were a
random subset of size 7 out of the 15. Let T be any statistic which
compares Arm 1 with Arm 2, calculated from the values returned by
those who went into Arm 1, and the values returned by those who
went into Arm 2.

The value of T, call it T0, obtained from the allocation that was
actually used in the trial, is one out of the 6435 possible results
that would have been obtained if the allocation had been different.
Since the 7 are a random 7 out of the 15, all 6435 allocations are
equally likely. So work out T for each of the 6435 allocations.

If the Null Hypothesis (H0) were true, then these are the possible
values that really would have been obtained if the allcoation had
been different, since by hypothesis it would make no difference to
an individual's returned value, whichever Arm he was allocated to.

Now suppose that large values of T0 would be considered as evidence
against H0. How large is significant (e.g. at P=0.05)?

Well, any value within the top 5% of the 6435 valoues of T would be
significant. Is T0 one of these values? If so, then P < 0.05.
A numerical P-value would be the proportion of the 6435 T-values
which are as large as, or exceed, T0.

A similar approach leads to a P-value for "Is T0 significantly
different (either way) from 0?" -- just use both the top and
bottom ends of the 6435 T-values.

One thing which is not straightforwardly available with the use
of Randomisation Tests is an approach to evaluating the Power of
the Trial, since Power depends on hypothesising a particular
numerical deviation from the Null Hypothesis; and when the test
is as described it is far from obvious how this should be expressed
so that it can be incorporated in a similar calculation. Nevertheless,
in practice various ad-hoc approaches can be adopted with reasonable
plausibility.

Hoping this helps,
Ted (One of the top 99% of statisticians)

On 29-Jun-09 18:56:56, Ryan wrote:
> Dear Bendix,
>
> I've never heard about using "permutations to assess the p-value."
> Would you mind elaborating on this a bit with a very simple, concrete
> example? If you don't have the time, would you mind sharing a
> reference or two on the rationale and how to do this?
>
> Thanks!
> Ryan
>

--------------------------------------------------------------------
E-Mail: (Ted Harding) <Ted.H...@manchester.ac.uk>
Fax-to-email: +44 (0)870 094 0861
Date: 29-Jun-09 Time: 21:28:50
------------------------------ XFMail ------------------------------

[Ryan]

Thank you Ted, Bendix and Bruce. Each of your responses were uniquely
helpful. I grasp the simple example of group 1 versus group 2, using
either a test statistic (i.e. t statistic) or, for instance, in one of
Bruce's links, the use of the sum of group 1.

I have two follow-up questions, if anyone has the time:

(1) Aside from demonstrating that I randomly selected participants
from the population of interest and randomly assigned (perhaps with
balancing) each participant to either condition A or B, what else
would I have to demonstrate for it to be considered an acceptable
approach to compute the p-value based on permutations?

(2) If I wanted to conduct an independent samples t test, what would
be the factors that would lead you to consider computing a p-value
using permutations. Certainly sample size is critical, but is that the
only reason? What would be the rules of thumb specific to this
independent samples t test example?

Any thoughts on these would be most appreciated.

Thanks again,

Ryan

On Jun 29, 1:28 pm, (Ted Harding) <Ted.Hard...@manchester.ac.uk>
wrote:

> E-Mail: (Ted Harding) <Ted.Hard...@manchester.ac.uk>

> Fax-to-email: +44 (0)870 094 0861
> Date: 29-Jun-09                                       Time: 21:28:50

> ------------------------------ XFMail ------------------------------- Hide quoted text -

[Christian Lerch]

Many thanks to Bendix, Bruce and Ted (the top 99% statistician...).

In addition to Ryan's question: Is there any possibility to use the
proposed procedure if randomisation was done after participants where
matched 1:1? (I don't think so).

Thanks to all contributors to this informative thread.
Best regards,
Christian

Ryan schrieb:

[Graham Smith]

These links may be useful for those who want to follow up on the
usefulness of permutations.

http://www.statistics101.net/

This provides the standalone version of Resample, as well as some tutorials

There is also a link on this site to Julian Simons book on Resampling,
but the direct link is here

http://www.resample.com/content/text/index.shtml

I also have a small list of other resampling/randoization books, not
dedicated to medicine (I'm an ecologist) but still with a biological
bias, if anyone is interested.

Graham