Beginner question and interpretation

[Mikaela Ceder]

Hej,

I am a bit confused, I have two datasets with 74 measurements in each group and I want to know if there is an effect size difference. I get the following results: The unpaired mean difference between x and y is -54.7 [95.0%CI -61.0, -47.9].The P value of the two-sided permutation t-test is 0.0.

This is completely new to me and I do not really understand how to interpret this. According to several publications the permutation test should never be zero. Can someone please explain this to me? 

Thank you!

Kindest regards,

Mikaela

[Adam Claridge-Chang]

Hi Mikaela,

Is it possible to post the Gardner-Altman plot here?

Do you have a citation for the permutation test never being zero? I would have thought that any time the effect size (and precision) are large, that it is possible to be zero.

Adam

[Mikaela]

Hi Adam,

Thank you for answering, I attach the plot. 

okay, then it might be that it is me that are reading the references (http://dx.doi.org/10.2202/1544-6115.1585)wrong maybe? I think this is one of the most difficult things (statistics, and of course correct statistics). 

Kindest regards,

Mikaela

[Adam Claridge-Chang]

Hi Mikaela,

OK I see, the effect size is huge, so the permutation p-value is zero by the method we are using. We haven't implemented (and likely won't) that paper's algorithm to adjust for the minuscule underestimate of P. As the authors say, "the implications can be serious in a multiple testing context." Certainly, suitability for multiple testing is not the intention of this tool, and, besides, we only give P for legacy reasons (reviewers demand them), not because we think it is relevant.

If you need a precise P value for these data, you could do a Mann-Whitney U test and report that instead. But the more interesting aspect of these data is the huge effect.

Kind regards,

Adam

[Michael Lotinga]

Hi Mikaela and Adam.

I've implemented the method for adjusting permutation p-values documented in the Phipson & Smyth 2010 article - my version can be seen in the master branch of my dev fork of dabest-python:

github.com/mlotinga/DABEST-python_devMJBL 

It seems to be working ok, although it is limited to the effsize_objects - I haven't used delta_objects, so I've not touched that yet.

I understand from the scipy documentation that the same basic adjustment approach is also applied in the permutation test implementation within scipy, although it is programmed in a nebulous manner that I can't follow!

Hope it helps.

[Adam Claridge-Chang]

Thank you Michael! We will review!