mann whitney vs permutation pvalue show different outcomes

[Hande Tunbak]

Hi, 

I have a some naive questions which are explained by different sets of measurement's so I'm going to post the questions separately with the relevant data in each case.

My knowledge on statistics is rather weak so please keep this in mind.

I have data that looks at the eye/body ratio of animals (see below).

I want to find out if the treatment has a greater effects size on mutant animals.

The default status for sample data is set to Genotype

The variables for the horizontal axis and colour are set to Treatment,Condition

1) Do these settings seem correct for investigating the effects on genotype?

Condition Genotype Treatment Value
Control wt Off 0.092267
Control wt Off 0.093487
Control wt Off 0.09148
Control wt Off 0.093055
Control wt Off 0.093639
Control wt Off 0.092473
Control wt Off 0.090499
Control wt Off 0.088853
Control wt Off 0.089868
Control wt Off 0.08826
Control wt Off 0.085833
Control wt Off 0.09401
Control wt Off 0.094043
Control wt Off 0.089872
Control wt Off 0.096584
Control wt Off 0.091483
Control wt Off 0.093683
Control wt Off 0.086852
Control wt Off 0.092845
Control wt Off 0.094503
Control wt Off 0.087177
Control wt Off 0.088264
Control wt Off 0.093634
Control wt Off 0.092272
Control wt Off 0.090255
Control wt Off 0.093037
Control wt Off 0.088613
Control wt Off 0.091774
Control wt Off 0.091939
Control wt Off 0.089692
Control wt Off 0.091909
Control wt Off 0.089546
Control wt Off 0.092505
Control wt Off 0.091091
Control wt Off 0.093317
Control wt Off 0.094736
Control wt Off 0.090532
Control wt Off 0.094695
Control wt Off 0.090148
Control wt Off 0.092456
Control wt Off 0.089232
Control wt Off 0.087391
Control wt Off 0.088678
Control wt Off 0.08661
Control wt Off 0.08521
Control wt Off 0.093471
Control wt Off 0.096193
Control wt Off 0.092051
Control wt Off 0.096917
Control wt Off 0.093224
Control wt Off 0.093049
Control wt Off 0.086349
Control wt Off 0.09197
Control wt Off 0.09481
Control wt Off 0.088265
Control wt Off 0.087995
Control wt Off 0.093768
Control wt Off 0.092452
Control wt Off 0.093554
Control wt Off 0.091013
Control wt Off 0.092197
Control wt Off 0.091564
Control wt Off 0.092555
Control wt Off 0.091568
Control wt Off 0.091733
Control wt Off 0.090646
Control wt Off 0.092359
Control wt Off 0.092906
Control mut Off 0.080744
Control mut Off 0.080259
Control mut Off 0.083163
Control mut Off 0.086617
Control mut Off 0.088843
Control mut Off 0.084012
Control mut Off 0.083491
Control mut Off 0.087721
Control mut Off 0.079138
Control mut Off 0.084372
Control mut Off 0.085647
Control mut Off 0.083235
Control mut Off 0.082303
Control mut Off 0.082718
Control mut Off 0.083421
Control mut Off 0.085423
Control mut Off 0.083907
Control mut Off 0.084434
Control mut Off 0.083579
Control mut Off 0.08739
Control mut Off 0.077767
Control mut Off 0.082712
Control mut Off 0.083855
Control mut Off 0.082008
Test wt On 0.092919
Test wt On 0.09262
Test wt On 0.093078
Test wt On 0.09881
Test wt On 0.093853
Test wt On 0.096927
Test wt On 0.099317
Test wt On 0.096221
Test wt On 0.089648
Test wt On 0.1029748574645363
Test wt On 0.093948
Test wt On 0.090152
Test wt On 0.097388
Test wt On 0.093304
Test wt On 0.098866
Test wt On 0.094205
Test wt On 0.1002179464073643
Test wt On 0.098311
Test wt On 0.097483
Test wt On 0.094112
Test wt On 0.091564
Test wt On 0.099103
Test wt On 0.093558
Test wt On 0.09489
Test wt On 0.094047
Test wt On 0.088951
Test wt On 0.09021
Test wt On 0.099451
Test wt On 0.089619
Test wt On 0.092389
Test wt On 0.09174
Test wt On 0.092366
Test wt On 0.091686
Test wt On 0.093976
Test wt On 0.088071
Test wt On 0.094398
Test wt On 0.092269
Test wt On 0.096504
Test wt On 0.096081
Test wt On 0.092522
Test wt On 0.099859
Test wt On 0.094489
Test wt On 0.089753
Test wt On 0.1015939484272903
Test wt On 0.091595
Test wt On 0.097142
Test wt On 0.09408
Test wt On 0.099405
Test wt On 0.09954
Test wt On 0.098229
Test wt On 0.094754
Test wt On 0.095153
Test wt On 0.10082
Test wt On 0.097235
Test wt On 0.096374
Test wt On 0.093771
Test wt On 0.089676
Test wt On 0.093124
Test wt On 0.093293
Test wt On 0.090029
Test wt On 0.086623
Test wt On 0.091985
Test mut On 0.084103
Test mut On 0.085283
Test mut On 0.088919
Test mut On 0.086686
Test mut On 0.085488
Test mut On 0.085284
Test mut On 0.085654
Test mut On 0.083617
Test mut On 0.086507
Test mut On 0.082866
Test mut On 0.090832
Test mut On 0.083236
Test mut On 0.083965
Test mut On 0.082022
Test mut On 0.0883
Test mut On 0.085009
Test mut On 0.082265
Test mut On 0.084906
Test mut On 0.085224
Test mut On 0.081131
Test mut On 0.083078
Test mut On 0.082238
Test mut On 0.091311
Test mut On 0.085612

I've copied the results below and also inserted the PNG and CSV files 

• The unpaired mean difference between Off wt and On wt is 0.00316 [95.0%CI 0.00209, 0.00424]. The P value of the two-sided permutation t-test is 0.0, calculated for legacy purposes only.
• The unpaired mean difference between Off mut and On mut is 0.00153 [95.0%CI 0.000178, 0.00311]. The P value of the two-sided permutation t-test is 0.0484, calculated for legacy purposes only.
• The delta-delta between wt and mut is -0.00163 [95.0%CI -0.00341, 0.000281]. The P value of the two-sided permutation t-test is 0.108, calculated for legacy purposes only.

2) The Mann-Whitney p-value is 0.08, but the t-test and permutation p-values are 0.0484. Which should I use? The interpretation changes depending on which I use.

3) Also, I have a 0.0 p-value for the mean difference between Off wt and On wt. I assume this is because the value is exceptionally small.

3) With regard to this data, the delta-delta between wt and mut is -0.00163, with a p-value of 0.108. Would I be correct in saying that mutants respond similarly to wt to treatment since ~ 51% of the effect size comes from the genotype and 49% from the effects of the treatment? 

4) Lastly, in terms of interpretation, when reporting the effect size, can/how could I report the effect size as a percentage? Would something like this work: ' Mutants displayed a 51% reduction in effects size for the parameter eye/body ratio in response to treatment compared to wt'?

[Jonathan Anns]

Hi, apologies for the delayed response!

1. If we understand your experiment correctly, the ‘Value’ column is eye/body ratio, the ‘Genotype’ column has the two genotypes Control and Mutant, and the ‘Condition’ and ‘Treatment’ columns both represent the addition (or not) of the treatment. You have then plotted in such a way as to get an effect size for the Control genotype (treatment on - treatment off) and the Mutant genotype (treatment on - treatment off). These therefore indicate the effect of the treatment on a given genotype. In addition, you have computed a delta-delta effect size (Mutant (treatment on - treatment off) - Control (treatment on - treatment off)). In your case, the delta-delta effect size indicates the difference in the effect of the treatment on the Mutant genotype versus the Control genotype - As far as we can see, this is in line with your desired intent.

2. Our aim with DABEST/Estimationstats is to guide users to focus on (and report) the effect size (and confidence of this effect size) of the experiment, and p-values are provided for legacy purposes only. Beyond this, there are resources online which describe permutation T-tests and Mann-Whitney U (E.g., even the Wikipedia page for permutation tests is pretty useful!). While there are differences between these two methods, both are valid non-parametric tests that make fewer assumptions about the distributions your data came from. To reiterate, the p-values we provide are for legacy purposes only, and we won't advise on how to interpret these or which to choose. 

P.s. If your interpretation changes based on the p-value threshold (p>=0.05 or p<0.05), maybe that should raise a question of the validity of the interpretation ;)

3. Due to the nature of a permutation t-test, it is possible to get a value of 0.0 (this is not due to rounding). 

4. We agree with your interpretation: It would be reasonable to suggest from the delta-delta effect size of -0.00163 [95.0%CI -0.00341, 0.000281] that the treatment does not have a meaningful difference in effect on Mutant versus Control genotypes. The fact that the treatment had a large effect on both the Control and Mutant animals indicates that it is causing non-specific changes to the phenotype unrelated to the mutation.

5. The mean of Mutant (Treatment on - off) does decrease by ~51% from the mean of Control (Treatment on - off), however, the purpose of the confidence interval visualisation for this effect size is to show whether this change is meaningful or not. From the graph and the answer to (4), it does not appear to be. 

~ Estimationstats team