Permutation p-value of 0.0 - further query

[Michael Lotinga]

Hi, I'm a user of DABEST-python, which is a great package.

First, I agree completely with the estimationstats philosophy concerning minimising the attention paid to p-values with focus instead on effect sizes and confidence intervals.

Second, I know there was a previous question on this issue, which I've already read — I'd just like to add something to the query

https://groups.google.com/g/estimationstats/c/AFoTa0rIHag/m/74cMbrTwCAAJ 

The response to that question was along the lines of 'the effect size is so huge that the permutation p-value is effectively 0'.

In my application I've found something a bit odd: For the same repeated measures dataset, I've noticed that one estimation (sequential pairing) produces a set of sizeable effect sizes, and the three permutation p-values for that effect are all calculated as '0.0'. However, another estimation (different grouping variables and baseline pairing) produces some larger effect sizes in the set, with permutation p-values output as '0.0000', the formatting of which is consistent with the other (larger) p-values in the same set.

So, is this just a formatting issue, or is there something else going on?

My effect size tables (apologies for formatting, I've highlighted the relevant values in underline and bold):

Set 1

control test control_N test_N effect_size is_paired difference ci bca_low bca_high pvalue_permutation pvalue_wilcoxon statistic_wilcoxon pvalue_paired_students_t statistic_paired_students_t
0 42 48 369 369 Cohen's d sequential 0.455334 95 0.344373 0.568954 0.0 1.012021e-15 8576.0 2.420381e-15 -8.273149
1 48 54 369 369 Cohen's d sequential 0.375490 95 0.267149 0.482397 0.0 2.516079e-13 10578.5 5.236929e-12 -7.133314
2 54 60 369 369 Cohen's d sequential 0.401178 95 0.301119 0.502650 0.0 1.623886e-12 10817.5 1.563889e-13 -7.669002

Set 2

control test control_N test_N effect_size is_paired difference ci bca_low bca_high pvalue_permutation pvalue_wilcoxon statistic_wilcoxon pvalue_paired_students_t statistic_paired_students_t

[omitted rows 0-3 as not relevant]

4 T150 Flyby M300 Takeoff 164 164 Cohen's d baseline 0.260034 95 0.120406 0.401062 0.0000 5.676448e-04 2424.0 4.015446e-04 -3.613871
5 T150 Flyby H520 Flyby 164 164 Cohen's d baseline 0.190719 95 0.064082 0.333781 0.0050 9.464494e-03 2706.5 7.273684e-03 -2.718222
6 T150 Flyby H520 Landing 164 164 Cohen's d baseline 0.326214 95 0.178402 0.478751 0.0000 2.140203e-05 2232.5 5.024485e-05 -4.165546
7 T150 Flyby H520 Takeoff 164 164 Cohen's d baseline 0.434757 95 0.283147 0.587144 0.0000 5.238470e-07 2117.5 1.629070e-07 -5.474587

[Michael Lotinga]

Sorry, the underlined heading in the tables should be 'difference' rather than 'effect size', but hopefully it's obvious what is meant!

[Jonathan Anns]

Hi, thanks for the question!

This difference in the number of zeros is most likely due to pandas formatting whereby it adjusts to the largest number of decimal places within a given column. 
See the attached image for an example!

~ Estimationstats team

[Michael Lotinga]

Ok, thank you for this response!