Q re permutation t-test & interpretation of data when estimation stats and significance testing don't agree

[Yajing Xu]

Hi everyone!

I'm fairly new to estimation stats and have a few questions regarding the web app as well as interpretation of data.

1. As I understand it, the permutation t-test offered on the website has itself nothing to do with the estimation stats, and is just a separate t-test, is that right? I.e. if my data is ordinal or does not have equal variance, can I use MannWhitney U or the Welch's t-test instead of the permutation t-test in addition to the estimation stats?
2. In one of my two-sample test, zero lied outside the 95% CI of estimation stats, but the p-value for the t-test (Mann Whitney) was 0.054, and non-significant. Now I know that significance thinking should be abandoned, but how would you best interpret/phrase the result for a journal? (Or in general when estimation stats and significance testing don't agree on the outcome...)

Many thanks for your help!

Best,

Yajing

[Adam Claridge-Chang]

Hi Yajing,

1. As I understand it, the permutation t-test offered on the website has itself nothing to do with the estimation stats, and is just a separate t-test, is that right? I.e. if my data is ordinal or does not have equal variance, can I use MannWhitney U or the Welch's t-test instead of the permutation t-test in addition to the estimation stats?

Yes, we chose permutation p-values that simply because it uses resampling like the bootstrap, and is fairly robust for non-normality. We strongly discourage significance testing (choosing alpha and reporting 'significance') and believe p-values of any kind should not be used to interpret data.

1. In one of my two-sample test, zero lied outside the 95% CI of estimation stats, but the p-value for the t-test (Mann Whitney) was 0.054, and non-significant. Now I know that significance thinking should be abandoned, but how would you best interpret/phrase the result for a journal? (Or in general when estimation stats and significance testing don't agree on the outcome...)

Ignore the p-value and focus on the effect size. Is the effect interesting? This puts responsibility in your hands where it belongs.

Adam

[Yajing Xu]

Thanks for the reply!

One more question: I have seen that sometimes that the permutation t-test returns the value 0 - is this because the the value is too small to be accurately calculated? If yes, what is the cut-off? I want to put the p-values in supplementary for my paper, so I was wondering how to report that, e.g. P <0.0001 or sth like that? Thanks again!