two questions on bootstrapping techniques to compare differences between independent groups

[Tom Franken]

Hi,

Thank you for developing and sharing the DABEST software, it is really helpful.

I have 2 questions:

I would like to use BCa correction to compute a bootstrap confidence interval on the difference of a complex statistic for independent groups.

 When I was looking at the Dabest Python code, I saw that you do the jackknifing for independent groups by leaving out one element for each group separately, and the use of zip() will limit the number of jackknife samples to the size of the smallest group. Thus, for example, if you compare a group with 19 samples and a group with 25 samples, the number of jackknife samples for the groups together will be 38 (2x19). Is this indeed the intended strategy? I guess this way you avoid over-weighing the group with more data points? 

Second, I was wondering if you could point me to some literature regarding using a permutation test versus bootstrap methods to compare two independent groups. 

I am finding that permutation tests can be less sensitive to detect differences between groups. I think this may be due to the strong requirement that both populations need to be identical under the null hypothesis to use a permutation test? Is it therefore correct that bootstrapping differences in effect size between groups is more widely applicable than a permutation test? Do you know of any papers/books that discuss this? I find very little literature on the use of bootstrapping techniques to compare differences between groups.

Thank you!

Tom Franken

Postdoctoral Fellow

Systems Neurobiology Laboratories - Reynolds

The Salk Institute for Biological Studies

[Adam Claridge-Chang]

Hi Tom,

Thanks for your questions. I'll have to let Joses answer the first question.

For your second question, TL;DR: I think it's more important to use some form of effect sizes and estimation graphics, than to get into the weeds on which specific method.

Longer answer:

We are coming from a position that p-values and significance testing are almost entirely useless. We included p-value calculations just so that users who need to satisfy legacy requirements (misinformed editors, reviewers, and bosses) can do so without having to use another package. We provide the permutation p as a default method that uses resampling like the bootstrap. Nevertheless I think discussion of p-values is beside the point. I don't think it's critical to use bootstrapping for estimation, but it is a nice way to calculate effect-size distribution curves and CIs with some degree of robustness for weird distributions. The aspect of methodology that I find interesting is the different flavors of effect size, e.g. mean difference, Hedges g, Cliff's delta, R^2, etc which can be useful in different contexts.

There are some textbooks on the bootstrap by Efron and Tibshirani you might find helpful.

Best wishes,

Adam