Multiple comparisons example, with questions

[Mike Lawrence]

Hi folks,

For some teaching I'm doing, I coded up an example of a scenario where multiple comparisons might be of concern. Andrew posted code to do this analytically on his blog back in the August (http://andrewgelman.com/2016/08/22/bayesian-inference-completely-solves-the-multiple-comparisons-problem/) but I thought it would be useful to show it using Stan. Code is here (https://gist.github.com/mike-lawrence/a124e7ae144ecdab93f36f87ef8c513e), but a couple questions: 

- The thing I initially found curious was that the "unpooled" model actually achieved the same "false positive" rate as the frequentist lm(). This contrasts with my expectation from the plot Andrew posted but looking at the actual paper I see that the Bayesian model used there took the mu/sigma/tau parameters as known where my "un-pooled effects" model doesn't; would this account for the discrepancy?

- As expected, the "partially_pooled" model achieves a 0% false positive rate when the true value of all predictors' influence on the outcome is zero. Then I add one predictor with a non-zero effect on the outcome to see how this affects the standard advice to explicitly model the distributions of beta values as a normal distribution with an mean and sd that are estimated. As I suspected, adding a non-zero effect causes the false-positive rate for the other predictors to jump up again. This motivated the "heavy_partially_pooled" model that uses a heavy-tail student-t distribution for the effects and explicitly estimates the necessary tail-heaviness. This drops the false positive rate back down to 0%, but I notice that the effective sample size for the df & sd parameters of the student-t are rather low for a reasonably high iteration count; is this a problem?

Mike

(p.s. trigger warning: I forgot to convert from tabs to spaces before finalizing the gist, and github won't let me change.)

--
Mike Lawrence
Graduate Student
Department of Psychology & Neuroscience
Dalhousie University

~ Certainty is (possibly) folly ~