fake frequentist statistics

[Ross Boylan]

More a Bayesian than a stan question:

I have estimates of effects from models estimated with stan. The PI is
very keen on (and is probably correct that the intended audience will be
keen) on getting p-values for effects, and also for questions like "do
the effects differ by race?"

How much justification is there for interpreting the posterior effect
estimates as frequentist estimate of the CI, and then doing frequentist
type tests on appropriate null hypotheses? BDA 3rd edition indicates
this is generally asymptotically OK for parameter estimates, but these
are effect estimates.

Yes, I know point null hypotheses are kind of senseless from a Bayesian
perspective (as BDA also argues; some frequentists have also argued that
it's silly), and I am pushing for estimates of the probability that the
effect is "small" in some substantive sense.

Ross Boylan

[Ross Boylan]

On 5/16/2014 12:03 PM, Ross Boylan wrote:
> More a Bayesian than a stan question:
>
> I have estimates of effects from models estimated with stan. The PI
> is very keen on (and is probably correct that the intended audience
> will be keen) on getting p-values for effects, and also for questions
> like "do the effects differ by race?"
>
> How much justification is there for interpreting the posterior effect
> estimates as frequentist estimate of the CI, and then doing
> frequentist type tests on appropriate null hypotheses? BDA 3rd
> edition indicates this is generally asymptotically OK for parameter
> estimates, but these are effect estimates.

One detail I suspect matters: the models are hierarchical and often have
low estimated variances for sets of parameters. The same section that
says the equivalence is asymptotically OK says this is for
non-informative priors; I think the hierarchical model makes the priors
informative.

My fake chi-square test for equality of a set of 15 effects had p = 0.97
and another test of 15 different effects had p = 0.98; these seem
implausibly high, particularly given that the smallest simulated value
for the difference between the highest and lowest parameter exceeded
10%. The single effect fake p-values (based on a normal approximation
with the mean and variance of the posterior) may not have been right,
but at least looked plausible.

[Michael Betancourt]

In general, no, posterior intervals will not have the correct frequentist coverage. The asymptotic
limit where everything becomes a Gaussian and the intervals do have the correct coverage is
a subtle one and unlikely to be reached in practice (otherwise we wouldn’t need all of this heavy
statistical machinery in the first place).

If you wanted to be formal then the closest you could get would be to build up posterior predictive
distributions for any transformation of the parameters of interest and compute p-values there. The
coverage won’t be correct because of the latent marginalization over parameters but it gives you
something you can at least attempt to understand from a frequentist perspective.

This also demonstrates why you’ll run into issues trying to estimate an overall effect — you’d need
a full joint distribution over the covariates in order to properly marginalize them out. I believe
post-stratification is the right approach if you did want to push ahead here.

In the end the answer you need is a political one and not a statistical one...

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ross Boylan]

On Sat, 2014-05-17 at 13:01 +0100, Michael Betancourt wrote:
> In general, no, posterior intervals will not have the correct frequentist coverage. The asymptotic
> limit where everything becomes a Gaussian and the intervals do have the correct coverage is
> a subtle one and unlikely to be reached in practice (otherwise we wouldn't need all of this heavy
> statistical machinery in the first place).
>
> If you wanted to be formal then the closest you could get would be to build up posterior predictive
> distributions for any transformation of the parameters of interest and compute p-values there. The
> coverage won't be correct because of the latent marginalization over parameters but it gives you
> something you can at least attempt to understand from a frequentist perspective.
>

> This also demonstrates why you'll run into issues trying to estimate an overall effect -- you'd need

> a full joint distribution over the covariates in order to properly marginalize them out. I believe
> post-stratification is the right approach if you did want to push ahead here.
>
> In the end the answer you need is a political one and not a statistical one...

Politically, there is a certain advantage to saying the requested
statistic is impossible to compute reliably.
Ross

[Michael Betancourt]

Politically, there is a certain advantage to saying the requested
statistic is impossible to compute reliably.

For what it’s worth I’ve found academics become strongly defensive

when you attempt this strategy.  People are ingrained with odd

assumptions about how statistics works and you have to either spend 

a long time convincing them otherwise or skirt the issue and produce

a ninja-statistic that sates their needs at the same superficial level

of their understanding.  I know that comes across as incredibly

impolite, but mathematically there really isn’t a compromise between

the two.

[Ross Boylan]

On Sun, 2014-05-18 at 21:36 +0100, Michael Betancourt wrote:
>
> > Politically, there is a certain advantage to saying the requested
> > statistic is impossible to compute reliably.
>
> For what it’s worth I’ve found academics become strongly defensive
> when you attempt this strategy. People are ingrained with odd
> assumptions about how statistics works and you have to either spend
> a long time convincing them otherwise or skirt the issue and produce
> a ninja-statistic that sates their needs at the same superficial level

What do you mean by a ninja-statistic?
Ross

> of their understanding. I know that comes across as incredibly
> impolite, but mathematically there really isn’t a compromise between
> the two.

[Michael Betancourt]

> What do you mean by a ninja-statistic?

Oh, just something that looks like a frequentist statistic
(say, a p-value) but doesn’t hold up upon mathematical
scrutiny (like a posterior tail probability).