quantifying compatibility of independent posterior distributions

[Luigi Acerbi]

Dear Stans,

I have n posterior distributions computed independently and from different sources for what I believe to be the same quantity of interest θ, that is f1(θ), f2(θ), ..., fn(θ).

Each fk(θ) is obtained by MCMC and smoothed with a KDE; but for this question let's assume that the fk(θ) are simply given and I cannot recompute them.

I would like to combine the information from these posteriors in a single posterior by multiplying together the fk(θ)

(let's assume that the prior contribution for each fk(θ) is absent or negligible so that there is no trouble with that).

Before combining the fk(θ), it seems to be a sane thing to check whether these posteriors are "compatible" in the first place.

Intuitively, I would plot the fk(θ) and check that they somewhat overlap (e.g., the 95% CI overlap).

If they don't, I may have to revise my assumption that the posteriors are actually measuring the same thing.

Is there a (somewhat) established statistical way of doing that & quantifying "compatibility" in this sense, besides the noble ancient technique of visual inspection?

(I am asking since I can think of many ways of quantifying this; I wonder if there is a standard.)

Thanks,

Luigi

[Michael Betancourt]

There is no statistically rigorous way of combining samples like this,

regardless of whether they look “compatible” or not.  You can combine

independent likelihood components this way, but not posteriors as at

the very least the prior correlates all of the components.  This means

that the total posterior does _not_ factor into independent components,

at least unless your posterior is exactly Gaussian.

There have been many papers trying to approximate this process when

the posterior is approximately Gaussian, but they all end up quite fragile

as even small perturbations away from a Gaussian distribution can skew

the results.

Unless you can fit everything jointly, we can’t offer any advice other than

“try it and see if it’s good enough for you”.

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Luigi Acerbi]

Hi Michael,

thanks for the answer, but the question was really about a compatibility statistic.

Please forget for now about combining the posteriors (or, more correctly, likelihoods -- I did say to ignore the prior :)).

My question is more basic -- I have these say two posteriors coming from two experiments (e.g., measurements of supposedly the same quantity of interest).

Intuitively, if they are extremely narrow, Gaussian-like and separated by say 20 SDs, I have to conclude that the sampling procedure is wrong, or the model is wrong (e.g., massively misspecified), or the two experiments are not measuring the same quantity.

So, I was looking for some empirical statistic to measure this compatibility (similar in spirit to e.g. the Rhat statistic for chain convergence).

What I'll do next is up to me -- for example, if I find that they are compatible, I can rerun the analysis on the joint dataset.

If the posteriors are "suspiciously" incompatible, I might go and re-check my assumptions.

So, nothing beyond visual inspection?

Thanks,

Luigi

[Bob Carpenter]

We use split R-hat to test whether two sets of draws from a
posterior are from the same distribution. This assumes they
have the same number of parameters.

What we'd usually recommend is that if you have shared parameters theta
and two sets of data, y1 and y2, then the full joint model is

p(y1, y2, theta) = p(y1 | theta) * p(y2 | theta) * p(theta)

That's all koshwer. But it's combinining likelihoods for different
groups of data, not combining posteriors.

- Bob

[Michael Betancourt]

Right, combine the likelihoods and then fit jointly.

All of our tools for comparing distributions, for example Rhat of
samples from the distributions, assume that the distributions are
all the same. In this case that would imply the same data and
the same parameters in _every_ model being considered.

But if that’s the case then you wouldn’t need to combine the
separate inferences at. I’m being cagey because I’m 90% that
what you want to do isn’t really a valid statistical procedure and
hence you’ll have to resort to approximations and hacks and
deal with the consequences yourself.

[Luigi Acerbi]

Thanks again.

I appreciate what's the correct way of combining priors and likelihoods, and there are issues with the multiplicity of priors if you have posteriors from multiple sources -- that's why I said to assume that the prior is flat or negligible (so that you are effectively multiplying likelihoods). What I wrote is correct under the specified assumptions, although I understand why it may have raised flags.

Anyhow, forget about combining these distributions -- honestly, the question was much simpler.

I just want to check whether these "posteriors under a flat prior" are compatible.

If they are compatible, I might do the kosher thing.

But first I want to check whether I should instead consider other models in which for example y1 and y2 are actually measuring different things.

And yes, I know that the truly kosher thing to test whether these posteriors are measuring the same thing would be to do some joint fits & model comparison in which theta is either the same for y1 and y2, or there are two distinct parameters theta1 and theta2 for respectively y1 and y2, and then I could do some model selection, or build a complete model and do inference within it (RJMCMC), or perhaps have another parameter that interpolates between these scenarios (unique theta vs. distinct thetas). All possible and statistically sound ways of dealing with this problem -- and expensive in my case.

To be honest, I don't think it's incorrect to use a heuristic check before embarking in an expensive analysis, but perhaps we see it differently here.

(I am not a statistician, just a former theoretical physicist turned to computational neuroscience -- I like to hack.)

We use split R-hat to test whether two sets of draws from a 
posterior are from the same distribution.  This assumes they 
have the same number of parameters. 

Saying that two sets of draws are drawn from the same distribution is much stronger than what I want to check for.

The posteriors I have might very well be different even if they are about the same quantity, since they come from different data (actually, there is really no reason why they should be the same).

The indirect conclusion from this discussion seems to be that there is no such (possibly heuristic) "compatibility statistic"...

Thanks!

Luigi

[Bob Carpenter]

> On Dec 8, 2016, at 4:15 PM, Luigi Acerbi <luigi....@gmail.com> wrote:
>
> Thanks again.
>
> I appreciate what's the correct way of combining priors and likelihoods, and there are issues with the multiplicity of priors if you have posteriors from multiple sources -- that's why I said to assume that the prior is flat or negligible (so that you are effectively multiplying likelihoods).

I don't see how the priors matter, but then I don't know what you
mean by "compatible".

> What I wrote is correct under the specified assumptions, although I understand why it may have raised flags.
> Anyhow, forget about combining these distributions -- honestly, the question was much simpler.
>
> I just want to check whether these "posteriors under a flat prior" are compatible.

What do you mean by "compatible"?

> If they are compatible, I might do the kosher thing.
> But first I want to check whether I should instead consider other models in which for example y1 and y2 are actually measuring different things.
>
> And yes, I know that the truly kosher thing to test whether these posteriors are measuring the same thing would be to do some joint fits & model comparison in which theta is either the same for y1 and y2, or there are two distinct parameters theta1 and theta2 for respectively y1 and y2, and then I could do some model selection, or build a complete model and do inference within it (RJMCMC), or perhaps have another parameter that interpolates between these scenarios (unique theta vs. distinct thetas). All possible and statistically sound ways of dealing with this problem -- and expensive in my case.
>
> To be honest, I don't think it's incorrect to use a heuristic check before embarking in an expensive analysis, but perhaps we see it differently here.

It depends on the false positive and false negative rate of your heuristic
and how important the problem is.

> (I am not a statistician, just a former theoretical physicist turned to computational neuroscience -- I like to hack.)
>
>
> We use split R-hat to test whether two sets of draws from a
> posterior are from the same distribution. This assumes they
> have the same number of parameters.
>
> Saying that two sets of draws are drawn from the same distribution is much stronger than what I want to check for.
> The posteriors I have might very well be different even if they are about the same quantity, since they come from different data (actually, there is really no reason why they should be the same).
>
> The indirect conclusion from this discussion seems to be that there is no such (possibly heuristic) "compatibility statistic"...

Do they have the same parameters? You can do something like
measure KL divergence. Or check interval coverage. But it's
not clear what you want to do from these messages.

- Bob

[Luigi Acerbi]

What do you mean by "compatible"?

Compatibility is how likely are two or more measurements to be about the same quantity of interest.

In Bayesian terms:

- H0 is that p(theta1|y1) and p(theta2|y2) represent posteriors about the same variable theta == theta1 = theta2.

- H1 is that theta1 and theta2 are distinct (physically, or because something went wrong in the measurement, or for whatever reason).

A fully Bayesian approach would compute the posterior ratio p(H0 | y1, y2) / p(H1| y1, y2) and then do something with it. The problem is that the full Bayesian approach is expensive.

The heuristic I have in mind is probably something along classical hypothesis testing.

The idea is that, for example, if p(theta1|y1) and p(theta2|y2) have near-zero overlap (and I am confident about the sampling procedure), I probably (barring weird priors or other caveats) don't even need to run the joint fit and model selection to say that we are in scenario H1 (theta1 and theta2 are different).

It depends on the false positive and false negative rate of your heuristic
and how important the problem is.

Good point, I agree -- that's what I am planning to investigate with some toy examples.

 

> The indirect conclusion from this discussion seems to be that there is no such (possibly heuristic) "compatibility statistic"...

Do they have the same parameters?  You can do something like
measure KL divergence.  Or check interval coverage.  But it's
not clear what you want to do from these messages.

Yep, I am working out something based on coverage.

Thanks,

Luigi

PS: Actually, it looks like I can be (approximately) Bayesian in answering the original question H0 vs H1, so I'll go along that way.

[Michael Betancourt]

To be honest, I don't think it's incorrect to use a heuristic check before embarking in an expensive analysis, but perhaps we see it differently here.

(I am not a statistician, just a former theoretical physicist turned to computational neuroscience -- I like to hack.)

I am a former experimental particle physicist turned statistician.  I’ve seen

all the excuses for hacks, but 1 - epsilon of the time they have poor

practical consequences.  When you’re having trouble trying to well-pose

your problem, like trying to define “compatibility” rigorously, it’s usually

a sign that you’re trying to do something ill-posed within probability theory.

If you want robust approximations that will be accurate in practice then

you need to start with a well-posed approach and _then_ start adding

principled approximation whose consequences you can understand.

Again, I’m not speaking out of any speculative zealousness but mounds

of empirical evidence.  Take of that what you will, but consider it a friendly

foreboding.

[Luigi Acerbi]

Hi Michael, thanks for the advice.

I think we actually agree after all -- apologies for being unclear. That's why I came here asking for your wisdom rather than coming up with something. :)

Talking to you and Bob was very helpful in teasing out my problem.

I think I do have a well-posed definition of compatibility (perhaps I should have called it consistency), as per my previous email, in terms of two causal scenarios H0 (all measurements are the same) and H1 (all, or some, measurements are independent -- perhaps strong assumption but I need to start somewhere), which -- once I settle on the priors -- I can deal with in Bayesian terms, and it boils down to a (log) posterior ratio which I can take as my measure of consistency. And then I can do approximations, if needed.

Thanks again for the patience!

Best wishes,

Luigi