Using of log predictive density rather than log posterior for measuring predictive accuracy

[Ben Lambert]

Hi,

I am trying to understand better a section in BDA3:

"Given that we are working with the log predictive density, the question may arise: why not use the log posterior? Why only use the data model and not the prior density in this calculation? The answer is that we are interested here in summarizing the fit of model to data, and for this purpose the prior is relevant for estimating the parameters, but not assessing a model's accuracy."

For some reason, I am finding it hard to understand this. I cannot seem to explain why we don't use the log posterior density for scoring a model. Why wouldn't we want to choose the model that had demonstrated the highest posterior predictive probability of generating a new data point?

Does anyone have any intuitive logic for why this is the case? Or any papers etc. I could read to further explain this? Sorry for my lack of understanding here!

Best,

Ben

[Andrew Gelman]

Hi, Ben.  We discuss this a bit more in my paper with Vehtari and Hwang.  The short answer is that for this person we are just trying to evaluate predictive accuracy, and the prior is not relevant to this.

A

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Lambert]

Hi Andrew,

Thanks for your message. I am still a bit 'at sea' though I'm afraid (I have read the Vehtari, Hwang paper). What I don't quite see is why the log-likelihood measures the predictive accuracy, whereas the posterior predictive distribution does not? If I were to predict new data I would use the latter since it incorporates all knowledge - prior and data. What am I missing?

Sorry again for the misunderstanding.

Ben

[Michael Betancourt]

The posterior is a distribution over _parameters_ so it doesn’t tell you

anything about prediction.  Prediction is defined by new measurements,

which means you need a distribution over _measurements_.  So

mathematically the posterior isn’t even applicable.

Really the only distribution over measurements that we have are the

distributions defined in the likelihood, but to make a prediction we

need to narrow this down to a single distribution.

In frequentist methods one selects a single “best” parameter and then

uses the corresponding distribution, L(data | theta_best), to make

predictions.  This is especially common in machine learning.

From a Bayesian perspective we can average over the likelihood.

Using the prior gives the prior predictive distribution,

p_{prior predictive} (data_new | data_meas) = \int dtheta L(data_new | theta) p(theta).

This was used by George Box back in the day but has some pathologies

that limits its use.  Instead it’s much more robust to average over

the posterior to incorporate what we’ve learned into the prediction,

p_{post predictive} (data_new | data_meas) = \int dtheta L(data_new | theta) p(theta | data_meas).

This is the formally correct “posterior probability for a new data point”

and is the basis for posterior predictive checks, WAIC, etc.

On Oct 7, 2015, at 1:28 AM, Ben Lambert <ben.c....@googlemail.com> wrote:

Hi Andrew,

Thanks for your message. I am still a bit 'at sea' though I'm afraid (I have read the Vehtari, Hwang paper). What I don't quite see is why the log-likelihood measures the predictive accuracy, whereas the posterior predictive distribution does not? If I were to predict new data I would use the latter since it incorporates all knowledge - prior and data. 

Sorry again for the misunderstanding.

Ben

On Wednesday, October 7, 2015 at 1:03:44 AM UTC+1, Andrew Gelman wrote:

Hi, Ben.  We discuss this a bit more in my paper with Vehtari and Hwang.  The short answer is that for this person we are just trying to evaluate predictive accuracy, and the prior is not relevant to this.

A

On Oct 6, 2015, at 7:50 PM, Ben Lambert <ben.c....@googlemail.com> wrote:

Hi,

I am trying to understand better a section in BDA3:

"Given that we are working with the log predictive density, the question may arise: why not use the log posterior? Why only use the data model and not the prior density in this calculation? The answer is that we are interested here in summarizing the fit of model to data, and for this purpose the prior is relevant for estimating the parameters, but not assessing a model's accuracy."

For some reason, I am finding it hard to understand this. I cannot seem to explain why we don't use the log posterior density for scoring a model. Why wouldn't we want to choose the model that had demonstrated the highest posterior predictive probability of generating a new data point?

Does anyone have any intuitive logic for why this is the case? Or any papers etc. I could read to further explain this? Sorry for my lack of understanding here!

Best,

Ben

--

You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

[Ben Lambert]

Hi Michael,

Thanks for your message. Sorry, I am still not understanding though. I do understand that the posterior is a distribution over _parameters_, but the posterior predictive distribution is a distribution over _data_ . What I don't understand is why we can't use the latter for new measurements?

Best,

Ben

[Bob Carpenter]

Michael's just defining the posterior predictive for you.

The new observations are going to be random variables, just
ones that we happen to observe. This whole data vs. parameter
thing is tricky --- by "data" in Stan we just mean the values
that are observed (constants or modeled data) and by "parameters"
we just mean values we're trying to infer. We try to clarify
these uses in the language section of the manual, following
Gelman and Hill's regression book's classification of variables.

- Bob

[Ben Lambert]

Of course! Thank you very much for clearing up my misunderstanding! Sorry for the protracted series of messages.

Best,

Ben

[Andrew Gelman]

Yes, we do use the posterior predictive distribution for new data.

On Oct 7, 2015, at 9:17 AM, Ben Lambert <ben.c....@googlemail.com> wrote:

[wor...@gmail.com]

Along the same topic, I have some questions related to BDA3.  I have a hard time understanding the log of a "density" as a score because 

1. Density functions are not dimensionless, but have a unit of 1/dy, so it's hard to interpret it.  If the sample contains many iid points, then p(y | theta) = product of p(y_i | theta), which would have a unit of 1/(y^n).

2. A density function could have point values greater than 1 (if it's narrow enough), meaning log(p|theta) can change sign.  That means removing 1 data point away from my y_i's could result in p(y | theta) changing sign.

3. In the broader context of KL divergence, it's hard to picture how the aforementioned sign change would impact the divergence value.  I can imagine situations where D(P || Q) ~ 0 when P and Q are discernibly distant from each other.

Thanks.

Best,

CW

[Michael Betancourt]

1. Density functions are not dimensionless, but have a unit of 1/dy, so it's hard to interpret it.  If the sample contains many iid points, then p(y | theta) = product of p(y_i | theta), which would have a unit of 1/(y^n).

Correct — the only well-posed calculation in probability theory are expectations

which reduce to integration.  The dimensionality washes away when you integrate 

a density over some region of parameter space.

2. A density function could have point values greater than 1 (if it's narrow enough), meaning log(p|theta) can change sign.  That means removing 1 data point away from my y_i's could result in p(y | theta) changing sign.

Yes, log p(y | theta) can oscillate between signs as more or less data is considered.

And the exact nature of that oscillation will depend on the parameterization and

hence density that you have chosen.  There’s nothing wrong with that provided

you keep (1) in mind.

3. In the broader context of KL divergence, it's hard to picture how the aforementioned sign change would impact the divergence value.  I can imagine situations where D(P || Q) ~ 0 when P and Q are discernibly distant from each other.

The KL divergence, really any of the divergences from information theory, are

parameterization-invariant — they are maps between probability distributions 

to a number, not densities.  More operationally, all of the odd paramterization-

dependent properties of densities cancels out when you evaluate the 

divergence (or you can write it purely measure-theoretically using 

Radon-Nikodym derivatives, if you’re into that kind of thing).  Really this has

to be true according to (1).

Now all of this gets muddied when people try to argue about quantifying

predictive performance.  The explicit values of the “log predictive density”

score depends on the parameterization, but importantly the _ordering_

of the log predictive density from different models is preserved.  When

the literature discusses log densities as valid “scores” they ultimately are

referring to its ability to determine the ordering between models up to

certain technical assumptions.

Many gory details at https://arxiv.org/abs/1506.02273.