question: can actual data be outside 95%CI of replicated data

[Linas Mockus]

Hi,

I am struggling with a very strange and non-intuitive result: actual data is outside 96% CI of replicated data. Is this possible. I am using hierarchical model (attached). y_rep is a replicated data while y is actual data (% dissolved).

Is this possible?

Thanks,
Linas

[Bob Carpenter]

Yes. I should happen roughly 4% of the time :-)

- Bob

P.S. We're shutting this list down soon and moving to:

http://discourse.mc-stan.org

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
> <doa.stan>

[lmockus]

Thanks a lot. Is there any explanation why?

Linas

>You received this message because you are subscribed to a topic in the

>Google Groups "Stan users mailing list" group.

>To unsubscribe from this topic, visit
>https://groups.google.com/d/topic/stan-users/Pft2DnUe4PE/unsubscribe.
>To unsubscribe from this group and all its topics, send an email to

[Ben B]

Usually when you define 95% intervals you're saying that something has a 95% chance of being in that interval. That means 5% of the time you're outside of it.

Your data being outside the 95% intervals of your posterior predictives can also mean that your model isn't doing a good job of explaining your data.

Is it 5% of your data that's outside of the 95% intervals? (this is perfectly normal)

Or is most of your data outside of the intervals? (this can easily happen when you're modeling, and usually means you need to change your model)

Ben

 

On Sat, May 13, 2017 at 4:40 PM, lmockus <linasm...@gmail.com> wrote:

Thanks a lot. Is there any explanation why?

Linas

------ Original Message ------
From: "Bob Carpenter" <ca...@alias-i.com>
To: stan-...@googlegroups.com
Sent: 5/13/2017 7:00:44 PM
Subject: Re: [stan-users] question: can actual data be outside 95%CI of replicated data

Yes.  I should happen roughly 4% of the time :-)

- Bob

P.S.  We're shutting this list down soon and moving to:

http://discourse.mc-stan.org

 On May 13, 2017, at 5:43 PM, Linas Mockus <linasm...@gmail.com> wrote:

 Hi,

 I am struggling with a very strange and non-intuitive result: actual data is outside 96% CI of replicated data. Is this possible. I am using hierarchical model (attached). y_rep is a replicated data while y is actual data (% dissolved).

 Is this possible?

 Thanks,
 Linas

 --
 You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

 To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.

 To post to this group, send email to stan-...@googlegroups.com.
 For more options, visit https://groups.google.com/d/optout.
 <doa.stan>

--

You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/Pft2DnUe4PE/unsubscribe.

To unsubscribe from this group and all its topics, send an email to stan-users+unsubscribe@googlegroups.com.

To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.

[lmockus]

Ben, thanks a lot. Apparently only 2.5% of data is outside. But still it looks bad.

To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.

[Ben B]

Thank Bob too! You should take the data and model over to the new discourse and see if anyone has any opinions on it (http://discourse.mc-stan.org). Looks neat, but there are more eyes over there.

Ben

[Marco Inacio]

In average, 5% of data will be outside of a 95% "CI" obtained directly the true data generating function itself (that is, an interval with 95% probability that a new data point will be inside the interval, given the true data generating function).

So an interval with estimated probability of 95% shouldn't be any better and in fact, if it is "better" than that, then it might be a bad feature... just imagine: you could just say that your CI is the whole line of real numbers (or whatever sample space you have), then your data would be 100% there (and that's why any sensible loss functions would penalize the "size" of the interval since otherwise, you could just choose the full sample space and forget statistics).

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Bob Carpenter]

See:

http://andrewgelman.com/2017/04/12/bayesian-posteriors-calibrated/

I'm working on a more detailed case study, but that and the linked
paper will give you the math.

- Bob

[Bob Carpenter]

> On May 13, 2017, at 7:59 PM, Ben B <bbba...@gmail.com> wrote:
>
> Usually when you define 95% intervals you're saying that something has a 95% chance of being in that interval. That means 5% of the time you're outside of it.
>
> Your data being outside the 95% intervals of your posterior predictives can also mean that your model isn't doing a good job of explaining your data.

It depends on what we're talking about here. This can happen even
when you're using exactly the same model as was used to generate
the data. In fact, it'll happen 5% of the time. In that case, it's
a "bad" draw of data.

This can also happen if the model's misspecified, which is usually
why we run these on real data.

- Bob