Testing Distributions

[Peter Li]

I noticed that in the distribution testing framework, cdfs (and now
cdf_log and ccdf_log) test the lower bound/upper bound of the random
variable, y, but log prob doesn't. Is there a reason for this?


Peter

[Daniel Lee]

On Jun 28, 2013, at 11:50 AM, Peter Li wrote:
> I noticed that in the distribution testing framework, cdfs (and now cdf_log and ccdf_log) test the lower bound/upper bound of the random variable, y, but log prob doesn't. Is there a reason for this?

Yes, I couldn't reason around what we consider upper and lower bounds. If we take the normal_log(y, mu, sigma) as an example, we can push in {-inf, +inf} for y and mu and expect the result to be -Inf (well -- off by constants, I guess), but what about sigma? If we push in sigma = {0, Inf} do we expect -Inf? Does this hold in general? How do we encode in the tests the lower and upper bounds for every parameter? (I guess I did it for the cdf's, so maybe it's not that difficult.)

We could always test just the first argument, y.

Feel free to change the tests. I built them a while ago and haven't really gone back to update them.


Daniel

[Peter Li]

I thought the upper/lower bound tests were only for the first argument
regardless, especially since it only takes in one lower bound (in the
cdf tests). If the lower bound were assigned to all the params, I think
errors would be thrown, since in the case of the normal distribution,
the lower bound of y is negative infinity (so has_lower_bound() returns
false) but sigma has a lower bound of 0.0, so an error would be thrown
if we tried to pass in negative infty for sigma.


Peter

[Peter Li]

Ah, I see the issue now. The pdfs do not all have the same end
behavior. I've modified the test framework to take in both the lower
bound and the values that should be obtained at the boundary.

Do we want to generalize this so that the lower bounds and upper bounds
of each param are tested too [not just the first param, y]?

Also, should I move this to a separate branch? [Currently, it's on
feature/add_rayleigh_distr]


Peter

[Daniel Lee]

On Jun 28, 2013, at 1:48 PM, Peter Li wrote:
> Also, should I move this to a separate branch? [Currently, it's on feature/add_rayleigh_distr]

Yes, please.

> Ah, I see the issue now. The pdfs do not all have the same end behavior. I've modified the test framework to take in both the lower bound and the values that should be obtained at the boundary.
>
> Do we want to generalize this so that the lower bounds and upper bounds of each param are tested too [not just the first param, y]?

If you can reason around it, please do. I got stuck and I moved on. I'm sure we'll all be grateful if you figured it out.



Daniel

>
>> On 6/28/13 12:24 PM, Peter Li wrote:
>>> I thought the upper/lower bound tests were only for the first argument regardless, especially since it only takes in one lower bound (in the cdf tests). If the lower bound were assigned to all the params, I think errors would be thrown, since in the case of the normal distribution, the lower bound of y is negative infinity (so has_lower_bound() returns false) but sigma has a lower bound of 0.0, so an error would be thrown if we tried to pass in negative infty for sigma.
>>>
>>>
>>> Peter
>>>
>>> On 6/28/13 12:07 PM, Daniel Lee wrote:
>>>> On Jun 28, 2013, at 11:50 AM, Peter Li wrote:
>>>>> I noticed that in the distribution testing framework, cdfs (and now cdf_log and ccdf_log) test the lower bound/upper bound of the random variable, y, but log prob doesn't. Is there a reason for this?
>>>> Yes, I couldn't reason around what we consider upper and lower bounds. If we take the normal_log(y, mu, sigma) as an example, we can push in {-inf, +inf} for y and mu and expect the result to be -Inf (well -- off by constants, I guess), but what about sigma? If we push in sigma = {0, Inf} do we expect -Inf? Does this hold in general? How do we encode in the tests the lower and upper bounds for every parameter? (I guess I did it for the cdf's, so maybe it's not that difficult.)
>>>>
>>>> We could always test just the first argument, y.
>>>>
>>>> Feel free to change the tests. I built them a while ago and haven't really gone back to update them.
>>>>
>>>>
>>>> Daniel
>>>>
>>>>
>>>>
>>>>
>>>
>>
>

> --
> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

[Bob Carpenter]

On 6/28/13 2:01 PM, Daniel Lee wrote:
>
> On Jun 28, 2013, at 1:48 PM, Peter Li wrote:
>> Also, should I move this to a separate branch? [Currently, it's on feature/add_rayleigh_distr]
>
> Yes, please.
>
>
>> Ah, I see the issue now. The pdfs do not all have the same end behavior. I've modified the test framework to take in both the lower bound and the values that should be obtained at the boundary.
>>
>> Do we want to generalize this so that the lower bounds and upper bounds of each param are tested too [not just the first param, y]?
>
> If you can reason around it, please do. I got stuck and I moved on. I'm sure we'll all be grateful if you figured it out.

There was some discussion around the behavior that CDFs should
have outside of their support. I would prefer to check the
arguments to CDFs just like the probability functions and have
the same configurable behavior.

For example, uniform(L,U) has support in [L,U] so in
unfirom_log(y,L,U) a check is made that y is in [L,U] and
if not, an exception is raised (or -infinity or NaN returned depending
on config). I'd like to do the same thing with uniform_cdf(y,L,U).

- Bob

[Peter Li]

Are you referring to the checks such as check_positive,
check_nonnegative, check_finite, etc.? I believe an exception is raised
when those aren't satisfied.


Peter

[Peter Li]

Then is the best path to just merge add rayleigh distr and then I'll add
the appropriate tests in the new branch for it?


Peter

[Bob Carpenter]

On 6/28/13 2:39 PM, Peter Li wrote:
> Are you referring to the checks such as check_positive, check_nonnegative, check_finite, etc.? I believe an exception is
> raised when those aren't satisfied.

Good.

Was your message just about the testing framework itself and
not what needed to be tested?

- Bob

[Peter Li]

On 6/28/13 2:43 PM, Bob Carpenter wrote:
>
> On 6/28/13 2:39 PM, Peter Li wrote:
>> Are you referring to the checks such as check_positive,
>> check_nonnegative, check_finite, etc.? I believe an exception is
>> raised when those aren't satisfied.
>
> Good.
>
> Was your message just about the testing framework itself and
> not what needed to be tested?
>

Correct. It was whether we should test the specific upper/lower bounds
of each param. I think I can modify the current framework to do such
testing.


Peter