Quick question about efficient internal parametrization
117 views
Skip to first unread message
PierGianLuca
Sep 13, 2022, 2:57:23 AM9/13/22
to nimble...@googlegroups.com
Dear Nimble devs & users,
Just a quick question to see if I got things right. Suppose I have a variate with gaussian probability distribution, whose variance in turn has an inverse-Gamma distribution (so there's conjugacy). Alternatively I can use the precision instead, with a Gamma distribution. Which of these two representation is computationally more efficient for Nimble? For my purposes either representation works well, so it makes sense to use the computationally most efficient one.
If I understand the manual correctly, Nimble internally uses the precision, so that's the representation I'm currently using. But I wanted to make sure that's really the case.
And what about the multivariate case with Wishart distributions?
Thank you very much!
Luca
Chris Paciorek
Sep 13, 2022, 7:54:26 PM9/13/22
to PierGianLuca, nimble-users
Hi Luca,
Short answer: I don't think it matters much what parameterization you use in the cases I can think of at the moment.
Longer answer:
We do some parameter transformations under the hood, ultimately using the 'canonical' parameterizations shown in the Table 5.2 of the manual.
Note that if the value of a given parameter has not changed, then nimble's algorithms will just used the cached value of the canonical parameter.
To the extent you can set up priors that are conjugate on the scale of the canonical parameterization, that will save some computation. E.g., if it were possible to set up a conjugate prior on the standard deviation of a Gaussian distribution that would avoid having to calculate the standard deviation based on the precision or the variance, which is what happens in the two cases you mention (we use standard deviation internally, not precision, because that is what Rmath's C code uses). But often this won't be possible. Whether a given non-canonical parameterization is more efficient than another non-canonical parameterization will depend on the computational cost of calculating the canonical parameter from the non-canonical.
Now in the case of scalars, this probably won't matter much. The case where it might is with multivariate distributions. For example with the Wishart, the canonical parameterization is in terms of the Cholesky decomposition of either the scale or inverse scale matrix. So we have to do a Cholesky decomposition whenever the scale/inverse scale changes. Then we have to compute the density, which depending on whether one is working with scale or inverse scale either involves matrix multiplication or solving a system of equations. It turns out the computational cost should be about the same regardless of whether working with scale or inverse scale.
Similar things come up with the multivariate normal, but again one either ends up doing matrix multiplication or solving systems of equations and the cost should be about the same.
-chris
--
You received this message because you are subscribed to the Google Groups "nimble-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/c0e6cafd-3fa4-f61d-5a39-8142048ad12c%40magnaspesmeretrix.org.
PierGianLuca
Sep 14, 2022, 12:11:14 AM9/14/22
to paci...@stat.berkeley.edu, nimble-users
Hi Chris,
Thank you so much for the clear explanation! I now realize that I misread chap. 5 of the manual (which is clearly written, my fault). So for the gaussian case using the variance as conjugate parameter should be just infinitesimally faster than using precision, as the latter also involves an inverse besides the square. For the multivariate case and the Cholesky decompositions involved, it makes much sense what you say. Thank you again!
Luca
On 220914 01:54, Chris Paciorek wrote:
> Hi Luca,
>
> Short answer: I don't think it matters much what parameterization you use in the cases I can think of at the moment.
>
> Longer answer:
>
> We do some parameter transformations under the hood, ultimately using the 'canonical' parameterizations shown in the Table 5.2 of the manual <https://r-nimble.org/html_manual/cha-writing-models.html#subsec:dists-and-functions>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/c0e6cafd-3fa4-f61d-5a39-8142048ad12c%40magnaspesmeretrix.org <https://groups.google.com/d/msgid/nimble-users/c0e6cafd-3fa4-f61d-5a39-8142048ad12c%40magnaspesmeretrix.org>.
>
Thank you so much for the clear explanation! I now realize that I misread chap. 5 of the manual (which is clearly written, my fault). So for the gaussian case using the variance as conjugate parameter should be just infinitesimally faster than using precision, as the latter also involves an inverse besides the square. For the multivariate case and the Cholesky decompositions involved, it makes much sense what you say. Thank you again!
Luca
On 220914 01:54, Chris Paciorek wrote:
> Hi Luca,
>
> Short answer: I don't think it matters much what parameterization you use in the cases I can think of at the moment.
>
> Longer answer:
>
> Note that if the value of a given parameter has not changed, then nimble's algorithms will just used the cached value of the canonical parameter.
>
> To the extent you can set up priors that are conjugate on the scale of the canonical parameterization, that will save some computation. E.g., if it were possible to set up a conjugate prior on the standard deviation of a Gaussian distribution that would avoid having to calculate the standard deviation based on the precision or the variance, which is what happens in the two cases you mention (we use standard deviation internally, not precision, because that is what Rmath's C code uses). But often this won't be possible. Whether a given non-canonical parameterization is more efficient than another non-canonical parameterization will depend on the computational cost of calculating the canonical parameter from the non-canonical.
>
> Now in the case of scalars, this probably won't matter much. The case where it might is with multivariate distributions. For example with the Wishart, the canonical parameterization is in terms of the Cholesky decomposition of either the scale or inverse scale matrix. So we have to do a Cholesky decomposition whenever the scale/inverse scale changes. Then we have to compute the density, which depending on whether one is working with scale or inverse scale either involves matrix multiplication or solving a system of equations. It turns out the computational cost should be about the same regardless of whether working with scale or inverse scale.
>
> Similar things come up with the multivariate normal, but again one either ends up doing matrix multiplication or solving systems of equations and the cost should be about the same.
>
> -chris
>
>
> To the extent you can set up priors that are conjugate on the scale of the canonical parameterization, that will save some computation. E.g., if it were possible to set up a conjugate prior on the standard deviation of a Gaussian distribution that would avoid having to calculate the standard deviation based on the precision or the variance, which is what happens in the two cases you mention (we use standard deviation internally, not precision, because that is what Rmath's C code uses). But often this won't be possible. Whether a given non-canonical parameterization is more efficient than another non-canonical parameterization will depend on the computational cost of calculating the canonical parameter from the non-canonical.
>
> Now in the case of scalars, this probably won't matter much. The case where it might is with multivariate distributions. For example with the Wishart, the canonical parameterization is in terms of the Cholesky decomposition of either the scale or inverse scale matrix. So we have to do a Cholesky decomposition whenever the scale/inverse scale changes. Then we have to compute the density, which depending on whether one is working with scale or inverse scale either involves matrix multiplication or solving a system of equations. It turns out the computational cost should be about the same regardless of whether working with scale or inverse scale.
>
> Similar things come up with the multivariate normal, but again one either ends up doing matrix multiplication or solving systems of equations and the cost should be about the same.
>
> -chris
>
> On Mon, Sep 12, 2022 at 11:57 PM PierGianLuca <lu...@magnaspesmeretrix.org <mailto:lu...@magnaspesmeretrix.org>> wrote:
>
>
> Dear Nimble devs & users,
>
> Just a quick question to see if I got things right. Suppose I have a variate with gaussian probability distribution, whose variance in turn has an inverse-Gamma distribution (so there's conjugacy). Alternatively I can use the precision instead, with a Gamma distribution. Which of these two representation is computationally more efficient for Nimble? For my purposes either representation works well, so it makes sense to use the computationally most efficient one.
>
> If I understand the manual correctly, Nimble internally uses the precision, so that's the representation I'm currently using. But I wanted to make sure that's really the case.
>
> And what about the multivariate case with Wishart distributions?
>
> Thank you very much!
> Luca
>
> --
> You received this message because you are subscribed to the Google Groups "nimble-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to nimble-users...@googlegroups.com <mailto:nimble-users%2Bunsu...@googlegroups.com>.
>
>
> Dear Nimble devs & users,
>
> Just a quick question to see if I got things right. Suppose I have a variate with gaussian probability distribution, whose variance in turn has an inverse-Gamma distribution (so there's conjugacy). Alternatively I can use the precision instead, with a Gamma distribution. Which of these two representation is computationally more efficient for Nimble? For my purposes either representation works well, so it makes sense to use the computationally most efficient one.
>
> If I understand the manual correctly, Nimble internally uses the precision, so that's the representation I'm currently using. But I wanted to make sure that's really the case.
>
> And what about the multivariate case with Wishart distributions?
>
> Thank you very much!
> Luca
>
> --
> You received this message because you are subscribed to the Google Groups "nimble-users" group.
> To view this discussion on the web visit https://groups.google.com/d/msgid/nimble-users/c0e6cafd-3fa4-f61d-5a39-8142048ad12c%40magnaspesmeretrix.org <https://groups.google.com/d/msgid/nimble-users/c0e6cafd-3fa4-f61d-5a39-8142048ad12c%40magnaspesmeretrix.org>.
>
Reply all
Reply to author
Forward
0 new messages