MCMC for doubly-intractable distributions in Stan

[Ellis Scharfenaker]

I was wondering if Stan is currently capable, or will be in the future, of posterior simulation for "doubly-intractable" distributions, such as those described in Murray et al. (2012) or Liang et al. (2016). 

[Andrew Gelman]

Can this be done either by introducing latent variables or by using the differential equation solver?  We recently used the ODE solver to fit a model that had an integral inside the likelihood function.

A

On Apr 6, 2017, at 1:38 PM, Ellis Scharfenaker <el.scha...@gmail.com> wrote:

I was wondering if Stan is currently capable, or will be in the future, of posterior simulation for "doubly-intractable" distributions, such as those described in Murray et al. (2012) or Liang et al. (2016). 

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<dist.pdf>

[Bob Carpenter]

No, nor do we have any plans to implement Murray et al.'s
method.

- Bob

> On Apr 6, 2017, at 1:38 PM, Ellis Scharfenaker <el.scha...@gmail.com> wrote:
>

> I was wondering if Stan is currently capable, or will be in the future, of posterior simulation for "doubly-intractable" distributions, such as those described in Murray et al. (2012) or Liang et al. (2016).
>
>
>

[Seth Flaxman]

Hey Ellis, what models are you hoping to fit? Various approximations exist that work pretty well in practice depending on the context, e.g. if the doubly intractable model is a Cox process than I might have some ideas for you.

Seth

On Fri, Apr 7, 2017 at 7:57 PM, Bob Carpenter <ca...@alias-i.com> wrote:

No, nor do we have any plans to implement Murray et al.'s
method.

- Bob


> On Apr 6, 2017, at 1:38 PM, Ellis Scharfenaker <el.scha...@gmail.com> wrote:
>

> I was wondering if Stan is currently capable, or will be in the future, of posterior simulation for "doubly-intractable" distributions, such as those described in Murray et al. (2012) or Liang et al. (2016).
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.

> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

> <dist.pdf>

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+unsubscribe@googlegroups.com.

[Elizaveta Semenova]

Hello Seth, Ellis and all,

I would be very interested in ideas for the Cox process since I am working on it now. Currently I am implementing the Kronecker approach suggested by Seth at some point ealier. If there is more imporvement potential to unlock, I would be more than happy to hear.

Regards,

Liza

[Elizaveta Semenova]

Hello Seth, Ellis and all,

I would be very interested in ideas for the Cox process since I am working on it now. For the one-dimensional case, I attempted the trick of computing the integral through ODE solving, but this approach does not seem sustainable for higher dimensions.

Currently, I am implementing the Kronecker methodology suggested by Seth at some point earlier. If there is more improvement potential to unlock (beyond Kronecker), I would be more than happy to hear.

Regards,

Liza

On Thu, Apr 13, 2017 at 11:15 AM, Seth Flaxman <fla...@gmail.com> wrote:

[Seth Flaxman]

Hi Liza.

Happy to hear how the Kronecker stuff works for you--hopefully the backend work that Rob Trangucci has been doing will mean some performance gains. How big is your dataset / how many grid cells do you have when you discretize?

Some other ideas for a Cox process, none of which I've quite implemented but seem like they should work okay based on some other projects in which I've tried similar things:

- Parameterize the intensity using a radial basis function network formulation, e.g. pick a set of "landmark" points and place kernels centered at these points. For mots kernel choices you should be able to work out the necessary integral in closed form, since the intensity is parametric. The reason I think this should work is that it's pretty similar to an standard version of the Hawkes process, which I have an implementation of in Stan (see: https://arxiv.org/abs/1611.06713) and will be cleaning up to share soon. 

- Parameterize the intensity using some other set of simple basis functions (linear + quadratic?) - same idea as above in terms of working out the integral. Maybe someone out there already has a Stan implementation of this?

- Related to radial basis networks would be an inducing points approach. I know someone had inducing points in Stan but you'll have to search the list.

- Use Random Fourier Features. I've done this in other places and can share code, but not for Cox processes in Stan...

- Do something Bayesian based on my AISTATS '17 paper: https://arxiv.org/abs/1610.08623 ; I'm not exactly sure what it'd mean to do a Bayesian version of this yet. Maybe it'd work to simply put priors on the relevant hyperparameters and learn by sampling rather than crossvalidation? If other people have thoughts, let me know.

Seth

[Andrew Gelman]

Hi all.  When you make progress on this, please inform the Stan list about this, and we also encourage you to prepare a case study which can be posted on the Stan website!

Andrew

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Ellis Scharfenaker]

Hi, Seth,

I'm attaching a description of the model I'm trying to fit. I have data x \in Reals for which I'm trying to fit a maximum entropy distribution that I derived from certain constraints. There are four parameters and the distribution is qualitatively similar to the Subbotin, but I can't find a way to solve the partition function in order to normalize it. I know the model fits the data quite well using a grid based approximation and finding the parameter values that minimize the KL-divergence between the model and the observed frequencies, but I would like to report full marginal posteriors for each parameter.

The attached description contains the derivation of the model, but really I'm just trying to fit Eq. 5 to my data x.

I'm not really familiar with Cox processes, but any suggestions you might have would be greatly appreciated. Thanks in advance,

Ellis

On Thursday, April 13, 2017 at 4:16:13 AM UTC-5, Seth Flaxman wrote:

Hey Ellis, what models are you hoping to fit? Various approximations exist that work pretty well in practice depending on the context, e.g. if the doubly intractable model is a Cox process than I might have some ideas for you.

Seth

On Fri, Apr 7, 2017 at 7:57 PM, Bob Carpenter <ca...@alias-i.com> wrote:

No, nor do we have any plans to implement Murray et al.'s
method.

- Bob


> On Apr 6, 2017, at 1:38 PM, Ellis Scharfenaker <el.scha...@gmail.com> wrote:
>

> I was wondering if Stan is currently capable, or will be in the future, of posterior simulation for "doubly-intractable" distributions, such as those described in Murray et al. (2012) or Liang et al. (2016).
>
>
>
> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

> <dist.pdf>

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.