gmo as possible solution?

[Stephen Martin]

Hey all,

I'm just curious what the state of the gmo estimator is, whether it'll be included into stan, and any vignettes about how to use it?

I do a lot of mixture-model based estimation in stan, and it seems like the MML approach may be a solution to many of my issues.
Mainly, running HMC on the models can take a long time, and I'd love to test my model out on several (e.g., 1000) datasets; this isn't feasible in HMC, but if a faster estimator is implemented (MML), it might be.

I've tried using optimizing(), but it's finding several local modes, none of which recover the parameters used to generate the data; HMC works fine, but it takes a long time.
I'm guessing MML may perform better than optimizing for my purposes, but I have no idea how to actually use it.

[Andrew Gelman]

The GMO paper is almost done.  Once we have proof of concept (i.e., once I'm convinced it works!) we can figure out how to put it into Stan.  Still not ready yet, but I'm hoping that we'll have something in R and Stan (not fully in Stan, but an R program that runs Stan as part of doing GMO) in January?

A

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

On Wednesday, December 7, 2016 at 5:30:07 PM UTC-5, Stephen Martin wrote:

I've tried using optimizing(), but it's finding several local modes, none of which recover the parameters used to generate the data; HMC works fine, but it takes a long time.
I'm guessing MML may perform better than optimizing for my purposes, but I have no idea how to actually use it.

You'll need a unimodal posterior after integrating out the local variables because GMO is assuming multivariate normality.
 

[Andrew Gelman]

GMO does not assume multivariate normality but it does use a normal approximation for the distribution of the local variables conditional on the hyperparameters.

[Stephen Martin]

This would probably be fine; I can constrain-out the multimodality for the most part, and the posteriors are typically approximately normal. The model is built up with normal or logistic priors, and the likelihood varies (either bernoulli_logit, ordered_logistic, or normal).

Is it working at all yet? I installed the package from github, but have no idea how to actually use it (the docs were scarce; no idea what  file and file_local are supposed to be).

[Andrew Gelman]

I'm ccing Dustin who should know whether it's working at all yet.

A

[Ben Goodrich]

On Wednesday, December 7, 2016 at 5:46:19 PM UTC-5, Andrew Gelman wrote:

GMO does not assume multivariate normality but it does use a normal approximation for the distribution of the local variables conditional on the hyperparameters.

I don't understand. The estimated standard errors of phi are based on the observed Fisher information. Is there a way that the observed Fisher information yields appropriate standard error estimates without the posterior distribution of phi | y being multivariate normal?

[Andrew Gelman]

I think of the main goal of GMO as being to get the point estimate of phi, so then the shape of p(phi|y) is not such a big deal.  But if we want posterior draws from phi (which, I agree, we do), I was thinking of the normal approx as a starting point, which could be improved via importance sampling.  This can work in various realistic settings where the dimensionality of phi is low.

A

[Stephen Martin]

And what is phi? A set of parameters of interest?

[Andrew Gelman]

I think Ben was using phi to denote the hyperparameters of the model.

[Stephen Martin]

So it's doing a bit of a two-step procedure (and this is why it's 'inspired' by EM)?

It'll maximize hyper parameters, then set those as data and maximize the parameters, etc, until convergence?
So the two files needed is essentially the same model, but with one that has the hyper parameters as data, and the other with hyper parameters as parameters?

[Andrew Gelman]

Yup.  But, when all's working well, it should run particularly efficiently because both internal and external optimizations are gradient-based.

[Stephen Martin]

Very cool. I may play with this soon. I have to figure out how to easily separate out what are hyperparameters vs parameters in my model.

Would any of you be willing to help convert my stan model to one that can be used with gmo, both for your testing and my benefit?
It's a model that relies heavily on mixture models, so it seems like a prime candidate for this sort of procedure.
The issue is that it's got several mixtures. It's a measurement model (think CFA, 2PL, GRM), but each individual's (i) response on an item (j) can either be produced from a "non-DIF" item model (item j's parameters equal across groups) or from a "DIF" item model (item j's parameters differ across groups). Moreover though, if their response is generated from a DIF item model, it will estimate which latent group the individual belongs to.
The likelihood is something like:


for a CFA.
So there are two 'layers' of mixtures that cross.
There is a mixture model per-item, and within one of the mixture components for each item is a latent group model. Eek. Works decently in NUTS, miserably in vb and optimizing.
mu, lambda, theta, and sigma vectors are all treated as unknown. Each individual's pi vector is estimated using [multinomial] logistic regression, or could just be estimated from an intercept-only model.
Priors are placed to make lambdas positive, and lambdas are ordered across latent groups for one item (to reduce label switching).

I'd love to give gmo a task like this, but I'm not sure what separates the parameters from hyperparameters in this case.

[Andrew Gelman]

Hi, there's no rule for what should be the "hyperparameters" and what should be the "parameters"; the general idea is that we'd want the hyperparmameters to be low-dimensional and we'd like the conditional posterior distribution of the other parameters to be approx normal.

I'm not quite sure what would be best in your model, also I'm not quite sure what your Delta_j's are.

A

[Stephen Martin]

Oops: this was supposed to go to stan.

Delta j is the probability of the second component to the mixture . That is, the probability that an item needs separate parameter estimates between groups.

Imagine your data are generated as such: you have a 10 item survey with 9 response options on each item. You sample from two populations, unbeknownst to you, who do not respond to items 3 and 4 in the same way. Given equal latent ability, those in group 1 have a different expected value than those in 2 on items 3 and 4. So there exist two populations for whom the item parameters of 3 and 4 are different.

This model attempts to estimate 1) the probability of each item having meaningfully different parameters between groups 2) the probability of each person belonging to each group that differs on those items 3) the latent abilities of those individuals and 4) the item parameters for all items.

So Delta j is the probability of the item having separate item parameters between groups, pi is the probability of each person belonging to each latent group, theta is latent ability for each person, lambda is the factor loading for the item j and mu is the intercept for item j.

On Dec 7, 2016 9:22 PM, "Stephen Martin" <hwki...@gmail.com> wrote:

Delta j is the probability of the second component to the mixture . That is, the probability that an item needs separate parameter estimates between groups.

Imagine your data are generated as such: you have a 10 item survey with 9 response options on each item. You sample from two populations, unbeknownst to you, who do not respond to items 3 and 4 in the same way. Given equal latent ability, those in group 1 have a different expected value than those in 2 on items 3 and 4. So there exist two populations for whom the item parameters of 3 and 4 are different.

This model attempts to estimate 1) the probability of each item having meaningfully different parameters between groups 2) the probability of each person belonging to each group that differs on those items 3) the latent abilities of those individuals and 4) the item parameters for all items.

So Delta j is the probability of the item having separate item parameters between groups, pi is the probability of each person belonging to each latent group, theta is latent ability for each person, lambda is the factor loading for the item j and mu is the intercept for item j.

[Dustin Tran]

hello,

i agree with all of what andrew said. the current state of the code is usable; algorithm hyper parameters will need to be tweaked for it to work well in practice. 

one challenge in GMO is how to scale it to high-dimensional parameters. we’d like to avoid explicit calculations of the importance weights while still remaining an exact MML algorithm (otherwise we’d just do variational EM). i have some ideas to get at this that are not in the current code base.

best,

dustin

[Bob Carpenter]

Are you trying to avoid explicit importance weighting because
the likelihood is prohibitive with big data sets?

- Bob

[Dustin Tran]

we’d like to avoid the poor scaling with respect to dimensionality in the importance weight. this can be a result of big data. i prefer to think of it as the number of parameters to integrate over, which can be independent of the data set size.

dustin

[Andrew Gelman]

Yeah, one of the challenges in getting the GMO paper finished is that there are so many different scenarios. One set of problems is the INLA-type situation in which the dimensionality of the hyperparameters if fairly low, and in that case iterative importance sampling could help. But I also agree with Dustin that if we're gonna do iterative importance sampling, we'd probably be better off doing something variational.

To put it another way, GMO is a version of ADVI, and marginal posterior mode is a version of variational inference, but in either case the key step is the partitioning of parameter space into local parameters (which get averaged over) and hyperparameters (which get optimized).

Once we get GMO working for real in its prototype version (i.e., demonstrating that we can use the GMO algorithm to get good answers for real problems much faster than NUTS), then we can talk about expanding the Stan language so Stan can do GMO internally. I don't think this will be much of a technical challenge, but I don't think it makes sense to ask this of the Stan development team until we have a algorithm that we're convinced is working. Which I hope we will have in the near future...
A