Is it possible to identify non-converged ADVI runs in rstan?

[Joergen B]

Hi all!

Thank you very much for the work you are putting into stan, it is really providing me with some powerful tools.

I am currently doing some Monte Carlo simulations of a model estimated with ADVI using rstan. Every now and then, it seems there will be an instance where I get the following: "Informational Message: The maximum number of iterations is reached! The algorithm may not have converged. This variational approximation is not guaranteed to be meaningful." (This message can easily be reproduced by setting the "iter"-option of the vb()-command very low.) 

I would like to identify the cases that produce this message, to inspect the results, and possibly exclude them from the main results. Is there a way to do this? I have looked at the content of the stanfit-object, but have not really been able to find anything useful for assessing the convergence status. Am I missing something? Is there some kind of hack I could do, if there is no readymade method for it?

Best,

Joergen

[Bob Carpenter]

I don't know about the RStan interface to ADVI. I'd hope you
could somehow get a return code or something that indicated whether
everything converged OK according to internal diagnostics.

The deeper problem is that we don't know how to diagnose what's
going wrong with ADVI and we don't have any developers willing
to work on making it more robust. That's why it's still labeled
as "experimental".

And of course we can't give you any model specific advice given that
you haven't said anything about your model. The only way to make sure
things are working that we know of is to run MCMC and compare results.
You probalby want to at least run multiple random inits for ADVI to
make sure you're converging to the same result.

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

On Monday, January 23, 2017 at 4:21:50 PM UTC-5, Bob Carpenter wrote:

I don't know about the RStan interface to ADVI.  I'd hope you
could somehow get a return code or something that indicated whether
everything converged OK according to internal diagnostics.

That is sadly not the case. The only thing ADVI produces is a CSV file (provided there is no error).

Ben

[Jonah Gabry]

On Monday, January 23, 2017 at 4:21:50 PM UTC-5, Bob Carpenter wrote:

The deeper problem is that we don't know how to diagnose what's
going wrong with ADVI and we don't have any developers willing
to work on making it more robust.  That's why it's still labeled
as "experimental".

Are Alp and Dustin not working on this at all anymore? 

[Bob Carpenter]

Nope. Their last commits to ADVI were in 2015.

We may just remove ADVI if we can't find anyone to fix it. I don't
want to leave algorithms in Stan that aren't robust and don't have
good failure diagnostics---it just frustrates our users no matter
how much we try to point out it's "experimental".

- Bob

[Jørgen Bølstad]

Well, for what it’s worth, ADVI seems to produce excellent results in my simulations. Any improvement in accuracy to be had with NUTS is very limited in my case, and as ADVI gives me estimates in a matter of seconds rather than hours, I think it is incredibly valuable. I really hope you won’t remove it (unless you it is because you have found something better).

Also, in case anyone else encounters the issue I originally posted about: My current, not-so-elegant workaround is to use sink() to capture the outputs, and the check whether it contains the non-convergence warning. Here’s an example:

out.file <- file(“output.Rout", open = "wt")
sink(out.file, append=F)
fit <- vb( … ) # Code to fit mode goes here.
sink()
filename <- “output.Rout"
output <- readChar(filename, file.info(filename)$size)
model.failed <- grepl("may not have converged”, output) # Logical indicating convergence issue.

Best,
Joergen

> --
> You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/NZseSPJ3XAo/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.

[Dustin Tran]

Andrew and a Master’s student are continuing off Rayleigh’s work on benchmarking ADVI. We (Alp and I) halted work on improving ADVI. The first step is to really understand what is causing the failures—we don’t even know that much. This is difficult theoretically, so instead we’re doing a massive empirical analysis. The hope is that these evaluations will then lead to solid heuristics, similar to the conception of NUTS for adapting hyper parameters and R-hat for diagnosing convergence.

dustin

You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Daniel Lee]

Just out of curiosity, what model is it? Can you share data and the program? Perhaps we can spot some things that make it work well.



Daniel

> On Jan 24, 2017, at 5:17 PM, Jørgen Bølstad <j.boe...@gmail.com> wrote:
>

> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Jonah Sol Gabry]

Hi Jorgen,

I'm glad it produces good results for you, but the problem is that you wouldn't know that without comparing to NUTS. So until we have some way of better diagnosing advi runs, in order to trust it you'll have to run NUTS anyway, in which case you might as well just use NUTS. 

Jonah 

[Bob Carpenter]

If we could isolate classes of problems for which ADVI worked
well, that'd be a start.

- Bob

> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.

[Andrew Gelman]

Jonah's statement is a bit extreme, I think. Another way to check ADVI is to simulate fake data from some versions of the fitted model and see if ADVI recovers the parameter values.
A

[Michael Betancourt]

Having estimated means around the true values, or even better using

the estimating variances to quantify that the estimates mean are close

to the true values, is not near sufficient.  That depends too strongly on

the actual model which could place posterior mass away from the true

values.  In other words, a failure could be due to model problems or

computation problems and even a success could be due to computational

problems canceling its modeling problems.

We need to isolate the computational accuracy from the modeling issues 

and test whether or not we are accurately computing posterior expectations.

This is what cannot be done in situ by ADVI where as we have lots of 

diagnostics and guarantees with HMC.

[Andrew Gelman]

I agree that the fake-data check is not perfect, but it's something.  I was pushing back against Jonah's statement, "in order to trust it you'll have to run NUTS anyway, in which case you might as well just use NUTS."  That's fine, but suppose you have a problem where Nuts is too slow to run.  Then you have to do some approx.  ADVI is one choice.

I should elaborate, though, that fake-data checking on the fitted model is not the only confidence-building tool out there.  Another useful approach is to build up a model step by step and check that the inferences make sense at each step.  Another approach is to evaluate errors using cross-validation.  And another is to run ADVI from multiple starting points and compare the different estimates.

These won't give full confidence in results but, again, it's something.  And I think these steps can help.  Recall that we put the birthday model on the cover of our book, and we didn't fit that with Nuts.  And, for that matter, that model isn't perfect.  But the inference for the imperfect model with the imperfect EP algorithm was still useful.

So I resist over-the-top statements such as, "you might as well just use NUTS." 

A

[Michael Betancourt]

The birthday approximate analysis was useful because it found a model

configuration that looked okay?  That’s one perspective, and the same

argument for many of the machine learning algorithms that “have good

predictive performance” against various point predictions like cross

validation.  

But all of that is based on finding _a_ good point and not accurately

quantifying the uncertainty inherent to the posterior.  The model can

no longer be separated from the computation, and you cannot identify

from which problems are arising.  Decisions are overconfident and 

fragile because of the lack of accurate uncertainty quantification.  In

general inferences become awkward.  Plus, it’s not fair to compare

that to NUTS and not some approximation where you run a Markov

chain for just a few iterations to get into the typical set.

And really the slow performance of NUTS was demonstrating that the

birthday model was highly non-identified, as we have since better

understood.  Again, this doesn’t affect the point approximations but

does slow down NUTS which is doing everything it can to get accurate

uncertainty quantification.  

If this is the argument for these unvalidated approximations then in

my opinion you have to fundamentally change the way you present

the results.  You have to make it very clear that you are presenting

a good point and not quantifying all of the good points, and not really 

comparing to full Bayesian inference at all.  This is an _extremely_

important step to not mislead users.

[Andrew Gelman]

We did the birthday model 3 years ago.  The model on the cover of the book is a culmination of models, each of which revealed more information and made more sense than what came before.  I think that including the model was a better choice than us waiting 4 years until we could fit it using Nuts.

A

[Michael Betancourt]

Sure, finding a model configuration consistent with the data can be informative.

Just don’t call it full Bayesian inference, which requires quantifying all of the model 

configurations consistent with the data!

[Jonah Sol Gabry]

On Tue, Jan 24, 2017 at 9:41 PM, Andrew Gelman <gel...@stat.columbia.edu> wrote:

I agree that the fake-data check is not perfect, but it's something.  I was pushing back against Jonah's statement, "in order to trust it you'll have to run NUTS anyway, in which case you might as well just use NUTS."  That's fine, but suppose you have a problem where Nuts is too slow to run.  Then you have to do some approx.  ADVI is one choice.

Yeah my comment doesn't really apply in the case of a problem that is still too time consuming for running NUTS to be feasible. 

On another note, one thing that does sometimes help when using advi is Ben's trick of applying a scaled QR decomposition to the X matrix in a regression model. 

Jonah

[Jørgen Bølstad]

Hi Daniel (and others)

Briefly put, I am working on a model to estimate latent positions of political actors from survey data, and I am using MC simulations to compare its performance to existing models, showing where the latter fall short. I would be happy to share the code, data, and paper, and would be very grateful for any comments and advice. I don’t want to make it public before it (hopefully) gets published (which would still be far into the future), but I could e-mail it those interested (unless you have a better suggestion for sharing privately). I should probably spend a few days cleaning up the code and the paper first though.

I hope the paper will illustrate that ADVI can be useful for simulations — although, in line with Michael’s point, I will only use ADVI to get approximate point estimates, and use NUTS when I need to sample the full posteriors. (For me the choice depends on whether I want to fit the model a few thousand times, and whether I can make do with the point estimates — I hope to be able to justify using ADVI for these situations.)

Best,
Joergen

[Bob Carpenter]

I think Andrew's making a good point here. Before Stan, I
was writing "machine learning" algorithms for natural language
processing (LDA, L1-penalized logistic regression, conditional random
fields, language models for spell checking, etc.) In almost all of
these cases, simpler models with more data have always outperformed
more complex models predictively in terms of 0/1 loss. We're talking gigabytes
of training data here. I don't see anything wrong with that. We need
to build spell checkers, named-entity finders, classifiers for sites
like Google news, etc.

But Michael also makes a good point that there should be some kind
of discussion of approximation error. Or at least an acknowledgement
that the posterior approximation is of unknown quality. In a lot of
what I've seen in the natural language literature, variational inference
is just being performed for a point estimate.

We didn't run with simulated data, but we ran with data where we
had the HMC fit, and saw that ADVI failed to recover the true posteiror
means within 1 posterior standard deviation in most cases. And the
real problem is that when it fails on real data, we don't know how to diagnose
that. Fitting with data simulated from a model is often insufficient because
the model is misspecified for real data.

We also know that variational inference underestimates posterior uncertainty
by construction, because of the direction of KL divergence minimized. So if
we do simulated data, we should have too low posterior coverage of p-intervals.

I think a lot of machine learning researchers only care about
predictions, which can also be calibrated and evaluated for
sharpness (how narrow the intervals are).

So we can test that calibration. And we can even do it in a believable
way if we have held out data, we can test that calibration on real
predictions.

- Bob

[Andrew Gelman]

Hi, again, I recommend checking your ADVI by re-running it from a few different starting points, and by checking it with fake-data simulations.
A

[Bob Carpenter]

Do you mean simulating from the posterior approximation and then
applying something like R-hat? And trying to test calibration of
coverage for that posterior?

- Bob

[Bob Carpenter]

And as Michael keeps saying, we can only do this all to within
MCMC error. So you have to run enough iterations to get a large
enough effective sample size.

- Bob

> On Jan 25, 2017, at 4:14 PM, Andrew Gelman <gel...@stat.columbia.edu> wrote:
>

[Andrew Gelman]

I mean 2 things:
1. Running ADVI on the data several times and checking that the resulting estimates are similar. Ultimately I think there should be an R-hat-like test here but I don't have that yet.
2. Fake-data simulation: Choosing reasonable hypothetical values for the hyperparameters of the model, then simulating local parameters and data, then running ADVI and checking to see if the resulting estimate is close, both for the hyperparameters and the true values.
A

[Bob Carpenter]

1. How do you test for similarity? As in, I have the simulated
value and I have the posterior mean and marginal posterior standard
deviation from the estimate, what do I do?

2. What do you mean by close? Close as in the true values are
in an appropriate number of posterior intervals?

We want to write code. We can't build a homunculus to replace
you unless we know what you'd do.

I get that you don't want to use MCMC results to calibrate.

But I don't even understand if you only want to test the
posterior mean point estimates, or if you also care about
posterior uncertainty. And I don't see why you wouldn't just
take the posterior approximation and simulate from that then
use R-hat. But then you could probably calculate R-hat analytically
from that since it's multivariate Gaussian.

- Bob

[Andrew Gelman]

Hi, the short answer to both questions is that I'd like to develop something formal, along the lines of R-hat (for item 1) or my paper with Cook and Rubin (for item 2), but for now the answer is that Jørgen Bølstad (the person who wrote that email) should look at the estimate and see if they're close enough for practical purposes, based on his own goals in fitting the model. Jørgen is his own homunculus.

Yes, though, we should definitely come up with some formulas for this and code them up, and then people can try them out.

I'm fine to use Nuts to calibrate. But that won't work for problems where Nuts is just too slow.

A