Evaluating mean-field ADVI: 1. Some results
37 views
Skip to first unread message
Andrew Gelman
Jun 9, 2016, 12:49:03 PM6/9/16
to stan development mailing list, Rayleigh Lei, Aki Vehtari, Dustin Tran, Alp Kucukelbir, Daniel Lee, Bob Carpenter, Ben Goodrich
Hi--as many of you know, Rayleigh wrote a script based on some code of Ben to run NUTS and then mean-field ADVI on a library of over 200 Stan programs (the Bugs examples, the Arm examples, and a bunch more: https://github.com/stan-dev/example-models/wiki). The idea, for each model, was to first run NUTS to convergence, and then run ADVI to check that the estimates from ADVI were close. To give ADVI the best chance of succeeding, we used the posterior mean estimates from NUTS as our starting points for ADVI.
Next it's time to summarize the evaluations. For each parameter in a model, we define "close" in terms of the z-score: ((ADVI estimate) - (posterior mean))/(posterior sd), where the posterior mean and sd are from NUTS. The idea is that this z-score should be close to zero. For each model, we computed the maximum of the z-scores for all the parameters in the model (which includes transformed parameters and generated quantities). We then had 200-plus max z-scores, which we plotted; see attached.
Next it's time to summarize the evaluations. For each parameter in a model, we define "close" in terms of the z-score: ((ADVI estimate) - (posterior mean))/(posterior sd), where the posterior mean and sd are from NUTS. The idea is that this z-score should be close to zero. For each model, we computed the maximum of the z-scores for all the parameters in the model (which includes transformed parameters and generated quantities). We then had 200-plus max z-scores, which we plotted; see attached.
Reply all
Reply to author
Forward
0 new messages