Stan v2.9.0 has also been tagged

[Daniel Lee]

Time to update the interfaces!

Daniel

[Rob J Goedman]

Hi Daniel,

Just cloned Cmdstan/develop, ran 'make stan-update' and 'make -j6 build'. No problems.

I removed the eta_adagrad keyword in the variational run command line (I didn't follow all ADVI discussions, but I expected a change).

I do see a few warnings still on calling stansummary() on the variational output files:

Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adaptation data

I guess 1 warning for each parallel chain.

Should I not use stansummary() here? Complete stansummary() output below.

Calling stansummary() allows me to easily plot the results, so it is a handy shortcut.

If not, will this be addressed in the future with the io discussion?

Regards,
Rob

Inference for Stan model: bernoulli_model
4 chains: each with iter=(10001,10001,10001,10001); warmup=(0,0,0,0); thin=(0,0,0,0); 40004 iterations saved.

Warmup took (0.00, 0.00, 0.00, 0.00) seconds, 0.00 seconds total
Sampling took (0.00, 0.00, 0.00, 0.00) seconds, 0.00 seconds total

Mean MCSE StdDev 5% 50% 95% N_Eff N_Eff/s R_hat
lp__ 0.00 0.0e+00 0.00 0.00 0.00 0.00 40004 inf nan
theta 0.34 4.2e-03 0.14 0.14 0.33 0.59 1079 inf 1.0e+00

Samples were drawn using meanfield with .
For each parameter, N_Eff is a crude measure of effective sample size,
and R_hat is the potential scale reduction factor on split chains (at
convergence, R_hat=1).

Iterations = 1:10000
Thinning interval = 1
Chains = 1,2,3,4
Samples per chain = 10000

Empirical Posterior Estimates:
Mean SD Naive SE MCSE ESS
theta 0.34043865 0.13915055 0.00069575273 0.0010007606 10000

Quantiles:
2.5% 25.0% 50.0% 75.0% 97.5%
theta 0.113212 0.23509775 0.3271405 0.4305215 0.6461084

[Daniel Lee]

Hi Rob,

On Fri, Dec 4, 2015 at 11:46 AM, Rob J Goedman <goe...@icloud.com> wrote:

Hi Daniel,

Just cloned Cmdstan/develop, ran 'make stan-update' and 'make -j6 build'. No problems.

Great. I'm going to be tagging and releasing CmdStan v2.9.0 soon, but there shouldn't be major changes.

 

I removed the eta_adagrad keyword in the variational run command line (I didn't follow all ADVI discussions, but I expected a change).

Where did you remove it from? Is this a Stan.jl thing?

I do see a few warnings still on calling stansummary() on the variational output files:

What's stansummary()?

 

Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adaptation data

I guess 1 warning for each parallel chain.

Should I not use stansummary() here? Complete stansummary() output below.

Calling stansummary() allows me to easily plot the results, so it is a handy shortcut.

If not, will this be addressed in the future with the io discussion?

I'm not sure I follow. I don't think there's been a change in behavior from CmdStan.

 

Regards,
Rob



Inference for Stan model: bernoulli_model
4 chains: each with iter=(10001,10001,10001,10001); warmup=(0,0,0,0); thin=(0,0,0,0); 40004 iterations saved.

Warmup took (0.00, 0.00, 0.00, 0.00) seconds, 0.00 seconds total
Sampling took (0.00, 0.00, 0.00, 0.00) seconds, 0.00 seconds total

        Mean     MCSE  StdDev    5%   50%   95%  N_Eff  N_Eff/s    R_hat
lp__    0.00  0.0e+00    0.00  0.00  0.00  0.00  40004      inf      nan
theta   0.34  4.2e-03    0.14  0.14  0.33  0.59   1079      inf  1.0e+00

Samples were drawn using meanfield with .
For each parameter, N_Eff is a crude measure of effective sample size,
and R_hat is the potential scale reduction factor on split chains (at
convergence, R_hat=1).

Iterations = 1:10000
Thinning interval = 1
Chains = 1,2,3,4
Samples per chain = 10000

Empirical Posterior Estimates:
         Mean        SD        Naive SE       MCSE      ESS
theta 0.34043865 0.13915055 0.00069575273 0.0010007606 10000

Quantiles:
        2.5%      25.0%     50.0%     75.0%     97.5%
theta 0.113212 0.23509775 0.3271405 0.4305215 0.6461084

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rob J. Goedman]

Thanks Daniel,

Inserted a few answers below.

On Dec 4, 2015, at 9:13 AM, Daniel Lee <bea...@alum.mit.edu> wrote:

Hi Rob,

On Fri, Dec 4, 2015 at 11:46 AM, Rob J Goedman <goe...@icloud.com> wrote:

Hi Daniel,

Just cloned Cmdstan/develop, ran 'make stan-update' and 'make -j6 build'. No problems.

Great. I'm going to be tagging and releasing CmdStan v2.9.0 soon, but there shouldn't be major changes.

 

I removed the eta_adagrad keyword in the variational run command line (I didn't follow all ADVI discussions, but I expected a change).

Where did you remove it from? Is this a Stan.jl thing?

If I’m correct, in Stan v2.8.x you could pass in eta_adagrad=0.1, e.g. page 49 of CmdStan user manual:

eta_adagrad=<double>

Stepsize weighting parameter for adaptive stepsize sequence Validvalues: 0<eta_adagrad≤1.0

(Defaults to 0.1) 

By default Stan.jl passes in all default values to CmdStan while there is an option to update these if needed, e.g. vanilla command lines for 4 chains:

julia> stanmodel.command

4-element Array{Base.AbstractCmd,1}:

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=1 data file=bernoulli_1.data.R output file=bernoulli_variational_1.csv refresh=100`

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=2 data file=bernoulli_2.data.R output file=bernoulli_variational_2.csv refresh=100`

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=3 data file=bernoulli_3.data.R output file=bernoulli_variational_3.csv refresh=100`

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=4 data file=bernoulli_4.data.R output file=bernoulli_variational_4.csv refresh=100`

In v2.8.0 eta_adagrad=0.1 was also present:

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 eta_adagrad=0.1 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=1 data file=bernoulli_1.data.R output file=bernoulli_variational_1.csv refresh=100`

I do see a few warnings still on calling stansummary() on the variational output files:

What's stansummary()?

This is the summary tool described on page 8 of the CmdStan manual:

• stansummary, a posterior analysis tool. 

The stansummary command summarizes the comma-separated values files that are generated from Stan program runs. 

For each parameter within the Stan program, stansummary reports the mean, standard deviation, quantiles, Rˆ, and other values. 

Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adaptation data

I guess 1 warning for each parallel chain.

Should I not use stansummary() here? Complete stansummary() output below.

Calling stansummary() allows me to easily plot the results, so it is a handy shortcut.

If not, will this be addressed in the future with the io discussion?

I'm not sure I follow. I don't think there's been a change in behavior from CmdStan.

Hope above inserts make sense now.

You received this message because you are subscribed to a topic in the Google Groups "stan development mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-dev/E6VqHzZCg0I/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-dev+u...@googlegroups.com.

[Daniel Lee]

Hi Rob,

Thanks for the clarification.

On Fri, Dec 4, 2015 at 12:34 PM, Rob J. Goedman <goe...@icloud.com> wrote:

Thanks Daniel,

Inserted a few answers below.

On Dec 4, 2015, at 9:13 AM, Daniel Lee <bea...@alum.mit.edu> wrote:

Hi Rob,

On Fri, Dec 4, 2015 at 11:46 AM, Rob J Goedman <goe...@icloud.com> wrote:

Hi Daniel,

Just cloned Cmdstan/develop, ran 'make stan-update' and 'make -j6 build'. No problems.

Great. I'm going to be tagging and releasing CmdStan v2.9.0 soon, but there shouldn't be major changes.

 

I removed the eta_adagrad keyword in the variational run command line (I didn't follow all ADVI discussions, but I expected a change).

Where did you remove it from? Is this a Stan.jl thing?

If I’m correct, in Stan v2.8.x you could pass in eta_adagrad=0.1, e.g. page 49 of CmdStan user manual:

eta_adagrad=<double>

Stepsize weighting parameter for adaptive stepsize sequence Validvalues: 0<eta_adagrad≤1.0

(Defaults to 0.1) 

By default Stan.jl passes in all default values to CmdStan while there is an option to update these if needed, e.g. vanilla command lines for 4 chains:

julia> stanmodel.command

4-element Array{Base.AbstractCmd,1}:

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=1 data file=bernoulli_1.data.R output file=bernoulli_variational_1.csv refresh=100`

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=2 data file=bernoulli_2.data.R output file=bernoulli_variational_2.csv refresh=100`

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=3 data file=bernoulli_3.data.R output file=bernoulli_variational_3.csv refresh=100`

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=4 data file=bernoulli_4.data.R output file=bernoulli_variational_4.csv refresh=100`

In v2.8.0 eta_adagrad=0.1 was also present:

 `./bernoulli variational algorithm=meanfield grad_samples=1 elbo_samples=100 eta_adagrad=0.1 iter=10000 tol_rel_obj=0.01 eval_elbo=100 output_samples=10000 random seed=-1 init=2 id=1 data file=bernoulli_1.data.R output file=bernoulli_variational_1.csv refresh=100`

Sorry -- are you asking what happened to eta_adagrad?

Alp / Dustin, can you chime in?

I do see a few warnings still on calling stansummary() on the variational output files:

What's stansummary()?

This is the summary tool described on page 8 of the CmdStan manual:

• stansummary, a posterior analysis tool. 

The stansummary command summarizes the comma-separated values files that are generated from Stan program runs. 

For each parameter within the Stan program, stansummary reports the mean, standard deviation, quantiles, Rˆ, and other values. 

Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adapation data
Warning: non-fatal error reading adaptation data

I guess 1 warning for each parallel chain.

Should I not use stansummary() here? Complete stansummary() output below.

Calling stansummary() allows me to easily plot the results, so it is a handy shortcut.

If not, will this be addressed in the future with the io discussion?

I'm not sure I follow. I don't think there's been a change in behavior from CmdStan.

Ok. The () made me think it was a function.

I think you should use stansummary. The io discussion doesn't apply here since you're calling CmdStan's executables. If you wanted to change your approach and have Stan tightly integrated with Julia, then it'd be a concern.

[Alp Kucukelbir]

hi rob,

correct: eta_adagrad is gone.

the default value for eta is now 1.0, which should trigger the "autotuning" heuristic feature.

i didn't put any effort into making stansummary work well with ADVI because of upcoming io changes.

also, we typically don't run multiple chains of ADVI. (although perhaps we should!) so i'm not sure how stansummary's behavior changes with multiple output.csv files.

thanks so much for your help in this!

cheers
alp

[Bob Carpenter]

[new subject]

See below.



> On Dec 4, 2015, at 4:52 PM, Alp Kucukelbir <akucu...@gmail.com> wrote:
>
> ...

> also, we typically don't run multiple chains of ADVI. (although perhaps we should!) so i'm not sure how stansummary's behavior changes with multiple output.csv files.

I'm with Andrew in thinking we should recommend running
multiple chains of ADVI. This will allow users to diagnose
non-convergence of the final multivariate approximations. Can
we come up with analytical versions of things like expected R-hat
for the final approximations? Or could we compute things like
KL divergence of two multinormals easily?

- Bob

[Andrew Gelman]

Yes, indeed. Especially given how fast ADVI is, it seems silly _not_ to do at least 2 or 4 runs of it in parallel. How to come up with R-hat etc is another story, but I have some tentative ideas which we’ve discussed in the meeting.
A

[Alp Kucukelbir]

sure! KL's and Gaussians play nicely; we could analytically compute
what bob suggests.

how is the parallelization done with NIPS? do we need to write some
additional code to make it work with ADVI?

[Bob Carpenter]

For CmdStan, you just fire up four processes from the shell
writing to different files and using different seeds --- the
CmdStan manual has an example. For the result, we can do R-hat
with the samples for all the runs.

RStan then wraps it all up in R's parallelization for
cross-platform support (at least I think they figured out
how to run it on Windows).

I keep forgetting that the draws get transformed back to
constrained space where they're no longer multinormal. So
maybe analytic stuff's not so good here. As tempting as
it is to diagnose convergence on the unconstrained scale,
non-linear transforms play merry hell with ESS and R-hat
calcs as they're based on linear correlations.

- Bob

> You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Dustin Tran]

this was something i was planning to think about this weekend. its also related to fake data checking/model-fit checking with advi.

good to know we can parallelize easily. it wouldn’t be useful to implement the standard r-hat unless changing it is easy, since we know it won’t be that good of a diagnostic

Dustin

You received this message because you are subscribed to the Google Groups "stan development mailing list" group.

To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.

[Bob Carpenter]

The split R-hat we use for MCMC comes for free.

Why isn't it a good diagnostic? It should at
least pick out when two runs haven't converged
to the same posterior mean or variance.

Of course, throw it out the window for something
combinatorially multimodal like LDA --- nothing's
going to work in that case because the posterior's
intractable.

- Bob

[Rob J. Goedman]

Thanks Alp,

Great, in that case it looks I’m set for Stan 2.9.0. I’ll publish the next tagged Stan.jl once Stan 2.9.0 is out.

On the stansummary front, I remember some discussions about .csv files with comments in there as being less than ideal. I’ll just hold off to see which direction the io refactoring goes as it works reasonably right now in Stan.jl.

Not being very familiar with ADVI I often I see mean/density line patterns for multiple chains like shown below. Does that look reasonable?

Regards,
Rob

[Daniel Lee]

I just released CmdStan 2.9.0!

--
You received this message because you are subscribed to the Google Groups "stan development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-dev+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Rob J. Goedman]

Hi Daniel & Alp,

Just installed 2.9.0 without problems.

For the variational test case I now get:

Exception: stan::variational::advi::adapt_eta: All proposed step-sizes failed. Your model may be either severely ill-conditioned or misspecified.

Diagnostic information: 

Dynamic exception type: std::domain_error

std::exception::what: stan::variational::advi::adapt_eta: All proposed step-sizes failed. Your model may be either severely ill-conditioned or misspecified.

This is exactly the same ./bernoulli run (same model, same data, etc.) as earlier today.

Is there something I can try?

Regards,

Rob

[Daniel Lee]

Does that fail if you run it multiple times? I think I've seen that fail and I just pieces it together that it fails intermittently. 

[Rob J. Goedman]

Yes, you are correct. Just ran it about 20 times, each time with 4 chains. Got 1 more failure. Must have been unlucky I hit it with the very first Pkg.test(“Stan”).

Regards,

Rob