Iter = 110000, n.eff = 100 or less, WHY?

[Chunlei Zhang]

Hello,

I used Rstan to do a Bayesian Hierarchical One-Level Model. The question I would ask is why the effective sample size (n.eff) from the Rstan output is much smaller then WinBUGS'. Rstan (3-7th columns) and WinBUGS (8-12th columns)  for the same dataset with exactly same number of iterations  and burn-in samples. Below is an output example.

I used R3.3.3, Rstan 2.14.2, Windows 8 OS. Call WinBUGS14 from R.

The Rstan model is the below.

model_1level_stan <- "data { int<lower=0> J;

                              real samSE[J];

                              real betahat[J];

                              real<lower=0> sigma;

                             }

                        parameters { real Nation;  

                                    real cityRisk[J];

                                    real<lower=0> sigma_unif;

                         }

                        model { 

                              for(j in 1:J) {

                                            betahat[j] ~ normal(cityRisk[j], samSE[J]);   

                  cityRisk[j] ~ normal(Nation,  sigma_unif);

                                            }

                              Nation ~ normal(0, 10);

                              sigma_unif ~ uniform(0,sigma);

                             }"

Then the below is the stan function I used.

 stan(model_code = model_1level_stan,

                data = data.info,

                init = rep(inits.info1, 3),

                chains = 3,

                iter =110000,

                warmup = 10000,

                control=list(adapt_delta=0.95),

                thin = 1, 

                verbose = FALSE, 

                open_progress = FALSE, 

                refresh = 1000

                )

Thanks for your time and consideration.

Chunlei Zhang

[Chunlei Zhang]

Q75	Q97.5	Rhat	n.eff
0.000196	0.001463	1.006882	447
0.001196	0.002969	1.000705	7075
0.001483	0.003207	1.000177	5973
0.001326	0.003473	1.000489	8455

The above is a sample from the RStan output. The iter=110000, but n.eff could be 447 or even less. Why? Use same model, same data, WinBUGS would have much larger n.eff.

[Bob Carpenter]

Two things.

First, these aren't consistent:

> real<lower=0> sigma_unif;

> sigma_unif ~ uniform(0,sigma);

You need to declare sigma_unif with upper=sigma in order to do this.
Every parameter value satisfying constraints should have a finite
log density.

But we don't really recommend those hard constraints for
statistical and computational reasons (you get bias when the
artificial bound isn't way out in the tail of a more diffuse
prior).

Second, you need to use a non-centered parameterization for a
hierarchical model like this (it's like an 8-schools/measurement
error model). That's described in the manual.
You'll want to do the same thing for WinBUGS, by the way.

Try starting WinBUGS from the same diffuse initializations
as Stan uses and then evaluate four chains using Stan's
Rhat calculations. They're much more conservative and use
cross-chain information to discount n_eff when there hasn't
been good convergence.

- Bob

> On Apr 21, 2017, at 5:06 PM, Chunlei Zhang <chun...@gmail.com> wrote:
>
> Q75 Q97.5 Rhat n.eff
> 0.000196 0.001463 1.006882 447
> 0.001196 0.002969 1.000705 7075
> 0.001483 0.003207 1.000177 5973
> 0.001326 0.003473 1.000489 8455
>
> The above is a sample from the RStan output. The iter=110000, but n.eff could be 447 or even less. Why? Use same model, same data, WinBUGS would have much larger n.eff.
>
>
> On Friday, April 21, 2017 at 4:40:56 PM UTC-4, Chunlei Zhang wrote:
> Hello,
>
> I used Rstan to do a Bayesian Hierarchical One-Level Model. The question I would ask is why the effective sample size (n.eff) from the Rstan output is much smaller then WinBUGS'. Rstan (3-7th columns) and WinBUGS (8-12th columns) for the same dataset with exactly same number of iterations and burn-in samples. Below is an output example.
>
>

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> To post to this group, send email to stan-...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Andrew Gelman]

We gotta set up pedantic mode!

[Chunlei Zhang]

Bob, thanks for the suggestion.

I declared sigma_unif with upper=sigma, and found n.eff in the new model sometimes larger sometimes smaller than the old model without declared sigma_unif with upper=sigma. It did not change the n.eff significantly.

My next question is how large (n.eff) is large enough? Generally speaking for exampling, the larger the example size is, the better result we would have.

Furthermore, why Rstan give a much smaller n.eff then WinBUGS under same scenario (same model, same input data, same computer OS)? The n.eff is at least 10 times smaller than the one from WinBUGS.

Thanks for your time and consideration.

Best regards,

Chunlei

[Bob Carpenter]

> On Apr 25, 2017, at 11:23 AM, Chunlei Zhang <chun...@gmail.com> wrote:
>
> Bob, thanks for the suggestion.
>
> I declared sigma_unif with upper=sigma, and found n.eff in the new model sometimes larger sometimes smaller than the old model without declared sigma_unif with upper=sigma. It did not change the n.eff significantly.
>
> My next question is how large (n.eff) is large enough? Generally speaking for exampling, the larger the example size is, the better result we would have.

Your MCMC standard error is posterior std deviation / sqrt(n_eff).
So it depends on how much certainty you need.

> Furthermore, why Rstan give a much smaller n.eff then WinBUGS under same scenario (same model, same input data, same computer OS)? The n.eff is at least 10 times smaller than the one from WinBUGS.

How are the convergence stats (R-hat)? And are you using
the same software to calculate n_eff in BUGS and Stan? We use
a much more conservative estimate that discounts n_eff if
you have evidence of non-convergence across chains. The standards
built into Coda and used in the BUGS R interfaces don't do that.
So you want to use our R-hat and n_eff to compare apples to apples.

But my main guess would be that you're not fitting the same
models in both systems.

-Bob

[Chunlei Zhang]

Hello Bob,

Thanks for the reply. Now we made the iteration number 5 times (or evern 10 times) larger, also have one more chain in order to have larger n.eff from Rstan result. It did made n.eff larger significantly. Do you have any idea to make n.eff larger for same Bayesian model by using Rstan? Thanks for your time and consideration.

Best regards,

Chunlei

[Bob Carpenter]

RStan runs the same Stan as all our other interfaces.

If you increase iterations by 10 (and don't prune) you
should get roughly 10 times as large an n_eff. If that's
happening, you're probably mixing OK, but slowly.

The only thing that'll make that better is reparameterizing.

- Bob