rstan: dropping chains

[Joseph Burling]

Hi all,

I was wondering if there was a method already existing to drop unwanted chains? I couldn't find any so I tried to implement this myself but I'm getting a few errors of 'the model object is not created or not valid.' Probably to do with stuff from the .MISC slot. Thanks!

stan_drop_chains <- function(object, chain_ids) {

    if (!is(object, "stanfit"))

        stop("object must be a fitted rstan object")

    

    if (missing(chain_ids))

        stop("please select chain numbers to drop for argument 'chain_ids'")

    

    ids <- unlist(lapply(object@stan_args, function(l) l$chain_id))

    

    drop <- ids %in% chain_ids

    

    if (!any(drop))

        stop("chain_ids not found in stanfit object")

    

    for (chain in seq_along(drop)) {

        if (drop[chain]) {

            object@sim$samples[ids[chain]] <- NULL

            object@inits[ids[chain]] <- NULL

            object@sim$permutation[ids[chain]] <- NULL

            object@stan_args[ids[chain]] <- NULL

        }

    }

    

    object@sim$chains <- length(which(!drop))

    object@sim$n_save <- object@sim$n_save[!drop]

    object@sim$warmup2 <- object@sim$warmup2[!drop]

    

    for (chain in seq_len(object@sim$chains)) {

        object@stan_args[[chain]]$chain_id <- chain

    }

    

    new_obj <- new(

        "stanfit",

        model_name = object@model_name,

        model_pars = object@model_pars,

        par_dims = object@par_dims,

        mode = 0L,

        sim = object@sim,

        inits = object@inits,

        stan_args = object@stan_args,

        stanmodel = object@stanmodel,

        date = date(),

        .MISC = new.env(parent = emptyenv())

    )

    

    return(new_obj)

}

[Michael Betancourt]

Please do not drop chains.  Misbehaving chains are serious indicators 

that the “good” chains are also biased.  The only valid reason to drop

a chain would be if you have too many samples for R to handle well, 

but in that case you can just rerun with fewer iterations.

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Joseph Burling]

Thanks for the suggestion, Michael. So is it always a good idea to run as many chains  as your machine can handle to detect problems with the model?

Also, I was wanting to drop chains because I fit a model with a lot of samples, and was wanting to do some some basic plots and working with a subsample to speed up the process without refitting a smaller model with less samples.

[Michael Betancourt]

Thanks for the suggestion, Michael. So is it always a good idea to run as many chains  as your machine can handle to detect problems with the model?

Correct.

Also, I was wanting to drop chains because I fit a model with a lot of samples, and was wanting to do some some basic plots and working with a subsample to speed up the process without refitting a smaller model with less samples.

Yeah, this would be the one okay use-case.  The Rstan devs will hopefully chime in.

[Bob Carpenter]

If not, please repost with a slightly different title
so the rstan devs will know it hasn't been answered.

We're trying to play more of a zone defense on these
questions these days, so we're not wasting time having all
the devs read all of the user messages.

- Bob

[Jonah Gabry]

Do you want to drop entire chains or take a subset of posterior draws? It's not easy to drop a chain from a stanfit object really, but you can easily take a sample from the array of posterior draws as you would with any array in R, or drop a chain from the array. The resulting array can still be used with shinystan (albeit with some limitations) although not with the plotting functions in RStan. However that functionality is coming soon when we standardize the plotting functionality across all of our R packages using a new and more flexible and dependency free (except for ggplot really) package. That should be ready at some point this summer.

Jonah

[Joseph Burling]

I guess if I randomly permute a subsample the length of one of the chains I should get something similar?

Anyway, I found that the issue I had with the function I was using seems to be with calculating the Bayesian Fraction of Missing Information portion during summary.

If I just run one of the example models, save the stanfit object, then reload it into R, I get the same error.

[Jonah Sol Gabry]

On Wed, Jul 13, 2016 at 2:39 PM, Joseph Burling <joseph...@gmail.com> wrote:

If I just run one of the example models, save the stanfit object, then reload it into R, I get the same error.

Thank you for mentioning this! I'm also seeing an error from the print method when using a saved stanfit object. This is definitely the result of adding the BFMI to the printed output in the latest version. Thankfully it still prints the rest of the output and just errors when it tries to print BFMI but that's still bad. I've opened an issue for this (https://github.com/stan-dev/rstan/issues/313) and we'll get it sorted out.

[Jonah Sol Gabry]

In fact this has actually already been fixed in the development version of RStan, so it will definitely be fixed in the next release. 

[Bob Carpenter]

What are you trying to accomplish? As Michael said,
if you want to drop chains that didn't converge, that
will potentially lead to biased estimates.

If you just want to subsample because there's too much
data, you want to thin existing chains rather than cut
out whole chains. That way, the convergence diagnostics
and effective sample size estimates are reasonable. If you
just permute, you lose all of this.

- Bob