Re: [stan-users] Wondering - How bad would it be to ignore the slowest converging chains? (Given the extra bias.)

[Michael Betancourt]

All of the conceptually incorrect.  Please search the list for

thorough explanations of why this is a bad idea.

On Nov 18, 2015, at 6:01 PM, David Manheim <davidm...@gmail.com> wrote:

Hi all,

As of now, my model has some identifiability problems, and some convergence problems. Those are an issue, and I will be trying to fix them but I had what is probably a bad idea, and I wanted to know how bad.

Dropping chains on the basis of how long it takes them to converge will bias the result. Given that, it seems there is some bias/computation time tradeoff. In my case, many chains converge fairly quickly, and give values near where I expect them, while others converge slowly, AND have values that are much less reasonable (this might not be a universal trait.) This probably has something to do with initial values, and it might be bad to suggest using a hack when better model specification would work, but it seems like there is a implicit bias/variance tradeoff - more samples, slower convergence, but we can increase bias by dropping the slowest / non-converging chains. (I can do this by running chains separately, on separate cores, then killing those that exceed some threshold, and re-combining the remaining chains outputs.) 

I assume it makes stats like the R-hat that is computed not conceptually reasonable, and the bias introduced would not be possible to quantify easily, but... how conceptually incorrect is this?

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
To post to this group, send email to stan-...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[David Manheim]

Sorry to bug you guys with this, I should have searched better first.

To reference the points I found;
Bob said "We [would] need to do more research on this, both theoretical and practical." (https://groups.google.com/forum/#!topic/stan-users/cBXUQBCi-LI)
You said "People can abuse MCMC against our recommendations at their own risk. We don’t have time to police bad statistics — we also don’t have to facilitate it." (https://groups.google.com/forum/#!topic/stan-users/2Yv4fAqVtew)