seeking help with reinforcement learning model

[Lei Zhang]

Hi Stan,

we've been trying to fit a variant of the simple reinforcement learning model on a trial-by-trial basis. The task is a social decision-making paradigm, in which on each trial the subject (1) makes a choice, (2) observes the choices of 4 other players, (3) based on this social information makes a second choice, and (4) the probabilistic outcome is reveals for all players. We encountered some problems and we wish to get some hints /feedback from the mailing list. We run our model on Stan 2.7.0 (rstan) with 16500 samples per chain, 4 chains, 4000 warming-up and 5 steps of thinning, which ends up with 10000 valid samples altogether. Problems are listed below and you might also want to check the model file attached.

(1) Does the low effective sample size (ESS) in the range of 5-50% of the nominal sample size indicate a pathology in the sampling process?

Though nearly all the R-hat of all parameters of interest are below 1.10, quite a few of them have really low ESS due to the fact that they have relatively high auto-correlation. Does increase the length of chains and increase thinning steps (perhaps even thinning=20) make sense to deal with this problem?

(2) Do parameter values near the range boundaries suggest an ill-specified model? 

The model uses a learning rate (range 0-1) to weight a reward prediction error, which is then use to update expected values. These are converted into action probabilities using a softmax function parameterized by a temperature parameter (range 0-3). However, the MAP/median of the posterior distribution of learning rate is close to 0 (for another model it's close to 1) and the temperature is also super low (~ 0.3, which indicates random choice). We were wondering if the model is ill-specified but so far we haven't come up with a more sophisticated idea.

(3) Is there a standard way to implement posterior predictive check (PPC) with reinforcement learning models?

We've checked previous entries on the mailing-list about PPC but couldn’t find something that seems to fit. In practice, we want to re-run the model 100 times to simulate data in the generated quantities section to generate simulated choice data. We are planning to divide each simulation of 100 trials into 6-10 bins and then compute the predictive accuracy for, say, let choice for each bin. (Just taking the overall frequency of left choice in all 100 trials or overall number switches as the target measure for a validation is losing too much information as as the overall number of left choices between simulations and data can be identical, yet the model can be wrong on every trial.) Does this binning idea sound like a valid approach to PPC? We would be eager to hear other suggestions from the community.

(4) If you check the attached model code you’l notice that we have 2 sampling statements involving choices in each trial:

statements: choice1 ~ categorical_logit(), and choice2 ~ categorical_logit(). Are both of them used for incrementing the log-probability in Stan?

As the 2nd choice is more important than the 1st one (because the 2nd choice leads to reward/punishment), is there a way to weight the importance of them? say w * sampling1 + (1-w) * sampling2 and then also fit 'w' as a free parameter?

Thanks a lot in advance for giving this some thoughts and we appreciate any input or feedback.

Best,

Lei

[Andrew Gelman]

Hi, I’d recommend you use the default of 50% warmup and 50% post-warmup, with no thinning.  Also it’s not clear why you are doing lower=0.01 on some parameters.  I think lower=0 would be fine.

Regarding your question about point estimates being close to 0, I don’t think that’s so important.  Better to just use the entire posterior distribution.

For posterior predictive check, just simulate replicates that are the same size and shape of the original data, then you can make whatever graphs you want.

A

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<RLcoh.stan>

[Lei Zhang]

Hi Andrew and Stan,

thanks a lot for your comments and suggestions. 

I've tried 10000 samples per chain (with 50% warmup) with thinning =1. However, the autocorrelation is still high. For some chains, even when lag =10, the autocorr is still as high as around 0.4. Probably I should run chains with more samples?

The concern regarding the point estimates is that, we somehow do need point estimates to correlate them with imaging data for conducting model-based analysis. In practice, it is hard to fit the whole posterior distribution with the brain data. 

For the ppcheck, I just found the shinyStan online demo pretty useful, I will then follow that. But what we want to have is a trial-by-trial ppcheck, so purely summarize from the posterior distribution could not provide some precise information. Does anybody have any experience with trial-by-trial ppcheck?  I'd highly appreciate.

Plus, with regard to our last point, is it possible to fit a model with a weighted average of the two sampling statement? 

Thanks a lot.

L.

[Andrew Gelman]

If the autocorrelation is .4, then you’re getting something like 50% efficiency, right?  So why 10,000 samples per chain?  Wouldn’t 100 or so be enough?

A

[Bob Carpenter]

> On Aug 7, 2015, at 10:07 AM, Lei Zhang <bnuzhan...@gmail.com> wrote:
>
> Hi Andrew and Stan,
>
> thanks a lot for your comments and suggestions.
>
> I've tried 10000 samples per chain (with 50% warmup) with thinning =1. However, the autocorrelation is still high. For some chains, even when lag =10, the autocorr is still as high as around 0.4. Probably I should run chains with more samples?

I'd suggest looking at the n_eff rather than autocorrelation.

> The concern regarding the point estimates is that, we somehow do need point estimates to correlate them with imaging data for conducting model-based analysis. In practice, it is hard to fit the whole posterior distribution with the brain data.

I don't understand why you need point estimates.

I'm also not sure what you mean by "fit the whole posterior".

> For the ppcheck, I just found the shinyStan online demo pretty useful, I will then follow that. But what we want to have is a trial-by-trial ppcheck, so purely summarize from the posterior distribution could not provide some precise information. Does anybody have any experience with trial-by-trial ppcheck? I'd highly appreciate.

I don't.

> Plus, with regard to our last point, is it possible to fit a model with a weighted average of the two sampling statement?

Check out the mixture model chapter in the manual.

- Bob

[Lei Zhang]

Hi Andrew, 

yes you are right. And for some individual level parameters, the ESS is even less than 10% of the samples. 

The reason why we use long chains (10,000) is that we are implementing a rather complex hierarchical model with 12 group-level parameters and 6 individual-level parameters. We also need to fit the model in a trial-by-trial manner. Altogether, we thought longer chains could yield better results. Or maybe I got this wrong?

Lei

[Lei Zhang]

Hi Bob,

right, I could look at n_eff instead. Is there a criteria in practice? How much percent is good enough?

Regarding the point estimates, in model-based fMRI, we will have to correlate some point estimates (eg. mode/median) per subject against the imaging data. I'm not sure if we could correlate the whole posterior distribution against the imaging data. 

Lei

[Daniel Lee]

On Mon, Aug 10, 2015 at 4:02 AM, Lei Zhang <bnuzhan...@gmail.com> wrote:

Hi Bob,

right, I could look at n_eff instead. Is there a criteria in practice? How much percent is good enough?

In practice, you want enough n_eff to make as accurate estimate as you need.

Regarding percent, I wouldn't really look at that. I would first check to see that there weren't any post-warmup divergent iterations. That's going to tell you whether you're having a  issue. If you do:

- increase delta

- decrease epsilon

- reparameterize
 

--

[Bob Carpenter]

Suppose you have a function f(theta), that takes a parameter vector
theta and returns something like your correlation vs. imaging data.

The fully Bayesian approach is to treat theta like a random variable,
condition on the observed data y, and compute the posterior f(theta).

In practice, all that means is that you take M draws theta(1), ..., theta(M)
from the posterior p(theta | y), and then f(theta(1)), ..., f(theta(M)) is
a sequence of M draws from the posterior on f(theta). You can compute estimates
for posterior means of f(theta), posterior medians, posterior intervals, etc.
It builds in the uncertainty in f(theta) due to the uncertainty for theta,
which is modeled in Bayesian stats using the posterior p(theta | y).

- Bob

[Jonah]

Regarding the question about the length of the chains: there's no point in running them that long unless you really need to (e.g. takes that long to converge). If the markov chain isn't geometrically ergodic then by running the chains for a really long time you might suddenly see undesirable behavior, which is a big red flag, but in general if there's convergence then you only need as many draws as will give you a good effective sample size for your goals.

[Lei Zhang]

Hi Daniel,

thanks for your comments. I'll definitely check the post-warmup iterations. 

But I don't really understand what do delta and epsilon stand for. Are they just arbitrary examples or specific parameters (sorry for the lack of enough knowledge)?

thanks again,

Lei 

[Lei Zhang]

Hi Jonah,

thanks for your suggestion. I do see the point that we are having too long chains. However, even with longer chains, the autocorrelation is still high, though all the R-hat is below 1.10. So do I have to worry about the autocorrelation, or just stick with the R-hat as the indicator for convergence?

Thanks a lot,

Lei

[Jonah Sol Gabry]

To answer your question to Daniel,

delta (or adapt_delta) is the target metropolis acceptance probability and epsilon is the integrator stepsize. If you're using RStan then check out the 'control' argument to the stan or sampling function.

You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/SKMzamqciqY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.

[Daniel Lee]

Thanks. I didn't have the reference off hand.

Each of the interfaces controls how to set the acceptance probability separately. This is documented in the CmdStan manual and probably in the RStan stan() function's help.

Daniel

[Jonah Sol Gabry]

Regarding the other question, it's really a matter of whether you have enough information/power/precision (I mean these informally, we don't have a great word for it) in your autocorrelated draws (i.e. n_eff is sufficient). You can also check at which lag the draws become essentially independent. 

On Tuesday, August 11, 2015, Lei Zhang <bnuzhan...@gmail.com> wrote:

--
You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/SKMzamqciqY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.

[Bob Carpenter]

Read the chapter of the manual on MCMC. If you have questions
about that, please repost.

- Bob

> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.