GRM model consulation

[Katherine Grace Jonas]

Hello All,

First off, thank you to everyone who's contributed to this list.  Searching through the questions and answers has helped me solve a number of problems with this model.

The model I'm working on (a version and simulated data are attached) is a graded response IRT model, where the difficulty parameter is decomposed into two components: an item component and a person component.  The person component would theoretically encompass response styles.  I am hoping to capture response style by having respondents answer items with regard to themselves, as well as a vignette (or three, in this case).  Response styles should remain constant across self-report, vignettes, and items.

The model shows poor convergence across parameters.  Rhat is large for most parameters, n_effs can be 2, and the chains wander about.  The problem seems to be caused by difficulties estimating the person difficulties (bi), because RStan frequently warns that the category cutpoints are not appropriately ordered.  The model will converge without bi, or if bi_scale is given a very tight prior.  My hunch is that there is a better way to parameterize bi.  The category cutpoints (bj) can be declared as ordered vectors, but the person parameters can't, so the sum of bi and bj is not always ordered.  The prior on bi_scale (uniform(0,3*N)) should keep bi in about the right range, given the distance between category cutpoints, but that doesn't seem to be enough.

I'd really appreciate your thoughts.  Thanks again to all the list contributors.

Katherine

[Joshua Rosenau]

Hmm, it looks like you’re scaling each cutpoint separately, which would seem like it would leave the vector bi[] not being ordered in at least some cases. Shouldn’t you be multiplying by a single value (or a single value per item, at least)?

Also, I think that’s not the best way to fix a value for bj[1,2]. Better to construct bj in the transformed parameters block, which would require a bit of reorganization of the code, but would allow you to simply assign 0 to bj[1,2]. You’d do something like declare parameters:

 

ordered[K-1] bj_raw[J-1]; 

real<upper=0> bj_init_a;

real<lower=0> bj_init_b;

Then in the transformed parameters, declare vector[K-1] bj[J], and:

bj[1,1]<-bj_init_a; bj[1,2]<-0; bj[1,3]<-bj_init_b;

for (j in 2:J) {

bj[j]<-bj_raw[j-1];

}

and put the necessary priors on all the declared parameters (bj_raw, bj_init_a, bj_init_b). 

Hope that helps.

Josh

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
<IRT bi 7SR14Vign model.txt><bi 7SR14Vign data.rdump>

[Katherine Grace Jonas]

Thanks for the suggestions.  Using only one scale parameter helped.  In this model (attached), the priors on bi_raw and bi_scale are constrained so that bi + bj will be appropriately ordered.  The item cutpoints are declared and fixed as per your suggestion.  With those changes, STAN is producing fewer errors (just one or two per warm up period per chain).

The traces look better than before, but still aren't mixing well (a few from bi are attached).  Rhats are still high, from 1.3 to 20.  However, the bi estimates themselves are generally in the right range.  At least, they tend to be of the same sign as the simulated values.  

Another possible hiccup is in the response data.  Some simulated respondents never use certain response categories, due to response style.  It would be very difficult to estimate the third threshold (bi[,3]) for someone who never uses the highest response category.  Simulating more extreme vignettes helps, but in the real world there are often respondents with this sort of response data.  It may call for a ragged array.

--
You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/TqkyAt8BkQo/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.

[Bob Carpenter]

On Jul 19, 2014, at 2:45 PM, Katherine Grace Jonas <katherine....@gmail.com> wrote:

> Thanks for the suggestions. Using only one scale parameter helped. In this model (attached), the priors on bi_raw and bi_scale are constrained so that bi + bj will be appropriately ordered. The item cutpoints are declared and fixed as per your suggestion. With those changes, STAN is producing fewer errors (just one or two per warm up period per chain).
>
> The traces look better than before, but still aren't mixing well (a few from bi are attached). Rhats are still high, from 1.3 to 20. However, the bi estimates themselves are generally in the right range. At least, they tend to be of the same sign as the simulated values.

If the model's converging properly, all R-hat values should be under 1.05,
and ideally very close to 1.

Very high R-hats often signal non-identifiabilities in the model, especially
if other parameters do converge reasonably.

> Another possible hiccup is in the response data. Some simulated respondents never use certain response categories, due to response style. It would be very difficult to estimate the third threshold (bi[,3]) for someone who never uses the highest response category. Simulating more extreme vignettes helps, but in the real world there are often respondents with this sort of response data. It may call for a ragged array.

You just want to be sure to avoid things like separability. You need
stronger priors to overcome them and still sample efficiently.

- Bob

> <IRT bi 7SR14Vign model.txt><bi traces 2014-07-19.pdf><bi 7SR21Vign data.rdump>

[Katherine Grace Jonas]

I'm sorry for the delayed response.  Thank you for your advice!  Stronger priors definitely helps convergence, and in this case there is a little more prior information to use.