IRT 2PL model in Stan (as in Curtis, 2010)

[t.wol...@ibe.edu.pl]

Curtis (2010: http://www.jstatsoft.org/v36/c01) has provided a BUGS code for IRT models. I re-coded the model from Table 1 in Stan:

data {
   int<lower=1> N;
   int<lower=1> K;
   int<lower=1> J;
   matrix<lower=0, upper=1>[N, K] Y;
}
parameters {
   real theta[N];
   real alpha<lower=0>[J];
   real beta[J];
}
model {
   beta ~ normal(0, 1);
   alpha ~ normal(0, 1);
   theta ~ normal(0, 1);

   for (i in 1:N) {
      for (j in 1:K) {
         Y[i, j] ~ bernoulli_logit(alpha[j] * (theta[i] - beta[j]));
      }
   }
}

I know that Bob Carpenter has provided the code for IRT in Stan (https://github.com/stan-dev/example-models/tree/master/misc/irt) but I would like to use the Curtis-like code. The question is: is it ok? Would it be better, for example, to use lognormal for alpha?

Thanks fot suggestions!

Tim

[t.wol...@ibe.edu.pl]

Sorry for typhos:

data {
  int<lower=1> N;
  int<lower=1> K;

  matrix<lower=0, upper=1>[N, K] Y;
}
parameters {
  real theta[N];

  real<lower=0> alpha[K];
  real beta[K];

[Daniel Lee]

Hi Tim,

Have you tried running this model? I'm not sure what you're asking. It seems like there are going to be unidentifiability issues, but you might have enough data to get over that.


Daniel

--
You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tymoteusz Wołodźko]

I think I already found an answer by Bob Carpenter in here:
https://groups.google.com/forum/#!searchin/stan-users/irt/stan-users/U_F1p7YSiro/1yWzoSX1MNEJ

Tim

--
You received this message because you are subscribed to a topic in the Google Groups "Stan users mailing list" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/stan-users/MmmkCFNVyX4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to stan-users+...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Tymoteusz Wołodźko

Zespół Analiz Osiągnięć Uczniów Instytut Badań Edukacyjnych | ul. Górczewska 8 | 01-180 Warszawa
tel.  +48 695 370 391 | www.ibe.edu.pl

[Bob Carpenter]

We need to rewrite the example models in the manual (I added
to the issue) knowing what we know now.

The approach we usually go with is to pin down student abilities

theta ~ normal(0,1)

with a fixed prior, which will (softly) fix the scale and location.
Then constrain discrimations gamma to be > 0, under the assumption
that there aren't questions that are harder for good students (that'll
sort the sign identification).

Then you can give alpha and gamma hierarchical priors with varying
means and scales, or just fix the mean of discrimination to 1.

- Bob

[t.wol...@ibe.edu.pl]

Thanks!

I decided to use the model that was quoted in the discussion:

data {
  int<lower=1> N;
  int<lower=1> K;

  int<lower=0, upper=1> Y[N, K];

}
parameters {
  real theta[N];
  real<lower=0> alpha[K];
  real beta[K];
}
model {

  beta ~ normal(0, 10);
  alpha ~ lognormal(0, 1);

  theta ~ normal(0, 1);
 
  for (i in 1:N)
    for (j in 1:K)
      Y[i, j] ~ bernoulli_logit(alpha[j] * (theta[i] - beta[j]));
}

it maches the one by Curtis.

However I must comment, Bob, that there actually are questions that are harder for good students. I have in mind few examples, that however would be hard to translate, of questions about complicated stuff, where you can choose between complicated answer and a simple answer, so that good students choose the complicated answer that seems correct while the simple one, that is choosen by worse students, is in fact the correct one. Those are very rare but happend in practice. So while in general discrimination > 1 is a good idea, but you have to remember that it will not always work with the real data. Maybe it woundn't be such a bad idea to have a prior that enables a very, very small chance of having a negative discrimination..?

Greetings,
Tim

[Bob Carpenter]

> On Nov 5, 2014, at 2:22 AM, t.wol...@ibe.edu.pl wrote:
>
> Thanks!
>
> I decided to use the model that was quoted in the discussion:
>
> data {
> int<lower=1> N;
> int<lower=1> K;
> int<lower=0, upper=1> Y[N, K];
> }
> parameters {
> real theta[N];
> real<lower=0> alpha[K];
> real beta[K];
> }
> model {
> beta ~ normal(0, 10);
> alpha ~ lognormal(0, 1);
> theta ~ normal(0, 1);
>
> for (i in 1:N)
> for (j in 1:K)
> Y[i, j] ~ bernoulli_logit(alpha[j] * (theta[i] - beta[j]));
> }
>
> it maches the one by Curtis.
>
> However I must comment, Bob, that there actually are questions that are harder for good students. I have in mind few examples, that however would be hard to translate, of questions about complicated stuff, where you can choose between complicated answer and a simple answer, so that good students choose the complicated answer that seems correct while the simple one, that is choosen by worse students, is in fact the correct one. Those are very rare but happend in practice. So while in general discrimination > 1 is a good idea, but you have to remember that it will not always work with the real data. Maybe it woundn't be such a bad idea to have a prior that enables a very, very small chance of having a negative discrimination..?

I can imagine non-linear effects, too, where poor to medium students
and really good students get the question right, but good students get it
wrong. We mainly work with well designed and well tested tests that
don't have this problem, but it can definitely come up.

You can code whatever kind of prior you want, but it should ideally
be continuous and differentiable at 0. It could even be a mixture. If
the negative effect's small, the prior may dominate, though, and force
an answer near 0.

If you just let the sign float (discimination unconstrained), you get
two posterior modes of the same likelihood contour with their signs
flipped. What you'll see it different chains exploring different ones
and messing up your final results. You can try pinning down some of the
questions if you know ones that aren't pathological (the approach often
taken for ideal point models where the sign isn't known, but you can
pin down one issue or one judge or something on the scale somewhere).

- Bob