Fitting a Multidimensional IRT Model in Stan

[Rick Farouni]

Hi Everyone,

I wrote two blog posts about fitting IRT models in Stan. I have provided code that I hope might be of benefit to people who are fitting similar models in Stan. Here are the links

Multidimensional IRT with plots rendered using the rbokeh package: http://rfarouni.github.io/code/2015/04/22/mirt/

2PL unidimensional IRT model with results of fit exported to ShinyStan: http://rfarouni.github.io/code/2015/03/20/irt/

I apologize for an errors you might in the posts. Feedback is very welcome.

Thanks,

Rick Farouni

[Bob Carpenter]

Thanks for sharing! I love these knitr docs that include
everything for a model from simulation to fit. It's the format
I'd like to adopt for all of our examples (at least those done
in R --- IPython Notebooks would do the same work).

Identification's a tricky business. Presumably Doug Rivers
was talking about MLE identification in his 2003 paper.
I'm not sure how your parameterization in terms of a product
minus a sum is identifiable --- it always hurts my head to
think about these.

We'd have made your life a lot easier if we had a lower-triangular
data type or even general ragged matrices!

That knitr-HTML-rendered LaTeX could use some work in spacing,
like between probablity terms in multiplications.

I didn't understand the hex plot. Why are there hexagons and
why are they all touching?

The print output's still too much to scan in a presentation, but
it's nice to see it so users know what to do if they follow your example.
I'm always torn about that, because a visualization of the means (and
perhaps uncertainties) would be easier to scan --- maybe even scatterplots
of the two alpha components.

The person order vs. number of correct responses is hard to
understand --- wouldn't a histogram have been easier?

- Bob

> --
> You received this message because you are subscribed to the Google Groups "Stan users mailing list" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to stan-users+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Ben Goodrich]

On Thursday, April 23, 2015 at 2:49:37 AM UTC-4, Bob Carpenter wrote:

We'd have made your life a lot easier if we had a lower-triangular
data type

You added such a type: cholesky_factor_cov[J,K]
 

or even general ragged matrices!

Yes, that would help too.

Ben

[Rick Farouni]

On Thursday, April 23, 2015 at 2:49:37 AM UTC-4, Bob Carpenter wrote:

Thanks for sharing!  I love these knitr docs that include
everything for a model from simulation to fit.  It's the format
I'd like to adopt for all of our examples (at least those done
in R --- IPython Notebooks would do the same work).

Identification's  a tricky business.  Presumably Doug Rivers
was talking about MLE identification in his 2003 paper.
I'm not sure how your parameterization in terms of a product
minus a sum is identifiable --- it always hurts my head to
think about these.   

 

I guess that  by MLE identification, you are referring to local identification. In that case, I think you are right.  I wrote a post summarizing identifiability of IRT models here  http://rfarouni.github.io/2015/04/23/identification/  

We'd have made your life a lot easier if we had a lower-triangular
data type or even general ragged matrices!

That knitr-HTML-rendered LaTeX could use some work in spacing,

like between probablity terms in multiplications.   True

I didn't understand the hex plot.  Why are there hexagons and
why are they all touching?  

 Probably, a two dimensional histogram could have been as effective, but I couldn't resist the colorful display.  I have only started using them, but  I guess that the hexagons are touching for the same reason rectangles in a histogram abut each other.

The print output's still too much to scan in a presentation, but
it's nice to see it so users know what to do if they follow your example.
I'm always torn about that, because a visualization of the means (and
perhaps uncertainties) would be easier to scan --- maybe even scatterplots
of the two alpha components.

 

 I agree. A very good point. 

 

The person order vs. number of correct responses is hard to
understand --- wouldn't a histogram have been easier?

I used this plot because  a histogram won't show any deviations from, or the variability of, the linear trend we expect to see. Namely, the higher the latent ability on one dimension, the higher the proportion of correct responses.

[Bob Carpenter]

> On Apr 24, 2015, at 4:59 AM, Rick Farouni <rfar...@gmail.com> wrote:
>
>
>
> On Thursday, April 23, 2015 at 2:49:37 AM UTC-4, Bob Carpenter wrote:

> ...

> I didn't understand the hex plot. Why are there hexagons and
> why are they all touching?
>
> Probably, a two dimensional histogram could have been as effective, but I couldn't resist the colorful display. I have only started using them, but I guess that the hexagons are touching for the same reason rectangles in a histogram abut each other.

Why hexagons? I take it the counts represent counts of places
where the theta values are in the hexagon? I'd have thought squares
would be much easier to understand because then you can read off
the axes.

> The person order vs. number of correct responses is hard to
> understand --- wouldn't a histogram have been easier?
>
>
> I used this plot because a histogram won't show any deviations from, or the variability of, the linear trend we expect to see. Namely, the higher the latent ability on one dimension, the higher the proportion of correct responses.

Ah, I'd forgottten that you'd sorted the students by
ability.

- Bob

[Rick Farouni]

On Thursday, April 23, 2015 at 10:21:37 PM UTC-4, Bob Carpenter wrote:

> On Apr 24, 2015, at 4:59 AM, Rick Farouni <rfar...@gmail.com> wrote:
>
>
>
> On Thursday, April 23, 2015 at 2:49:37 AM UTC-4, Bob Carpenter wrote:
> ...
> I didn't understand the hex plot.  Why are there hexagons and
> why are they all touching?  
>
>  Probably, a two dimensional histogram could have been as effective, but I couldn't resist the colorful display.  I have only started using them, but  I guess that the hexagons are touching for the same reason rectangles in a histogram abut each other.

Why hexagons?  I take it the counts represent counts of places
where the theta values are in the hexagon?  I'd have thought squares
would be much easier to understand because then you can read off
the axes.

 

I see your point. This information would be lost indeed, but perhaps, and I am speculating here, if the goal is to obtain a discrete approximation to a density surface, then hexagons might be appropriate since if I am not mistaken, hexagons are an optimal pattern in some respect for tessellating a plane (e.g. number of edges required).

[Michael Betancourt]

The idea is that hexagonal tilings don’t introduce a visual

bias to the axes like a square tiling does (in a square tiling

diagonal moves are awkward and so the eye tends to follow

horizontal and vertical lines).  In practice that’s way more

relevant to tabletop games (http://en.wikipedia.org/wiki/Hex_map)

and I’ve never found it particularly compelling as a

visualization argument.  If the appearance of the plot is that 

sensitive to the tiling then make the tiles smaller, and if you

can’t make them any smaller than you don’t really have enough

data yield a reasonable histogram estimator of the density.