IRT model fit indices

[Augustin Mutak]

Dear Phil,

I am trying to perform an IRT analysis of a questionnaire using the graded response model. The questionnaire consists of 9 items on a 1-7 Likert type scale and there are 243 subjects in the dataset. However, when I calculate the model and call the G2 fit statistic or its associated p-value:

extract.mirt(fit.iaq1, what = "G2") or extract.mirt(fit.iaq1, what = "p")

I get "NaN" as the output. I also tried using the M2 function, but it tells me that it cannot be calculated since df is too low. I tried adding 

calcNull = T

to the mirt function when I run it to obtain CFI and TLI, but those two indicators are also NaN. Item-level fit, which I accessed using the itemfit() function works fine. Is there anything else I can do to get some "global" fit indices?

Best,

AM

[Phil Chalmers]

On Sun, Nov 26, 2017 at 11:27 AM, Augustin Mutak <mut...@gmail.com> wrote:

Dear Phil,

I am trying to perform an IRT analysis of a questionnaire using the graded response model. The questionnaire consists of 9 items on a 1-7 Likert type scale and there are 243 subjects in the dataset. However, when I calculate the model and call the G2 fit statistic or its associated p-value:

extract.mirt(fit.iaq1, what = "G2") or extract.mirt(fit.iaq1, what = "p")

I get "NaN" as the output.

This makes sense. There's theoretically 7^9 unique response patterns required to saturate the G2 statistic (each of which needs 3+ observations to be stable). So mirt doesn't compute this statistic when the data is too sparse. 

 

I also tried using the M2 function, but it tells me that it cannot be calculated since df is too low. I tried adding 

calcNull = T

to the mirt function when I run it to obtain CFI and TLI, but those two indicators are also NaN.

Right, it looks like you're in the no-mans land for goodness of fit statistics. There is a fix for this via a hybrid approach for M2, but it requires a pretty big rewrite that I haven't found the time or inspiration to do yet. 

 

Item-level fit, which I accessed using the itemfit() function works fine. Is there anything else I can do to get some "global" fit indices?

Not so much no. Though given that you're using the graded response model you could get a decent proxy with the lavaan package and it's model-fit statistics. HTH.

Phil

 

Best,

Augustin

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[AM]

Dear Phil,

Thank you for your answer! I didn't know it was OK to use CFA fit indices as a proxy. Makes sense though, if the simpler model fits good, then the more complex model should fit at least equally good.

Best,
Augustin

[ecme...@gmail.com]

One more point related to the above discussion: in a graded response model, it seems really difficult to obtain an RMSEA above 0 (i.e., at .01 or above). Is RMSEA really so insensitive for this model? I especially noticed it in my class projects this semester where, regardless of the data, it seemed like everyone had a good-fitting GRM (as judged by RMSEA).

In exploring it, I notice that mirt 1.26.3 reports a much smaller RMSEA than the old stackexchange post below, so I wonder whether some handling of sparsity (small sample size, many response categories) has changed.

https://stats.stackexchange.com/questions/95785/how-to-check-goodness-of-fit-for-a-graded-response-model-in-r

Ed

[Phil Chalmers]

Hi Ed,

I get the following using mirt 1.26.3 on my R 3.4.3. The only thing that really changes is that seeds are slightly different across R versions. 

> library(mirt)

> set.seed(1)

> a <- matrix(rlnorm(20))

> diffs <- t(apply(matrix(runif(20*4, .3, 1), 20), 1, cumsum))

> diffs <- -(diffs - rowMeans(diffs))

> d <- diffs + rnorm(20)

> dat <- simdata(a, d, N=1000, itemtype = 'graded')

> mod <- mirt(dat, 1)

Iteration: 42, Log-Lik: -23030.209, Max-Change: 0.00010

> M2(mod)

            M2  df         p       RMSEA RMSEA_5   RMSEA_95      SRMSR       TLI       CFI

stats 113.5443 110 0.3892452 0.005679219       0 0.01731935 0.02078929 0.9991159 0.9992519

The RMSEA calculations are performed with the non-exported mirt:::rmsea() function, and as far as I can tell that gives the correct results too, even when passing the values from the CV post. 

mirt:::rmsea(126.155, 110, 1000)

Phil

--

[Ed Merkle]

Thanks for looking at it. It seemed like a fairly big difference to me to move from RMSEA of .006 (1.26.3) to .012 (the stackexchange output), when the old RMSEA rules of thumb tell you to look for something under .05. But it may well be that those rules of thumb just don't work here.

Ed