model fit using M2 and anova

[Jeanne Sinclair]

Hello mirt forum,

First, I would like to express my thanks to Dr. Chalmers for his help last week.

I'm attempting to analyze dichotomous reading comprehension data (n=10,521) for DIF with 1 reference group and 5 focal groups. I am quite new to MIRT so I apologize in advance for any obvious (or subtle) errors and would appreciate your feedback. 

My understanding is that as a very first step I need to determine the dimensionality of the data. To do so I ran this code (not including multipleGroup at this stage):

mod1 <- mirt(read_mirt, 1, verbose=TRUE)

mod2 <- mirt(read_mirt, 2, verbose=TRUE)

mod3 <- mirt(read_mirt, 3, verbose=TRUE)

mirt::M2(mod1)

mirt::M2(mod2)

mirt::M2(mod3) 

Here are the model fit statistics produced by M2:

> mirt::M2(mod1)

            M2  df p      RMSEA    RMSEA_5   RMSEA_95      SRMSR       TLI       CFI

stats 1522.236 405 0 0.01619336 0.01533095 0.01706289 0.01842013 0.9853581 0.9863679

> mirt::M2(mod2)

           M2  df p      RMSEA    RMSEA_5   RMSEA_95      SRMSR       TLI       CFI

stats 1035.82 376 0 0.01291549 0.01198092 0.01385578 0.01546339 0.9906858 0.9919491

> mirt::M2(mod3)

            M2  df p      RMSEA     RMSEA_5 RMSEA_95      SRMSR       TLI       CFI

stats 784.1172 348 0 0.01091451 0.009897989 0.011932 0.01353918 0.9933483 0.9946786

I've read that SRMSR less than .05 is acceptable, and that TLI and CFI greater than .95 are acceptable. (I've also heard that RMSEA itself is not ideal for model selection.) However, I'm unsure if these are cut-offs, or if relatively smaller SRMSR and greater TLI/CFI are considered better fits (along the lines of BIC/AIC interpretations).  Therefore, I am not sure which of these models is a better fit. 

Although I am not entirely sure this is how I should proceed, I calculated AIC and BIC using the anova method. This appears to show that 2 or 3 dimensions are preferable to 1. Comparing 2 to 3 dimensions, 3 dimensions appears preferable for all indices except non-sample size adjusted BIC.

> anova (mod1, mod2)

Model 1: mirt(data = read_mirt, model = 1, verbose = TRUE)

Model 2: mirt(data = read_mirt, model = 2, verbose = TRUE)

       AIC     AICc    SABIC      BIC    logLik      X2  df   p

1 268046.1 268046.8 268291.1 268481.8 -133963.1     NaN NaN NaN

2 267718.3 267719.9 268081.7 268364.6 -133770.2 385.808  29   0

> anova (mod1, mod3)

Model 1: mirt(data = read_mirt, model = 1, verbose = TRUE)

Model 2: mirt(data = read_mirt, model = 3, verbose = TRUE)

       AIC     AICc    SABIC      BIC    logLik      X2  df   p

1 268046.1 268046.8 268291.1 268481.8 -133963.1     NaN NaN NaN

2 267554.0 267556.6 268031.7 268403.5 -133660.0 606.143  57   0

> anova (mod2, mod3)

Model 1: mirt(data = read_mirt, model = 2, verbose = TRUE)

Model 2: mirt(data = read_mirt, model = 3, verbose = TRUE)

       AIC     AICc    SABIC      BIC    logLik      X2  df   p

1 267718.3 267719.9 268081.7 268364.6 -133770.2     NaN NaN NaN

2 267554.0 267556.6 268031.7 268403.5 -133660.0 220.335  28   0

>  

It appears the best-fitting IRT model has three dimensions. If the forum could provide advice about model fit, I would appreciate it very much.

(Note: is it necessary to run 4-dimensional model for comparison?)

Thank you!

Jeanne

[Jeanne Sinclair]

I have an additional question and would appreciate any advice the forum can offer. While it looks like multidimensional model is a better fit, the 2-dimensional discrimination coefficients are negative, which I do not know how to interpret. The one-dimensional model, however, has positive discrimination (please see below). Thank you in advance for your help! 

coef1 (unidimensional)

$rmc01

       a1     d g u

par 0.965 1.951 0 1

$rmc02

       a1     d g u

par 1.415 3.407 0 1

$rmc03

       a1     d g u

par 1.411 2.822 0 1

> coef2 (2 dimensions unrotated)

$rmc01

        a1     a2     d g u

par -0.901 -0.342 1.949 0 1

$rmc02

        a1     a2     d g u

par -1.308 -0.525 3.401 0 1

$rmc03

        a1     a2     d g u

par -1.226 -0.744 2.841 0 1

> coef2r (orthogonal rotation with 2 dimensions)

$rmc01

        a1     a2     d g u

par -0.688 -0.675 1.949 0 1

$rmc02

        a1     a2     d g u

par -0.987 -1.007 3.401 0 1

$rmc03

        a1     a2     d g u

par -0.823 -1.175 2.841 0 1

[Phil Chalmers]

Hi Jeanne,

On Mon, Oct 2, 2017 at 11:46 AM, Jeanne Sinclair <jeanneh...@gmail.com> wrote:

Hello mirt forum,

First, I would like to express my thanks to Dr. Chalmers for his help last week.

I'm attempting to analyze dichotomous reading comprehension data (n=10,521) for DIF with 1 reference group and 5 focal groups. I am quite new to MIRT so I apologize in advance for any obvious (or subtle) errors and would appreciate your feedback. 

My understanding is that as a very first step I need to determine the dimensionality of the data. To do so I ran this code (not including multipleGroup at this stage):

mod1 <- mirt(read_mirt, 1, verbose=TRUE)

mod2 <- mirt(read_mirt, 2, verbose=TRUE)

mod3 <- mirt(read_mirt, 3, verbose=TRUE)

mirt::M2(mod1)

mirt::M2(mod2)

mirt::M2(mod3) 

Here are the model fit statistics produced by M2:

> mirt::M2(mod1)

            M2  df p      RMSEA    RMSEA_5   RMSEA_95      SRMSR       TLI       CFI

stats 1522.236 405 0 0.01619336 0.01533095 0.01706289 0.01842013 0.9853581 0.9863679

> mirt::M2(mod2)

           M2  df p      RMSEA    RMSEA_5   RMSEA_95      SRMSR       TLI       CFI

stats 1035.82 376 0 0.01291549 0.01198092 0.01385578 0.01546339 0.9906858 0.9919491

> mirt::M2(mod3)

            M2  df p      RMSEA     RMSEA_5 RMSEA_95      SRMSR       TLI       CFI

stats 784.1172 348 0 0.01091451 0.009897989 0.011932 0.01353918 0.9933483 0.9946786

I've read that SRMSR less than .05 is acceptable, and that TLI and CFI greater than .95 are acceptable. (I've also heard that RMSEA itself is not ideal for model selection.) However, I'm unsure if these are cut-offs, or if relatively smaller SRMSR and greater TLI/CFI are considered better fits (along the lines of BIC/AIC interpretations).  Therefore, I am not sure which of these models is a better fit. 

These fit statistics have generally been established for SEMs, not IRT models.  So take the cut-offs with a grain of salt.

Although I am not entirely sure this is how I should proceed, I calculated AIC and BIC using the anova method. This appears to show that 2 or 3 dimensions are preferable to 1. Comparing 2 to 3 dimensions, 3 dimensions appears preferable for all indices except non-sample size adjusted BIC.

> anova (mod1, mod2)

Model 1: mirt(data = read_mirt, model = 1, verbose = TRUE)

Model 2: mirt(data = read_mirt, model = 2, verbose = TRUE)

       AIC     AICc    SABIC      BIC    logLik      X2  df   p

1 268046.1 268046.8 268291.1 268481.8 -133963.1     NaN NaN NaN

2 267718.3 267719.9 268081.7 268364.6 -133770.2 385.808  29   0

> anova (mod1, mod3)

Model 1: mirt(data = read_mirt, model = 1, verbose = TRUE)

Model 2: mirt(data = read_mirt, model = 3, verbose = TRUE)

       AIC     AICc    SABIC      BIC    logLik      X2  df   p

1 268046.1 268046.8 268291.1 268481.8 -133963.1     NaN NaN NaN

2 267554.0 267556.6 268031.7 268403.5 -133660.0 606.143  57   0

> anova (mod2, mod3)

Model 1: mirt(data = read_mirt, model = 2, verbose = TRUE)

Model 2: mirt(data = read_mirt, model = 3, verbose = TRUE)

       AIC     AICc    SABIC      BIC    logLik      X2  df   p

1 267718.3 267719.9 268081.7 268364.6 -133770.2     NaN NaN NaN

2 267554.0 267556.6 268031.7 268403.5 -133660.0 220.335  28   0

>  

It appears the best-fitting IRT model has three dimensions. If the forum could provide advice about model fit, I would appreciate it very much.

(Note: is it necessary to run 4-dimensional model for comparison?)

Probably not. That said, the test structure seems to be fairly well fit by even a uni-dimensional model according to the fit statistics. You may do just as well using that model and avoiding multidimensionality all together. Whether that's valid or not is up to your theoretical structure (the reasons you're seeing such differences in anova() is largely a function of your sample size). Higher-dimensional models will almost always fit the data better because they use considerably more parameters. 

For DIF testing though, you really should establish the pattern of loadings first (think exploratory factor analysis to get to confirmatory factor analysis). Otherwise, you'll be performing DIF analyses that are largely meaningless. Check out the loadings pattern via summary() and see if it makes sense theoretically for these models.

Phil

  

 

Thank you!

Jeanne

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Phil Chalmers]

See the previous reply. This has to do with using an exploratory (rotationally indeterminate) model versus a confirmatory model. 

Phil

--

[Jeanne Sinclair]

Thank you so much! This is very helpful. I will review the loadings in summary and try to figure out the confirmatory approach.