Possible bug while estimating X2*/X2*_df statistics in itemfit() for multidimensional model results

[Giuseppe Carteny]

Dear Phil,

First, thank you for developing the mirt package. It is a really powerful, flexible and complete tool that I am extensively using for my PhD research.

I guess that I encountered a bug with the itemfit().

I was trying to "extract" from the results of my model different goodness-of-fit statistics. 

However when I try to estimate the X2* or the X2*_df statistics I encounter the following error:

  

  Error: Rows in supplied and starting value data.frame objects do not match. Were the data or itemtype input arguments modified?

  

Since I was trying different uni- and bidimensional models, I noticed that the error occurs when I switch from unidimensional to bidimensional ones.

I didn't try with additional dimensions, thus I don't know if the problem occurs also with multidimensional(>2) models.

In any case, thanks in advance and I look forward to hearing from you. 

Cheers,

Giuseppe

# Code ==========================================================================================

library(mirt)
library(mvtnorm)


# Simulated data - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
a <- matrix(c(rep(1, 20), rep(0,40), rep(1,20)), 40)
d <- matrix(rnorm(40*4), 40)
d <- t(apply(d, 1, sort, decreasing=TRUE))
dat <- simdata(a, d, 1000, Theta=Theta, itemtype = 'graded')


dat <- data.frame(poly=dat[,1:6])


# Model specification - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
model1 <- mirt.model('F1 = 1-6')
model2 <- mirt.model('F1 = 1-3  
                    F2 = 4-6
                    COV = F1*F2')


# Estimation - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
res1 <- mirt(model=model1, method="MHRM", SE=T, SE.type="MHRM", data=dat)  
res2 <- mirt(model=model2, method="MHRM", SE=T, SE.type="MHRM", data=dat)  



# Function to wrap up some fit statistics - - - - - - - - - - - - - - - - - - - - - - - - - - - -
fit.fun <- function(res) {
 
  items.fit <- list(S_X2=itemfit(x=res,
                                 fit_stats = "S_X2"),
                    X2_star=itemfit(x=res,
                                    fit_stats="X2*",
                                    boot=1000),
                    X2_star_df=itemfit(x=res,
                                       fit_stats = "X2*_df",
                                       boot_dfapprox=1000)
                    )
 
  return(items.fit)
}


fit.res1 <- fit.fun(res1) # Here the function works
fit.res2 <- fit.fun(res2) # Here it doesn't.


# Error: Rows in supplied and starting value data.frame objects do not match. Were the
# data or itemtype input arguments modified?
# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

[Phil Chalmers]

Hi Giuseppe,

Dear Phil,

First, thank you for developing the mirt package. It is a really powerful, flexible and complete tool that I am extensively using for my PhD research.

Thanks for the kind words. Sadly, 'bug free' wasn't part of the description.... maybe one day!

 

I guess that I encountered a bug with the itemfit().

I was trying to "extract" from the results of my model different goodness-of-fit statistics. 

However when I try to estimate the X2* or the X2*_df statistics I encounter the following error:

  

  Error: Rows in supplied and starting value data.frame objects do not match. Were the data or itemtype input arguments modified?

  

Since I was trying different uni- and bidimensional models, I noticed that the error occurs when I switch from unidimensional to bidimensional ones.

I didn't try with additional dimensions, thus I don't know if the problem occurs also with multidimensional(>2) models.

In any case, thanks in advance and I look forward to hearing from you. 

Models fit via bfactor() are not very well supported by itemfit() since mirt() is the main function being called for the bootstrapping, but those models originally fit via mirt() all should have been. In this case, the X2* was just never implemented for multidimensional models (integration grid was never made to multidimensional). I have now sent a patch to the dev version of Github to add support for this type of input object. Please install from their for the new feature, and thanks for the report! Cheers.

Phil

 

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mirt-package/51724ecc-0607-4579-b356-2239a8479177%40googlegroups.com.

[Giuseppe Carteny]

Hi Phil,

First, don't worry, a world without bugs I guess that it would be quite boring (or better... an Orwellian one)
Secondly, thanks for the fix, I just tried the function and it seems working (It is still computing so... I'll tell you later)!

Stepping out from the package discussion and touching briefly the methods one: 
I understand that the X2* has never implemented for multidim models, indeed for what I know the only references that I found (e.g. Stone 2000) refer to unidimensional model. 
Unfortunately, these statistics are quite important when you apply them in other fields than educational or psychometric ones, because more often than not you have to deal with few items per dimension.
Since (a) other fit indices (such as G2) are almost useless (inflated Type I errors) in these situations, and (b), at least for what I understood, global/test fit indices are not that reliable (at least as sole statistics to evaluate the model gof) in the IRT framework,... X2* indices seem to be the best option to have reliable estimates. 

Do you know any other reliable alternative to apply in these kind of situations (n>1000; n(items)~3 per dimension; dimensions>1)?

Of course, given that is a broad discussion and definitely not an emergency one, take your time to reply. 

Thanks again and cheers.

Giuseppe