Discordant item parameter estimates due to sorting or machine

[Cecilia Fiorini]

Hi,

I'm experiencing an unexpected behavior with mirt where changing only the data ordering produces different item parameter estimates, even though the code and dataset are identical. Additionally, running the same code with the same ordering on different computers also produces different estimates.

Context:
> Using mirt for 3PL IRT calibration with educational assessment data
> Model specification includes priors for discrimination and guessing parameters
> The same ordering always produces same results ion the same computer/R environment
> However, different orderings produce different item parameter estimates (a, b, c), which consequently lead to different theta estimates
> Also, the same code and data ordering produce different estimates on different computers
>Both models converge successfully

Code:
dados_calibracao <- dados_brutos %>%
    arrange(CO_INSCRICAO) %>% # <- Only difference
    filter(TP_PRES == 555)

Questions:
1. Is this behavior expected? Could it be due to how initial values are computed from the data?
2. Are the different solutions statistically equivalent (e.g., different local maxima with similar fit)?
3. Is there a recommended approach to ensure stable and reproducible item parameter estimates regardless of data ordering or computing environment (e.g., manually setting starting values)?

Thank you for any insights!

Best Regards,

Cecilia Fiorini

[Phil Chalmers]

Hi Cecilia,

Can you provide a reproducible instance of this? In general the order of the rows makes no difference to the program. Here's an example.

```

library(mirt)
library(dplyr)

mod1 <- mirt(Science)
mod2 <- mirt(arrange(Science, Comfort))
anova(mod1, mod2)

```

which prints

          AIC    SABIC       HQ      BIC   logLik X2 df   p
mod1 3249.739 3262.512 3274.922 3313.279 -1608.87          
mod2 3249.739 3262.512 3274.922 3313.279 -1608.87  0  0 NaN

Phil

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/mirt-package/c8411693-2585-4182-8959-dbaa4ec7f196n%40googlegroups.com.

[Heliton Tavares]

Hi Cecilia and Phil, here's an example of the differences when the order of the data is changed.

  library(mirt)
  library(dplyr)
  head(Science)
  Science2 <- 1*(Science=="strongly disagree") + 2*(Science=="disagree") + 3*(Science=="agree") + 4*(Science=="strongly agree")
    
  mod1 <- mirt(Science2)
  pars1 <- coef(mod1,simplify=TRUE,IRTpars=T)[[1]]
 
  mod2 <- mirt(Science2[sample(1:nrow(Science2)),]) # here we change the order of data
  pars2 <- coef(mod2,simplify=TRUE,IRTpars=T)[[1]]
 
  pars1-pars2

Running for the first time the differences are:

                        a            b1            b2            b3
Comfort     -2.796940e-11 -7.618617e-11 -4.179368e-11  2.131784e-11
Environment -8.801473e-12 -5.732836e-09 -1.869825e-09  1.632122e-09
Work         1.073031e-11  1.503775e-11  4.024225e-12 -1.257594e-11
Future       1.188316e-10  4.542189e-11  1.771272e-11 -1.975187e-11
Technology  -9.240810e-12 -8.098851e-09 -2.544144e-09  1.994763e-09
Industry    -1.062250e-11 -3.082654e-10 -1.498712e-10  2.777401e-11
Benefit     -2.309286e-11 -4.181944e-11 -1.279810e-11  1.858069e-11

 Running again (another order of data):

                        a            b1            b2            b3
Comfort      1.405385e-09  3.722014e-09  2.089823e-09 -1.091237e-09
Environment  4.361756e-10  2.839556e-07  9.315075e-08 -8.100153e-08
Work        -5.354475e-10 -7.562519e-10 -2.115669e-10  6.260594e-10
Future      -5.816666e-09 -2.251620e-09 -8.859237e-10  9.756997e-10
Technology   4.567079e-10  4.002170e-07  1.258890e-07 -9.844765e-08
Industry     5.179588e-10  1.500456e-08  7.307524e-09 -1.359096e-09
Benefit      1.144096e-09  2.076975e-09  6.151102e-10 -9.372609e-10

__________________________

HELITON RIBEIRO TAVARES

Faculty of Statistics

Federal University of Para

Belem-Para-Brazil

É (+55) 91 3201-8949 / (+55) 91 99144-9229

› hel...@ufpa.br,  heliton...@gmail.com

  www.helitontavares.com

To view this discussion visit https://groups.google.com/d/msgid/mirt-package/CAKYNACGo9WgvG_Kd__HZyzmfR8EXZs0QUrCAVkM6vafLTTZa9Q%40mail.gmail.com.

[Phil Chalmers]

Hi Heliton,

I get this error when trying to run the model as the data doesn't look valid:

> mod1 <- mirt(Science2)
Error: The following items have only one response category and cannot be estimated: Comfort Work Future Benefit

> head(Science2)
  Comfort Work Future Benefit
1       0    0      0       0
2       0    0      0       0
3       0    0      0       0
4       0    0      0       0
5       0    0      0       0

6       0    0      0       0

Could you fix the example so I recan reproduce the issue? Thanks.

 

Phil

[Heliton Tavares]

Hi Phil, in the code below I'm reordering by several columns from Science file.

library(mirt)
library(dplyr)

mod1 <- mirt(Science)

pars1 <- coef(mod1,simplify=TRUE,IRTpars=T)[[1]]

mod2 <- mirt(arrange(Science, Comfort))

pars2 <- coef(mod2,simplify=TRUE,IRTpars=T)[[1]]
pars1-pars2

mod3 <- mirt(arrange(Science, Work))
pars3 <- coef(mod3,simplify=TRUE,IRTpars=T)[[1]]
pars1-pars3

mod4 <- mirt(arrange(Science, Future))
pars4 <- coef(mod4,simplify=TRUE,IRTpars=T)[[1]]
pars1-pars4

These are rounding errors that shouldn't occur.

Heliton

Interestingly, our Science files from Mirt are different.

Error: Input data must be integer or numeric values only

>   head(Science)
         Comfort       Environment           Work         Future
1 strongly agree    strongly agree strongly agree          agree
2          agree    strongly agree          agree          agree
3          agree          disagree       disagree       disagree
4          agree             agree       disagree       disagree
5          agree strongly disagree strongly agree strongly agree
6 strongly agree             agree strongly agree          agree
      Technology       Industry           Benefit
1 strongly agree          agree          disagree
2          agree          agree             agree
3 strongly agree strongly agree             agree
4 strongly agree strongly agree             agree
5       disagree          agree strongly disagree
6          agree strongly agree             agree

__________________________ 

HELITON RIBEIRO TAVARES

Faculty of Statistics

Federal University of Para

Belem-Para-Brazil

É (+55) 91 3201-8949 / (+55) 91 99144-9229

› hel...@ufpa.br,  heliton...@gmail.com

  www.helitontavares.com

[Phil Chalmers]

The data does look unusual, so it's likely you have a conflict here. Try using this in your code to make the specific version of the Science dataset active:

data(Science, package='mirt')

Phil