A few questions about DIF, negative LRT statistics

[Erin717]

Dear Phil, 

Hope everything is going well!

I have a few questions about multiple group DIF analysis. I got some odd results that I could not figure out why.

Question 1: the model has mixed item types. Is it still okay to use the DIF() function? 

Here is what I did: First I estimated a fully constrained model,

spec.invar.full <- c(names(tam.dat[,-c(1:3,24)]), "free_var", "free_means")

mg.full <- multipleGroup(dat[,-c(1:3,24)], model=1, itemtype = c(rep("2PL",14), rep("gpcm",6)),  SE=TRUE,group=group, invariance=spec.invar.full,survey.weights=dat$wt,

                         technical=list(NCYCLES=2000))

Then I used the DIF() function 'drop' scheme to identify potential anchor items

dif.full <- DIF(mg.full, which.par = c("a1","d"), 

                scheme="drop",p.adjust="BH",technical=list(NCYCLES=2000))

However, because the item type is mixed, I tried the following too, 

dif.full.dich <- DIF(mg.full, which.par = c("a1","d"), items2test = 1:14,

                scheme="drop",p.adjust="BH",technical=list(NCYCLES=2000))

dif.full.poly <- DIF(mg.full, which.par = c("a1","d1","d2","d3"), items2test = 15:20,

                     scheme="drop",p.adjust="BH")

Do they look sensible?

Question 2: the LRT has negative statistics for all the dif models. 

For example: the result from dif.full.dich resulted large X2 statistics. Did I do somethign wrong here or does it mean the chi square test is wrong? It seems really odd to me. 

Question 3: Does it have anything to do with convergence issues? Because I got warnings that goes like "EM cycles terminated after 2000 iterations" and "Log-likelihood was decreasing near the ML solution. EM method may be unstable".

Question 4: I also tried to specify a configural model in which everything was freely estimated. Then I ran an anova() between the fully constrained model and configural model, it was statistically not significant. Does it mean there is no non-invariance? However, it is counter-intuitive because according to RMSD_DIF I did get quite some high RMSD values (e.g., > .30). Is it because of high df?

mg.config <- multipleGroup(dat[,-c(1:3,24)], model=1, itemtype = c(rep("2PL",14), rep("gpcm",6)), SE=TRUE,group=group, survey.weights=dat$wt, invariance = c("free_var","free_means")) ###may not be identified, just to have a look

anova(mg.config, mg.full)

      AIC   AICc    SABIC      HQ     BIC  logLik    X2  df   p

1 160.461 44.389  -51.887   7.360  87.259 -24.230   NaN NaN NaN

2 265.161 41.126 -151.949 -35.572 121.373 -22.581 3.299  54   1

The change of AIC looks substaintial, but small X2. Why would this happen?

Applogise in advance if these questions are naive. Thank you for all of your help!

Best,

Erin

[Erin717]

Sorry, looks like the screen shot to Question 2 was not attached successfully. I have attached it here.

Thank you!

[Phil Chalmers]

Hi Erin,

I can't quite see what's going on in the code without some data, so unfortunately I can only guess based on your responses (feel free to provide a reproducible instance of your problem). 

1. Input looks fine to me in terms of setting up an anchor-constrained model with user-defined weights. But I don't think the weights are being used in DIF(), which could explain the weird results (I'm skeptical that likelihood ratio tests should be used alongside survey weights anyway). 

2-3. I'd need to reproduce this first hand to give a more definitive answer, but it sounds as though it could be an estimation problem where the models do not converge to a reasonable MLE location. That can cause all kinds of weird problems.

4. That's correct, your configural model is not identified when you free the latent mean/variance. So comparing to the constrained identified model doesn't make much sense.

HTH, and sorry for the late reply.

Phil

--
You received this message because you are subscribed to the Google Groups "mirt-package" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mirt-package...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mirt-package/750089e4-5f53-4e70-8489-71a0914357d0n%40googlegroups.com.