Lavaan and Mplus Chi square difference

[Tomás Arriaza]

I am running a 3 factor solution with categorical and pairwise complete observations using the lavaan package and Mplus. I get equal results in the matrix correlation, factor loadings, factor correlation, residual matrix, and so on, in both programs, except for Chi squared and modification indices.

I was wondering if there is a different default calculation of Chi Square in lavaan (X^2 =2337.638) in comparison to Mplus (X^2 = 2341.213). The differences in modification indices is greater and it could be because of this little difference in Chi square.

I checked documentation of fitMeasures function, but it does not specify which other alternatives there are for standard.test and scaled.test arguments besides the "default" option.

My model in R.

model1 <- cfa(syntax,
             data = db2,
             ordered = c("ics1_1","ics1_2","ics1_3","ics1_4","ics1_5",
                         "ics2_1","ics2_2","ics2_3","ics2_4_r","ics2_5",
                         "ics2_6_r","ics2_7_r","ics2_8","ics2_9","ics2_10_r",
                         "ics3_1","ics3_2","ics3_3","ics3_4_r","ics3_5",
                         "ics3_6_r","ics3_7_r","ics3_8","ics3_9","ics3_10_r"),
             estimator = "WLSMV", missing = "pairwise")

measures_model1 <- fitMeasures(model1,
                              fit.measures = c("chisq.scaled", "df.scaled",
                                               "rmsea.scaled","rmsea.ci.lower.scaled",
                                               "rmsea.ci.upper.scaled","rmsea.pvalue.scaled",
                                               "cfi.scaled","tli.scaled"))

My model in Mplus (for extension I omitted some arguments like NAMES ARE, USE VARIABLES, MISSING ARE, etc, but fell free to ask if help to answer):

CATEGORICAL ARE ics1_1 ics1_2 ics1_3 ics1_4 ics1_5 ics2_1 ics2_2 ics2_3 ics2_4_r ics2_5 ics2_6_r ics2_7_r ics2_8 ics2_9 ics2_10_r ics3_1 ics3_2 ics3_3 ics3_4_r ics3_5 ics3_6_r ics3_7_r ics3_8 ics3_9 ics3_10_r;

ANALYSIS: STARTS=20;

MODEL:

ICS1 BY ics1_1 ics1_2 ics1_3 ics1_4 ics1_5;

ICS2 BY ics2_1 ics2_2 ics2_3 ics2_4_r ics2_5 ics2_6_r ics2_7_r ics2_8 ics2_9 ics2_10_r;

ICS3 BY ics3_1 ics3_2 ics3_3 ics3_4_r ics3_5 ics3_6_r ics3_7_r ics3_8 ics3_9 ics3_10_r;

OUTPUT: TECH1 residual cinterval sampstat modindices (ALL) standardized;

[Tomás Arriaza]

The difference in Chi-square is due to the lack of the mimic argument. With mimic = "Mplus" a different scaling factor correction or Chi square scaled is calculated (I don't know which is first). However, Modification indices still are different between both programs.

[Terrence Jorgensen]

Modification indices still are different between both programs

A modification index is a score test, which is asymptotically equivalent to a LRT that compares 2 fitted models (one representing the null hypothesis H0 that the restricted model is correct, and another representing the alternative hypothesis HA that the parameter in question should be freely estimated or fixed to a different value). 

Buse, A. (1982). The likelihood ratio, Wald, and Lagrange multiplier tests. The American Statistician, 36(3), 153-157. http://www.jstor.org/stable/2683166 

 

The advantage of the score test is that you only have to fit the H0 model, but then you cannot calculate the scale and shift parameters used to correct the chi-squared difference test, as described in the Mplus Web Note 10. 

However, Mplus applies the scale and shift parameters from the H0 model to the modification indices, on the assumption that they would not be very different from the ones calculated using their DIFFTEST procedure if you had fitted the HA model as well.  That assumption is highly dubious given how uncommon it is for the H0 model to fit the data well (in the sense that the chi-squared test is not significant).

Therefore, lavaan does not make that assumption, so modindices() only returns the unscaled score test stats.  Given how uncommon it is for people to rely on the score-test p values, it does not seem necessary to scale them anyway.  The common practice is to only consult them when the model fits poorly (which is when the H0 model's scale/shift parameters are probably not the same as you would obtain from a chi-squared difference testing procedure), and that the modification indices are rank-ordered to look for the largest sources of local misspecification.  

Terrence D. Jorgensen    (he, him, his)
Assistant Professor, Methods and Statistics
Research Institute for Child Development and Education, the University of Amsterdam
http://www.uva.nl/profile/t.d.jorgensen

[Tomás Arriaza]

Thank you very much for your answer. Just to check, does that mean that is not possible with lavaan to get same indices of modification as Mplus?

[Terrence Jorgensen]

does that mean that is not possible with lavaan to get same indices of modification as Mplus?

That means lavaan will not provide them for you.  But I think it should be pretty easy to apply the scale/shift correction to each modification index yourself.  

chisq.scaled = (chisq × scaling.factor) + shift.parameter

The summary() output shows the scale/shift parameters below the robust test.  

example(lavTables)

fit

You can access those values more directly (as R objects) using the lavTest() function

(TEST <- lavTest(fit, test = "scaled.shifted"))

Then apply those values to the $mi column of modindices() output.

MI <- modindices(fit)

MI$mi.scaled <- MI$mi * TEST$scaling.factor + TEST$shift.parameter

MI

Notice that the rank-order of indices is identical before/after transformation.

cor(MI$mi, MI$mi.scaled, method = "spearman")

So the same "most important indices" would be flagged.  But using the common arbitrary "only indices > 10" criterion, only the scaled/shifted ones have 2 flagged parameters (which is consistent with the naive test not being significant, p = .156, whereas the scaled/shifted test has p = .031)

[Tomás Arriaza]

Doesn't work that transformation, but if rank order is the same I will work with the unscaled MI. Thank you very much.

I have other questions now. When I use the "mimic = Mplus" argument "simple second-order correction (WLSMV)" is estimate, but when I don't "simple second-order correction" is. What is more "correct" to use? What is the difference between both?