Measurement invariance results-minimal fit index changes but significant chi-square difference
Blair Burnette
Hi,
I am still working on a measurement invariance in lavaan in R. I am comparing a measure of sociocultural appearance pressures across White and Black women. Because this is a 5-point scale, I am treating the items as ordinal. So, I'm presenting/analyzing the robust estimations.
I have large, but uneven samples. There are 553 White and 273 Black women. I'm evaluating a combo of change in x2 and fit indices. The fit indices are generally adequate for the initial cfa, though the RMSEA is a bit high: RMSEA=.085, 95% CI [.081, .090], TLI=.983; SRMR=.054, χ2(199)=1346.800, p<.001.
The configural model is somewhat similar: RMSEA=.088, 95% CI [.083, .092]; TLI=.982; SRMR=.061, χ2(398)=1612.698, p<.001.
The metric model had the following changes: ΔRMSEA=-.009 (i.e., .79), ΔTLI=.001 (i.e. .983), and ΔSRMR=.002 (i.e., 063). Only the SRMR suggested a decrement in model fit, but the change (.002) was below the recommended cut-off of .025 by Chen (2007). The adjusted chi-square difference test was significant, Δχ2(22)=38.62308, p=.016 (I used the scaling correction factors, etc. to calculate the change in the robust x2 estimates).
This is a tricky one for me. I believe I have seen on this forum that the change in robust chi-square estimates IS notable and something to explore. However, my co-authors are advising that I ignore the change in chi-square because of the large sample size and the minimal changes of the other fit indices. Given research and theory in the area, I would have hypothesized factor loadings would differ by group, so I feel uncomfortable not probing the significant change in chi-square.
I start to really question my data when I run the scalar model. In the scalar model, I constrained loadings and thresholds to equality. With this model, I was surprised to see that chi-square goes down, from 1620.556 (420 df) in the metric model to 1462.593 (481 df) in the scalar model. When I run the x2 difference test, I get a p value of 1 because the chi-square went down instead of up. Again, the fit indices changes are minimal, compared to the metric model: ΔRMSEA=-.002 (.077), ΔTLI=.003 (.986), and ΔSRMR=-.001 (.062). I question this also because latent factor means are clearly non-equivalent across groups. Would that not affect scalar invariance?
So, I guess my questions are: do you all agree that I should ignore the significant change in x2 from the configural to metric model? And is it OK for the chi-square to decrease from the metric model to the scalar model?
Thank you!
Terrence Jorgensen
I question this also because latent factor means are clearly non-equivalent across groups. Would that not affect scalar invariance?
So, I guess my questions are: do you all agree that I should ignore the significant change in x2 from the configural to metric model? And is it OK for the chi-square to decrease from the metric model to the scalar model?