How to interpret change of scaled.chi-square values between models?

[岡林秀樹]

Hi, All !

I have a question about scaled.chi-square.

When I conducted CFA with WLSMV, configural and factorial invariance models, in turn. The values of chi-square and scaled chi-square were follows:

                                                     chi-square      df        scaled.chi-square

configural invariance  model    2216.165      1232     2896.27

factorial  invariance  model      2745.157      1295     2658.23

  

Usually, the more constraint model such as factorial invariance shows lower chi-square than simpler model such as configural invariance. 

The values of normal chi-square is in line with the rule.

But, the values of scaled.chi-square in factorial invariance model is larger than in configural constraint model.  I feel it very strange. 

Would you please tell me how this happen and how we interpret this?

Sincerely,

Hideki

[Terrence Jorgensen]

Would you please tell me how this happen and how we interpret this?

The more constrained model has a higher standard chi-squared statistic, but the scaled one is obtained by multiplying a scaling variance (and in the case of *MV adjustment, also adding a shift parameter).  In this case, it looks like the scaling factor is substantially lower than 1 for the more constrained model (more so than for the less constrained model).  

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

http://www.uva.nl/profile/t.d.jorgensen

[岡林秀樹]

Dear Terrence,

 

Thank you very much for your response.

 

I reconstructed the table adding scaling correction factors and shift parameters. Then, I calculated (standard chi-square/scaling correction factor) + shift parameter in the right edge column. These numbers are about the same as the scaled.chi-square values.

 

Then, I conducted anova between these two models.

 

 The scaled chi-squared difference test, using the difference value of standard chi-square between the two models, showed that the factorial invariance model was significantly worse than the configural invariance model. Is my interpretation right?

 

We should use standard chi-square values for model comparison instead of using scaled chi-square values.

Is my understanding right?

 

Further, I calculated scaled cfi, tli, and rmsea in the two models.

 

 

These values showed that the factorial invariance model was better than the configural invariance model.

Although these findings contradicted the previous ones of the chi-square difference test, the values such as cfi, tli, and rmsea have similar meaning to effect size and they do not always produce similar suggestions to the statistical test. In addition to the two findings, we should judge which model is better comprehensively with theoretical consideration. Is my understanding right?

 

Sincerely,

Hideki 

2021年4月29日(木) 10:13 Terrence Jorgensen <tjorge...@gmail.com>:

--
You received this message because you are subscribed to a topic in the Google Groups "lavaan" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lavaan/nxdMucHbgLs/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/80a6f4bc-7153-4397-9890-afa1e574a967n%40googlegroups.com.

[Terrence Jorgensen]

factorial invariance model was significantly worse than the configural invariance model. Is my interpretation right? 

Yes.

We should use standard chi-square values for model comparison instead of using scaled chi-square values.

Is my understanding right?

Yes, on the contingency that you do a robust correction by scaling the difference between standard chi-squared values.

 

These values showed that the factorial invariance model was better than the configural invariance model.

The chi-squared test stat shows you the factorial invariance fits worse, to a statistically significant degree, than the configural model.  The RMSEA shows that decrement in fit is less than is expected given the increase in df.  The CFI/TLI show that factorial invariance improves on the independence model relatively more than the configural model (where "relative" is in terms of the change in df).  

Although these findings contradicted the previous ones of the chi-square difference test

It is not a contradiction.  If these fit measures all quantified the same thing, there would not be multiple fit measures.  They are quantifying different aspects of (mis)fit, and they are all vague because they try to boil down everything that is wrong with your model into a single number.  Try instead looking at local indices of misfit, by running lavResiduals(), which gives you an idea how poorly your model(s) reproduce each mean, variance, and correlation in your data.

we should judge which model is better comprehensively with theoretical consideration. Is my understanding right?

That plays an important role, but the data can provide evidence against your theory.  Your theory should guide your choice of what evidence you look for in the data to challenge your preconceptions (null hypotheses).

[岡林秀樹]

Dear Terrence,

 

Thank you very much for your response.

 

According to your suggestion, “Try instead looking at local indices of misfit by running lavResidual(),”

I run lavResiduals(). 

I have four groups, each of which has a standardized residual matrix for covariance (cov.z).

But, it seems to be difficult for me to deal with all of them.

 

Is lavResiduals() similar to modificationIndices()?

It seems to me that these functions give us only the misfit information in each group.

I think that both lavResiduals() and modificationIndices() cannot show which equal constraint worsen the fit.

Is my understanding right?

 

I need the information of misfit between groups.

Would you please tell me where I can get such information?

 

Sincerely yours,

Hideki

2021年5月14日金曜日 12:05:33 UTC+9 Terrence Jorgensen: