Measurement Invariance and CFI change_Construct validity

[Torres]

Dear everyone,

I am testing the measurement invariance by group of school. I used my syntax as followings:

configschool<-cfa(model, data = data1, group="School",estimator = "mlm")
weekschool <- cfa(mode, data = data1, group="School",estimator = "mlm", group.equal = c("loadings"))
strongschool<-cfa(model,data = data1,group = "School", estimator = "mlm",group.equal = c("loadings","intercepts"))
strictschool <- cfa(model,data = data1,group = "School", estimator = "mlm",group.equal = c("loadings","intercepts", "residuals"))

measurementInvariance(model, data = data1, group = "School", estimator = "mlm")

Then my output is below:

Measurement invariance models:

Model 1 : fit.configural
Model 2 : fit.loadings
Model 3 : fit.intercepts
Model 4 : fit.means

Scaled Chi Square Difference Test (method = "satorra.bentler.2001")

                Df   AIC   BIC  Chisq Chisq diff Df diff Pr(>Chisq)   
fit.configural 126 14645 15061 321.95                                 
fit.loadings   144 14658 15003 370.53     14.916      18   0.667741   
fit.intercepts 162 14650 14924 398.80     41.853      18   0.001159 **
fit.means      166 14643 14901 400.07      1.306       4   0.860438   
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


Fit measures:

               cfi.scaled rmsea.scaled cfi.scaled.delta rmsea.scaled.delta
fit.configural      0.908        0.039               NA                 NA
fit.loadings        0.930        0.032            0.022              0.007
fit.intercepts      0.895        0.037            0.034              0.005
fit.means           0.899        0.036            0.003              0.001

I made use of CFI change to decide if there are some measurement invariances. In the section of Fitmeasure, row "fit.loadings" the cfi.scaled.delta = .02 which is greater than the cutpoit .01 indicating that there is no measurement invariance between config model and week model. However, I found there is no significant difference between config model and week model in the section of "Scaled Chi Square Difference Test..." (p=.68). 

In addition, I tested the differences between each pair of model and found no significant difference between them
> anova(configschool, weekschool)
Scaled Chi Square Difference Test (method = "satorra.bentler.2001")

              Df   AIC   BIC  Chisq Chisq diff Df diff Pr(>Chisq)
configschool 126 14645 15061 321.95                              
weekschool   144 14658 15003 370.53     14.916      18     0.6677
> anova(configschool, strongschool)
Scaled Chi Square Difference Test (method = "satorra.bentler.2001")

              Df   AIC   BIC  Chisq Chisq diff Df diff Pr(>Chisq)
configschool 126 14645 15061 321.95                              
strongschool 162 14650 14924 398.80     39.085      36      0.333
> anova(configschool, strictschool)
Scaled Chi Square Difference Test (method = "satorra.bentler.2001")

              Df   AIC   BIC  Chisq Chisq diff Df diff Pr(>Chisq)
configschool 126 14645 15061 321.95                              
strictschool 184 14735 14921 527.40     56.597      58     0.5276

So, according to the results from the testings of anova, there is no significant difference across models. But according to the results displayed in the section of "Fit measures" it seems to have significant differences across models.
Those results above seem inconsistent. Because I am new to measurement invariance, I am a bit confused about those results. Could anyone please help me exlain:

1. Are there any differences between those models?
2. Is there difference between results from the testings of anova and of measurementInvariance?

Because I want to test the construct validity, could anyone please tell me if there is measurement invariance then which level of measurement invariance is sufficient enough to conclude the construct validity of my scale (config, week or strong, etc,...)?

Thank you very much in advance!
Kind regards,
Torres.

[kma...@aol.com]

Torres,
This kind of question is better suited to SEMNET because it is not lavaan specific.  However, the question is probably also too broad for any email distribution list.

Note that your final model uses group.equal = c("loadings","intercepts", "residuals") where the last term constrains the unique variances, not the means.

CFI stands for comparative fit index and compares your model to the fit of a baseline model whereas chi-square is based only on your model.  So, they do not quantify the same thing.  Also, the rule of thumb applied to CFI does not constitute a test of statistical significance but rather offers a rough guide to practical significance.  (Notice that your CFI values are not monotonic across models but the chi-square values are.)

I would encourage you to think of both measurement invariance and construct validity as matters of degree rather than categorical properties that are either fully present or fully absent.

Construct validity depends upon your construct theory.  If the construct theory is so weak as to allow different factor structures in different schools, then construct validity is consistent with the absence of any form of measurement invariance.  If the construct theory asserts the same number of dimensions and the same mapping of items onto dimensions across schools, then construct validity requires configural invariance.  Likewise, you can construct stronger and stronger construct theories to entail stronger and stronger forms of invariance.

Rather than thinking of consruct validity as a property of your scale, think about the inferences that you wish to draw using the test scores.  Then think about the validity of those inferences under different levels of measurement invariance.  The level that you need depends upon the kinds of inferences you wish to draw.  Almost any introductory source on measurement invariance will spell out the relationships between measurement invariance and the kinds of hypotheses that the test scores can be used to evaluate in more detail than I can here.

Conversely, establishing an acceptable level of construction validity will require additional lines of evidence beyond just measurement invariance.  A scale can be invariantly invalid across schools by assessing the wrong thing the same way in each school.  So, failure to show the required level of invariance can offer useful disconfirming evidence with respect to construct validity.  However, demonstrating the expected level of invariance does not provide sufficient evidence for construct validity.

Keith
------------------------
Keith A. Markus
John Jay College of Criminal Justice, CUNY
http://jjcweb.jjay.cuny.edu/kmarkus
Frontiers of Test Validity Theory: Measurement, Causation and Meaning.
http://www.routledge.com/books/details/9781841692203/

[Torres]

Thanks Keith. Your explanations make the problems more clearly to me ;)

Vào 07:31:32 UTC+1 Thứ Sáu, ngày 25 tháng 11 năm 2016, kma...@aol.com đã viết: