Discriminant validity – Chi-square difference tests – goodness of fit
516 views
Skip to first unread message
Paul Schreuder
Jun 22, 2022, 4:52:42 AM6/22/22
to lavaan
Hi! To test the discriminant validity, I want to compare the measuring instrument we developed for consumer brand relationships (cbr) with existing measuring instruments for the constructs brand love (bl), brand attachment (ba) and brand hate (bh).
All alpha’s, AVE and composite reliabilities of constructs have good values in Lavaan (see Table 1).
When applying Fornell Larcker criterion for discriminant reliability it shows that all squared correlations (r2) between constructs are less than the AVE of the construct (see Table 2).
When performing Chi-square difference tests for discriminant validity results show that all the two-factor models (allowing both constructs to correlate) have a significant better fit than their respective single-factor model (force the two factors to be perfectly correlated) (see Table 3).
QUESTIONS:
- I notice that Chi-square difference values do not reflect the actual difference in Chi-square values? Is this because I use the MLM-estimator and the lavaan NOTE: The “Chisq” column contains standard test statistics, not the robust test that should be reported per model. A robust difference test is a function of two standard (not robust) statistics. Should I do something about this?
- Pr(>Chisq) values are the same for each pair of constructs; I’m not sure what the Pr(>Chisq) means. Can this equality of values be correct?
Not all existing measuring instruments have a good fit to the data (see Table 4). TLI, CFI and SRMR are generally good, but RMSEA sometimes indicates mediocre or poor fit. Chi square values are mostly significant, but I’m assuming this is because of sample size (N=502).
QUESTIONS:
- Can I still apply Chi-square difference tests as in Table 3, despite bad model fit? Would a significant improvement in Chi2 from the one-factor model to the two-factor model still indicate that the concept of cbr is distinct from the other (bl, ba, or bh) construct?
Or is it necessary to adjust the existing model until it has a good fit with the data (which doesn't seem appropriate to me as it is an existing model)?
- Sidenote: Should instruments have the same scales for a Chi-square difference test or can for example instruments with 5-pt scales and 11-pt scales be combined?
As the BL, BA and BH models are published models/instruments in respected journals and because of my relative inexperience with the use of Lavaan I’m questioning if I executed analyses in a correct manner, if my syntax is in order? See Table 5 and Table 6.
Help is much appreciated. I’ve put in quit some effort to find literature on the relationship between Chi-square difference tests and overall goodness of fit indices, but can’t find an answer to it.
All alpha’s, AVE and composite reliabilities of constructs have good values in Lavaan (see Table 1).
When applying Fornell Larcker criterion for discriminant reliability it shows that all squared correlations (r2) between constructs are less than the AVE of the construct (see Table 2).
When performing Chi-square difference tests for discriminant validity results show that all the two-factor models (allowing both constructs to correlate) have a significant better fit than their respective single-factor model (force the two factors to be perfectly correlated) (see Table 3).
QUESTIONS:
- I notice that Chi-square difference values do not reflect the actual difference in Chi-square values? Is this because I use the MLM-estimator and the lavaan NOTE: The “Chisq” column contains standard test statistics, not the robust test that should be reported per model. A robust difference test is a function of two standard (not robust) statistics. Should I do something about this?
- Pr(>Chisq) values are the same for each pair of constructs; I’m not sure what the Pr(>Chisq) means. Can this equality of values be correct?
Not all existing measuring instruments have a good fit to the data (see Table 4). TLI, CFI and SRMR are generally good, but RMSEA sometimes indicates mediocre or poor fit. Chi square values are mostly significant, but I’m assuming this is because of sample size (N=502).
QUESTIONS:
- Can I still apply Chi-square difference tests as in Table 3, despite bad model fit? Would a significant improvement in Chi2 from the one-factor model to the two-factor model still indicate that the concept of cbr is distinct from the other (bl, ba, or bh) construct?
Or is it necessary to adjust the existing model until it has a good fit with the data (which doesn't seem appropriate to me as it is an existing model)?
- Sidenote: Should instruments have the same scales for a Chi-square difference test or can for example instruments with 5-pt scales and 11-pt scales be combined?
As the BL, BA and BH models are published models/instruments in respected journals and because of my relative inexperience with the use of Lavaan I’m questioning if I executed analyses in a correct manner, if my syntax is in order? See Table 5 and Table 6.
Help is much appreciated. I’ve put in quit some effort to find literature on the relationship between Chi-square difference tests and overall goodness of fit indices, but can’t find an answer to it.
Terrence Jorgensen
Jun 23, 2022, 10:40:17 AM6/23/22
to lavaan
- I notice that Chi-square difference values do not reflect the actual difference in Chi-square values? Is this because I use the MLM-estimator and the lavaan NOTE: The “Chisq” column contains standard test statistics, not the robust test that should be reported per model. A robust difference test is a function of two standard (not robust) statistics. Should I do something about this?
No, lavaan is explaining what you noticed, so you don't incorrectly think it is a mistake.
- Pr(>Chisq) values are the same for each pair of constructs; I’m not sure what the Pr(>Chisq) means. Can this equality of values be correct?
It is the p value for the test statistic. If they are equal (in the number of decimals report), that is probably a coincidence.
Terrence D. Jorgensen
Assistant Professor, Methods and Statistics
Research Institute for Child Development and Education, the University of Amsterdam
Paul Schreuder
Jun 24, 2022, 4:04:20 AM6/24/22
to lavaan
Dear Terrence, thank you for your answer. So does this mean I should report the Chi-square difference value as produced by the Scaled Chi-Squared Difference Test and given in the table (since I used MLM-method for estimating)? Or should I calculate the difference between the Chisq values (which means to perform the standard chi-square difference test including a different Pr(>Chisq) value)?
Op donderdag 23 juni 2022 om 16:40:17 UTC+2 schreef Terrence Jorgensen:
Jošt Bartol
Jun 24, 2022, 9:54:19 AM6/24/22
to lav...@googlegroups.com
Deal Paul Schreuder,
the differences in the chi-square values do not match because of scaling correction. Scaling correction is used when you use MLM, MLMV or MLR etc estimators. For more information how the scaled chi-square difference test is conducted, see the attached article and excel file. Note, that lavaan adjusts the chi-square difference for the scaling automatically, while e.g. Lisrel does not (and in this case you would need the excel file). Hope this clarifies your question.
Regards,
Jošt
V V pet., 24. jun. 2022 ob 10:04 je oseba Paul Schreuder <paulsc...@hotmail.com> napisala:
--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/3e2b2e32-ac0f-46a6-97be-a38fe5aab865n%40googlegroups.com.
Paul Schreuder
Jun 27, 2022, 10:41:41 AM6/27/22
to lavaan
Dear Jošt, it does. Thank you!
As I use the Chisquare difference test for a discriminant validity test I have another question. For one of the construct pairs both the one-factor model (forcing the two constructs to be perfectly correlated) and the two factor model (allowing both constructs to correlate) indicate not a good fit of these models to the sample data. The (scaled) Chi-square difference test for discriminant validity result show that the two-factor model has a significant better fit than the single-factor model. Would this still indicate that the one construct is distinct from the other construct?
Happy to hear from you again. Or anyone else with helpful insights :-)
Thanks, Paul
Op vrijdag 24 juni 2022 om 15:54:19 UTC+2 schreef Jošt Bartol:
Jošt Bartol
Jun 28, 2022, 2:23:12 AM6/28/22
to lav...@googlegroups.com
Dear Paul Schreuder,
Regarding the discriminant validity: I cannot really give you a definitive answer, but I have seen this used in such a way even with suboptimal fit (e.g.,
https://dl.acm.org/doi/abs/10.1145/3449218). In any case, I would explore why the fit is bad (check residual covariance or modification indices) and decide based on that if such an analysis is acceptable.
Regards,
Jošt
V V pon., 27. jun. 2022 ob 16:41 je oseba Paul Schreuder <paulsc...@hotmail.com> napisala:
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/1e718984-77d3-42c1-8f09-64c184115a6an%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages