Satorra-Bentler 2010 error?

[Hynek Cígler]

Hello, I'm using lavTestLRT to compare model. I has been always using Satorra-Bentler 2010 method , but I have strong feeling its function results of the function has been changed in last months. I'm strongly afraid that the test statistic has an inverse value.

An example: I have two nested models, one vs. two dimensional catCFA. Their fit is terrible, but it does not matter in that case:

cfa1: "x2(27) = 133.29, p < .001, TLI = 0.859, RMSEA = 0.134 (95% CI = [0.112; 0.158]), SRMR = 0.084"

cfa2: "x2(25) = 79.13, p < .001, TLI = 0.922, RMSEA = 0.1 (95% CI = [0.075; 0.125]), SRMR = 0.064"

And now all the three methods for LRT:

> lavTestLRT(cfa1, cfa2, method = "satorra.2000")

Scaled Chi Square Difference Test (method = "satorra.2000")

     Df AIC BIC  Chisq Chisq diff Df diff Pr(>Chisq)    

cfa2 25         53.314                                  

cfa1 27         93.555     41.276       2  1.089e-09 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

> lavTestLRT(cfa1, cfa2, method = "satorra.bentler.2001")

Scaled Chi Square Difference Test (method = "satorra.bentler.2001")

     Df AIC BIC  Chisq Chisq diff Df diff Pr(>Chisq)    

cfa2 25         53.314                                  

cfa1 27         93.555     44.415       2  2.267e-10 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

> lavTestLRT(cfa1, cfa2, method = "satorra.bentler.2010")

Scaled Chi Square Difference Test (method = "satorra.bentler.2010")

     Df AIC BIC  Chisq Chisq diff Df diff Pr(>Chisq)

cfa2 25         53.314                              

cfa1 27         93.555    -46.193       2          1

[Terrence Jorgensen]

To track down the possible bug, can you share your R script and enough data to reproduce the problem?

Terrence D. Jorgensen

Postdoctoral Researcher, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

UvA web page: http://www.uva.nl/profile/t.d.jorgensen

[Hynek Cígler]

Dear Terrence,

I am sorry for the delay, I did not receive notice about your response. Data and script is here: https://is.muni.cz/de/hynek.cigler/satorra-bentler-2010.zip . Unfortunately, I used the chi2 difference test for ordinal CFAs, which could be problematic. So I included also interval CFAs to the script and result is still the same. (Models have terrible fit, but it should not bias the results of lavTestLRT function).

Best, Hynek

Dne sobota 18. srpna 2018 16:27:27 UTC+2 Terrence Jorgensen napsal(a):

[Terrence Jorgensen]

I used the chi2 difference test for ordinal CFAs, which could be problematic.

I was under the impression that the Satorra-Bentler corrections were only for continuous nonnormal data.  Actually, the Satorra (2000) correction was also developed for continuous data, but I think the pertinent difference is that it can apply to mean- and variance-adjusted statistics (e.g., MLMV, WLSMV), as opposed to the Satorra-Bentler corrections applying to only mean-adjusted statistics (e.g., MLM, MLR, WLSM).  

[Hynek Cígler]

Yes, so I said it could be problematic and I therefore re-run the analyses using MLR estimator. The chi-square difference using Satorra-Bentler (2010) is still negative (compared to e.g. Satorra, 2000, or Satorra-Bentler, 2001). The problem is not therefore based on the WLSMV estimator, but definitely in the way how the Satorra-Bentler (2010) scaled chi2 difference is computed. 

Dne sobota 25. srpna 2018 14:10:48 UTC+2 Terrence Jorgensen napsal(a):

[Terrence Jorgensen]

Yes, so I said it could be problematic and I therefore re-run the analyses using MLR estimator. The chi-square difference using Satorra-Bentler (2010) is still negative (compared to e.g. Satorra, 2000, or Satorra-Bentler, 2001). The problem is not therefore based on the WLSMV estimator, but definitely in the way how the Satorra-Bentler (2010) scaled chi2 difference is computed. 

Thanks for sending data and syntax.  Actually, it does not return an error when using MLR, and it returns a positive test statistic (5.4147).  In the script you sent, you forgot to add "b" to the model names when checking the test under MLR.

lavTestLRT(cfa1b, cfa2b, method = "satorra.2000") ## OK
lavTestLRT(cfa1b, cfa2b, method = "satorra.bentler.2001") ## OK
lavTestLRT(cfa1, cfa2, method = "satorra.bentler.2010") ## produces strange result (negative x2 statistic)
lavTestLRT(cfa1b, cfa2b, method = "satorra.bentler.2010") ## produces positive x2 statistic

[Hynek Cígler]

Shame on me. Thank you for help and patience! :-)

Dne úterý 28. srpna 2018 12:50:28 UTC+2 Terrence Jorgensen napsal(a):