Comparing nested CFA models with ordered = TRUE

[Chris Evans]

I fear that I'm being stupid and I'm no SEM/LVM expert but I am puzzled by this output:

lavTestLRT(SingleFitDWLS, TwoFitDWLS, method = "satorra.bentler.2001")
##
## Scaled Chi-Squared Difference Test (method = "satorra.bentler.2001")
##
## lavaan NOTE:
##     The "Chisq" column contains standard test statistics, not the
##     robust test that should be reported per model. A robust difference
##     test is a function of two standard (not robust) statistics.
##  
##               Df AIC BIC  Chisq Chisq diff Df diff Pr(>Chisq)    
## TwoFitDWLS    34         29.161                                  
## SingleFitDWLS 35         46.009     11.642       1   0.000645 ***
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
lavTestLRT(SingleFitDWLS, TwoFitDWLS, method = "satorra.bentler.2010")
##
## Scaled Chi-Squared Difference Test (method = "satorra.bentler.2010")
##
## lavaan NOTE:
##     The "Chisq" column contains standard test statistics, not the
##     robust test that should be reported per model. A robust difference
##     test is a function of two standard (not robust) statistics.
##  
##               Df AIC BIC  Chisq Chisq diff Df diff Pr(>Chisq)
## TwoFitDWLS    34         29.161                              
## SingleFitDWLS 35         46.009    -1.7381       1          1

The fit results are from:

singleFac.model  <- 'F1  =~ NEdgy + NTalk + PCope + NHurt + PHelp + NThFe + NProb + NSlee + NUnha + PDone'
SingleFitDWLS <- lavaan::cfa(singleFac.model,
                             ordered = TRUE,
                             std.lv = TRUE,
                             data = tibYPdatRenamed)

and

twoFac.model  <- 'NF  =~ NEdgy + NTalk + NHurt + NThFe + NProb + NSlee + NUnha
                     PF  =~ PCope + PHelp + PDone
                     NF ~~ PF'
TwoFitDWLS <- lavaan::cfa(twoFac.model,
                        ordered = TRUE,
                        std.lv = TRUE,
                        data = tibYPdatRenamed)

The dataset is of responses on a ten item questionnaire with item responses all 0:4 and clearly ordinal, n = 263 and seven of the items are negatively cued and three positively cued hence the a priori factor model.

Both models seem to fit quickly and without complaints.  The fit is slightly better for the two factor model but what's puzzling me is the output from lavTestLRT() with method = satorra.bentler.2010.  I thought that the method from that paper was simply offering a method to ensure that the test values are always positive but here it's come out negative and looks very odd to me. 

A couple of tangential things:

1) I am sure I have recently seen a post here clarifying what tests lavaan uses for model comparisons when ordered = TRUE has been used with cfa() but I'm failing to find that and failing to search it up on the web.  Can someone point me to the details?

2) Am I the only one who finds:
## lavaan NOTE:
##     The "Chisq" column contains standard test statistics, not the
##     robust test that should be reported per model. A robust difference
##     test is a function of two standard (not robust) statistics.
a bit cryptic and fairly scary?!  I wonder if it could be expanded a bit to be clearer to non-experts?

TIA,

Chris

--
Chris Evans (he/him)
Visiting Professor, UDLA, Quito, Ecuador & Honorary Professor, University of Roehampton, London, UK.
CORE site: http://www.coresystemtrust.org.uk/
Other work web site: https://www.psyctc.org/psyctc/
Personal site: https://www.psyctc.org/pelerinage2016/

[Terrence Jorgensen]

what's puzzling me is the output from lavTestLRT() with method = satorra.bentler.2010.  I thought that the method from that paper was simply offering a method to ensure that the test values are always positive

There is probably a mismatch with the default robust test you implicitly requested when treating data as ordinal.  The Satorra Bentler stats are mean-adjustments, but the default for DWLS estimation is a mean- and variance-adjusted statistic.  By default, lavTestLRT() will already select the "satorra.2000" method when it inspects your models. 

1) I am sure I have recently seen a post here clarifying what tests lavaan uses for model comparisons when ordered = TRUE has been used with cfa() but I'm failing to find that and failing to search it up on the web.  Can someone point me to the details?

The Satorra (2000) paper is on the help page References, but the method currently employed is described here:

https://www.statmodel.com/download/WLSMV_new_chi21.pdf 

2) Am I the only one who finds:
## lavaan NOTE:
##     The "Chisq" column contains standard test statistics, not the
##     robust test that should be reported per model. A robust difference
##     test is a function of two standard (not robust) statistics.
a bit cryptic and fairly scary?!  I wonder if it could be expanded a bit to be clearer to non-experts?

You can find numerous posts on this forum from people who were scared because the Chisq diff column did not match the Chisq column.  This message was added to assuage that fear.  It just means what it says: You first calculate the difference between 2 non-robust chi-squared statistics (in the Chisq column), then robustify it (which appears in the Chisq diff column).

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

http://www.uva.nl/profile/t.d.jorgensen