chi-square difference test, satorra-bentler 2010, negative scaling case

[Njål Foldnes]

Hi,  I am trying to formally test whether a two-factor model is better than a one-factor model. These models may be considered (covariance) nested. 

The two factor model has a not-too-bad fit:

> mod2

lavaan (0.5-22) converged normally after  39 iterations

  Number of observations                          1171

  Estimator                                       DWLS      Robust

  Minimum Function Test Statistic              200.315     245.176

  Degrees of freedom                                53          53

  P-value (Chi-square)                           0.000       0.000

  Scaling correction factor                                  0.846

  Shift parameter                                            8.483

    for simple second-order correction (WLSMV)

While the one-factor model has poor fit:

Estimator                                       DWLS      Robust

  Minimum Function Test Statistic              681.205     838.320

  Degrees of freedom                                54          54

  P-value (Chi-square)                           0.000       0.000

  Scaling correction factor                                  0.821

  Shift parameter                                            8.455

    for simple second-order correction (WLSMV)

So there is no doubt that the two-factor model is substantially better. 

However, we have a case where the 2001 satorra-bentler procedure leads to negative scaling factor. Hence, I requested the 2010 procedure, which is supposed

to fix this:

> lavTestLRT(mod1, mod2, method = "satorra.bentler.2010")

Scaled Chi Square Difference Test (method = "satorra.bentler.2010")

     Df AIC BIC  Chisq Chisq diff Df diff Pr(>Chisq)

mod2 53         200.32                              

mod1 54         681.21    -59.551       1          1

I was expecting a p-value of 0 here, since the nested model (mod 1) is much worse than the parent model (model2).

Indeed, in Mplus the DIFFTEST results in 

MODEL FIT INFORMATION

Number of Free Parameters                       48

Chi-Square Test of Model Fit

          Value                            838.319*

          Degrees of Freedom                    54

          P-Value                           0.0000

Chi-Square Test for Difference Testing

          Value                            199.683

          Degrees of Freedom                     1

          P-Value                           0.0000

Can anyone figure out what I have done wrong here? Or is there som issue with lavTestLRT, maybe?

[Terrence Jorgensen]

Can anyone figure out what I have done wrong here? Or is there som issue with lavTestLRT, maybe?

I think the 2010 procedure involves fitting the model again with constraints imposed. Just out of curiosity, do you get the same result if you compare your 2-factor model to a more restricted 2-factor model, in which the factor correlation is constrained to 1?  That constrained model is statistically equivalent to a 1-factor model, so you should get the same chi-squared, but I wonder whether the 2010 method will produce a different result due to how it calculates the scale/shift parameters for delta-chi-squared.

If you get same result, it would probably help if you posted (or sent Yves) your syntax with enough data to reproduce the problem.

Terrence D. Jorgensen

Postdoctoral Researcher, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

UvA web page: http://www.uva.nl/profile/t.d.jorgensen

[Yves Rosseel]

Well, the two models are nested at the covariance level, but not nested
at the parameter level. As the SatorraBentler2010 method requires the
fitting of one model with parameter values from the other, it does not
surprise me that it goes wrong somewhere (although a warning of some
sorts would have been welcome).

In general, lavTestLRT was not designed to handle models that are not
nested at the parameter level. The 'safer' way to use lavTestLRT is to
convert the models in such a way so that they become nested at the
parameter level. For example:

mod2<- 'f1 =~ I1 + I2 + I4 + I6+ I9 + I11 + I12
f2 =~ I3 + I5 + I7 + I8 + I14'
fit2 = sem(mod2, data=mysample, ordered=colnames(mysample), std.lv = TRUE)

mod3 <- 'f1 =~ I1 + I2 + I4 + I6 + I9 + I11 + I12
f2 =~ NA*I3 + I5 + I7 + I8 + I14
f1 ~~ 1*f2'
fit3 <- sem(mod3, data=mysample, ordered=colnames(mysample), std.lv = TRUE)

lavTestLRT(fit2, fit3, method="satorra.bentler.2010")

Scaled Chi Square Difference Test (method = "satorra.bentler.2010")

Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)

fit2 53 200.32
fit3 54 681.21 115.66 1 < 2.2e-16 ***

which looks reasonable.

Mplus gives a result, but is it correct? It looks like Mplus is using
the Satorra2000 method (which assumes the models are nested at the
parameter level!!), as I can mimic this by using:

lavaan:::lav_test_diff_Satorra2000(fit2, fit1)
$T.delta
[1] 199.6835

$scaling.factor
[1] 2.408262

$df.delta
[1] 1


But the Satorra2000 method should be identical to the SatorraBentler2010
method, and this is indeed ok for lavaan:

lavTestLRT(fit2, fit3, method="satorra2000")
Scaled Chi Square Difference Test (method = "satorra.2000")

Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)

fit2 53 200.32
fit3 54 681.21 115.56 1 < 2.2e-16 ***


Yves.

[Njål Foldnes]

Thank you Terrence, for your clarification here. Indeed, things work better with the reparameterisation

mod1 = "f =~  I1 +I2+ I4 +I6 +I9+ I11+ I12+ I3+ I5+ I7+ I8+ I14"

mod2 = "f1 =~ I1 +I2 +I4+ I6+ I9+ I11 +I12; f2 =~ I3 +I5+ I7+ I8+ I14"

mod1.alternative = "f1 =~ I1 +I2 +I4+ I6+ I9+ I11 +I12; f2 =~ I3 +I5+ I7+ I8+ I14; f1~~1*f2"

f1 = sem(mod1,  data=mysample, ordered=colnames(mysample))

f2 = sem(mod2,  data=mysample, ordered=colnames(mysample), std.lv=T)

f1.alternative = sem(mod1.alternative, data=mysample, ordered=colnames(mysample), std.lv=T)

lavTestLRT(f1.alternative, f2)

lavTestLRT(f1.alternative, f2, method="satorra.bentler.2010")

gives

> lavTestLRT(f1.alternative, f2, method="satorra.bentler.2010")

Scaled Chi Square Difference Test (method = "satorra.bentler.2010")

               Df AIC BIC  Chisq Chisq diff Df diff Pr(>Chisq)    

f2             53         200.32                                  

f1.alternative 54         681.21     115.66       1  < 2.2e-16 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

> lavTestLRT(f1.alternative, f2)

Scaled Chi Square Difference Test (method = "satorra.2000")

               Df AIC BIC  Chisq Chisq diff Df diff Pr(>Chisq)    

f2             53         200.32                                  

f1.alternative 54         681.21     115.56       1  < 2.2e-16 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

So things work out fine, after having reparameterised the constrained model as a 2-factor model.

the important thing is to remember to set std.lv=T and "f1~~1*f2"

PS:  I tried to set the correlation between f1 and f2 to one by using the constraints option, but that did not work as expected:

> f=sem("f1 =~ I1 +I2 +I4+ I6+ I9+ I11 +I12; f2 =~ I3 +I5+ I7+ I8+ I14", data=mysample, ordered=colnames(mysample), constraints="f1~~1*f2")

There were 13 warnings (use warnings() to see them)

> parameterestimates(f)[63,]

   lhs op rhs   est   se     z pvalue ci.lower ci.upper

63  f1 ~~  f2 0.102 0.02 5.043      0    0.062    0.141