LRT Test

[Rizqy Amelia Zein]

Hi all,

I'm comparing three models of CFA and when I ran a LR test, I got a below warning message:

Warning message:
In lavTestLRT(object = object, ..., model.names = NAMES) : lavaan WARNING:
    Some restricted models fit better than less restricted models;
    either these models are not nested, or the less restricted model
    failed to reach a global optimum. Smallest difference =
    -94.8244397480173

Could anyone please explain what this error means?

Best

Amelia

--

Rizqy Amelia Zein

https://linktr.ee/ameliazein

[Shu Fai Cheung]

May you post the call to lavTestLRT()? Are there any two models in the

comparison, say Model A and Model B, for which Model A is lower than

Model B on df but higher than Model B on model chi-square? That is:

Model A df < Model B df

Model A chi-square > Model B chi-square

There are several possible ways for this to happen. One of them is that

the two models do not have a nested-within relation (one model being

nested within another).

Following is a reproducible example of how to produce that warning:

# Adapted from the example of semTools::net()
library(semTools)
#> Loading required package: lavaan
#> This is lavaan 0.6-15
#> lavaan is FREE software! Please report any bugs.
#> 
#> ###############################################################################
#> This is semTools 0.5-6
#> All users of R (or SEM) are invited to submit functions or ideas for functions.
#> ###############################################################################

m1 <- ' visual  =~ x1 + x2 + x3
        textual =~ x4 + x5 + x6
        speed   =~ x7 + x8 + x9'
m2 <- ' visual  =~ x1 + x2 + x3 + x6
        textual =~ x4 + x5 + x2
        speed   =~ x7 + x8 + x9'
fit1 <- cfa(m1, data = HolzingerSwineford1939)
fit2 <- cfa(m2, data = HolzingerSwineford1939)

lavTestLRT(fit2, fit1)
#> Warning in lavTestLRT(fit2, fit1): lavaan WARNING:
#>     Some restricted models fit better than less restricted models;
#>     either these models are not nested, or the less restricted model
#>     failed to reach a global optimum. Smallest difference =
#>     -84.3542470701474
#> 
#> Chi-Squared Difference Test
#> 
#>      Df    AIC    BIC   Chisq Chisq diff RMSEA Df diff Pr(>Chisq)
#> fit2 23 7603.8 7685.4 169.660                                    
#> fit1 24 7517.5 7595.3  85.305    -84.354     0       1          1

The models m1 and m2, though having a difference of one on df, are not nested.
Model m2 is formed from Model m1 by adding two paths *and* removing one
path. LR test cannot be used for two models not nested.

If the models are too complicated to be determined by inspection nested or not,
the function net() from semTools can be used to test whether one model is nested
within another.

tests <- net(fit1, fit2)
tests
#> 
#>         If cell [R, C] is TRUE, the model in row R is nested within column C.
#> 
#>         If the models also have the same degrees of freedom, they are equivalent.
#> 
#>         NA indicates the model in column C did not converge when fit to the
#>         implied means and covariance matrix from the model in row R.
#> 
#>         The hidden diagonal is TRUE because any model is equivalent to itself.
#>         The upper triangle is hidden because for models with the same degrees
#>         of freedom, cell [C, R] == cell [R, C].  For all models with different
#>         degrees of freedom, the upper diagonal is all FALSE because models with
#>         fewer degrees of freedom (i.e., more parameters) cannot be nested
#>         within models with more degrees of freedom (i.e., fewer parameters).
#>         
#>                fit2  fit1
#> fit2 (df = 23)           
#> fit1 (df = 24) FALSE
summary(tests)

Hope this helps.

Regards,

Shu Fai Cheung (張樹輝)

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAJiC1p-vh4hwMz0_k2eBMbYSmuj%2B5W9v7MAqedMd3c4_JroLzw%40mail.gmail.com.

[Shu Fai Cheung]

Sorry. My previous post may unintentionally imply that using lavTestLRT()

on two models not nested necessarily leads to that warning. This is

not what I intended to imply. The different orders on df and model

chi-square are *possible* for two models not nested, but not

not necessarily.

This is an example using lavTestLRT() on two models not nested, without

that warning:

# Adapted from the example of semTools::net()
library(semTools)
#> Loading required package: lavaan
#> This is lavaan 0.6-15
#> lavaan is FREE software! Please report any bugs.
#> 
#> ###############################################################################
#> This is semTools 0.5-6
#> All users of R (or SEM) are invited to submit functions or ideas for functions.
#> ###############################################################################

library(lavaan)

m1 <- ' visual  =~ x1 + x2 + x3
        textual =~ x4 + x5 + x6
        speed   =~ x7 + x8 + x9'

m2 <- ' visual  =~ x1 + x2 + x3 + x9
        textual =~ x4 + x5 + x6 + x2
        speed   =~ x7 + x8 + x3
        x3 ~~ x9
        x8 ~~ x9'

fit1 <- cfa(m1, data = HolzingerSwineford1939)
fit2 <- cfa(m2, data = HolzingerSwineford1939)

lavTestLRT(fit2, fit1)
#> 

#> Chi-Squared Difference Test
#> 
#>      Df    AIC    BIC  Chisq Chisq diff    RMSEA Df diff Pr(>Chisq)

#> fit2 20 7520.2 7612.9 80.004                                       
#> fit1 24 7517.5 7595.3 85.305     5.3018 0.032881       4     0.2577

tests <- net(fit1, fit2)
tests
#> 
#>         If cell [R, C] is TRUE, the model in row R is nested within column C.
#> 
#>         If the models also have the same degrees of freedom, they are equivalent.
#> 
#>         NA indicates the model in column C did not converge when fit to the
#>         implied means and covariance matrix from the model in row R.
#> 
#>         The hidden diagonal is TRUE because any model is equivalent to itself.
#>         The upper triangle is hidden because for models with the same degrees
#>         of freedom, cell [C, R] == cell [R, C].  For all models with different
#>         degrees of freedom, the upper diagonal is all FALSE because models with
#>         fewer degrees of freedom (i.e., more parameters) cannot be nested
#>         within models with more degrees of freedom (i.e., fewer parameters).
#>         
#>                fit2  fit1

#> fit2 (df = 20)           

#> fit1 (df = 24) FALSE

Model m2 is lower than Model m1 on both df and model chi-square,

but Model m2 and Model m1 are not nested.

Anyway, checking whether two models are nested is still one way to

see what might have led to that warning.

Sorry for the confusion that I might have caused.

Regards,

Shu Fai Cheung (張樹輝)

[Rizqy Amelia Zein]

Shu Fai,

Thanks so much for the reply!

Ah, yes that indeed the case. I was wondering if x2 and df may be the reason, now I got to understand, yes, this is the case.

So I'm comparing three different models with different structures. First, I estimated a bifactor model:

bifactor <- '
mach =~ mach1 + mach2 + mach3 + mach4 + mach5 + mach6 + mach7
narc =~ narc1 + narc2 + narc3 + narc4 + narc5 + narc6 + narc7
psyc =~ psyc1 + psyc2 + psyc3 + psyc4 + psyc5 + psyc6 + psyc7
sad =~ sad1 + sad2 + sad3 + sad4 + sad5 + sad6 + sad7
d =~ mach1 + mach2 + mach3 + mach4 + mach5 + mach6 + mach7 + narc1 + narc2 + narc3 + narc4 + narc5 + narc6 + narc7 + psyc1 + psyc2 + psyc3 + psyc4 + psyc5 + psyc6 + psyc7 + sad1 + sad2 + sad3 + sad4 + sad5 + sad6 + sad7

d ~~ 0*mach
d ~~ 0*narc
d ~~ 0*psyc
d ~~ 0*sad
mach~~0*narc
mach~~0*psyc
mach~~0*sad
narc~~0*psyc
narc~~0*sad
psyc~~0*sad
'

...then a model assuming that the latents are correlated.

correlated_latent <- '
mach =~ mach1 + mach2 + mach3 + mach4 + mach5 + mach6 + mach7
narc =~ narc1 + narc2 + narc3 + narc4 + narc5 + narc6 + narc7
psyc =~ psyc1 + psyc2 + psyc3 + psyc4 + psyc5 + psyc6 + psyc7
sad =~ sad1 + sad2 + sad3 + sad4 + sad5 + sad6 + sad7
mach~~narc
mach~~psyc
mach~~sad
narc~~psyc
narc~~sad
psyc~~sad
'

...and at last, a single latent model.

single_latent <- '
d =~ mach1 + mach2 + mach3 + mach4 + mach5 + mach6 + mach7 + narc1 + narc2 + narc3 + narc4 + narc5 + narc6 + narc7 + psyc1 + psyc2 + psyc3 + psyc4 + psyc5 + psyc6 + psyc7 + sad1 + sad2 + sad3 + sad4 + sad5 + sad6 + sad7
'

And this is the comparison between these three model:

################### Nested Model Comparison #########################

Chi-Squared Difference Test

                       Df AIC BIC   Chisq Chisq diff   RMSEA Df diff Pr(>Chisq)    
cfa_bifactor          322          848.16                                          
cfa_correlated_latent 344          753.34     -94.82 0.00000      22          1    
cfa_single_latent     350         2840.33    2086.99 0.68003       6     <2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

####################### Model Fit Indices ###########################
                          chisq  df pvalue rmsea   cfi   tli  srmr
cfa_bifactor           848.164  322   .000 .047  .977  .973  .059 
cfa_correlated_latent  753.340† 344   .000 .040† .982† .980† .055†
cfa_single_latent     2840.332  350   .000 .097  .891  .882  .113 

################## Differences in Fit Indices #######################
                                          df  rmsea    cfi    tli   srmr
cfa_correlated_latent - cfa_bifactor      22 -0.007  0.005  0.007 -0.004
cfa_single_latent - cfa_correlated_latent  6  0.058 -0.091 -0.098  0.058

As seen there, bifactor and correlated-latent models show a strange situation where the bifactor model has a lower df but higher x2 compared to the correlated latent.

I truly have no idea what's happening here.

Amelia 

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAPVJd2XTBh0xSPf4UGcSNdmZPaGO4BK7ownX53RwZaFh%3D872Hw%40mail.gmail.com.

[Shu Fai Cheung (張樹輝)]

Dear Amelia,

Given the models you compared, even if the warning did not appear, you may still need to think about an alternative way to compare cfa_bifactor and cfa_correlated_latent.

You have already used compareFit() from semTools. You can try net() on the following two models:

net(cfa_bifactor, cfa_correlated_latent)

It should return FALSE but it is better to check using your own data.

I am not familiar with bifactor models. I believe there are other ways to compare them.

By the way, which estimator did you use? I noticed that the columns of AIC and BIC are blank.

-- Shu Fai

[Rizqy Amelia Zein]

Dear Shu Fai,

I ran net() and the output looked like this:

 If cell [R, C] is TRUE, the model in row R is nested within column C.

        If the models also have the same degrees of freedom, they are equivalent.

        NA indicates the model in column C did not converge when fit to the
        implied means and covariance matrix from the model in row R.

        The hidden diagonal is TRUE because any model is equivalent to itself.
        The upper triangle is hidden because for models with the same degrees
        of freedom, cell [C, R] == cell [R, C].  For all models with different
        degrees of freedom, the upper diagonal is all FALSE because models with
        fewer degrees of freedom (i.e., more parameters) cannot be nested
        within models with more degrees of freedom (i.e., fewer parameters).
        
                                 cfa_bifactor cfa_correlated_latent
cfa_bifactor (df = 322)                                            
cfa_correlated_latent (df = 344) FALSE  

Looks like the problem lies in the estimator. I initially used DWLS, but the problem disappeared after I changed it to MLR -- the df and x2 now look alright. Here is the output of compare() when I changed the estimator to MLR.

################### Nested Model Comparison #########################

Scaled Chi-Squared Difference Test (method = “satorra.bentler.2001”)

lavaan NOTE:
    The “Chisq” column contains standard test statistics, not the
    robust test that should be reported per model. A robust difference
    test is a function of two standard (not robust) statistics.
 
                       Df   AIC   BIC  Chisq Chisq diff Df diff Pr(>Chisq)    
cfa_bifactor          322 58890 59278 1246.5                                  
cfa_correlated_latent 344 58908 59194 1308.4      42.05      22   0.006163 ** 
cfa_single_latent     350 61538 61797 3950.4    2222.62       6  < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

####################### Model Fit Indices ###########################
                      chisq.scaled df.scaled pvalue.scaled rmsea.robust cfi.robust tli.robust  srmr
cfa_bifactor             1132.217†       322          .000        .061       .901†      .884  .059 
cfa_correlated_latent    1163.324        344          .000        .060†      .898       .888† .058†
cfa_single_latent        3508.824        350          .000        .116       .606       .574  .111 
                             aic        bic
cfa_bifactor          58889.992† 59278.079 
cfa_correlated_latent 58907.950  59194.395†
cfa_single_latent     61537.867  61796.591 

################## Differences in Fit Indices #######################
                                          df.scaled rmsea.robust cfi.robust tli.robust   srmr
cfa_correlated_latent - cfa_bifactor             22       -0.001     -0.003      0.004 -0.001
cfa_single_latent - cfa_correlated_latent         6        0.057     -0.292     -0.314  0.052
                                               aic      bic
cfa_correlated_latent - cfa_bifactor        17.958  -83.684
cfa_single_latent - cfa_correlated_latent 2629.917 2602.197

Amelia

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/52311c5f-a5f8-43cd-9748-f44dfeb537cfn%40googlegroups.com.

[Rizqy Amelia Zein]

Hi Shu Fai,

I just found Vuong's test as an alternative to LRT to compare non-nested models (1402.6720.pdf (arxiv.org). Do you think this is an appropriate solution?

Amelia 

[Shu Fai Cheung (張樹輝)]

Dear Amelia,

Good to know that you have found the source of the problem. I am not familiar with the common practice of model comparison that involves bifactor models. I believe other group members can help on this.

Btw, I believe the following paper is the final published version of the arXiv paper you found:

Merkle, E. C., You, D., & Preacher, K. J. (2016). Testing nonnested structural equation models. Psychological Methods, 21(2), 151–163. https://doi.org/10.1037/met0000038

I did a quick search and find papers on bifactor models that cited this paper. You can take a look at them.

-- Shu Fai