How to compare non-nested models (with different number of items) in CFA?

[Tina]

I investigate the factor structur of an existing scale in a data set with N = 714 (using lavaan, R). In this context I would like to compare to alternative models:

Model 1: Two factors with 5 items each and in addition one item that is allowed to load on both factors (11 items)
Model 2: Two factors with 5 items each without the additional item (10 items).
For the "standard indices" the following results are obtained:

Model 1: RMSEA=0.135 [0.125; 0.145], SRMR=0.059, CFI=0.871, TLI=0.831, AIC=18633.493, BIC=18743.194
Model 2: RMSEA=0.141 [0.130; 0.153], SRMR=0.058, CFI=0.875, TLI=0.834, AIC=16891.818, BIC=16987.807

(As you can see the fit for both models is not very good. This is due to a number of correlated errors, which may be caused by the formulation of questions and dependencies between questions. If I include these correlated errors in the model (which I know to be considered with caution for various reasons and therefore I'm still undecided how to handle it) the fit improves clearly. With correlated errors, model 2 seems superior to model 1 in terms of all indices)
But after further reading, I don’t think it’s appropriate to use these indices to compare my models, i.e. non nested models with different numbers of items (even though I've seen it in literature several times). Finally I came across the vuong-test and got the following results:

Model 1 
 Class: lavaan 
 Call: lavaan::lavaan(model = m1, data = data.set, model.type = "cfa", ...

Model 2 
 Class: lavaan 
 Call: lavaan::lavaan(model = m2, data = data.set, model.type = "cfa", ...

Variance test 
  H0: Model 1 and Model 2 are indistinguishable 
  H1: Model 1 and Model 2 are distinguishable 
    w2 = 0.559,   p = 1.8e-09

Non-nested likelihood ratio test 
  H0: Model fits are equal for the focal population 
  H1A: Model 1 fits better than Model 2 
    z = -132.171,   p = 1
  H1B: Model 2 fits better than Model 1 
    z = -132.171,   p = < 2.2e-16

Model 1 
 Class: lavaan 
 Call: lavaan::lavaan(model = m1, data = data.set, model.type = "cfa", ...
 AIC: 25452.965 
 BIC: 25562.666 

Model 2 
 Class: lavaan 
 Call: lavaan::lavaan(model = m2, data = data.set, model.type = "cfa", ...
 AIC: 20167.351 
 BIC: 20263.340 

95% Confidence Interval of AIC difference (AICdiff = AIC1 - AIC2) 
  5207.322 < AICdiff < 5363.905

95% Confidence Interval of BIC difference (BICdiff = BIC1 - BIC2) 
  5221.035 < BICdiff < 5377.617

Do you think that in this case the vuong-test is actually a reasonable way to compare the models? The extreme p-values appear suspicious... And if no, what could you possibly recommend to me?

[Terrence Jorgensen]

Model 1: Two factors with 5 items each and in addition one item that is allowed to load on both factors (11 items)
Model 2: Two factors with 5 items each without the additional item (10 items).

You cannot compare models that are fitted to different data.  Period.

Do you think that in this case the vuong-test is actually a reasonable way to compare the models?

Yes, but only when the models are fitted to the same data.  If not, the likelihoods from the 2 models are not on the same scale, so they are not comparable.  From the ?vuongtest help page:

Users should take care to ensure that the two models have the same dependent variable (or, for lavaan objects, identical modeled variables), with observations ordered identically within each model object. Assuming the same data matrix is used to fit each model, observation ordering should generally be identical. There are currently no checks for this, however.

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

http://www.uva.nl/profile/t.d.jorgensen

 

[Tina]

Thank you very much! It's good to have a clear answer!