Comparing fit of 2 non-nested models - DWLS estimator used

[ann-marie...@nicholls.edu]

Hi folks - I am trying to compare the fit of 2 non-nested CFAs. I used DWLS because my data violated MVN. I am understanding that I cannot obtain AiC when DWLS is used. Does anyone have any advice or sources that explains the appropriate way of comparing the fit of these 2 models?

[Terrence Jorgensen]

I cannot obtain AiC when DWLS is used. Does anyone have any advice or sources that explains the appropriate way of comparing the fit of these 2 models?

I am aware of a likelihood-ratio test and AIC-difference test for nonnested models (in the nonnest2 package, which can work on lavaan models), but as you said, only when a likelihood-based estimation method is used, not a least-squares estimator.  But AIC could still be calculated with least-squares information, although this has not been published on in the context of SEM.  Here is a link to a SEMNET post by Cam McIntosh (Feb 2, 2018, Re: "Path analysis with categorical vbles: both AIC and Index of fit?") discussing how to do so when using DWLS.  

https://listserv.ua.edu/cgi-bin/wa?A2=SEMNET;3f485938.1802

I think you might need to register with SEMNET to view it, but that is a good idea anyway, because it is a more general SEM forum, and more appropriate for questions like this (i.e., not software-specific).

Terrence D. Jorgensen

Postdoctoral Researcher, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

UvA web page: http://www.uva.nl/profile/t.d.jorgensen

[Amonet]

Hi Terrence, 

Is the test you are referring to Vuong's test? In the nonnest2 package. 

I noticed that it pretty much chooses any model with fewer manifest variables in my analysis, so I thought it might not be appropriate to use. 

For example, would have a 1-factor model with 6 manifest variables / indicators and would throw an indicator out with low factor loading, residual correlations high and high MI scores. 

Then compare the 6-item model to the 5-item model using Vuong's test, in any case it would give me that the 5-item model was preferred over the 6 item (also when I would e.g. throw a high factor loading out with minor local fit issues). 

[Stas Kolenikov]

What Cam wrote there was voodoo, honestly. The log-likelihood in "plain" AIC, developed for i.i.d. regression models, is a measure of misfit; the p is the trace of the information matrix. I am sure both can be adapted to SEM with DWLS, but not by bluntly plugging in whatever output measures your SEM package spits out. My two cents.

-- Stas Kolenikov, PhD, PStat (ASA, SSC)  @StatStas
-- Senior Scientist, Abt Associates @AbtDataScience
-- Program Chair (2018), Survey Research Methods Section of the American Statistical Association
-- Opinions stated in this email are mine only, and do not reflect the position of my employer
-- http://stas.kolenikov.name
 

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+unsubscribe@googlegroups.com.
To post to this group, send email to lav...@googlegroups.com.
Visit this group at https://groups.google.com/group/lavaan.
For more options, visit https://groups.google.com/d/optout.

[Terrence Jorgensen]

I noticed that it pretty much chooses any model with fewer manifest variables in my analysis, so I thought it might not be appropriate to use. 

Well, the reason it is not appropriate to use in this case is that you are not comparing models fit to the same data.

Then compare the 6-item model to the 5-item model using Vuong's test, in any case it would give me that the 5-item model was preferred over the 6 item

Can't compare those models.  One has 6 outcomes, the other has 5.  The likelihoods are not on the same metric.

[Terrence Jorgensen]

What Cam wrote there was voodoo, honestly. The log-likelihood in "plain" AIC, developed for i.i.d. regression models, is a measure of misfit; the p is the trace of the information matrix. I am sure both can be adapted to SEM with DWLS, but not by bluntly plugging in whatever output measures your SEM package spits out. My two cents.

I was curious about whether those quantities were really analogous.  At some point, Mijke Rhemtulla also alerted me to the fact that AICc was only developed for univariate regression models, so should not be applied to SEMs (even though it is in semTools::moreFitIndices(), now with a warning in the documentation).

Thanks for the feedback, Stas!  I think I learn something important every time you post.