EFA and CFA with dichotomous items

[Francisco José Álvarez-Montero]

Dear lavaan community:

                         I am trying to run an EFA and a CFA in lavaan. on a unidimensional test  that has 10 dichotomous items in which the response options are incorrect (1) and correct (2).  The code that I am using is the following:

ExpFA<-lavaan::efa(data=mydata,nfactors=1,rotation = "none", estimator="WLSMV", ordered=F, output="efa")

ConFA<-lavaan::cfa(model=cfaModel,data=mydata,estimator="WLSMV",ordered=F)

                       My questions is the following: should I set the "ordered=" to be TRUE or should I set to it to be FALSE? 

                       

  I've executed/run the code with "ordered=F" and then, with "ordered=T" and I get very different results. I hope someone could give me a hand with this and explain which is the right choice and the difference.

Best Regards

Francisco Alvarez-Montero

[Yago Luksevicius de Moraes]

Dear Francisco.

When you say "very different results", what exactly do you mean?

I'm doing a simulation study about what happens if we treat a six-item dichotomous scale as intervalar, and so far what I'm finding is that:

• If you are interested in the values of the factor loadings, you MUST set "ordered = TRUE". There is no discussion. Treating items as intervalar always results in underestimated factor loadings.
• If you are interested in the fit indices, there should not be much difference, so you can use whatever you prefer or is easier to you. If you are getting different results, maybe you are underpowered and should try to increase your sample size.

Hope this helps,

Yago

[Francisco José Álvarez-Montero]

Dear Yago:

 

                      Thanks for your answer.  My real world data (N=303) and your simulated data coincide.  When I set the "ordered" argument  to "TRUE", in the "efa()" and “cfa()” functions, factor loadings are higher that when I set the "ordered" argument to "FALSE", and there isn't much difference in the fit indices, except, and I believe this is important, in the RMSEA 90% Confidence Interval,  which becomes much wider when "ordered=TRUE" , as it is shown  in the image below. 

 

 

 

                       Therefore, it seems that you "ordered=TRUE" will produce higher factor loadings but at the cost of a wide CI for RMSEA. However, according to a recent article (https://doi.org/10.3758/s13428-020-01479-0): the lower value of the 90% CI should include or be very near zero and the upper value should not be not very large, i.e., less than 0.08 (I use 0.10). Since the upper bound of the CI is 0.11, and the test of not-close fit has p=0.180 when "ordered=TRUE", close fit has to be rejected:

 

 

 

 

                          I will think about this problem over the weekend and try running the EFA/CFA using the MLMV estimator. 

 

Best Regards

Francisco Alvarez-Montero

[Edward Rigdon]

Right. If your dichotomous items are reflections of underlying normal continuous variables, then the dichotomization attenuated correlations by in effect discarding information. Treating your dichotomous items as ordered, the Pearson correlations used for continuous items are replaced by polychoric correlations, which under assumptions provide unbiased estimates of the correlations among the underlying continuous items. Estimated correlations are generally stronger, which results in stronger loadings but often worse fit. Declaring that your dichotomous items are not ordered, but rather continuous, means choosing an empirical correlation matrix that is a biased estimate of the population correlation matrix. Yes, sometimes biased estimators, as in ridge regression, can yield reduced mean squared error, trading off bigger bias for lower sampling variance. I don't think that you will find a methodological reference recommending modeling dichotomous indicators as if they were continuous.

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/ad0a57ac-6afc-40bc-8e7e-2e4025fc1837n%40googlegroups.com.

[Francisco José Álvarez-Montero]

Dear Edward:

                          Thanks for your help. Do you know of any references in the literature where it is recommended to model dichotomous indicators as ordered? That would help a lot. 

Best Regards

Francisco Alvarez-Montero

[Keith Markus]

Francisco,

To the best of my knowledge, you will not find literature addressing that question because threshold models (and IRT models before them) were introduced precisely for the purpose of modeling ordinal and dichotomous items.  Well before SEM and CFA became feasible, it was understood that the common factor model was not appropriate for dichotomous items.  Think about the analogy to another linear model: regression.  Economists sometimes apply linear regression to dichotomous outcomes and call it a linear probability model.  This model has several shortcomings: (a) residuals are not normally distributed, they always equal 0 - Y^ or 1 - Y^, (b) residual variance is not homoskedastic, the variance of a dichotomous variable varies with its mean, (c) the probability of Y^ being a valid value of Y is infinitesimal, (d) predictions are not bounded by 0 and 1.  Likewise, when you apply linear factor analysis models to dichotomous variables, analogous difficulties arise (not identical because different estimators make different assumptions).  So, it is not a choice between two equally appropriate models that optimize different criteria.  Linear factor analysis is simply the wrong model for dichotomous observed variables.  Using a threshold model in factor analysis is comparable to using probit regression (or else logistic regression) instead of linear regression.  It provides a link function to make the linear predictor conform to the distribution of the observed variable.  Think of it this way:  If you were playing an honest carnival game in which a peanut was under one of two shells, would you ever guess that the peanut was somewhere in between the two shells, or anywhere other than under one or under the other, 0 or 1?  In CFA you do not want to model a normal distribution of item scores around the predicted value but rather a distribution of zeros and ones conditional on the predicted value.

So, I would just cite an introductory reference or canonical reference for threshold models in CFA.  That should satisfy a skeptical reviewer.  You can always explain the problems with fitting a linear model to dichotomous data in the responses to reviewers' comments.  One option is Edwards, Wirth, Houts and Xi's chapter in Hoyle's Handbook of Structural Equation Modeling.

Keith

------------------------
Keith A. Markus
John Jay College of Criminal Justice, CUNY
http://jjcweb.jjay.cuny.edu/kmarkus
Frontiers of Test Validity Theory: Measurement, Causation and Meaning.
http://www.routledge.com/books/details/9781841692203/

[Francisco José Álvarez-Montero]

Dear Colleagues:

                                   I have found the answer in two old posts made in this group in  2013, which were answered by Yves Rosseel:

https://groups.google.com/g/lavaan/c/QPn4UBGCH48/m/BLeltKD_lQYJ

https://groups.google.com/g/lavaan/c/LxqIagOPmRU/m/3G1LHPB0tHYJ

                                    The problem lied in the approach used by the estimator to 'adjust' the chi-square test statistic (and/or its degrees to freedom), to better approximate the chi-square distribution. In this sense  If you use the "WLSMV" tag in the "Estimator=" argument,   lavaan uses the "simple second-order correction" implemented in Mplus  (https://www.statmodel.com/download/WLSMV_new_chi21.pdf) and  which has never been published as an article or book chapter. 

                                     

                                    However, if you choose the "WLSMVS" tag in the "Estimator=" argument,   lavaan utilizes the "mean and variance" corrections proposed by Satorra & Benter (1994) and Satorra (2000) called the "Satterthwaite"  approach/method -the S at the end of WLSMVS stands for Satterthwaite-, which, as opposed to the Mplus one, has been published in the scientific literature. Yves Rosseel, in one of the posts above (https://groups.google.com/g/lavaan/c/LxqIagOPmRU/m/3G1LHPB0tHYJ) stated that  the  formulas used in the "WLSMVS" tag are the "correct ones". It seems that the "WLSMV" tag remains in lavaan as a kind of "fan service" to Mplus users and should be avoided.

                                     

                                    I tried both approaches/methods, explicitly specifying the estimator, the type of method to calculate standard errors, and the correction formulas to approximate the chi-square distribution:

1.Mplus approach: lavaan::cfa(model=cfaModel,data=mydata,ordered=T,estimator="DWLS",se="robust",test="scaled.shifted")

2.Satterwaite method:  lavaan::cfa(model=cfaModel,data=mydata,ordered=T,estimator="DWLS",se="robust",test="mean.var.adjusted")

                                  You can see the results of both approaches/methods in the image below:

​

                                  As it can be seen, the factor loadings are the same, the fit indices values are pretty much the same too, with the exception of RMSEA that has improved, particularly with respect to its confidence interval. Hence, for future references if you have categorical/ordered data use the "WLSMVS" tag. On the other hand, if your data is continuous or can be treated as such, and you want to use a Maximum Likelihood estimator, with  a "mean and variance adjustment" use the "MLMVS" one. 

References

Satorra, A., & Bentler, P. M. (1994). Corrections to test statistics and standard errors in covariance structure analysis. In A. von Eye & C. C. Clogg (Eds.), Latent variables analysis: Applications for developmental research (pp. 399–419). Sage Publications, Inc.

Satorra, A. (2000). Scaled and adjusted restricted tests in multi-sample analysis of moment structures. In Innovations in multivariate statistical analysis: A Festschrift for Heinz Neudecker (pp. 233-247). Boston, MA: Springer US.

[Yves Rosseel]

Hello Francisco,

On 7/17/23 03:10, Francisco José Álvarez-Montero wrote:
> https://groups.google.com/g/lavaan/c/QPn4UBGCH48/m/BLeltKD_lQYJ

Right. This posts explains the origin of the *MV and *MVS labels.

> https://groups.google.com/g/lavaan/c/LxqIagOPmRU/m/3G1LHPB0tHYJ

But this one is about the (robust) chi-square difference test, comparing
two nested models.

> Yves Rosseel, in one of the posts above
> (https://groups.google.com/g/lavaan/c/LxqIagOPmRU/m/3G1LHPB0tHYJ)

> stated that the formulas used in the "*WLSMVS*" tag are the
> "*correct ones*".

Not really. The latter post is about comparing two models. And I also
write in the same post:

"Mplus is using a slightly different approach (described in webnote 10
on the Mplus website). AFAIK, both approaches are correct."

There is no need to switch to WLSMVS (unless you need to replicate older
results). The WLSMV implementation (which is using the 'scaled-shifted
approach) works just fine. WLSMVS and WLSMV should lead to the same
p-value for the chi-square test statistic. The newer "scaled-shifted"
version has the advantage that we do not need to adjust the
degrees-of-freedom.

However, I do recommend that users avoid using estimator = "WLSMV" (or
anything with the *M, *MV, *MVS extension). They are just (Mplus)
shortcuts for specifying three things: the estimator=, se= and test=
arguments. It is more clear if you specify them explicitly, as in

estimator = "DWLS", se = "robust", test = "scaled.shifted"

> As it can be seen, the factor loadings are the same, the fit indices
> values are pretty much the same too, with the exception of RMSEA that
> has improved, particularly with respect to its confidence interval.

Unfortunately, I think that the 'rmsea.robust' version (used for
scaled.shifted) is more trustworthy (and usually much more pessimistic)
than the 'rmsea.scaled' version (still used for test =
"mean.var.adjusted"). The latter is still using the 'old' approach to
get a 'scaled' RMSEA, while the former is using the approach described here:

Savalei, V. (2018). On the computation of the RMSEA and CFI from the
mean-and-variance corrected test statistic with nonnormal data in SEM.
Multivariate behavioral research, 53(3), 419-429.

It may be possible to use this approach for WLSMVS, but that has not
(yet) been developed, most likely because we should all use WLSMV (or
test = "scaled.shifted") anyway.

Yves.

[Francisco José Álvarez-Montero]

Thanks   Yves, your comments are much appreciated.

Best Regards

Francisco Alvarez-Montero