High factor loadings, low reliability (omega)

[e.s.tw...@gmail.com]

I have a model with 2 factors with binary items and n>2000:

m1 <- 'f1 =~ BCSPEA00 + BCMCON00 + BCPRAI00 + BCKISS00
            f2 =~ BCSCOL00 + BCPHYS00 + BCSLAP00'
fact_m1 <- cfa(m1, data=MCS_PTFT_matched, std.lv=TRUE, ordered=TRUE)
summary(fact_m1, fit.measures=TRUE, standardized=TRUE)
compRelSEM(fact_m1)

Good fit:

Chi-square p-value = .16, CFI=.997, TLI=.996, RMSEA=.014

High factor loadings:

F1:

BCSPEA00  0.954

BCMCON00  0.766

BCPRAI00  0.810

BCKISS00  0.842

F2:

BCSCOL00  0.897

BCPHYS00  0.742

BCSLAP00  0.872

Poor reliability:

F1: 0.673

F2: 0.593

What could explain these low omega values, given the relatively high factor loadings? Your insights are much appreciated!

[Jošt Bartol]

Not an expert on reliability, but the estimate of reliability depends on the way you calculate it. If you were to calculate it based on the standardized solution (as, e.g., proposed in Hair et al., 2018, Multivariate data analysis, p. 676), then your omega would change quite substantially. And, in this case, the high factor loadings would hint at the size of the reliability estimate. However, as much as I am aware, the compRelSEM function calculates omega by using the unstandardized loadings, factor variance, and observed or model implied covariance matrix. Although I have yet to find which of these approaches is better, some comparison has been done by Hancock & Mueller, 2001, Rethinking construct reliability within latent variable systems.

Hope this helps,

Jošt

V V sre., 31. maj 2023 ob 16:33 je oseba e.s.tw...@gmail.com <e.s.tw...@gmail.com> napisala:

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/bea7f288-b84b-44bb-b1da-388795103892n%40googlegroups.com.

[Keith Markus]

e.s.tw...,

  If I am reading your post correctly, you standardized the latent variables but not the observed variables.  As a result, the factor loadings (squared) do not represent the proportion of common factor variance.  You could have very large unique variance and the variances of the observed variables could exceed unity.  In that case, factor loadings that look high compared to standardized loadings may in fact produce a smaller proportion of variance attributable to the common factor.

  You did request the standardized output but appear to have deleted it form what you posted.  Compare the raw estimates to the factor loading estimates for the fully standardized solution.  Also, directly inspect the unique variance estimates. 

  Also consider, however, that these are very short tests.  Reliability comes from errors cancelling out over repeated observations.  That process normally takes more than 3 or 4 observations.

  I believe that the primary difference between reliability estimates based on raw scores and standardized scores has to do with the relative weight of the items in the total score.  If the item variances are homogeneous, then they will be similar.  If the are heterogeneous, then the items with larger variances will carry more weight in the estimate based on the raw scores.  (However, the same words get recycled with different meanings in different contexts and I am not sure that I interpreted everything as intended in this regard.)  However, there is nothing inconsistent about fitting raw scores and then inspecting the standardized solution to evaluate proportions of variance.  Generally, the reliability estimate for the scale scores you intend to be used is the appropriate estimate.

Keith

------------------------
Keith A. Markus
John Jay College of Criminal Justice, CUNY
http://jjcweb.jjay.cuny.edu/kmarkus
Frontiers of Test Validity Theory: Measurement, Causation and Meaning.
http://www.routledge.com/books/details/9781841692203/

[e.s.tw...@gmail.com]

Thank you for your responses. Sorry, the reported factor loadings above are the standardized ones. I indeed standardized the latent variables but not the observed variables because I believe that is not relevant for binary/ordinal data.

But I think I overlooked something else. I believe it has to do with the fact that the Green & Yang formula is applied to take into account both item covariances and thresholds to calculate omega with categorical indicators. This way, the coefficient is calculated on the observed ordinal scale instead of the latent-response scale, which would result in an overestimation of the reliability. I guess this explains the discrepancy between the high factor loadings and low omega values?

And indeed I agree that the small number of items may play a role as well.

Thank you for your insights! 

Op donderdag 1 juni 2023 om 16:14:41 UTC+1 schreef kma...@aol.com:

[Jošt Bartol]

Upon a quick inspection of Green & Yang's (2009), I would (1) calculate alpha (KR20) to compare the estimates, and (2) check the distributions of the items. In Table 1, Green & Young compare a 4-item scale with symmetric and asymmetric distributions (in both cases loadings = 0.8) and find that when distribution is asymmetrical, reliability is lower. Maybe this is the reason in your case?

Green & Yang's (2009), RELIABILITY OF SUMMED ITEM SCORES USING STRUCTURAL EQUATIOMODELING: AN ALTERNATIVE TO COEFFICIENT ALPHA. Psychometrika

V V čet., 1. jun. 2023 ob 18:05 je oseba e.s.tw...@gmail.com <e.s.tw...@gmail.com> napisala:

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/f7282c8c-cbf2-46d5-94eb-2cee166b709en%40googlegroups.com.