Reliability of a Second-Order Factor and Its Subdimensions – Which Values Should I Report?

[lavaanrookie99]

Hi everyone,

I conducted a CFA with a second-order factor for Wellbeing to validate my measurement model for an SEM. My model is structured as follows:

model_secondorder <- 

 ' Wellbeing =~ Engagement_LV + Perseverance_LV + Optimism_LV + Connectedness_LV + Happiness_LV 

Engagement_LV =~ EPOCH5_E1 + EPOCH7_E2 + EPOCH11_E3 + EPOCH12_E4 Perseverance_LV =~ EPOCH2_P1 + EPOCH9_P2 + EPOCH16_P3

Optimism_LV =~ EPOCH3_O1 + EPOCH13_O2 + EPOCH17_O3

Connectedness_LV =~ EPOCH1_C1 + EPOCH10_C2 + EPOCH14_C3 + EPOCH15_C4 Happiness_LV =~ EPOCH4_H1 + EPOCH6_H2 + EPOCH8_H3 + EPOCH18_H4 ' 

fit_secondorder <- cfa(model_secondorder, data = data, estimator = "MLR", orthogonal = TRUE, missing = "fiml", std.lv = TRUE) 

For the descriptive statistics, I used sum scores of the manifest variables for each subscale. I calculated Cronbach’s alpha and McDonald’s omega using ci.reliability. Additionally, I computed a global Wellbeing score as the average of the five subdimension sum scores.

Now, I want to report reliability values for Wellbeing as a higher-order construct. I tried the following functions:

compRelSEM(fit_secondorder, higher = "Wellbeing", tau.eq = FALSE)

This gives me the following values:

Engagement_LV: 0.683

Perseverance_LV: 0.766

Optimism_LV: 0.714

Connectedness_LV: 0.748 

Happiness_LV: 0.893

Wellbeing: 0.787

And with:

reliabilityL2(fit_secondorder, "Wellbeing")

I get:

omegaL1: 0.7946842

omegaL2: 0.9017477

partialOmegaL1: 0.9037898

My Questions:

1. The reliability values differ depending on the function used. Also, the subscale reliabilities are different from when I compute them separately with ci.reliability. Which values should I report?

2. Can I compute omega for the global Wellbeing sum score (the average of the subdimensions)? If so, which function should I use?

Thanks in advance for your help! :)

[Terrence Jorgensen]

compRelSEM(fit_secondorder, higher = "Wellbeing", tau.eq = FALSE)

Wellbeing: 0.787

reliabilityL2(fit_secondorder, "Wellbeing")

omegaL1: 0.7946842

omegaL2: 0.9017477

partialOmegaL1: 0.9037898

My Questions:

1. The reliability values differ depending on the function used.

The reason they differ is because the deprecated reliabilityL2() only calculated model-implied (rather than observed) total variance for the denominator, whereas the more general function gives you the choice.  The highlighted values above would match if you set compRelSEM(fit_secondorder, higher = "Wellbeing", obs.var = FALSE), which tells it to use model-implied total variance in the denominator.

1. Also, the subscale reliabilities are different from when I compute them separately with ci.reliability. Which values should I report?

You didn't provide your ci.reliability() output, but that is only in the MBESS package now (it hasn't been part of semTools for several years).  But the point estimates are still provided by the deprecated reliability() function, which confusingly outputs several options that compRelSEM() makes possible but only if explicitly requested.  The reliability() output's "omega2" (also "omega" if the model has simple structure and no residual covariances) is what compRelSEM(obs.var=FALSE) provides, reliability() output's "omega3"  output is what compRelSEM() provides by default.  Also, unlike the deprecated functions, compRelSEM() provides both the higher-order and subscale reliabilities in the same output:

Engagement_LV: 0.683

Perseverance_LV: 0.766

Optimism_LV: 0.714

Connectedness_LV: 0.748 

Happiness_LV: 0.893

Wellbeing: 0.787

 

1. Can I compute omega for the global Wellbeing sum score (the average of the subdimensions)?

That is what the highlighted values above represent: reliability of the sum or average of all observed indicators (i.e., the proportion of that composite's variance that is attributable to the higher-order factor).

Terrence D. Jorgensen    (he, him, his)
Assistant Professor, Methods and Statistics
Research Institute for Child Development and Education, the University of Amsterdam
http://www.uva.nl/profile/t.d.jorgensen

[lavaanrookie99]

Dear Terrence,

Thank you very much for your clarification! Indeed, the values match when I set obs.var = FALSE.

I read the paper by Flora (2020) a few months ago and recall that it includes the option of calculating model-implied total variance. However, I must admit that I had forgotten about this option.

Best regards,
lavannrookie99