Reliability of a Second-Order Factor and Its Subdimensions – Which Values Should I Report?

[lavaanrookie99]

Hi everyone,

I conducted a CFA with a second-order factor for Wellbeing to validate my measurement model for an SEM. My model is structured as follows:

model_secondorder <- 

 ' Wellbeing =~ Engagement_LV + Perseverance_LV + Optimism_LV + Connectedness_LV + Happiness_LV 

Engagement_LV =~ EPOCH5_E1 + EPOCH7_E2 + EPOCH11_E3 + EPOCH12_E4 Perseverance_LV =~ EPOCH2_P1 + EPOCH9_P2 + EPOCH16_P3

Optimism_LV =~ EPOCH3_O1 + EPOCH13_O2 + EPOCH17_O3

Connectedness_LV =~ EPOCH1_C1 + EPOCH10_C2 + EPOCH14_C3 + EPOCH15_C4 Happiness_LV =~ EPOCH4_H1 + EPOCH6_H2 + EPOCH8_H3 + EPOCH18_H4 ' 

fit_secondorder <- cfa(model_secondorder, data = data, estimator = "MLR", orthogonal = TRUE, missing = "fiml", std.lv = TRUE) 

For the descriptive statistics, I used sum scores of the manifest variables for each subscale. I calculated Cronbach’s alpha and McDonald’s omega using ci.reliability. Additionally, I computed a global Wellbeing score as the average of the five subdimension sum scores.

Now, I want to report reliability values for Wellbeing as a higher-order construct. I tried the following functions:

compRelSEM(fit_secondorder, higher = "Wellbeing", tau.eq = FALSE)

This gives me the following values:

Engagement_LV: 0.683

Perseverance_LV: 0.766

Optimism_LV: 0.714

Connectedness_LV: 0.748 

Happiness_LV: 0.893

Wellbeing: 0.787

And with:

reliabilityL2(fit_secondorder, "Wellbeing")

I get:

omegaL1: 0.7946842

omegaL2: 0.9017477

partialOmegaL1: 0.9037898

My Questions:

1. The reliability values differ depending on the function used. Also, the subscale reliabilities are different from when I compute them separately with ci.reliability. Which values should I report?

2. Can I compute omega for the global Wellbeing sum score (the average of the subdimensions)? If so, which function should I use?

Thanks in advance for your help! :)

[Terrence Jorgensen]

compRelSEM(fit_secondorder, higher = "Wellbeing", tau.eq = FALSE)

Wellbeing: 0.787

reliabilityL2(fit_secondorder, "Wellbeing")

omegaL1: 0.7946842

omegaL2: 0.9017477

partialOmegaL1: 0.9037898

My Questions:

1. The reliability values differ depending on the function used.

The reason they differ is because the deprecated reliabilityL2() only calculated model-implied (rather than observed) total variance for the denominator, whereas the more general function gives you the choice.  The highlighted values above would match if you set compRelSEM(fit_secondorder, higher = "Wellbeing", obs.var = FALSE), which tells it to use model-implied total variance in the denominator.

1. Also, the subscale reliabilities are different from when I compute them separately with ci.reliability. Which values should I report?

You didn't provide your ci.reliability() output, but that is only in the MBESS package now (it hasn't been part of semTools for several years).  But the point estimates are still provided by the deprecated reliability() function, which confusingly outputs several options that compRelSEM() makes possible but only if explicitly requested.  The reliability() output's "omega2" (also "omega" if the model has simple structure and no residual covariances) is what compRelSEM(obs.var=FALSE) provides, reliability() output's "omega3"  output is what compRelSEM() provides by default.  Also, unlike the deprecated functions, compRelSEM() provides both the higher-order and subscale reliabilities in the same output:

Engagement_LV: 0.683

Perseverance_LV: 0.766

Optimism_LV: 0.714

Connectedness_LV: 0.748 

Happiness_LV: 0.893

Wellbeing: 0.787

 

1. Can I compute omega for the global Wellbeing sum score (the average of the subdimensions)?

That is what the highlighted values above represent: reliability of the sum or average of all observed indicators (i.e., the proportion of that composite's variance that is attributable to the higher-order factor).

Terrence D. Jorgensen    (he, him, his)
Assistant Professor, Methods and Statistics
Research Institute for Child Development and Education, the University of Amsterdam
http://www.uva.nl/profile/t.d.jorgensen

[lavaanrookie99]

Dear Terrence,

Thank you very much for your clarification! Indeed, the values match when I set obs.var = FALSE.

I read the paper by Flora (2020) a few months ago and recall that it includes the option of calculating model-implied total variance. However, I must admit that I had forgotten about this option.

Best regards,
lavannrookie99

[ryan chesnut]

Following up on this thread to confirm that compRelSEM() no longer provides the reliability estimate for second-order factors, along with reliability estimates for the first-order factors, by including the higher = argument. Rather, to get a reliability estimate for the second-order factor, you need to use the W = argument with the custom weight syntax (https://www.rdocumentation.org/packages/semTools/versions/0.5-9/topics/compRelSEM). Also, when the compRelSEM() function is used on a second-order CFA, are the values returned for each first-order factor interpreted as the omega reliability estimates for that first-order factor accounting for the second-order factor? 

[Terrence Jorgensen]

Following up on this thread to confirm that compRelSEM() no longer provides the reliability estimate for second-order factors, along with reliability estimates for the first-order factors, by including the higher = argument. Rather, to get a reliability estimate for the second-order factor, you need to use the W = argument with the custom weight syntax (https://www.rdocumentation.org/packages/semTools/versions/0.5-9/topics/compRelSEM).

Confirmed.

 

Also, when the compRelSEM() function is used on a second-order CFA, are the values returned for each first-order factor interpreted as the omega reliability estimates for that first-order factor accounting for the second-order factor? 

I don't know what you mean by "accounting for".  If you mean "controlling for", I don't think that conforms to the higher-order model.  

By default, any detected higher-order factor(s) would be assumed the source of true-score variance.  The default output format is verbose, indicating what source(s) of variance constitutes the numerator.

From the help-page example:

mod.hi <- ' visual  =~ x1 + x2 + x3

            textual =~ x4 + x5 + x6
            speed   =~ x7 + x8 + x9
            general =~ visual + textual + speed '
fit.hi <- cfa(mod.hi, data = HolzingerSwineford1939)
compRelSEM(fit.hi)

$visual

Composite `visual` is composed of observed variables:
x1, x2, x3
True-score variance is represented by common factor(s):
general
Total variance of composite `visual` determined from the unrestricted model.
The proportion attributable to "true" scores is its model-based estimate of reliability ("omega"):

[1] 0.467

$textual

Composite `textual` is composed of observed variables:
x4, x5, x6
True-score variance is represented by common factor(s):
general
Total variance of composite `textual` determined from the unrestricted model.
The proportion attributable to "true" scores is its model-based estimate of reliability ("omega"):

[1] 0.244

$speed

Composite `speed` is composed of observed variables:
x7, x8, x9
True-score variance is represented by common factor(s):
general
Total variance of composite `speed` determined from the unrestricted model.
The proportion attributable to "true" scores is its model-based estimate of reliability ("omega"):

[1] 0.199

If you still want to consider lower-order factors to be true scores (i.e., the lower- and higher-order factors are both components of a true score), you could name them using the true= argument:

compRelSEM(fit.hi, true = list(visual  = "visual",
                               textual = "textual",
                               speed   = "speed"))

$visual

Composite `visual` is composed of observed variables:
x1, x2, x3
True-score variance is represented by common factor(s):
visual
Total variance of composite `visual` determined from the unrestricted model.
The proportion attributable to "true" scores is its model-based estimate of reliability ("omega"):

[1] 0.612

$textual

Composite `textual` is composed of observed variables:
x4, x5, x6
True-score variance is represented by common factor(s):
textual
Total variance of composite `textual` determined from the unrestricted model.
The proportion attributable to "true" scores is its model-based estimate of reliability ("omega"):

[1] 0.885

$speed

Composite `speed` is composed of observed variables:
x7, x8, x9
True-score variance is represented by common factor(s):
speed
Total variance of composite `speed` determined from the unrestricted model.
The proportion attributable to "true" scores is its model-based estimate of reliability ("omega"):

[1] 0.686

Despite not stating it, that implicitly includes "general" variance in the numerator because it is calculated from lavInspect(fit.hi, "cov.lv")
        visual textul speed general
visual   0.809                    
textual  0.408  0.979            
speed    0.262  0.173 0.384      
general  0.617  0.408 0.262   0.617 

[Chesnut, Ryan]

Thank you, Dr. Jorgensen. I greatly appreciate the thorough response. I do have one additional follow-up question. Using the output for the visual factor from the help page as a guide:

$visual

Composite `visual` is composed of observed variables:
x1, x2, x3
True-score variance is represented by common factor(s):
general
Total variance of composite `visual` determined from the unrestricted model.
The proportion attributable to "true" scores is its model-based estimate of reliability ("omega"):

[1] 0.467

Is the last line before the 0.467 estimate is provided referring to the proportion of variance in the visual composite that is explained by "true" scores (i.e., the general common factor)? If so, would it be correct to interpret the result as approximately 46.7% of the variance in the visual composite is attributable to the general common factor?

Thank you,

Ryan P. Chesnut, PhD

Assistant Research Professor

Clearinghouse for Military Family Readiness

The Pennsylvania State University

402 Marion Place

University Park, PA 16802

Phone: 814-865-9637

Email: rpc...@psu.edu

Website: www.militaryfamilies.psu.edu

---

From: lav...@googlegroups.com <lav...@googlegroups.com> on behalf of Terrence Jorgensen <tjorge...@gmail.com>
Sent: Friday, November 13, 2026 8:05 AM
To: lavaan <lav...@googlegroups.com>
Subject: Re: Reliability of a Second-Order Factor and Its Subdimensions – Which Values Should I Report?

 

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/lavaan/79d6b47f-f29c-4703-ba8d-c0134fc1d8f8n%40googlegroups.com.

[Terrence Jorgensen]

Yes 👍🏼