semTools::reliability estimates greater than 1?

[Stephen Martin]

Hello,

I'm helping a colleague of mine with a latent variable model and recommended computing omega coefficients for a scale.
The scale has two factors, one of which is the construct measured by the scale and the other is a reverse-wording method effect. Essentially:

rf =~ item1 + item2 + item3 + item 4r + item5r + item6r
negItems =~ item4r + item5r + item6r

When he runs semTools::reliability() on the cfa output from the above, it results in some weird omega coefficients:

              rf  negItems     total
alpha  0.9459844 0.9134936 0.9459844
omega  0.9744239 0.9049808 0.9593466
omega2 1.5488518 0.7885791 0.9593466
omega3 1.5433199 0.7885505 0.9559202
avevar        NA        NA 0.6950781

What's up with that?

[Terrence Jorgensen]

What's up with that?

I don't know, but posting your model parameter estimates might provide clues.  Looking at the formulas in the documentation, it looks like omega 2 and 3 estimates could exceed 1 if you have negative variance estimates, or any omega estimates might be able to exceed 1 if there are negative residual covariances or if there are cross-loadings.  But I am not aware of a paper that explains this.  You could post on SEMNET to see if anyone has encountered omega > 1 before.

www2.gsu.edu/~mkteer/semnet.html

Terrence D. Jorgensen

Postdoctoral Researcher, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

UvA web page: http://www.uva.nl/profile/t.d.jorgensen

[Stephen Martin]

Thanks for your reply!

We observed no negative variance estimates, but there are cross-loadings.

We initially did the following (intuitively):

RF =~ rf_1 + rf_2 + ... rf_10r + rf_11r + rf_12r # All RF items
NegItems =~ rf_10r + rf_11r + rf_12r # Reversed items

Which resulted in that > 1 omega coefficient.

Doing it another way:

RF =~ rf_1 + rf_2 + ... rf_10r + rf_11r + rf_12r # All RF items
rf_10r ~~ rf_11r + rf_12r
rf11r ~~ rf_12r

Resulted in a nearly equal fit, but omegas < 1.
I'm not entirely sure why. So the method-factor approach produced weird omegas, but the residual covariance approach did not.

[Stephen Martin]

A followup to this; I had my colleague re-run the method-factor approach, but force 0 correlation between NegItems and RF, and the reliability estimates are now sensible. It dawned on me that to ensure that the negItems factor is in fact just explaining variance due to reverse items and NOT variance due to RF, we had to force the two to be non-correlated. That intuition was seemingly correct.

I searched on semnet and didn't see any mention of omega estimates greater than 1, but some decent discussion about what reliability really even means in latent variable contexts, which was nevertheless interesting.

Thank you again; I think this issue is resolved.

[Benny Salo]

I have a semTools::reliability issue that is not necessarily related to this - but I notice these similarities: there are dual loadings in the model and the ‘avevar’ figures are missing.

I recently went through the results for one of my publications and noted that I did not get the same reliability figures as I did when I wrote the manuscript. The issue seems to originate from my use of a model with several dual loadings. Also in version 0.4-14 some ‘avevar’ values are not calculated - apparently for those factors that are involved in dual loadings. I find that this change happens between versions 0.4-11 and 0.4.12.

The missing ‘avevar’ figures leeds me to believe that I should use the figures produced by the earlier versions of semTools - but maybe the missing figures are a feature rather than a bug?

I am using:

R version 3.4.0 Patched (2017-05-18 r72697)

lavaan 0.5-23.1097

semTools 0.4-14

I reproduce the issue below using the HolzingerSwineford1939 data.

Benny

# HS.model with an arbitrary dual loading (speed =~ x1)

HS.model.dual <- ' visual =~ x1 + x2 + x3

textual =~ x4 + x5 + x6

speed =~ x7 + x8 + x9 + x1'

# Run model

tested.fit <- cfa(model = HS.model.dual, data = HolzingerSwineford1939)

# Test reliability - 'avevar' is NA for the factors involved in the dual loading

reliability(tested.fit)

# Install and load semTools version  0.4-11

require(devtools)

install_version("semTools", version = "0.4-11", repos = "https://cloud.r-project.org")

# Test reliability again - gives different results

reliability(tested.fit)

[Terrence Jorgensen]

but maybe the missing figures are a feature rather than a bug?

Indeed.  From the help page:

"Note that AVE will not be provided for factors consisting of items with dual loadings. AVE is the property of items but not the property of factors."

[Benny Salo]

A (very) belated thank you for your answer!

I recently got a reason to return to this question, and it seems I missed a part of the problem of differing outputs with different versions of semTools::reliability. Not only is not average variance calculated in newer version (which is by design) but the omega numbers are different with semtools 0.4-11 and later versions. The issue arises with categorical items - and dual loadings. So I make a new attempt at reproducing the problem below. Now using r 3.5.1, lavaan 0.6-1, and semTools 0.5-0.

I am assuming that the newer version gives more accurate results - but would like to check this with someone who can tease apart code and formulas better than I can. Thank you!

Benny

 

 # Script

 

install.packages("semTools")

 

library(lavaan)   # lavaan_0.6-1 

library(semTools) # semTools_0.5-0

 

# Make a copy of `HolzingerSwineford1939` and coerce x-items to three level

# ordered items

 

HS_categorical <- HolzingerSwineford1939

 

categorical_items <- c("x1", "x2", "x3", "x4", "x5",

                       "x6", "x7" ,"x8", "x9")

 

HS_categorical[, categorical_items] <-

  lapply(HS_categorical[, categorical_items],

         cut,

         breaks = 3,

         labels = c(1, 2, 3))

 

HS_categorical[, categorical_items] <-

  lapply(HS_categorical[, categorical_items], 

         ordered)

 

 

 

# HS_model with an arbitrary dual loading (speed =~ x1)

 

HS_model_dual <- ' visual =~ x1 + x2 + x3

 

textual =~ x4 + x5 + x6

 

speed =~ x7 + x8 + x9 + x1'

 

 

 

# Run model

 

tested_fit <- cfa(model = HS_model_dual, data = HS_categorical)

 

 

 

# Test reliability - 'avevar' is NA for the factors involved in the dual loading

 

reliability(tested_fit)

 

# OUTPUT:

 

# visual   textual     speed     total

# alpha  0.6442547 0.9027799 0.6815202 0.7755001

# omega  0.5161615 0.8286978 0.5267941 0.7886961

# omega2 0.5146708 0.8286978 0.4935335 0.7886961

# omega3 0.5131706 0.8304090 0.5040632 0.8071102

# avevar        NA 0.7629496        NA 0.5548325

 

 

# Install and load semTools version  0.4-11

 

library(devtools)

 

install_version("semTools", version = "0.4-11", 

                          repos = "https://cloud.r-project.org")

 

 

 

# Test reliability again - gives different results

 

reliability(tested_fit)

 

# OUTPUT:

 

# visual   textual     speed     total

# alpha  0.6442547 0.9027799 0.6815203 0.7755001

# omega  0.5377516 0.8286978 0.6500490 0.7886961

# omega2 0.5377516 0.8286978 0.6500490 0.7886961

# omega3 0.5361841 0.8304090 0.6639180 0.8071102

# avevar 0.4145274 0.7629496 0.5308705 0.5548325

 

[Terrence Jorgensen]

numbers are different with semtools 0.4-11 and later versions. 

I have only been maintaining semTools since version 0.4-13, so I am not aware of what precisely changed after 0.4-11.  But because the current output is consistent with what the help page says to expect, I can only assume that the change was a correction.  The formulas are available on the help page, so you can work these out manually to check them yourself.  If you find an inconsistency, please let me know by starting a new thread rather than continuing one from a different topic.