Identifying sources of non-invariance when measurement invariance doesn't hold

[christophe...@nicholls.edu]

Hello,

I'm conducting a test of measurement invariance using ordinal data (using the "measurementInvarianceCat()" function from semTools. Measurement invariance does not hold and I'd like to identify the sources or contributors to non-invariance.

Here is a little more context for the study. The design involves, for instance, randomly assigning participants to one of two conditions. In one condition, items are placed in a random order for each participant (the which we believe will reduce the influence of serial item-order effects). Lets call this the "treatment" condition. In the control condition, individuals are given the same scale but in a serial order (i.e., the order of item presentation is held constant throughout the survey).

Ideally, I can find out two things:

1. What parameters are non-invariant (i.e., what is driving a significant chi-square comparison)?

2. What is the associated effect size?

On the second point, lets assume that factor loadings are non-invariant and we can identify which factor loadings vary across condition. Lets assume for the sake of simplicity that all items vary in their factor loadings. How can I estimate or calculate the effect of item randomization on the factor loadings?

Any guidance is much appreciated. 

Thanks in advance!

Chris

[Mauricio Garnier-Villarreal]

You can use the function lavTestScore to test each constraint parameter, do univariate null tests for each invariance constraint. This way you can see which constraints failed to hold

I havent seen anything about effect size from factor loading invariance. I guess you could use the lavTestScore statistic as a measure of magnitude.

Invariance is different than item randomization, not sure how you are relating these 2

[christophe...@nicholls.edu]

Research suggests that randomizing items might disrupt serial item-order effects, which (in the case of no item randomization) might relate to inflated inter-construct relationships. The study I'm running tests this idea.  

[Terrence Jorgensen]

Lets assume for the sake of simplicity that all items vary in their factor loadings.

Then you cannot draw any conclusions whatsoever.  At least 2 items must be invariant in order to test any other items, and unless you are lucky enough to know which ones certainly are invariant, then research shows you will have a much harder time finding the invariant items if the majority of items have DIF.  Interesting point to consider: If all items have DIF in the same direction (e.g., all loadings are higher in Group 2 than Group 1), then that is equivalent to having full invariance and a different latent variance instead.

How can I estimate or calculate the effect of item randomization on the factor loadings?

So the 2 groups you are comparing are a group whose items were randomized to a group who saw all items in the same order?  Well, Liu et al. (2017) suggested comparing the expected frequencies in each response category between the constrained (i.e., failed invariance at a particular) and less constrained models.  They wrote an R function for the longitudinal case, but perhaps you could use it (or adapt it) for the multiple-group case as well.

http://dx.doi.org/10.1037/met0000075

But since you said you expect consecutively viewed items to have correlated residuals, which would inflate the latent covariances, then I would say that is your effect size of theoretical interest.  You can compare the latent covariances between more-constrained and less-constrained models when invariance fails to see how those estimates are affected.  For example, if you residual invariance fails, try freeing them for one item at a time and compare the latent parameters of interest between that model and the full-invariance model to see the impact.  This is the same idea as Oberski's EPC-interest, which you could get also get from a fully constrained model by repeated calls to lavTestScore().

https://doi.org/10.1093/pan/mpt014

Terrence D. Jorgensen

Postdoctoral Researcher, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

UvA web page: http://www.uva.nl/profile/t.d.jorgensen

[Edward Rigdon]

In a multiple group analysis, you could define new parameters for the differences in loadings across treatments for the same items. Then lavaan's output would give you mean, standard deviation and p-value for each of the new parameters

'

factor1 =~ c(a1, a2)*y1 + c(b1, b2)*y2 + c(c1, c2)* y3

factor1 ~~ 1*factor1                                    # setting scale, achieving identification

a3:=a1-a2

b3:=b1-b2

c3:=c1-c2

'

Then results for a3, b3, c3 would give you the distribution of differences across the two groups.

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To post to this group, send email to lav...@googlegroups.com.
Visit this group at https://groups.google.com/group/lavaan.
For more options, visit https://groups.google.com/d/optout.

[Christopher Castille]

Thanks folks!I have a follow up question for Ed and Terrence. 

Historically, researchers have assumed that method effects have what we might call for our purposes a fixed effect on substantive parameters of interest. If this is true, then we would expect the same measure administered under two conditions [(1) we account for serial-item-order effects by randomizing items, and (2) keep the serial item-order the same] to result in a fixed difference in the factor loadings. To Terrence’s point, this is the same as full invariance but non-invariant latent variances (which I would like an article to read on that point so I can make sure that I understand it). In other words, the differences in the factor loadings would be equivalent.

My question is this: could I constrain the differences in the factor loadings to be equivalent to test the assumption of a fixed effect? I take it that I would have to augment Ed’s code with equality constraints such as “a3  ==b3;b3==c3” or am I mistaken?

I am asking because I have been asked to offer an estimate to help researchers “correct” for serial-item-order effects, which assumes a fixed effect. For instance, if someone hasn’t randomised their items, then they should correct their factor loadings by “x” amount, and correspondingly the latent covariances by “y” amount (if that makes sense). Please note that I don’t expect this hypothesis to pan out, but would like to test it explicitly so I can address these reviewers’ concerns.

As always, thanks in advance. You folks are really quite helpful (and make me wish I sought out your guidance years ago).

Chris

Sent from my iPhone

[Terrence Jorgensen]

could I constrain the differences in the factor loadings to be equivalent to test the assumption of a fixed effect? I take it that I would have to augment Ed’s code with equality constraints such as “a3  ==b3;b3==c3” or am I mistaken?

 Yes, you can apply constraints to user-defined parameters.  Adapting the ?lavTestWald help-page example:

HS.model <- '
    visual  =~ x1 + b1*x2 + x3
    textual =~ x4 + b2*x5 + x6
    speed   =~ x7 + b3*x8 + x9
## arbitrary user-defined parameters
d1 := b2 - b1
d2 := b3 - b2
## force both differences to be equal
d1 == d2
'

fit <- cfa(HS.model, data=HolzingerSwineford1939)
summary(fit)

Note that you can also just estimate the model with unconstrained user-defined parameters, then use the Wald test of the constraints, as shown on the ?lavTestWald help page.

I am asking because I have been asked to offer an estimate to help researchers “correct” for serial-item-order effects, which assumes a fixed effect. For instance, if someone hasn’t randomised their items, then they should correct their factor loadings by “x” amount, and correspondingly the latent covariances by “y” amount (if that makes sense). Please note that I don’t expect this hypothesis to pan out, but would like to test it explicitly so I can address these reviewers’ concerns.

I agree, I would expect that any such correction would be highly sensitive to the (unknown) model parameters, making it unlikely that a correction based on a particular data set would generalize to other cases.