Testing measurement invariance with multi-level SEM in lavaan?

[Nikola Ćirović]

Dear all, 

Is it possible to use multilevel SEM in lavaan to test for measurement invariance across groups (since the number of them is 7 to 9, or even more). If so - how? I am trying to grasp the limitations of MG CFA (and different strategies within MG CFA) and thought that maybe multilevel could provide a more straight-forward solution.

[Nikola Ćirović]

Also, Could anyone tell how do I inspect for invariant indicators (either loadings or intercepts) in the MG CFA across so many groups since lavTestScore doesn't seem to work?

Any paper that demonstrates this in lavaan will be gold to me.

[Terrence Jorgensen]

Is it possible to use multilevel SEM in lavaan to test for measurement invariance across groups (since the number of them is 7 to 9, or even more)

Only in principle.  In practice, you can't run a ML-SEM with N = 9 and expect to trust the results.  There are some single-group options 

since lavTestScore doesn't seem to work?

What does that mean?  It works fine for me in this 12-group example:

HS.model <- ' visual  =~ x1 + x2 + x3
              textual =~ x4 + x5 + x6
              speed   =~ x7 + x8 + x9 '
fit <- cfa(HS.model, data=HolzingerSwineford1939,
           # 12 "groups": table(HolzingerSwineford1939$agemo)
           group = "agemo", group.equal = "loadings")
lavTestScore(fit)

The list of univariate tests refer to constraints labeled in the parTable(fit)

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

http://www.uva.nl/profile/t.d.jorgensen

[Nikola Ćirović]

Dear professor,

Yes, I see now that I have made a mistake by specifying release = TRUE instead of NULL in the brackets.

But now I am having a hard time wrapping my head around what group is being compared against another group. 

This function lavTestScore seems to compare all other groups to the first one. That would be ok but as I put another group in the place of the first group - the modification indices remain the same (the order changes). And I am puzzled as to the appropriate way to test which indicator is invariant and which is not in so many groups.

Would the approach of merging all the groups except for the first one yield better results with this function?

субота, 29. јун 2019. 02.41.03 UTC+2, Nikola Ćirović је написао/ла:

[Nikola Ćirović]

And thank you for your reply.

субота, 29. јун 2019. 02.41.03 UTC+2, Nikola Ćirović је написао/ла:

[Terrence Jorgensen]

This function lavTestScore seems to compare all other groups to the first one.

It is not a comparison, it is a constraint.  If you are using the group.equal argument to constrain parameters, lavaan defines constraints the simplest way: set all other groups to share the first group's parameters.  

And I am puzzled as to the appropriate way to test which indicator is invariant and which is not in so many groups.

Look through the $uni table to see which row numbers are associated with all the across-group constraints for 1 indicator.  Then send those row numbers to lavTestScore() to get the omnibus score test for that set of constraints.  Following my example above:

allScores <- lavTestScore(fit)
allScores$uni # the automatic constraints put the first group always on the left-hand side.

ind2 <- which(allScores$uni$lhs == ".p2.") # second indicator's constraints
ind3 <- which(allScores$uni$lhs == ".p3.") # third indicator's constraints
...
lavTestScore(fit, release = ind2, uni = FALSE) # omnibus test for second indicator
lavTestScore(fit, release = ind3, uni = FALSE) # omnibus test for third indicator
...

If you have enough indicators (ideally > 4 per factor) and you have at least 2 indicators (per factor) with nonsignificant omnibus score tests, you could run a less constrained model in which the constraints across groups are only placed on the 2 indicators whose omnibus tests have the lowest chi-squared values (see https://doi.org/10.1177/0146621607314044).  Continue to label your remaining parameters for other indicators, but they get different labels across groups (e.g., using a .g1, .g2, ... .g8 suffix).  Then, you can use lavTestWald() to test any (set of) constraint(s) you want.  But if you really want to test pairwise comparisons of groups, you should adjust your alpha level for so many exploratory follow-up tests (or save your p values in a vector to send to the p.adjust() function).

[Nikola Ćirović]

Dear professor Jorgensen,

Thank you again. This code you have provided me with seems to work (almost like magic). 

I, however, did run into this error message when I did this omnibus test for the first indicator (out of nine):" 1: In max(r.idx) : no non-missing arguments to max; returning -Inf". and the associated x2 value is 525.136 which makes it the largest but I am worried as to can I trust the result.

At first, I thought it to be because the first indicator is used as a metric indicator (to scale the latent var) but the same thing happens if I use the std.lv = TRUE argument in the metric invariance model (which also confusingly worsens the fit a lot) and the same thing happened when I respecified the model so that the e.g. the sixth indicator was put in the place of the first (so I figure lavaan would take it to be a marker indicator and fix across groups it for identification). I may be rambling.

But when I used this as a trustworthy finding and released the indicator's loading with goup.partial argument the fit of the metric model did improve (and was no worse than .01 of CFI diff which I used as criteria).

Continue to label your remaining parameters for other indicators, but they get different labels across groups (e.g., using a .g1, .g2, ... .g8 suffix).  

Does this apply to the two-group MI testing? (because the code you wrote, as I fully continued it based on the principle did the job)

субота, 29. јун 2019. 02.41.03 UTC+2, Nikola Ćirović је написао/ла:

[Nikola Ćirović]

Actually, I was wrong, when I use std.lv in the metric invarice model this chi square value for the first indicator appears more normal but the fit of the entire model goes way down.

субота, 29. јун 2019. 02.41.03 UTC+2, Nikola Ćirović је написао/ла:

[Terrence Jorgensen]

I thought it to be because the first indicator is used as a metric indicator (to scale the latent var)

Yes, that is why.

when I use std.lv in the metric invarice model this chi square value for the first indicator appears more normal but the fit of the entire model goes way down. 

Because you need to free latent variances in all but 1 group.  The development version of lavaan now does this automatically when group.equal includes "loadings"

install.packages("lavaan", repos = "http://www.da.ugent.be", type = "source")

 but you have to free variances in your metric-model syntax manually if you constrain loadings using labels instead.

foo ~~ c(1, NA, NA, ..., NA)*foo # ... however many groups you have

[Nikola Ćirović]

Dear professor, 

Yes. I used this specification in the model syntax you gave me

'Y =~x1+ x2+ x3...

 foo ~~ c(1, NA, NA, ..., NA)*foo'

...along with std.lv and group.equal to constrain loadings across groups (cfa function for metric invariance). And this made the metric-invariance model fit as it is supposed to and assured me that the metric invariance model is good. However, it still doesn't allow for the omnibus tests you gave me a code for to run when loadings of all indicators are constrained.

So, I just swapped the first indicator (the one that passes the metric to the latent var) with the one that was shown to be least invariant in order to assess the invariance of that first indicator (because it turns out the first one is the most non-invariant).

Finally, the syntax you gave me for omnibus tests of releasing the constraints really saved the day it seems. I just wonder is there a specific reference for it. Does it work in the principle of "all others as anchors" while releasing the "studied" items (mentioned in the article by Woods you recommended) and giving the chi-square diff for that would result if the studied indicator is freed?. Because as I understand the manual, the code you wrote is the for the multivariate (omnibus) LM test.

This is the code I was referring to (you provided me with).

allScores <- lavTestScore(fit)
allScores$uni # the automatic constraints put the first group always on the left-hand side.
ind2 <- which(allScores$uni$lhs == ".p2.") # second indicator's constraints
ind3 <- which(allScores$uni$lhs == ".p3.") # third indicator's constraints
...
lavTestScore(fit, release = ind2, uni = FALSE) # omnibus test for second indicator
lavTestScore(fit, release = ind3, uni = FALSE) # omnibus test for third indicator

субота, 29. јун 2019. 02.41.03 UTC+2, Nikola Ćirović је написао/ла:

[Terrence Jorgensen]

I just wonder is there a specific reference for it.

See the References section of the ?lavTestScore help page.

Does it work in the principle of "all others as anchors" while releasing the "studied" items (mentioned in the article by Woods you recommended) and giving the chi-square diff for that would result if the studied indicator is freed?

Yes.

[Nikola Ćirović]

Dear professor, 

Thank you. You have been kind and helpful!

субота, 29. јун 2019. 02.41.03 UTC+2, Nikola Ćirović је написао/ла: