No scalar (full or partial) equivalence in evidence - what next?!

[Victoria McRitchie]

Hi everyone,

Have been toiling over a SEM for close to three months for a masters thesis, and admittedly, I have made it rather complicated (using WLSMV estimation on World Values Survey data, having different levels for the indicators, some are 9, some are 10, some are 4)! 

My model is 7 countries (as the group indicator) with 5 latent variables representing 23 indicators. I have not been able to use lavtestscore except by using it in a piecemeal fashion (so releasing 10 parameters at a time, and there are around 1900, so you can imagine how time-consuming this has been!) so have taken to omitting an item one-by-one and then returning to configural, then metric, then scalar testing, in an effort to prove partial scalar invariance.

Every time, I get really good results on configural and metric, but a big degradation when moving to scalar testing. (So config > CFI.scaled = 0.945; RMSEA.scale = 0.072, then metric > CFI.scaled = 0.941; RMSEA.scaled = 0.072, but then scalar > CFI.scaled = 0.867; RMSEA.scaled = 0.087).

As I said, I can't just run lavTestScore() on the whole scalar model as it won't converge, and running it a bit at a time is not practical, so I don't know where to go from here. I am happy to state in my thesis that the MGCFA is configural and metric invariant, but evidence for scalar invariance is lacking. But I want to then regress these latent factors on a dependent variable, and of course without scalar invariance, I fear that is not a sound approach. Would it therefore be reasonable to create 7 separate SEMs (one per country)?

Thanks.

[Terrence Jorgensen]

I get really good results on configural and metric, but a big degradation when moving to scalar testing. (So config > CFI.scaled = 0.945; RMSEA.scale = 0.072, then metric > CFI.scaled = 0.941; RMSEA.scaled = 0.072, but then scalar > CFI.scaled = 0.867; RMSEA.scaled = 0.087)

I don't think comparing these fit indices amounts to an omnibus test of invariance.  But assuming (1) metric invariance is at least approximately true, (2) residual variances are also approximately equal, and (3) common-factor variances are also approximately equal, then you could switch to a single-group MIMIC model that regresses the common factors on 6 dummy codes for the 7 countries.  Then estimation is less problematic, and you could look at score tests for the regression of individual indicators (e.g., one at a time) on the 6 dummy codes.  That's 23 omnibus tests, and then estimate paths for significant score tests so you can quantify impact. 

As I said, I can't just run lavTestScore() on the whole scalar model as it won't converge, and running it a bit at a time is not practical, so I don't know where to go from here. I am happy to state in my thesis that the MGCFA is configural and metric invariant, but evidence for scalar invariance is lacking. But I want to then regress these latent factors on a dependent variable, and of course without scalar invariance, I fear that is not a sound approach. 

Scalar invariance across countries is not required to make valid comparisons of covariance-structure parameters across countries; only metric invariance is required.  Scalar invariance would be required to compare (adjusted-)means/intercepts across countries.

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

http://www.uva.nl/profile/t.d.jorgensen

[Victoria McRitchie]

Thank you Professor Jorgensen, this is enormously helpful! And I have to say, this sort of information is so elusive on the net; everything I've found has been on the lines of "if no scalar invariance, try to have partial scalar invariance" but with no account of pragmatically what to do if you can't achieve partial scalar invariance.

Your point about making valid comparisons of covariance-structure parameters across countries (but not means/intercepts) - I don't suppose you have a reference for this? Or an example of this being applied in practice to a multi-group dataset?

Thanks

Victoria

[Terrence Jorgensen]

a reference for this?

"Weak factorial invariance constrains the whole of the factor loading matrix to equality across groups, making all group differences in observed covariances attributable to differences in the factor covariance structure" (Wu & Estabrook, 2016, p. 1017)

[lynx kitten]

Hi,

I have a question on testing measurement invariance, too. During my studies, I have learned that strict measurement invariance should be given for comparing latent variances.

Tere are some differences in articles and book chapters . One chapter suggests:

1. configural invaraince
2. metric/weak invariance: equal factor loadings as a precondition for comparing covariances between groups
3. scalar/strong invariance: plus equal intersepts of manifest indicators as a precondition for comparing latent means
4. strict factorial invariance: plus equal variances and covariances of latent variables
5. strict measurement error invariance/strict resiudal variance: plus equal residual variance

Can someone explain to me if the model five is necessary to compare variances between groups or not?

Kind regards

[Terrence Jorgensen]

strict measurement invariance should be given for comparing latent variances.

That is not the case.  Only metric invariance is needed to link the scales of common factors, making it valid to compare their variances.

Can someone explain to me if the model five is necessary to compare variances between groups or not?

Strict measurement invariance is assumed when comparing variances of a scale composite (i.e., without a latent-variable model to disaggregate common- and unique-factor variances).