Unable to estimate measEq() model using marker variable method of identifying common factor variance

[Thomas McCauley]

Hi all,

I am trying to run a measEq() test using the marker variable method of model identification. When I try to do this, my model fails. But when I try to run the invariance tests in measEq() using the reference-indicator method, the model can't be identified and I can't estimate standard errors. The loadings look right: the first indicator is indeed estimated to be equal to 1, and the reported degres of freedom are > 0, so the model should be identified. 

The model failure seems to be entirely due to the reference variable method as it is applied in the measEq() function, because when I run a multi-group model in CFA without the measEq() syntax the model is identified just fine. 

Here is my code:

## Specify unidimensional model


# Model syntax
Unidim <- '
empathy =~ sympathetic + empathic + compassionate + softhearted + tender
'


# Fit the model
UnidimModel <- as.character(measEq.syntax(configural.model = Unidim,
                                     data = data,
                                     ordered = c(
                                       "compassionate",
                                       "sympathetic",
                                       "empathic",
                                       "softhearted",
                                       "tender"),
                                     parameterization = "delta",
                                     ID.fac = "auto.fix.first",
                                     ID.cat = "Wu",
                                     group = "SexCode",
                                     group.equal = "configural"))


# Model fit
UnidimFit <- cfa(as.character(UnidimModel),
                              data = data,
                              group = "SexCode",
                              group.label = c("Male", "Female"),
                              ordered = c(
                                "compassionate",
                                "sympathetic",
                                "empathic",
                                "softhearted",
                                "tender"),
                              test = "Satorra-Bentler")


summary(UnidimFit,
        fit.measures = TRUE)


Warning messages:
1: In lav_model_vcov(lavmodel = lavmodel2, lavsamplestats = lavsamplestats,  :
  lavaan WARNING:
    Could not compute standard errors! The information matrix could
    not be inverted. This may be a symptom that the model is not
    identified.
2: In lav_test_satorra_bentler(lavobject = NULL, lavsamplestats = lavsamplestats,  :
  lavaan WARNING: could not invert information matrix needed for robust test statistic

The model still fails when I change other parameters, like switching the parameterization to theta and estimating the model without the satorra-bentler test statistic. The model does run using measEq() if I remove the ordered indicators argument, but that's an important part of the model. I can also estimate standard errors if I change ID.cat to the methods "default" or "LISREL", but I still get an error telling me the following:

Warning message:
In lav_model_vcov(lavmodel = lavmodel2, lavsamplestats = lavsamplestats,  :
  lavaan WARNING:
    The variance-covariance matrix of the estimated parameters (vcov)
    does not appear to be positive definite! The smallest eigenvalue
    (= 1.027876e-18) is close to zero. This may be a symptom that the
    model is not identified

Thanks!

Best,
Thomas

[Nickname]

Thomas,
  The standard advice with the fixed loading method is to choose the best indicator.  However, software cannot make that determination, and so substitutes the first indicator.  It could be that your first indicator is not a good choice.  You could try either specifying the model manually to fix a different loading, or reordering your indicators so that a different indicator is listed first.

Keith
------------------------
Keith A. Markus
John Jay College of Criminal Justice, CUNY
http://jjcweb.jjay.cuny.edu/kmarkus
Frontiers of Test Validity Theory: Measurement, Causation and Meaning.
http://www.routledge.com/books/details/9781841692203/

[Terrence Jorgensen]

The Wu & Estabrook based approach that I implemented is not really compatible with assigning a reference indicator, although I never programmed a check for this circumstance.  The argument ID.fac = "auto.fix.first" assumes that because the first intercept is fixed to zero, the factor mean can be estimated.  However, with ordinal indicators, the intercepts already begin as fixed to zero, so there is no information available to identify your estimated factor means.  An arbitrary single threshold of the marker variable could be fixed to zero to identify the factor mean, but you would have to assign that afterward by adjusting the generated syntax or by using the update() method as shown in an example here:  https://github.com/simsem/semTools/issues/73

Is there a reason you do not want to use the statistically equivalent method of fixing factor means/variances in the configural model (which is the default setting)?

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

http://www.uva.nl/profile/t.d.jorgensen

[Thomas McCauley]

Thank you Keith and Terrence for your replies!

Keith: I tested out all of the indicators to see if any of them would make the algorithm work, and none of them did. I'm pretty sure the indicators are ok - I selected them using the AOAA approach described by Adam Meade here, and my tests indicated that the items are all invariant. I selected my marker variable, the item "sympathetic", based on the nonsigMaxA criteria described by Woods & Wang (2017). 

Terrence: This makes sense to me, thank you for the explanation. Ordinal model estimation is still an adventure for me. The method you described looks great, but I had trouble implementing it in R. However, I also could not get the code that you linked in github to work (issue 73), even after re-installing the latest dev tools version of semTools. Here is the code I am trying to run to fix an arbitrary threshold of my preferred anchor item:

devtools::install_github("simsem/semTools/semTools")
library("semTools")

## Specify unidimensional model


# Model syntax
Unidim <-
'
empathy =~ sympathetic + empathic + compassionate + softhearted + tender
'

# Fit the model specifying that "sympathetic" serves as the marker variable
UnidimModel <- measEq.syntax(configural.model = Unidim,
                                     data = data,
                                     ordered = c(

                                       "compassionate",
                                       "sympathetic",
                                       "empathic",
                                       "softhearted",
                                       "tender"),
                                     parameterization = "delta",
                                     ID.fac = "auto.fix.first",
                                     ID.cat = "Wu",
                                     group = "SexCode",
                                     group.equal = "configural")

# fix an arbitrary threshold of the specified marker variable to zero
UnidimModelUpdated <- update(UnidimModel,
                              change.syntax = c('sympathetic | 0*t1'))

# Print syntax to see if the syntax changed. I didn't notice any change in my code
cat(mod.metric <- as.character(UnidimModelUpdated))


# Model fit
UnidimFit <- cfa(as.character(UnidimModelUpdated),

                              data = data,
                              group = "SexCode",
                              group.label = c("Male", "Female"),
                              ordered = c(
                                "compassionate",
                                "sympathetic",
                                "empathic",
                                "softhearted",
                                "tender"),
                              test = "Satorra-Bentler")

# Summary
summary(dddUnidimFit,
        fit.measures = TRUE)

Specifically, when I print the code for my model after running the update() function in the above lines of code, this is what I get:

> cat(mod.metric <- as.character(UnidimModelUpdated)) # print syntax first
## LOADINGS:


empathy =~ c(1, 1)*sympathetic + c(lambda.1_1.g1, lambda.1_1.g2)*sympathetic
empathy =~ c(NA, NA)*empathic + c(lambda.2_1.g1, lambda.2_1.g2)*empathic
empathy =~ c(NA, NA)*compassionate + c(lambda.3_1.g1, lambda.3_1.g2)*compassionate
empathy =~ c(NA, NA)*softhearted + c(lambda.4_1.g1, lambda.4_1.g2)*softhearted
empathy =~ c(NA, NA)*tender + c(lambda.5_1.g1, lambda.5_1.g2)*tender


## THRESHOLDS:


sympathetic | c(NA, NA)*t1 + c(sympathetic.thr1.g1, sympathetic.thr1.g2)*t1
sympathetic | c(NA, NA)*t2 + c(sympathetic.thr2.g1, sympathetic.thr2.g2)*t2
sympathetic | c(NA, NA)*t3 + c(sympathetic.thr3.g1, sympathetic.thr3.g2)*t3
sympathetic | c(NA, NA)*t4 + c(sympathetic.thr4.g1, sympathetic.thr4.g2)*t4
sympathetic | c(NA, NA)*t5 + c(sympathetic.thr5.g1, sympathetic.thr5.g2)*t5
sympathetic | c(NA, NA)*t6 + c(sympathetic.thr6.g1, sympathetic.thr6.g2)*t6
empathic | c(NA, NA)*t1 + c(empathic.thr1.g1, empathic.thr1.g2)*t1
empathic | c(NA, NA)*t2 + c(empathic.thr2.g1, empathic.thr2.g2)*t2
empathic | c(NA, NA)*t3 + c(empathic.thr3.g1, empathic.thr3.g2)*t3
empathic | c(NA, NA)*t4 + c(empathic.thr4.g1, empathic.thr4.g2)*t4
empathic | c(NA, NA)*t5 + c(empathic.thr5.g1, empathic.thr5.g2)*t5
empathic | c(NA, NA)*t6 + c(empathic.thr6.g1, empathic.thr6.g2)*t6
compassionate | c(NA, NA)*t1 + c(compassionate.thr1.g1, compassionate.thr1.g2)*t1
compassionate | c(NA, NA)*t2 + c(compassionate.thr2.g1, compassionate.thr2.g2)*t2
compassionate | c(NA, NA)*t3 + c(compassionate.thr3.g1, compassionate.thr3.g2)*t3
compassionate | c(NA, NA)*t4 + c(compassionate.thr4.g1, compassionate.thr4.g2)*t4
compassionate | c(NA, NA)*t5 + c(compassionate.thr5.g1, compassionate.thr5.g2)*t5
compassionate | c(NA, NA)*t6 + c(compassionate.thr6.g1, compassionate.thr6.g2)*t6
softhearted | c(NA, NA)*t1 + c(softhearted.thr1.g1, softhearted.thr1.g2)*t1
softhearted | c(NA, NA)*t2 + c(softhearted.thr2.g1, softhearted.thr2.g2)*t2
softhearted | c(NA, NA)*t3 + c(softhearted.thr3.g1, softhearted.thr3.g2)*t3
softhearted | c(NA, NA)*t4 + c(softhearted.thr4.g1, softhearted.thr4.g2)*t4
softhearted | c(NA, NA)*t5 + c(softhearted.thr5.g1, softhearted.thr5.g2)*t5
softhearted | c(NA, NA)*t6 + c(softhearted.thr6.g1, softhearted.thr6.g2)*t6
tender | c(NA, NA)*t1 + c(tender.thr1.g1, tender.thr1.g2)*t1
tender | c(NA, NA)*t2 + c(tender.thr2.g1, tender.thr2.g2)*t2
tender | c(NA, NA)*t3 + c(tender.thr3.g1, tender.thr3.g2)*t3
tender | c(NA, NA)*t4 + c(tender.thr4.g1, tender.thr4.g2)*t4
tender | c(NA, NA)*t5 + c(tender.thr5.g1, tender.thr5.g2)*t5
tender | c(NA, NA)*t6 + c(tender.thr6.g1, tender.thr6.g2)*t6


## INTERCEPTS:


sympathetic ~ c(0, 0)*1 + c(nu.1.g1, nu.1.g2)*1
empathic ~ c(0, 0)*1 + c(nu.2.g1, nu.2.g2)*1
compassionate ~ c(0, 0)*1 + c(nu.3.g1, nu.3.g2)*1
softhearted ~ c(0, 0)*1 + c(nu.4.g1, nu.4.g2)*1
tender ~ c(0, 0)*1 + c(nu.5.g1, nu.5.g2)*1


## SCALING FACTORS:


sympathetic ~*~ c(1, 1)*sympathetic
empathic ~*~ c(1, 1)*empathic
compassionate ~*~ c(1, 1)*compassionate
softhearted ~*~ c(1, 1)*softhearted
tender ~*~ c(1, 1)*tender




## LATENT MEANS/INTERCEPTS:


empathy ~ c(NA, NA)*1 + c(alpha.1.g1, alpha.1.g2)*1


## COMMON-FACTOR VARIANCES:


empathy ~~ c(NA, NA)*empathy + c(psi.1_1.g1, psi.1_1.g2)*empathy

As for my selection of the mark variable method: I want to use the marker variable method because I am unsure whether it is reasonable to assume that the latent factor variances are equal for the groups in my data. I can't get evidence for that without estimating more constrained versions of my model. But I can use anchor tests to figure out which items are noninvariant, keeping in mind that anchor tests are prone to false positives. So a data driven identification method seems to be the most principled way to go about making decisions, and the marker variable method appears to offer that. Maybe this bastardized mixture of IRT methods and CFA estimation doesn't make sense - if so, I'd be happy to learn of a better way to do things, though!  

Thanks again for your patience and help, all.

Best,
Thomas

[Terrence Jorgensen]

I also could not get the code that you linked in github to work 

# fix an arbitrary threshold of the specified marker variable to zero
UnidimModelUpdated <- update(UnidimModel,
                              change.syntax = c('sympathetic | 0*t1'))

It is a multiple group model, so make sure to pass the correct number of fixed parameter values (1 per group), e.g., 'sympathetic | c(0,0)*t1'

But the reason it didn't work was a silly bug I just found, where I switched the lhs and rhs of the parameter label (i.e., it was looking to update a parameter 't1 | sympathetic' rather than 'sympathetic | t1').  This is fixed in the latest semTools (0.5-2.925), so your code should work after installing and restarting. 

As for my selection of the mark variable method: I want to use the marker variable method because I am unsure whether it is reasonable to assume that the latent factor variances are equal for the groups in my data.

You don't assume that when you use the fixed-factor approach.  Those are arbitrary identification constraints, which get released once they are no longer needed (i.e., when loadings are constrained to equality).  The estimated variances are not comparable in a configural model just because a reference indicator is used, unless you can somehow verify that the reference indicator is indeed invariant.  But by doing that, you will have gone beyond the configural model anyway, and you can draw your inferences about differences in latent variance from a more restrictive model.

 

So a data driven identification method seems to be the most principled way to go about making decisions, and the marker variable method appears to offer that. 

Not unless you can verify it is invariant.  Look into Woods (2009) method for empirically selecting an anchor set.  It works pretty well, as long as the majority of items are invariant.  You only have 5 items, so fingers crossed.

https://doi.org/10.1177/0146621607314044

[Thomas McCauley]

But the reason it didn't work was a silly bug I just found, where I switched the lhs and rhs of the parameter label (i.e., it was looking to update a parameter 't1 | sympathetic' rather than 'sympathetic | t1').  This is fixed in the latest semTools (0.5-2.925), so your code should work after installing and restarting. 

That worked! I was able to estimate my model without a problem. 

You don't assume that when you use the fixed-factor approach.  Those are arbitrary identification constraints, which get released once they are no longer needed (i.e., when loadings are constrained to equality).  The estimated variances are not comparable in a configural model just because a reference indicator is used, unless you can somehow verify that the reference indicator is indeed invariant.  But by doing that, you will have gone beyond the configural model anyway, and you can draw your inferences about differences in latent variance from a more restrictive model.

Thanks very much for this clarification, and the link to the Woods (2009) paper. Although I'm glad to know how to use the marker variable method, my plan is now to standardize the common factor. I think the idea that the arbitrariness of choosing fixed factor or marker variables as identification constraints is probably explained in the technical details of Wu & Estabrook (2016), but if not you would you mind providing a citation? 

I'm veering into semnet territory with my last question, and if you think I should take it there I understand. If the constraints for identification are arbitrary then why do people make so much of a fuss about anchor items when they can just use common factor models? Is the idea that identifying anchors is optimal but difficult to do, or is anchoring just particularly important for IRT but not necessary for CFA? That doesn't really make sense to me since the two are so closely aligned, but I do find it curious that IRT heavily emphasizes the importance of anchors, but I rarely see anchoring mentioned in MI studies using CFA.

Thanks again for taking the time to answer my queries, Terrence.

Best,

Thomas 

[Nickname]

Thomas,
  Roger Millsap characterized measurement invarance as P(item | latent, group) = P(item | latent) which we can re-express as P[group1](item | latent) = P[group2](item | latent).  This latter expression emphasizes the need for referential consistency of 'latent' in both groups.  Or, if it is easier and more concrete, we can index these as latent1 and latent2 and then assert latent1 = latent2 as an assumption of the analysis.  Now suppose that your items all include the word 'quite' in the stem such that respondents in the US consistently interpret this as meaning very much and respondents in the UK consistently interpret this as meaning not very much.  You can use identification constraints to fix the scale of the latent variable in both groups.  You might even find that all the parameters are invariant across groups.  However, you have not actually met the criteria for measurement invariance because latent[US] =/= latent[UK].  You need at least some invariant items to serve as anchor items to assure that latent1 = latent2.  This holds true no matter whether you use IRT or CFA, and no matter whether you use unit variances or unit loadings.  I believe that unit variances in CFA correspond to standard practice of assuming a standard normal ability variable in IRT.  The arbitrary identification constraints merely remove overparameterization in the model to give the variable some determinate scale regardless of what it represents.  The anchor items serve a different purpose, to give the latent variables in different groups a common interpretation, that is, to provide a basis for the assumption that the latent variables represent the same thing in each group.  The choice of anchor items is not arbitrary because non-invariant anchor items would undermine that purpose.  However, anchor items can only do their work if one has already fixed the scales of the latent variables with arbitrary identification constraints.

[William M]

Thomas, Terrence, Keith:

This is an incredibly helpful/interesting thread. Keith, is it fair to interpret you as saying that once you have identified an invariant anchor item (using, e.g., AOAA), you do *not* need to constrain it across groups in the configural invariant and threshold invariant models, so long as you have identified the model one way or the other? And, just to make sure I really am grasping what you are saying, would you say my conclusions to the below hypothetical situations are cogent?

1) Say you have an anchor item that definitely represents the same construct in both groups, but it is nevertheless non-invariant (perhaps self-presentation bias is stronger in one group than the other, causing its intercept to be higher in the group more pre-occupied with appearing to land on the desirable pole of the construct). Metric and scalar invariance of the non-anchor items is thereby not assured evidence that latent group comparisons are unbiased because (a) the non-anchor items are valid to the extent that they have the same common cause as the anchor item, but (b) are non-invariant to the extent that the causal relationships between the construct and the response processes for the non-anchor items are similar to the response process for the anchor item.

2) Now imagine you have anchor item that is invariant, but has the "quite" confound that Keith mentioned above. Again,  metric and scalar invariance of the non-anchor items is not assured evidence of invariant latent group comparisons because you may be measuring different constructs in each group that just so happen to have the same parameter values.

3) AOAA, or some other procedure, indicates that several items could serve as anchors. You want to use one of them to identify the model using the marker variable method. To decide which should serve as the anchor, you could follow the literature's advice in using the item with the largest discrimination parameter. However, let's also imagine that your item set is biased away from the centroid of the construct (e.g., you are measuring a construct that is situation-general, but several of your items are about the same situation) such that the "gold standard" item (an indicator that is situation-general, and is definitely caused by the intended construct) does not have the highest discrimination parameter. In this case, it would still be best to use the gold standard item (provided that it's one of the invariant items) as the anchor.

Thanks in advance for any insight you can offer!

Will 

William H.B. McAuliffe, Ph.D.

Postdoctoral Research Fellow

Division on Addiction

Cambridge Health Alliance

Harvard Medical School Teaching Hospital

[Nickname]

William,
   We are now at some distance from the core focus of this list.  I will try to respond in a way that is both helpful and brief.

1. If anchor items are not invariant, then they are not good anchor items.  Using equality constraints is probably good practice but also should have little effect when the anchors are invariant and the samples are large.  It just makes the rest of the analysis cleaner.  If there is misspecification elsewhere in the model, this can propagate into free parameters in the well-specified portions of the model.

2. There is a bit of inconsistency in the standard terminology in that we distinguish constrained parameters from fixed parameters, but also refer to these collectively as model constraints.

3. Your numbered scenarios seem accurately characterized.  I think that it is best to think of testing measurement invariance as seeking disconfirming evidence.  The underlying hypothesis goes beyond the data.  However, tests of measurement invariance provide a fairly strong test nonetheless.  It would require and extraordinary set of coincidences for a population to fit the constraints of full measurement invariance when it was absent.  But it is not impossible.

4. There could be something that I am not aware of, but it seems unlikely that it is ever optimal to rely on a single anchor item.

5. Using the best discriminating items as anchors offers a general strategy (still requiring additional evidence that these are good anchors) but is not a requirement hat cannot be over-ridden by other considerations.

6. Measurement invariance studies cannot provide one-stop shopping for all forms of validity evidence.  Combining a measurement invariance study with other studies providing other forms of validity evidence can help reinforce the assumptions regarding what the items measure in each group.

  You might find Roger Millsap's book very helpful as a systematic development of the underlying issues.

[William M]

Keith,

Thank you for your thoughtful response. My apologies for veering away from lavaan issues in particular.  My understanding is that just one anchor will indeed lead to low power in detecting non-invariance in other items, but that having anchors of very high validity is a countervailing consideration that could lead one to justifiably use fewer than using the ~20% of indicators as anchors, a figure recommended by some authors. I will reflect more on these issues by consulting the Millsap book.

Cheers,

Will

--
You received this message because you are subscribed to a topic in the Google Groups "lavaan" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lavaan/9nvVozLbBng/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/0bd6028b-75e5-4169-8c87-0947eb55aed4o%40googlegroups.com.