Issue with pooled fit indices for longitudinal invariance model with multiply imputed data

[dist...@umn.edu]

Hi all, 

I am attempting to fit a series of models to test for longitudinal measurement invariance guided by a recent article by Liu et al. (2017; http://supp.apa.org/psycarticles/supplemental/met0000075/met0000075_supp.html). Everything looks fine when I fit the baseline model with either the raw data or a single imputed dataset, but things go awry when I attempt to pool my imputed data with cfa.mi() in semTools. The pooled RMSEA looks similar to that of the CFA with the raw data, but the TLI is negative and the CFI is zero... Any ideas about what might be going wrong with the pooling? Below is the syntax I am using the fit the baseline CFA with the imputed data. 

fit.baseline = cfa.mi(baseline.model, data = mice.imp,

                   ordered = c("fm21ags.1_r" , "fm22ags.1_r" , "fm23ags.1_r" , "fm24ags.1_r",

                   "fm25ags.1_r" , "fm26ags.1_r", "fm27ags.1_r" , "fm28ags.1_r",

                   "fm29ags.1_r" , "fm30ags.1_r", "fm21ags.2_r" , "fm22ags.2_r" , "fm23ags.2_r" , 

                   "fm24ags.2_r", "fm25ags.2_r" , "fm26ags.2_r", "fm27ags.2_r" , "fm28ags.2_r",

                   "fm29ags.2_r" , "fm30ags.2_r", "fm21ags.3_r" , "fm22ags.3_r" , "fm23ags.3_r" ,

                   "fm24ags.3_r","fm25ags.3_r" , "fm26ags.3_r", "fm27ags.3_r" , "fm28ags.3_r",

                   "fm29ags.3_r" , "fm30ags.3_r"), parameterization = "theta", estimator = "wlsmv", optim.method = "BFGS")

Thanks!

Rebecca

[Mauricio Garnier-Villarreal]

Rebecca

you need to use the anova function for .mi objects to ask for the fit indices. You can change the baseline model with yours like this

anova(fit_mi, indices=T, baseline.model=fit.baseline)

[Terrence Jorgensen]

 Below is the syntax I am using the fit the baseline CFA with the imputed data. 

fit.baseline = cfa.mi(baseline.model, data = mice.imp,

                   ordered = c("fm21ags.1_r" , "fm22ags.1_r" , "fm23ags.1_r" , "fm24ags.1_r",

                   "fm25ags.1_r" , "fm26ags.1_r", "fm27ags.1_r" , "fm28ags.1_r",

                   "fm29ags.1_r" , "fm30ags.1_r", "fm21ags.2_r" , "fm22ags.2_r" , "fm23ags.2_r" , 

                   "fm24ags.2_r", "fm25ags.2_r" , "fm26ags.2_r", "fm27ags.2_r" , "fm28ags.2_r",

                   "fm29ags.2_r" , "fm30ags.2_r", "fm21ags.3_r" , "fm22ags.3_r" , "fm23ags.3_r" ,

                   "fm24ags.3_r","fm25ags.3_r" , "fm26ags.3_r", "fm27ags.3_r" , "fm28ags.3_r",

                   "fm29ags.3_r" , "fm30ags.3_r"), parameterization = "theta", estimator = "wlsmv", optim.method = "BFGS")

That doesn't reveal anything about your model.  Assuming you are using semTools 0.5-0, it would be helpful to see your custom baseline model specification, as well as the full R syntax that shows how you apply it to retrieve fit indices for your hypothesized model.  FYI, in addition to anova(), you can also use fitMeasures() to return fit indices for lavaan.mi objects, although anova() gives you more control.

Terrence D. Jorgensen

Postdoctoral Researcher, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

UvA web page: http://www.uva.nl/profile/t.d.jorgensen

[Rebecca Distefano]

Below is the full syntax. Let me know if you have ideas for why the model looks okay (factor loadings and fit indices) with just one complete imputed dataset, but not when I try to use cfa.mi to pool the fit indices. 

#Baseline model

baseline.model = 'ASmom1 =~ NA*fm21ags.1_r + fm22ags.1_r + fm23ags.1_r + fm24ags.1_r + 1*fm25ags.1_r + fm26ags.1_r + fm27ags.1_r + fm28ags.1_r + fm29ags.1_r + fm30ags.1_r

ASmom2 =~ NA*fm21ags.2_r + fm22ags.2_r + fm23ags.2_r + fm24ags.2_r + 1*fm25ags.2_r + fm26ags.2_r + fm27ags.2_r + fm28ags.2_r + fm29ags.2_r + fm30ags.2_r

ASmom3 =~ NA*fm21ags.3_r + fm22ags.3_r + fm23ags.3_r + fm24ags.3_r + 1*fm25ags.3_r + fm26ags.3_r + fm27ags.3_r + fm28ags.3_r + fm29ags.3_r + fm30ags.3_r

ASmom1 ~~ ASmom1 + ASmom2 + ASmom3

ASmom2 ~~ ASmom2 + ASmom3

ASmom3 ~~ ASmom3

ASmom1 ~ 0*1

ASmom2 ~ 1

ASmom3 ~ 1

fm21ags.1_r | fm21ags.1_rt1*t1 + fm21agst2*t2 + fm21ags.1_rt3*t3

fm21ags.2_r | fm21ags.2_rt1*t1 + fm21agst2*t2 + fm21ags.2_rt3*t3

fm21ags.3_r | fm21ags.3_rt1*t1 + fm21agst2*t2 + fm21ags.3_rt3*t3

fm22ags.1_r | fm22ags.1_rt1*t1 + fm22agst2*t2 + fm22ags.1_rt3*t3

fm22ags.2_r | fm22ags.2_rt1*t1 + fm22agst2*t2 + fm22ags.2_rt3*t3

fm22ags.3_r | fm22ags.3_rt1*t1 + fm22agst2*t2 + fm22ags.3_rt3*t3

fm23ags.1_r | fm23ags.1_rt1*t1 + fm23agst2*t2 + fm23ags.1_rt3*t3

fm23ags.2_r | fm23ags.2_rt1*t1 + fm23agst2*t2 + fm23ags.2_rt3*t3

fm23ags.3_r | fm23ags.3_rt1*t1 + fm23agst2*t2 + fm23ags.3_rt3*t3

fm24ags.1_r | fm24ags.1_rt1*t1 + fm24agst2*t2 + fm24ags.1_rt3*t3

fm24ags.2_r | fm24ags.2_rt1*t1 + fm24agst2*t2 + fm24ags.2_rt3*t3

fm24ags.3_r | fm24ags.3_rt1*t1 + fm24agst2*t2 + fm24ags.3_rt3*t3

fm25ags.1_r | fm25ags.1_rt1*t1 + fm25agst2*t2 + fm25agst3*t3

fm25ags.2_r | fm25ags.2_rt1*t1 + fm25agst2*t2 + fm25agst3*t3

fm25ags.3_r | fm25ags.3_rt1*t1 + fm25agst2*t2 + fm25agst3*t3

fm26ags.1_r | fm26ags.1_rt1*t1 + fm26ags.1_rt2*t2 + fm26agst3*t3

fm26ags.2_r | fm26ags.2_rt1*t1 + fm26ags.2_rt2*t2 + fm26agst3*t3

fm26ags.3_r | fm26ags.3_rt1*t1 + fm26ags.3_rt2*t2 + fm26agst3*t3

fm27ags.1_r | fm27ags.1_rt1*t1 + fm27agst2*t2 + fm27ags.1_rt3*t3

fm27ags.2_r | fm27ags.2_rt1*t1 + fm27agst2*t2 + fm27ags.2_rt3*t3

fm27ags.3_r | fm27ags.3_rt1*t1 + fm27agst2*t2 + fm27ags.3_rt3*t3

fm28ags.1_r | fm28ags.1_rt1*t1 + fm28agst2*t2 + fm28ags.1_rt3*t3

fm28ags.2_r | fm28ags.2_rt1*t1 + fm28agst2*t2 + fm28ags.2_rt3*t3

fm28ags.3_r | fm28ags.3_rt1*t1 + fm28agst2*t2 + fm28ags.3_rt3*t3

fm29ags.1_r | fm29ags.1_rt1*t1 + fm29agst2*t2 + fm29ags.1_rt3*t3

fm29ags.2_r | fm29ags.2_rt1*t1 + fm29agst2*t2 + fm29ags.2_rt3*t3

fm29ags.3_r | fm29ags.3_rt1*t1 + fm29agst2*t2 + fm29ags.3_rt3*t3

fm30ags.1_r | fm30ags.1_rt1*t1 + fm30agst2*t2 + fm30ags.1_rt3*t3

fm30ags.2_r | fm30ags.2_rt1*t1 + fm30agst2*t2 + fm30ags.2_rt3*t3

fm30ags.3_r | fm30ags.3_rt1*t1 + fm30agst2*t2 + fm30ags.3_rt3*t3

fm21ags.1_r + fm22ags.1_r + fm23ags.1_r + fm24ags.1_r + fm25ags.1_r + fm26ags.1_r + fm27ags.1_r + fm28ags.1_r + fm29ags.1_r + fm30ags.1_r ~ 0*1

fm21ags.2_r + fm22ags.2_r + fm23ags.2_r + fm24ags.2_r + fm25ags.2_r + fm26ags.2_r + fm27ags.2_r + fm28ags.2_r + fm29ags.2_r + fm30ags.2_r ~ 0*1

fm21ags.3_r + fm22ags.3_r + fm23ags.3_r + fm24ags.3_r + fm25ags.3_r + fm26ags.3_r + fm27ags.3_r + fm28ags.3_r + fm29ags.3_r + fm30ags.3_r ~ 0*1

fm21ags.1_r ~~ 1*fm21ags.1_r

fm22ags.1_r ~~ 1*fm22ags.1_r

fm23ags.1_r ~~ 1*fm23ags.1_r

fm24ags.1_r ~~ 1*fm24ags.1_r

fm25ags.1_r ~~ 1*fm25ags.1_r

fm26ags.1_r ~~ 1*fm26ags.1_r

fm27ags.1_r ~~ 1*fm27ags.1_r

fm28ags.1_r ~~ 1*fm28ags.1_r

fm29ags.1_r ~~ 1*fm29ags.1_r

fm30ags.1_r ~~ 1*fm30ags.1_r

fm21ags.2_r ~~ NA*fm21ags.2_r

fm22ags.2_r ~~ NA*fm22ags.2_r

fm23ags.2_r ~~ NA*fm23ags.2_r

fm24ags.2_r ~~ NA*fm24ags.2_r

fm25ags.2_r ~~ NA*fm25ags.2_r

fm26ags.2_r ~~ NA*fm26ags.2_r

fm27ags.2_r ~~ NA*fm27ags.2_r

fm28ags.2_r ~~ NA*fm28ags.2_r

fm29ags.2_r ~~ NA*fm29ags.2_r

fm30ags.2_r ~~ NA*fm30ags.2_r

fm21ags.3_r ~~ NA*fm21ags.3_r

fm22ags.3_r ~~ NA*fm22ags.3_r

fm23ags.3_r ~~ NA*fm23ags.3_r

fm24ags.3_r ~~ NA*fm24ags.3_r

fm25ags.3_r ~~ NA*fm25ags.3_r

fm26ags.3_r ~~ NA*fm26ags.3_r

fm27ags.3_r ~~ NA*fm27ags.3_r

fm28ags.3_r ~~ NA*fm28ags.3_r

fm29ags.3_r ~~ NA*fm29ags.3_r

fm30ags.3_r ~~ NA*fm30ags.3_r

fm21ags.1_r ~~ fm21ags.2_r + fm21ags.3_r

fm21ags.2_r ~~ fm21ags.3_r

fm22ags.1_r ~~ fm22ags.2_r + fm22ags.3_r

fm22ags.2_r ~~ fm22ags.3_r

fm23ags.1_r ~~ fm23ags.2_r + fm23ags.3_r

fm23ags.2_r ~~ fm23ags.3_r

fm24ags.1_r ~~ fm24ags.2_r + fm24ags.3_r

fm24ags.2_r ~~ fm24ags.3_r

fm25ags.1_r ~~ fm25ags.2_r + fm25ags.3_r

fm25ags.2_r ~~ fm25ags.3_r

fm26ags.1_r ~~ fm26ags.2_r + fm26ags.3_r

fm26ags.2_r ~~ fm26ags.3_r

fm27ags.1_r ~~ fm27ags.2_r + fm27ags.3_r

fm27ags.2_r ~~ fm27ags.3_r

fm28ags.1_r ~~ fm28ags.2_r + fm28ags.3_r

fm28ags.2_r ~~ fm28ags.3_r

fm29ags.1_r ~~ fm29ags.2_r + fm29ags.3_r

fm29ags.2_r ~~ fm29ags.3_r

fm30ags.1_r ~~ fm30ags.2_r + fm30ags.3_r

fm30ags.2_r ~~ fm30ags.3_r'

fit.baseline = cfa.mi(baseline.model, data = mice.imp,

                   ordered = c("fm21ags.1_r" , "fm22ags.1_r" , "fm23ags.1_r" , "fm24ags.1_r",

                   "fm25ags.1_r" , "fm26ags.1_r", "fm27ags.1_r" , "fm28ags.1_r",

                   "fm29ags.1_r" , "fm30ags.1_r", "fm21ags.2_r" , "fm22ags.2_r" , "fm23ags.2_r" , 

                   "fm24ags.2_r", "fm25ags.2_r" , "fm26ags.2_r", "fm27ags.2_r" , "fm28ags.2_r",

                   "fm29ags.2_r" , "fm30ags.2_r", "fm21ags.3_r" , "fm22ags.3_r" , "fm23ags.3_r" ,

                   "fm24ags.3_r","fm25ags.3_r" , "fm26ags.3_r", "fm27ags.3_r" , "fm28ags.3_r",

                   "fm29ags.3_r" , "fm30ags.3_r"), parameterization = "theta", estimator = "wlsmv", optim.method = "BFGS")

summary(fit.baseline, standardized = TRUE)

fitMeasures(fit.baseline)

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+unsubscribe@googlegroups.com.
To post to this group, send email to lav...@googlegroups.com.
Visit this group at https://groups.google.com/group/lavaan.
For more options, visit https://groups.google.com/d/optout.

[Terrence Jorgensen]

Regarding my earlier post, I was confused by your calling the configural model "baseline.model", which is the name of the model used to calculate incremental fit indices like CFI & TLI.  By default, the baseline model will be an independence model (covariances are all fixed to zero), but you can pass a custom baseline model using the baseline.model= argument in fitMeasures().  Because your "baseline.model" is actually your hypothesized configural model, and you aren't specifying a custom baseline model, I might need access to your data to track down the problem.  I don't see any problems with your model specification in the syntax, despite how prone-to-human-error the Millsap & Tein (2004) & Liu et al. (2017) identification constraints are. 

First, though, you can see the fit indices from each imputed data set without running each imputation separately.  Just use the FUN= argument, as shown in the last example on the ?runMI help page.

fit.baseline <- cfa.mi(baseline.model, data = mice.imp,

                       ordered = c("fm21ags.1_r" , "fm22ags.1_r" , "fm23ags.1_r" , "fm24ags.1_r",
                                   "fm25ags.1_r" , "fm26ags.1_r", "fm27ags.1_r" , "fm28ags.1_r",
                                   "fm29ags.1_r" , "fm30ags.1_r", "fm21ags.2_r" , "fm22ags.2_r" , "fm23ags.2_r" ,
                                   "fm24ags.2_r", "fm25ags.2_r" , "fm26ags.2_r", "fm27ags.2_r" , "fm28ags.2_r",
                                   "fm29ags.2_r" , "fm30ags.2_r", "fm21ags.3_r" , "fm22ags.3_r" , "fm23ags.3_r" ,
                                   "fm24ags.3_r","fm25ags.3_r" , "fm26ags.3_r", "fm27ags.3_r" , "fm28ags.3_r",
                                   "fm29ags.3_r" , "fm30ags.3_r"),

                       parameterization = "theta", estimator = "wlsmv", optim.method = "BFGS",
                       FUN = function(fit) list(fitIndices = lavaan::fitMeasures(fit, c("cfi","tli"))))


sapply(fit.baseline@funList, function(x) x$fitIndices[["cfi"]])
sapply(fit.baseline@funList, function(x) x$fitIndices[["tli"]])

If the pooled indices are outside the sampling distribution of the indices from each imputation, something is possibly going on with the pooled chi-squared statistic from either the hypothesized configural model or the (actual) baseline model.  Feel free to send a sample of data that reproduces the problem to me in a private email, if you are able to.