semTools Pooled CFI and RMSEA Values Nonsensical

[Kara Styck]

Hello,

I have run into a problem that I'm hoping you can help me resolve. I'm getting pooled CFI and RMSEA values of 1.000 and 0.000, respectively, for several models even when the CFI and RMSEA values for each multiply imputed dataset are nowhere near the boundary values. Why is this occurring? Relatedly, what exactly is happening when pool.robust=T is specified?

For example: 

> fitMeasures(object=jsticsa.2factor_stO_fit, test=c("D2"), pool.robust=T) 

                 chisq                     df                 pvalue                   npar                 ntotal           chisq.scaled              df.scaled 

              1030.698                819.000                  0.000                168.000               1066.000                418.266                819.000 

         pvalue.scaled         baseline.chisq            baseline.df        baseline.pvalue  baseline.chisq.scaled     baseline.df.scaled baseline.pvalue.scaled 

                 1.000               9851.538                861.000                  0.000               8987.571                861.000                  0.000 

                   cfi             cfi.scaled                    rni             rni.scaled                   nnfi                    tli            nnfi.scaled 

                 0.976                  1.000                  0.976                  1.049                  0.975                  0.975                  1.052 

            tli.scaled                    rfi             rfi.scaled                    nfi             nfi.scaled                   pnfi            pnfi.scaled 

                 1.052                  0.890                  0.951                  0.895                  0.953                  0.852                  0.907 

                   ifi             ifi.scaled                    mfi                  rmsea         rmsea.ci.lower         rmsea.ci.upper           rmsea.pvalue 

                 0.977                  1.049                  0.905                  0.016                  0.012                  0.018                  1.000 

          rmsea.scaled  rmsea.ci.lower.scaled  rmsea.ci.upper.scaled    rmsea.pvalue.scaled               gammaHat            adjGammaHat        gammaHat.scaled 

                 0.000                  0.000                     NA                  1.000                  0.991                  0.990                  1.018 

    adjGammaHat.scaled                    rmr                   crmr                   srmr 

                 1.020                  0.000                  0.000                  0.000 

But... 

> sapply(jsticsa.2factor_stO_fit@funList, function(x) x$rmsea.scaled)

rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled 

   0.2158968    0.2259944    0.2278022    0.2132439    0.2121430    0.2259891    0.2206711    0.2193162    0.2159049    0.2112507    0.2209127    0.2162484 

rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled 

   0.2111190    0.2233463    0.2211086    0.2205104    0.2306268    0.2199048    0.2179947    0.2102073    0.2274641    0.2154875    0.2206737    0.2176311 

rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled rmsea.scaled 

   0.2132035    0.2091318    0.2115988    0.2214289    0.2072208    0.2225555 

> sapply(jsticsa.2factor_stO_fit@funList, function(x) x$cfi.scaled)

cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled 

 0.4634382  0.4208696  0.4088181  0.4825586  0.4738953  0.4222029  0.4500795  0.4643118  0.4704056  0.4819605  0.4501178  0.4601005  0.4773078  0.4386025 

cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled cfi.scaled 

 0.4345141  0.4492719  0.4009215  0.4566944  0.4613692  0.4731023  0.4253971  0.4696354  0.4373136  0.4546292  0.4749670  0.5076013  0.4752904  0.4325955 

cfi.scaled cfi.scaled 

 0.4846419  0.4419735 

[Edward Rigdon]

The scaled chi-square is less than DF. So your scaled noncentrality is 0, scaled CFI = 1 (almost always) and scaled RMSEA = 0 by formula. Your scaled chi square is much lower than the ML chi-square, which may mean that there is a lot of excess kurtosis in your data. Check multivariate kurtosis--I would expect it to be fairly extreme. If it is not, then you may need to dig a little deeper.

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To post to this group, send email to lav...@googlegroups.com.
Visit this group at https://groups.google.com/group/lavaan.
For more options, visit https://groups.google.com/d/optout.

[Terrence Jorgensen]

Why is this occurring?

See Ed's response (thanks Ed!)

what exactly is happening when pool.robust=T is specified?

When pool.robust=TRUE, the chisq.scaled value from each imputation is plugged into the D2 formula to pool the already-robust statistics.  When pool.robust=FALSE, the naïve chisq value from each imputation is plugged into the D2 formula to pool those values, then that pooled naïve statistic is robustified using the average chisq.scaling.factor across imputations.

?lavTestLRT.mi

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

http://www.uva.nl/profile/t.d.jorgensen

[Terrence Jorgensen]

CFI and RMSEA values for each multiply imputed dataset are nowhere near the boundary values

That is counterintuitive that each imputation indicates such undeniably poor fit.  You could check yourself that the pooled chi-squared is so different from each imputation:

(X2 <- sapply(jsticsa.2factor_stO_fit@funList, function(x) x$chisq.scaled))
calculate.D2(X2, DF = 819, asymptotic = TRUE)

[Kara Styck]

Thank-you! This makes perfect sense. Yes, the dataset is not multivariate normal. Responses are gathered on an ordinal scale with 4 ordered response categories. I used WLSMV and thought that would help, but it looks like I need to collapse categories. 

[Kara Styck]

Thank-you! I have a related question. I can extract each multiply imputed dataset from the mids object I created with the mice package into a data.frame and then collapse categories, but cfi.mi() requires a list. Can I force a data.frame into a list? Pardon me if this is a simple task. 

[Kara Styck]

Excellent, thank-you so much! 

[Terrence Jorgensen]

Can I force a data.frame into a list? 

It is the set of data.frames that need to be stored in a list.  If you are collapsing categories in a for-loop, you can store the updated data.frame in a list at the end of each iteration.

impList <- list()
for (i in 1:nImps) {
  temp <- complete(midsObject, i)
  # collapse categories here
 
  ## store in list
  impList[[i]] <- temp
}

Alternatively, you can use the mitml package's within() method:  

library(mitml)
?within.mitml.list

impList <- mids2mitml.list(midsObject)

impList2 <- within(impList, {
  # collapse categories here
})

[Kara Styck]

Thank-you! I realize this is not a stats help forum and I apologize for taking up more of your time, but this did not work. I'm attempting to locate outliers/aberrant response patterns (e.g., "flatliners") and rerunning the models without much success thus far. Collapsing categories and removing outliers/aberrant response patterns that I have identified has improved multivariate kurtosis somewhat, but is making multivariate skewness much worse. This is increasing my zero cell counts and wreaking havoc on the estimation of polychorics. Any hints on next steps? Things I can look at in lavaan? I've abandoned mice for the time being and I am working on the complete case dataset until I can figure out a workable solution. Thanks in advance for any help you are able to provide!