Fit indices with WLSMV

users...@gmail.com

unread,

Jun 27, 2014, 6:54:23 AM6/27/14

to lav...@googlegroups.com

I fit a model with categorical variables using WLSMV estimator. How do I have to interpret CFI and TLI in the context of WLSMV (they works worse than DWLS estimates)? Do I have to consider the robust values preferable to the DWLS values? I was not able to find literature on this topic. Thanks in advance for your help!

---------------------------------------------------------------------------------------

lavaan (0.5-16) converged normally after 126 iterations

Number of observations 1147

Estimator DWLS Robust

Minimum Function Test Statistic 6734.312 5181.513

Degrees of freedom 1405 1405

P-value (Chi-square) 0.000 0.000

Scaling correction factor 1.525

Shift parameter 766.004

for simple second-order correction (Mplus variant)

Model test baseline model:

Minimum Function Test Statistic 45090.699 13797.213

Degrees of freedom 1485 1485

P-value 0.000 0.000

User model versus baseline model:

Comparative Fit Index (CFI) 0.878 0.693

Tucker-Lewis Index (TLI) 0.871 0.676

Root Mean Square Error of Approximation:

RMSEA 0.058 0.048

90 Percent Confidence Interval 0.056 0.059 0.047 0.050

P-value RMSEA <= 0.05 0.000 0.966

yrosseel

unread,

Jul 1, 2014, 4:04:53 AM7/1/14

to lav...@googlegroups.com

On 06/27/2014 12:54 PM, users...@gmail.com wrote:
> I fit a model withcategorical variables using WLSMV estimator. How do I

> have to interpret CFI and TLI in the context of WLSMV (they works worse
> than DWLS estimates)?

The reason is the baseline model. Note that the 'robust' test statistic
for the baseline model (13797.213) is much smaller than the naive test
statistic (45090.699). Since CFI/TLI is comparing the model test
statistic with this baseline test statistic, the CFI/TLI values are
rather low.

Do you get a similar pattern if you refit with test="mean.var.adjusted"?

Do I have to consider the robust values preferable
> to the DWLS values?

The robust values should be preferable.

I was not able to find literature on this topic.

True indeed.

Yves.

Message has been deleted

Colleen McClain

unread,

May 4, 2015, 9:04:51 AM5/4/15

to lav...@googlegroups.com

Hello-

I have a follow up question on this thread. Is the robust CFI still preferable over the DWLS CFI no matter how many categorical variables are included in the model? I am trying to assess exactly when the robust CFI needs to be considered rather than the naive statistic.

For example, I am estimating a measurement model with six latent variables measured by observed continuous variables, but then estimate paths from those latent variables to a (binary) categorical outcome. In this case, the robust CFI is .75, but the DWLS CFI is just over .9.

Thanks for any help you can provide! Happy to provide more details, syntax/output, etc. as well.

Best-

Colleen

Seongho Bae

unread,

May 5, 2015, 2:36:57 AM5/5/15

to lav...@googlegroups.com

There is no matter what do you put in.

But, I want to know your syntax.

--

Seongho

2015년 5월 4일 월요일 오후 10시 4분 51초 UTC+9, Colleen McClain 님의 말:

yrosseel

unread,

May 5, 2015, 3:21:06 AM5/5/15

to lav...@googlegroups.com

On 05/04/2015 04:14 AM, Colleen McClain wrote:
> For example, I am estimating a measurement model with six latent
> variables measured by observed continuous variables, but then estimate
> paths from those latent variables to a (binary) categorical outcome. In

> this case, the robust CFI is just over .9, but the DWLS CFI is .75.

I do not think you can go wrong with the 'Robust' column. You should
always you the 'Robust' column, if it is shown in the summary output.

Yves.

David Hügli

unread,

Dec 6, 2016, 4:14:17 AM12/6/16

to lavaan

Hi everyone,

any idea, why my robust chisquare is not available?

lavaan (0.5-21) converged normally after 73 iterations

Number of observations                          1837

Estimator                                       DWLS      Robust
Minimum Function Test Statistic              595.507          NA
Degrees of freedom                               186         186
P-value (Chi-square)                           0.000          NA
Scaling correction factor                                     NA
Shift parameter

for simple second-order correction (Mplus variant)

thx a lot!

Terrence Jorgensen

unread,

Dec 6, 2016, 3:24:05 PM12/6/16

to lavaan

any idea, why my robust chisquare is not available?
lavaan (0.5-21) converged normally after 73 iterations

No, but try updating to the latest lavaan software first (0.5-22 on CRAN, or 0.5-23 development version). If the problem persists, you may need to post your script and enough data to reproduce the problem.

Terrence D. Jorgensen

Postdoctoral Researcher, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

UvA web page: http://www.uva.nl/profile/t.d.jorgensen

David Hügli

unread,

Dec 14, 2016, 8:48:12 AM12/14/16

to lavaan

Now using 0.5-22 i have this issue:

ModellWls <- 'fac1H =~ var8 + var32 + var36 + var41

fac2H =~ var17 + var24 + var27

fac3H =~ var12 + var29 + var37

fac4H =~ var10 + var15 + var23 + var53

fac5H =~ var1 + var31 + var45

fac6H =~ var9 + var28 + var50

fac7H =~var6 + var22 + var33'

#making data ordinal

for (i in 1:ncol(mydata)) {

mydata[,i] <- ordered(mydata[,i])

}

fit_lavaan_wlsmv <- cfa(ModellWls, data = mydata, conditional.x = "default", estimator = "wlsmv")

summary(fit_lavaan_wlsmv, fit.measures = TRUE, modindices = TRUE)

The I get this:

lavaan (0.5-22) converged normally after 71 iterations

Number of observations 1837

Estimator DWLS Robust

Minimum Function Test Statistic 539.732 737.049

Degrees of freedom 209 209

P-value (Chi-square) 0.000 0.000

Scaling correction factor 0.804

Shift parameter 65.717

for simple second-order correction (Mplus variant)

Model test baseline model:

Minimum Function Test Statistic 76030.048 30362.375

Degrees of freedom 253 253

P-value 0.000 0.000

User model versus baseline model:

Comparative Fit Index (CFI) 0.996 0.982

Tucker-Lewis Index (TLI) 0.995 0.979

Robust Comparative Fit Index (CFI) NA

Robust Tucker-Lewis Index (TLI) NA

Root Mean Square Error of Approximation:

RMSEA 0.029 0.037

90 Percent Confidence Interval 0.026 0.032 0.034 0.040

P-value RMSEA <= 0.05 1.000 1.000

Robust RMSEA NA

90 Percent Confidence Interval NA NA

Standardized Root Mean Square Residual:

SRMR 0.029 0.029

Weighted Root Mean Square Residual:

WRMR 1.211 1.211

Why dont I get the robust fit-indizes? as i want to report the scaled chisq. i also want to report the robust indices.

Thanks a los!

David Hügli

unread,

Dec 14, 2016, 9:01:49 AM12/14/16

to lavaan

my variables got 6 categories each

Terrence Jorgensen

unread,

Dec 16, 2016, 5:30:22 AM12/16/16

to lavaan

Why dont I get the robust fit-indizes? as i want to report the scaled chisq. i also want to report the robust indices.

No one has figured out how to calculate population-consistent fit indices with scaled/shifted test statistics, so they aren't available with WLSMV. Only the naïve calculations are available, which simply use the scaled test statistics in the formulas for the unscaled CFI, RMSEA, etc.

http://lavaan.ugent.be/history/dot5.html\

David Hügli

unread,

Dec 16, 2016, 7:28:23 AM12/16/16

to lavaan

Oh well, apparently I wasn't aware of that. Thx Terrence! So if I report the Chisq, CFI & RMSEA, would you suggest the robust values (as I thought you did suggest in other threads) or the DWLS values (which seem to be slightly better, for whatever reason I dont understand).

David Hügli

unread,

Dec 16, 2016, 7:31:45 AM12/16/16

to lavaan

my data doesnt follow the normal distribution...which would imply to choose the robust indices, as I understand

Message has been deleted

Terrence Jorgensen

unread,

Dec 17, 2016, 6:27:07 PM12/17/16

to lavaan

So if I report the Chisq, CFI & RMSEA, would you suggest the robust values

The chi-squared test in the Robust column is more valid (it has closer to nominal Type I error rates). There isn't enough research to know whether the naïve or scaled fit indices have greater limitations, but those aren't tests anyway, so I wouldn't recommend giving too much credence to anyone's rules of thumb.

Anh Hua

unread,

Aug 2, 2019, 5:07:42 PM8/2/19

to lavaan

Hi all,

I recently ran a CFA with a likert scale measure and I've learned that, for ordinal data, WLSMV estimator is a viable option. I specified this estimator within lavaan using the following script:

fit.cfa.<- cfa(cfa, data=data, estimator="DWLS",se="robust.sem",test="scaled.shifted")

summary(fit.cfa.el, fit.measures=TRUE)

I have 4 questions for you. (1) Does my syntax look correct? I think I took this script based on Dr. Terrence D. Jorgensen's suggestion elsewhere, but i have no idea what "test=scaled.shifted" and se="robust.sem" exactly means. I looked at lavaan's website to see but I am still unclear. What am I doing when I specified test="scaled.shifted" and se=robust.sem?

My output looks very similar to this one below. Elsewhere, people recommend to report robust statistics even though robust CFI/TLI tend to be lower than DWLS statistics. Question #2: What are these statistics robust to and why are they preferred over the DWLS statistics?

Question #3: How should I interpret the shift parameter of 65.717, as shown in someone's output below?

Question #4: How should I interpret the p value of 1 for RMSEA for both DWLS and robust statistics? I have the same exact p value for rmsea in my own data.

This is my first time running a CFA with ordinal data. I have a lot to learn and read but I would greatly appreciate some answers to my guestions to help get me started. And if you have some literature that explains, for example, what shift parameter is, etc, that would be incredibly helpful to me.

Thank you and I look forward to learning more about CFAs!

Anh

Terrence Jorgensen

unread,

Aug 6, 2019, 11:58:52 AM8/6/19

to lavaan

Does my syntax look correct?

Yes

What am I doing when I specified test="scaled.shifted" and se=robust.sem?

Requesting a robust test and robust SEs, respectively. They are the default when setting estimator="WLSMV"

What are these statistics robust to and why are they preferred over the DWLS statistics?

DWLS for categorical indicators involves multi-step estimation. First the thresholds then polychoric correlations are estimated. Then those are treated as known values when fitting the model to the polychoric correlations, even though they are estimated rather than known. The robust correction accounts for that uncertainty in the first steps.

Question #3: How should I interpret the shift parameter of 65.717, as shown in someone's output below?

That is the amount that the naive statistic is shifted to the left (after multiplying by the scaling factor) relative to what a true chi-squared statistic should be. Thus, the naive stat is shifted that much to the right (after multiplying by the scaling factor).

Question #4: How should I interpret the p value of 1 for RMSEA for both DWLS and robust statistics?

Better to interpret the 90% CI for RMSEA.

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

http://www.uva.nl/profile/t.d.jorgensen

ANH HUA

unread,

Aug 6, 2019, 12:22:40 PM8/6/19

to lavaan

Hi Dr. Jorgensen,

Thank you so much for your helpful responses and for reassuring me that I did my analyses correctly. Just to follow up, I have a couple of questions for you.

(1) regarding your answer to my question #3, the shift parameter of 65.717 refers to how much the naive statistics is shifted to the right (or to the left?). I saw both directions mentioned in your responses and I am confused.

(2). Do you have any references that explain a bit more on the shift parameter? I just need a bit more context to understand your definition of what the shift parameter is, i.e, you wrote "naive statistic is shifted to the left (after multiplying by the scaling factor) relative to what a true chi-squared statistic should be".

Thank you again so much for your time and assistance!

Anh

Terrence Jorgensen

unread,

Aug 7, 2019, 7:09:30 AM8/7/19

to lavaan

(1) regarding your answer to my question #3, the shift parameter of 65.717 refers to how much the naive statistics is shifted to the right (or to the left?). I saw both directions mentioned in your responses and I am confused.

If something is shifted to the left relative to where it should be, you shift it to the right to correct it.

(2). Do you have any references that explain a bit more on the shift parameter? I just need a bit more context to understand your definition of what the shift parameter is, i.e, you wrote "naive statistic is shifted to the left (after multiplying by the scaling factor) relative to what a true chi-squared statistic should be".

You can read the references on the ?lavTestLRT help page, as well as this tutorial:

https://doi.org/10.1080/10705511.2013.824793

But they don't really "focus" on the shift parameter per se. It's just a derived quantity that is used to obtain a robust test statistic, it is not necessary to interpret it directly when you report your results.

Alexandre Jaloto

unread,

Jul 3, 2020, 9:35:45 AM7/3/20

to lavaan

Hi everyone!

Still on this topic, I want to share a question. When I run this code, my robust RMSEA is not available. Do I have to use the scaled RMSEA in a situation like this? What is the difference between them? The data is attached.

Thanks a lot!

Alexandre Jaloto

data = read.table('data_SEM.csv', sep = ',', header = FALSE)

names (data) = paste0 ('i', 1:10)

sem.mod = '

lat1 =~ i1 + i2 + i3 + i4 + i5

lat2 =~ i6 + i7 + i8 + i9 + i10

lat1 ~~ lat2

'

sem = sem (sem.mod, data, ordered = names (data))

summary(sem, standardized = TRUE)

fitMeasures (sem)

data_SEM.csv

Patrick (Malone Quantitative)

unread,

Jul 3, 2020, 10:14:48 AM7/3/20

to lav...@googlegroups.com

Alexandre,

As far as I know, there's no analog of the scaled maximum likelihood
fit statistic for non-ML estimators.

Pat

> --
> You received this message because you are subscribed to the Google Groups "lavaan" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/05cfb021-d735-4add-a463-99c1e9a6d752o%40googlegroups.com.

--
Patrick S. Malone, Ph.D., Malone Quantitative
NEW Service Models: http://malonequantitative.com

He/Him/His

Alexandre Jaloto

unread,

Jul 3, 2020, 3:40:32 PM7/3/20

to lav...@googlegroups.com

Thanks, Patrick!

I was comparing my results in lavaan and Mplus. I noticed that Mplus does not specify which indices are presented. In lavaan, we have three (also the .robust and .scaled ones). Which should I report with ordinal variables? In this case, for example, rmsea or rmsea.scaled? chisq or chisq.scaled? cfi or cfi.scaled?

> fitMeasures (sem)
npar fmin
40.000 0.033
chisq df
19.530 34.000
pvalue chisq.scaled
0.978 37.693
df.scaled pvalue.scaled
34.000 0.304
chisq.scaling.factor baseline.chisq
0.589 4372.334
baseline.df baseline.pvalue
45.000 0.000
baseline.chisq.scaled baseline.df.scaled
2730.656 45.000
baseline.pvalue.scaled baseline.chisq.scaling.factor
0.000 1.611
cfi tli
1.000 1.004
nnfi rfi
1.004 0.994
nfi pnfi
0.996 0.752
ifi rni
1.003 1.003
cfi.scaled tli.scaled
0.999 0.998
cfi.robust tli.robust
NA NA
nnfi.scaled nnfi.robust
0.998 NA
rfi.scaled nfi.scaled
0.982 0.986
ifi.scaled rni.scaled
0.999 0.999
rni.robust rmsea
NA 0.000
rmsea.ci.lower rmsea.ci.upper
0.000 0.000
rmsea.pvalue rmsea.scaled
1.000 0.019
rmsea.ci.lower.scaled rmsea.ci.upper.scaled
0.000 0.047
rmsea.pvalue.scaled rmsea.robust
0.967 NA
rmsea.ci.lower.robust rmsea.ci.upper.robust
0.000 NA
rmsea.pvalue.robust rmr
NA 0.037
rmr_nomean srmr
0.041 0.041
srmr_bentler srmr_bentler_nomean
0.037 0.041
crmr crmr_nomean
0.041 0.045
srmr_mplus srmr_mplus_nomean
NA NA
cn_05 cn_01
745.079 859.265
gfi agfi
0.997 0.993
pgfi mfi
0.458 1.024

Best regards

Alexandre Jaloto

You received this message because you are subscribed to a topic in the Google Groups "lavaan" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lavaan/7_rL3eEfb4U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAJc%3DyOHBW_rbnbDVVJ7YEU4jmL%2BchEF7fNQYshD%3DYj0_atA_LA%40mail.gmail.com.

Patrick (Malone Quantitative)

unread,

Jul 3, 2020, 3:46:49 PM7/3/20

to lav...@googlegroups.com

Does Mplus have any notifications by the chi-square? That should
indicate (though possibly not clearly) that it's a scaled test
statistic.

> To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAF27OARXp53auaVxa4fAXnDLOx5Gs-APW-2zh4rM6ckg2Djqxw%40mail.gmail.com.

Alexandre Jaloto

unread,

Jul 3, 2020, 6:09:49 PM7/3/20

to lav...@googlegroups.com

Hi Patrick, thanks again!

No, I don't think Mplus has the notifications. I suppose its indices are the robust (or scaled) ones. They are very similar to lavaan's scaled. My aim here is to understand which one in lavaan I should use. Also, to understand the difference between them.

In Mplus I got this:

MODEL FIT INFORMATION

Number of Free Parameters 40

Chi-Square Test of Model Fit

Value 37.804*
Degrees of Freedom 34
P-Value 0.2997

* The chi-square value for MLM, MLMV, MLR, ULSMV, WLSM and WLSMV cannot be used
for chi-square difference testing in the regular way. MLM, MLR and WLSM
chi-square difference testing is described on the Mplus website. MLMV, WLSMV,
and ULSMV difference testing is done using the DIFFTEST option.

RMSEA (Root Mean Square Error Of Approximation)

Estimate 0.019
90 Percent C.I. 0.000 0.048
Probability RMSEA <= .05 0.966

CFI/TLI

CFI 0.999
TLI 0.998

Chi-Square Test of Model Fit for the Baseline Model

Value 2739.734
Degrees of Freedom 45
P-Value 0.0000

WRMR (Weighted Root Mean Square Residual)

Value 0.515

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAJc%3DyOFQ8JyKoPYP4_eLT93obJfELY1%2BtcEq6RJr%2BNZTwvAO9w%40mail.gmail.com.

Patrick (Malone Quantitative)

unread,

Jul 3, 2020, 6:35:12 PM7/3/20

to lav...@googlegroups.com

The "*" is the notification I meant. That's telling you, though not in
so many words, that is a scaled test statistic.

On Fri, Jul 3, 2020 at 6:09 PM Alexandre Jaloto

> To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAF27OAQeTckKuAFPsfwohFywBP1v7T5H0NauuqNpVTch78obzw%40mail.gmail.com.

Alexandre Jaloto

unread,

Jul 3, 2020, 9:58:49 PM7/3/20

to lav...@googlegroups.com

Ok! Thanks!

:)

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAJc%3DyOGWcRewVUot1DJzXoWSG%3DyQsJ--U9aHApo4exi-Dk_k%2BA%40mail.gmail.com.

Terrence Jorgensen

unread,

Jul 4, 2020, 3:30:09 PM7/4/20

to lavaan

robust RMSEA is not available. Do I have to use the scaled RMSEA in a situation like this? What is the difference between them?

rmsea.robust was not proposed for variance-adjusted test statistics:

https://doi.org/10.1080/00273171.2012.715252

Their follow-up article suggests that a similar extension for DWLS would not be logical:

https://doi.org/10.1080/00273171.2018.1455142

Alexandre Jaloto

unread,

Jul 4, 2020, 8:15:17 PM7/4/20

to lav...@googlegroups.com

Thanks, Terrence!

--
You received this message because you are subscribed to a topic in the Google Groups "lavaan" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/lavaan/7_rL3eEfb4U/unsubscribe.
To unsubscribe from this group and all its topics, send an email to lavaan+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/e5393312-59db-4582-b048-28d821c1b673o%40googlegroups.com.

Reply all

Reply to author

Forward