chi-square difference tests for nested models with categorical variables

[sebasti...@gmail.com]

Hello, 

Has anyone done this kind of test using lavaan? As far as I know, it's not been implemented yet, but maybe there is an alternative way to do it, at least for now.

Any suggestion will be welcome. 

Best, 

Sebastian

[yrosseel]

On 03/11/2013 07:39 PM, sebasti...@gmail.com wrote:
> Hello,
> Has anyone done this kind of test using lavaan? As far as I know, it's
> not been implemented yet, but maybe there is an alternative way to do
> it, at least for now.

Why do you think it doesn't work? Did you get errors. If so, a
reproducible example would help. This works for me:

Data <- read.table("http://www.statmodel.com/usersguide/chap5/ex5.2.dat")
names(Data) <- c("u1","u2","u3","u4","u5","u6")
Data <- as.data.frame(lapply(Data, ordered))

model <- ' f1 =~ u1 + u2 + u3
f2 =~ u4 + u5 + u6 '

fit <- cfa(model, data=Data)
fit2 <- cfa(model, data=Data, orthogonal=TRUE)

anova(fit,fit2)

Scaled Chi Square Difference Test (test = scaled.shifted)

Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)
fit 8 2.4538
fit2 9 3.3715 0.18654 1 0.6658


Yves.

[sebasti...@gmail.com]

Thanks Yves. I just wanted to be sure that I was doing the right thing. I read that: 

The difference in chi-squared values for nested models estimated with WLSMV is not distributed as chi-squared.

In his syntax examples of the book "Confirmatory Factor Analysis for Applied Research", Jan Lammertyn pointed out that the difference test was not yet implemented in lavaan for categorical variables.

Data <- read.fwf("http://people.bu.edu/tabrown/Ch9/BINARY.DAT", width = c(1,1,1,1,1,1), n = 750)

names(Data) <- c(paste("y", 1:6, sep = ""))

model1 <- '

etoh =~ y1 + y2 + y3 + y4 + y5 + y6

'

fit1 <- cfa(model1, data = Data, ordered = names(Data))

summary(fit1, fit.measures = TRUE, standardized=TRUE)

model2 <- '

etoh =~ y1 + l1*y2 + l1*y3 + l1*y4 + l1*y5 + l1*y6

'

fit2 <- cfa(model2, data = Data, ordered = names(Data), mimic = "mplus")

summary(fit2, fit.measures = TRUE)

anova(fit1, fit2)

Scaled Chi Square Difference Test (test = scaled.shifted)

     Df AIC BIC   Chisq Chisq diff Df diff Pr(>Chisq)    

fit1  9          5.6512                                  

fit2 13         42.1617     28.238       4  1.116e-05 ***

---

Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

So, I guest that that chisq diff (28.238) is right, isn't?

In the book, using Mplus, it is reported: 

Chi-Square Test for Difference Testing Value 27.958 Degrees of Freedom 4** P-Value 0.0000

Best, 

[yrosseel]

On 03/11/2013 11:00 PM, sebasti...@gmail.com wrote:
> Thanks Yves. I just wanted to be sure that I was doing the right thing.
> I read that:
>
> The difference in chi-squared values for nested models estimated with
> WLSMV is not distributed as chi-squared.

True, but lavaan uses a 'scaled' difference test, using the (correct)
formulas from Satorra (2000). (reference below).

> In his syntax examples of the book "Confirmatory Factor Analysis for
> Applied Research", Jan Lammertyn pointed out that the difference test
> was not yet implemented in lavaan for categorical variables.

It is now. Note that the results reported in Brown are based on an older
version of Mplus. To get identical results for the two model fits, you
can use estimator="WLSMVS" (with S for Satterthwaite) and mimic="Mplus".
The latter will round-off the fractional degrees of freedom to integers.

> In the book, using Mplus, it is reported:
>
> Chi-Square Test for Difference Testing Value 27.958 Degrees of Freedom
> 4** P-Value 0.0000

Well, if estimator="WLSMVS" and mimic="Mplus" is used, anova(fit1 ,fit2)
gives

> anova(fit1, fit2)
Scaled Chi Square Difference Test (test = mean.var.adjusted)

Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)
fit1 9 5.6512

fit2 13 42.1617 26.199 3.7111 2.077e-05 ***


Still not the 27.958 repored by Mplus, and note the fractional degrees
of freedom. lavaan is using Satorra (2000). Mplus is using a slightly
different approach (described in webnote 10 on the Mplus website).
AFAIK, both approaches are correct (and one day, I will add a
mimic="Mplus" to the anova() function too to get the exact Mplus result).

Yves.



Satorra, A. (2000). Scaled and adjusted restricted tests in multi-sample
analysis of moment structures. In Innovations in Multivariate
Statistical Analysis: A Festschrift for Heinz Neudecker. Heijmans,
D.D.H., Pollock, D.S.G. and Satorra, A. (edts.), pp. 233-247, Kluwer
Academic Publishers, Dordrecht.
http://www.econ.upf.edu/docs/papers/downloads/395.pdf

[sebasti...@gmail.com]

Thank you! Nice and complete answer.

Sebastian

[Jan Lammertyn]

On Monday, March 11, 2013 11:00:01 PM UTC+1, sebasti...@gmail.com wrote:

Thanks Yves. I just wanted to be sure that I was doing the right thing. I read that: 

In his syntax examples of the book "Confirmatory Factor Analysis for Applied Research", Jan Lammertyn pointed out that the difference test was not yet implemented in lavaan for categorical variables.

I have updated the example:

https://github.com/janlammertyn/lavaan-material/tree/master/brown-cfa 

Jan

[Cathy Zhang]

Hi Yves, 

Thanks so much for these answers. This may be a stupid question but it seems that when we use the anova() to do the difference test, the DWLS results were used. Shouldn't we use the chi-square results in the "Robust" column? 

Cathy 

[Yves Rosseel]

On 12/19/2014 05:39 AM, Cathy Zhang wrote:
> Hi Yves,
>
> Thanks so much for these answers. This may be a stupid question but it
> seems that when we use the anova() to do the difference test, the DWLS
> results were used. Shouldn't we use the chi-square results in the
> "Robust" column?

> > anova(fit1, fit2)
> Scaled Chi Square Difference Test (test = mean.var.adjusted)
>
> Df AIC BIC Chisq Chisq diff Df diff Pr(>Chisq)
> fit1 9 5.6512
> fit2 13 42.1617 26.199 3.7111 2.077e-05 ***

No. It is not a difference of the 'robust' test statistics. The formulas
start from the original test statistics. Note that 42.1617 - 5.6512 is
not 26.199

Yves.

[Cathy Zhang]

Thank you very much for your reply! 

[Maria Usacheva]

Hi Yves,

I see that this is a very old thread, but I'll hope you might still be able to help.

I'm using the exact same anova test and it gives me this error.

Below is my code:
> fit1.1.cfaA <- cfa(model = Model1, data = data, estimator= "MLR", orthogonal = TRUE, missing = "FIML")

> fit1.1.cfaB <- cfa(model = Model1, data = data, estimator= "MLR", missing = "FIML")

> anova(fit1.1.cfaA, fit1.1.cfaB, SB.classic = TRUE)

Error in if (cd < 0) { : missing value where TRUE/FALSE needed". I tried to add SB.Classic but it didn't resolve the issue.

Would you be able to tell why it doesn't work for me?

Thank you so much!

Best,

Maria

[Terrence Jorgensen]

Would you be able to tell why it doesn't work for me?

He may need to see some data and an script that reproduces the problem in order to investigate it.

Terrence D. Jorgensen

Postdoctoral Researcher, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam

UvA web page: http://www.uva.nl/profile/t.d.jorgensen

[Yves Rosseel]

I would need more context. Can you show us the output of the two models?
Instead of 'anova()', try the lavTestLRT() function directly.

Yves.

> --
> You received this message because you are subscribed to the Google
> Groups "lavaan" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to lavaan+un...@googlegroups.com
> <mailto:lavaan+un...@googlegroups.com>.
> To post to this group, send email to lav...@googlegroups.com
> <mailto:lav...@googlegroups.com>.
> Visit this group at https://groups.google.com/group/lavaan.
> For more options, visit https://groups.google.com/d/optout.