Testing model fits (of the same model) of two samples against each other

[Stephan Frederic]

I have to different samples which i used to calculate models fits. I would like to compare the fits of both samples.

I thought i would be nice to use the Santorra-Bentler difference test. However, i do not get the p values due to same degrees of freedom (which is obvious if i use the same model). 

> lavTestLRT(fit_PP_MTMM, fit_PC_MTMM, method="satorra.bentler.2010")

Scaled Chi-Squared Difference Test (method = “satorra.bentler.2010”)

lavaan NOTE:

    The “Chisq” column contains standard test statistics, not the

    robust test that should be reported per model. A robust difference

    test is a function of two standard (not robust) statistics.

 

             Df   AIC   BIC  Chisq Chisq diff Df diff Pr(>Chisq)

fit_PP_MTMM 555 14163 14534 1152.4                              

fit_PC_MTMM 555 14582 14955 1404.5     252.07       0           

Warning message:

In lavTestLRT(fit_PP_MTMM, fit_PC_MTMM, method = "satorra.bentler.2010") :

  lavaan WARNING: some models have the same degrees of freedom

Am I doing something wrong here? Are there other options to test this in lavaan? 

[Yves Rosseel]

On 6/2/20 11:35 AM, 'Stephan Frederic' via lavaan wrote:
> I have to different samples which i used to calculate models fits. I
> would like to compare the fits of both samples.

lavTestLRT() is for comparing (nested) models using the same sample.

If the model is the same, but the samples are different, you need a
multigroup analysis.

Yves.

[Stephan Frederic]

Hi Yves, thank you for the very quick response.

For the group analysis i found this description: https://lavaan.ugent.be/tutorial/groups.html

However, it seems like this is just to test the model fit for each group which i already accomplished by making two different datasets.

I am interested in testing the two model fits against each other (just the way the lavTestLRT tests two model fits against each other).
In your example data this would mean: Is the X² of Grant-White significantly better than the X² of Pasteur?
 Is there a way to do this?

Cheers,

Stephan

[car...@web.de]

Without wanting to interfere :-) If you want to test configural invariance, you can do so using Terrence's equivalence test: https://rdrr.io/cran/semTools/man/permuteMeasEq.html

There are a very good publications on this subject.

Am 03.06.20, 09:25 schrieb 'Stephan Frederic' via lavaan <lav...@googlegroups.com>:

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/6a987134-03ed-4fd1-9fb7-181e0e49b3d8%40googlegroups.com.

[Stephan Frederic]

Hello ...car,

thank you for this suggestions.

I have tried some calculations with permuteMEasEq. However, i am not sure what is tested against what.

This was my best intent i think:

out.config <- permuteMeasEq(nPermute = 100, 

                            con = fit.config, 

                            param = "configural", 

                            AFIs = c("chisq","cfi","rmsea","srmr","aic") )

I expected to find the difference in the first line, which was not the case.

Für chisq and aic i got the sum of the two values of both groups. For cfi, rmsea, and srmr something different.

It appears to me that this does not test the value of group A against the value of group B, right?

 

Am Dienstag, 2. Juni 2020 11:35:54 UTC+2 schrieb Stephan Frederic:

[car...@web.de]

Take a look at this: https://www.researchgate.net/publication/319274851_Applying_Permutation_Tests_and_Multivariate_Modification_Indices_to_Configurally_Invariant_Models_That_Need_Respecification (especually ILLUSTRATIVE EXAMPLE ). From what I understand your question, this should address your problem pretty precisely. Best. Carl.

Am 03.06.20, 22:07 schrieb 'Stephan Frederic' via lavaan <lav...@googlegroups.com>:

--

You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/51c8b508-6d31-435f-96a6-e23d7d3305e0%40googlegroups.com.