Modification indices

[David Kaplan]

Greetings all,

Hopefully a quick question about MIs before I spring lavaan on my class.  When running a path analysis with observed variables, I am not getting the EPC, but only the sepc.all and spec.nox.  Why am I not getting the EPC as well?  

Second, I can't find anything about the meaning of decision column.  For the model I am demonstrating, I only see the following: *** or (I), or epc associated with some, but not all fixed parameters.  What do these symbols mean and what criteria are being used to make these designations?

Thanks

David

[yrosseel]

On 02/04/2014 11:19 PM, David Kaplan wrote:
> Greetings all,
>
> Hopefully a quick question about MIs before I spring lavaan on my class.
> When running a path analysis with observed variables, I am not getting
> the EPC, but only the sepc.all and spec.nox. Why am I not getting the
> EPC as well?

I don't know. Can you show us the R script + output (and perhaps a
sample covariance matrix so we can reproduce it?).

> Second, I can't find anything about the meaning of decision column. For
> the model I am demonstrating, I only see the following: *** or (I), or
> epc associated with some, but not all fixed parameters. What do these
> symbols mean and what criteria are being used to make these designations?

Ah, you discovered the power=TRUE option. This was an attempt to mimic
the functionality of JruleMplus (see
https://github.com/daob/JruleMplus/wiki): to use both the EPC and the
power of the MI test to detect model misspecification.

I coded this a few years back, and it was not ready yet, because I could
(at the time) only provide unstandardized values for 'delta'
(representing the effect size). So consider this option as 'experimental'.

The 'flags' in the decision column are as follows:

- "***": significant MI and low power
- "epc": significant MI but also high power
- "(i)": non-significant MI and low power
- (nothing): non-significant MI and high power

in decreasing order of needing your attention...

Yves.

[dka...@education.wisc.edu]

Hi Yves,

Here is the program and a slice of the MI output.  I am please that you are using ideas from my old work  :-)

Kaplan, D (1990) Evaluating and modifying covariance structure models: A review and recommendation (with commentary).Multivariate Behavioral Research, 25, 137-155.

scimodel <- read.csv(file.choose(),header=T)

summary(scimodel)

## Science Achievement Example

## Kaplan (2009, pg. 15)

install.packages("lavaan")

require(lavaan)

pathmodel <- ' 

# regressions

sciach ~ scigrade10

scigrade10 ~ scigrade6+ses+certsci+understand+challenge

challenge~understand

understand ~ certsci

'

fit <- sem(pathmodel,data=scimodel)

summary(fit, fit.measures=TRUE)

modindices(fit,standardized=TRUE,power=TRUE,delta=0.1,alpha=.05,high.power=.80)

--------

          lhs op        rhs      mi sepc.all sepc.nox delta          ncp power decision

1      sciach ~~     sciach   0.000    0.000    0.000   0.1          NaN   NaN         

2      sciach ~~ scigrade10 764.680   -0.570   -0.570   0.1 2.040000e-01 0.074      ***

3      sciach ~~  challenge   1.053   -0.010   -0.010   0.1 2.009000e+00 0.294      (i)

4      sciach ~~ understand  15.548    0.043    0.043   0.1 1.334000e+00 0.211      ***

5      sciach ~~  scigrade6 198.227    0.160    0.907   0.1 2.410000e+00 0.342      ***

6      sciach ~~        ses 578.057    0.251    1.097   0.1 4.801000e+00 0.591      ***

7      sciach ~~    certsci   4.917   -0.024   -0.068   0.1 1.059500e+01 0.902      epc

8  scigrade10 ~~ scigrade10   0.000    0.000    0.000   0.1          NaN   NaN         

9  scigrade10 ~~  challenge      NA       NA       NA   0.1           NA    NA         

10 scigrade10 ~~ understand      NA       NA       NA   0.1           NA    NA         

[yrosseel]

On 02/05/2014 05:43 PM, dka...@education.wisc.edu wrote:
> Hi Yves,
>
> Here is the program and a slice of the MI output. I am please that you
> are using ideas from my old work :-)

Of course ;-)

> modindices(fit,standardized=TRUE,power=TRUE,delta=0.1,alpha=.05,high.power=.80)

It is the power=TRUE argument. Apparently, adding this argument removes
the epc and sepc.lv columns. I'm sure there was a reason for this, but I
do not remember. I will restore them in the next update.

If you just type

modindices(fit)

you do get the epc (and sepc.lv) columns.

Yves.

[Jan Henning-Kahmann]

Dear Yes,

I was wondering if the power=TRUE option of modindices () is still to be considered "experimental" as you stated in 2014?

My concern is that results from lavaan with mimic="Mplus" differ with respect to the modindices:

- the mi values are always lower than those from Mplus

- the epc values, however, are exactly the same!!

- again, NCP and power values are always lower than those from Mplus

Omitting mimic="Mplus" does not alter this differences.

As I want to use the approach by Saris and colleagues (2009) for my CFA model evaluation I am puzzled which results one can trust.

Can rely on JRule by D. Oberski (2009) with Mplus or e.g. miPowerFit () from semTools?

Any advice would be highly appreciated! :)

Kind regards,

Jan

[Yves Rosseel]

> I was wondering if the power=TRUE option of modindices () is still to be
> considered "experimental" as you stated in 2014?

No. Although the effect size (delta) still needs to be provided in an
unstandardized metric.

> - the mi values are always lower than those from Mplus

Are you using a robust estimator? Try using plain ML.

I am happy to have a look a this, but I need a reproducible example.

Yves.

[Jan Henning-Kahmann]

Dear Yves (sorry for the typo last time ;)

thanks for the fast reply, which I did not expect and therefore didn`t notice before I left my office for some weeks.

Yes, I am using a robust estimator, i.e. WLSMV, as I treat my items (5 point-Likert) as categorical.

Following your advice, I tried ML with an example dataset from:

Pendergast et al. (2017). Measurement equivalence: A non-technical primer on categorical multi-group confirmatory factor analysis in school psychology,
Journal of School Psychology, 60, 65–82 (http://dx.doi.org/10.1016/j.jsp.2016.11.002).

I used their example model (see Mplus input), which I wrote in lavaan as:

model <- ' F1 =~ ss1r + ss2 + ss3r + ss4r + ss5 + ss6r
           F2 =~ sa7 + sa8 + sa9 + sa10r + sa12

        ss1r ~~ ss3r
        ss1r ~~ ss4r
        ss1r ~~ ss6r
        ss3r ~~ ss4r
        ss3r ~~ ss6r
        ss4r ~~ ss6r
        ss6r ~~ sa10r'

(the correlated residuals are relevant for me as I also have some in my own data/model..)

To make it reproducible I attach the data and Mplus input file here.

From my first checks it seems that indeed all values (MI, EPC, power, NCP) are identical to Mplus with plain ML.

However, using MLR, small deviations are noticeable.

The stronger deviations arise with estimator WLS, again even more intense using WLSMV, which is the scenario I first encountered this issue.

Thus, it would be nice to have some explanation and/or solution for it that does also apply to categorical items (WLS/WLSMV) and not only to ML.

I`m curiuos about your "findings" for this case... ;)

Thx in advance & kind regards,

Jan

[Yves Rosseel]

Hello Jan,

My apologies for a late reply. I had a look at these differences, and
this is my hypothesis: lavaan is showing the MIs in the 'standard'
metric, while Mplus is showing the MIs in the 'corrected/robust' metric.
For example, when using WLSMV, all MIs seem to be scaled and shifted,
just like the 'robust' test statistic. But that is speculation. I could
not find any documentation about this on their website.

Of course, if you scale/shift all the MIs using the same shift and scale
parameters, you are only changing the 'metric', but the relative
ordering will remain the same.

Yves.

> --
> You received this message because you are subscribed to the Google
> Groups "lavaan" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to lavaan+un...@googlegroups.com
> <mailto:lavaan+un...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/lavaan/e766f36d-1d00-48a5-af73-7fef6aceac4do%40googlegroups.com
> <https://groups.google.com/d/msgid/lavaan/e766f36d-1d00-48a5-af73-7fef6aceac4do%40googlegroups.com?utm_medium=email&utm_source=footer>.

[jan.h...@ph-freiburg.de]

Hi Yves,

no problem as I also reply late. ;)

Thanks a lot for the clarification! In fact this seems to be in accord with a reply by Bengt Muthen stating that "MI for WLSMV also takes into account the correct asymptotic covariance matrix for the estimates."

However, now I have to decide on which 'metric' I consider as the appropriate for using 'the detection of misspecification’-procedure (Saris, Satorra, & van der Veld, 2009) since in lavaan the lower MI also result in lower NCP values and, more importantly, thus lower power (than in Mplus) which of course is not really desirable....

Nevertheless, thanks for taking the time and having a look at this!

Best,

Jan

[jan.h...@ph-freiburg.de]

Dear all,

as the Mplus discussion forum seems to be disabled at the moment (for whatever reason) and most of you are also quite familiar with Mplus too, I would like to post the following question here:

Still using modification indices (for WLSMV) in my CFA, I recognized that standardizing latent variables in the MODEL statement results in identical estimates under the MODEL RESULTS and STANDARDIZED MODEL RESULTS sections, which is in line with my expectations.
However, under MODEL MODIFICATION INDICES the StdYX E.P.C. differs from E.P.C. and/or Std E.P.C., but only for WITH Statements (ON/BY exhibit uniform EPCs).
Concerning correlated error terms (which I focus my analyses on) now I wonder which of these EPC values can or should be regarded as standardized?

I did not inspect this issue in other software like e.g. lavaan, but I would expect it to be identical as it seems to me rather a "fundamental" modeling/estimation/parameterization aspect than a software feature.

Thanks (again) in advance & kind regards,
Jan