Different modification indeces for r-lavaan and Mplus

49 views
Skip to first unread message

Manolo Cabran

unread,
Aug 14, 2018, 2:48:55 AM8/14/18
to lavaan
Hi,
this is a copy of what I posted on StackOverflow, for which, as of today 14 August 2018, I have not received any answer.

I am attending a class on Confirmatory Factor Analysis `CFA' with `MPlus`; however, I am very keen in learning how to do it with `R-lavaan`. I have run a model with `MPLUS` and the same model with `R lavaan`. The model has three latent variables and ten indicators. All parameters are set as free. The model estimation terminated normally in both `R` and `MPlus`. The `R` code, the `MPlus` input file and the `.dat` files are available in this GitHub repository.

When I look at the outputs, everything looks the same. In most cases, the values are the same; when they are not, they differ to the second or third decimal. This is true for:

 - Number of groups 
 - Number of observations
 - Number of dependent, independent and latent variables
 - Number of missing data patterns
 - Number of Free Parameters
 - Loglikelihood
 - Information Criteria (Chi-Square Test of Model Fit,
   RMSEA, CFI/TLI, Chi-Square Test of Model Fit for the Baseline Model,
   and SRMR)
 - All model results (e.g. latent and independent variables,
   both unstandardized and fully standardized)  
 - R-SQUARE

One important difference is the estimator: in `MPlus` is `ML`, in `R` is `FIML`. I first started with `ML` in both, but in `R` I got all estimates different from `MPlus`. I guess this is due to the different way `R` and `MPlus` handle `NAs`.

However, when I look at the modification indexes, `MPlus` has many more indexes than `R` and the values are significantly different.  If I fix the minimum value of the modification indexes in both `R` and MPlus at 10, in `R` I get 8 indexes, in MPlus I get 29. 

This is what I get from MPlus:

    ON/BY Statements
    
    IPEQOPT  ON TRACO    /
    TRACO    BY IPEQOPT               25.739    -0.337     -0.210       -0.226
    IMPENV   ON TRACO    /
    TRACO    BY IMPENV                33.457     0.446      0.278        0.295
    IMPENV   ON ATTI     /
    ATTI     BY IMPENV                24.846    -0.203     -0.126       -0.133
    
    ON Statements
    
    TRACO    ON IPEQOPT               22.308    -0.131     -0.210       -0.195
    TRACO    ON IMPENV                19.577     0.168      0.270        0.254
    ATTI     ON IMPENV                19.133    -0.117     -0.189       -0.178
    UNIV     ON IPEQOPT               16.226     0.133      0.292        0.272
    IMPTRAD  ON IMSMETN               10.895    -0.112     -0.112       -0.070
    IMPTRAD  ON IPUDRST               12.675    -0.106     -0.106       -0.081
    IMPTRAD  ON IMPENV                31.976     0.176      0.176        0.133
    IPEQOPT  ON IMPTRAD               22.316    -0.089     -0.089       -0.120
    IPEQOPT  ON IPBHPRP               15.872    -0.098     -0.098       -0.119
    IPUDRST  ON IMPTRAD               15.505    -0.076     -0.076       -0.100
    IPUDRST  ON IMDFETN               10.021     0.090      0.090        0.077
    IPUDRST  ON IMPENV                16.234    -0.197     -0.197       -0.194
    IMPENV   ON IMPTRAD               68.801     0.161      0.161        0.214
    IMPENV   ON IMSMETN               15.721    -0.114     -0.114       -0.095
    IMPENV   ON IMDFETN               23.385    -0.144     -0.144       -0.124
    IMPENV   ON IMPCNTR               12.216    -0.091     -0.091       -0.083
    IMPENV   ON IPUDRST               16.231    -0.170     -0.170       -0.172
    
    WITH Statements
    
    IPEQOPT  WITH TRACO               22.309    -0.086     -0.138       -0.170
    IPEQOPT  WITH UNIV                16.228     0.088      0.193        0.237
    IPEQOPT  WITH IMPTRAD             12.171    -0.080     -0.080       -0.094
    IPUDRST  WITH IPMODST             11.775     0.085      0.085        0.091
    IPUDRST  WITH IMPTRAD             16.337    -0.094     -0.094       -0.110
    IMPENV   WITH TRACO               19.576     0.098      0.157        0.206
    IMPENV   WITH ATTI                19.135    -0.068     -0.110       -0.144
    IMPENV   WITH IMPTRAD             52.625     0.166      0.166        0.209
    IMPENV   WITH IPUDRST             16.230    -0.114     -0.114       -0.183

And this is what I get from `R`:

    modificationindices(cfa4, minimum.value = 10)

    > modificationindices(cfa4, minimum.value = 10)
            lhs op     rhs     mi    epc sepc.lv sepc.all sepc.nox
    43    TraCo =~ ipeqopt 25.742 -0.337  -0.210   -0.226   -0.226
    45    TraCo =~  impenv 33.458  0.446   0.278    0.295    0.295
    52     Atti =~  impenv 24.847 -0.203  -0.126   -0.133   -0.133
    67  ipmodst ~~ ipudrst 11.776  0.085   0.085    0.091    0.091
    74  imptrad ~~ ipeqopt 12.171 -0.080  -0.080   -0.094   -0.094
    75  imptrad ~~ ipudrst 16.336 -0.094  -0.094   -0.110   -0.110
    76  imptrad ~~  impenv 52.626  0.166   0.166    0.209    0.209
    104 ipudrst ~~  impenv 16.227 -0.114  -0.114   -0.183   -0.183


If I lower the minimum value in `R` to 4, I have 21 indexes

    modificationindices(cfa4, minimum.value = 4)
    
    > modificationindices(cfa4, minimum.value = 4)
            lhs op     rhs     mi    epc sepc.lv sepc.all sepc.nox
    43    TraCo =~ ipeqopt 25.742 -0.337  -0.210   -0.226   -0.226
    45    TraCo =~  impenv 33.458  0.446   0.278    0.295    0.295
    47     Atti =~ imptrad  6.279 -0.114  -0.071   -0.056   -0.056
    49     Atti =~ ipbhprp  4.088  0.085   0.052    0.046    0.046
    50     Atti =~ ipeqopt  6.187  0.094   0.058    0.062    0.062
    51     Atti =~ ipudrst  9.056  0.117   0.072    0.076    0.076
    52     Atti =~  impenv 24.847 -0.203  -0.126   -0.133   -0.133
    57     Univ =~ imsmetn  7.344 -0.094  -0.043   -0.054   -0.054
    59     Univ =~ impcntr  4.496  0.084   0.038    0.045    0.045
    65  ipmodst ~~ impcntr  7.733  0.046   0.046    0.071    0.071
    67  ipmodst ~~ ipudrst 11.776  0.085   0.085    0.091    0.091
    71  imptrad ~~ imsmetn  5.735 -0.032  -0.032   -0.063   -0.063
    74  imptrad ~~ ipeqopt 12.171 -0.080  -0.080   -0.094   -0.094
    75  imptrad ~~ ipudrst 16.336 -0.094  -0.094   -0.110   -0.110
    76  imptrad ~~  impenv 52.626  0.166   0.166    0.209    0.209
    86  ipbhprp ~~ impcntr  6.967 -0.034  -0.034   -0.079   -0.079
    87  ipbhprp ~~ ipeqopt  4.281 -0.041  -0.041   -0.067   -0.067
    97  imdfetn ~~ ipudrst  5.444  0.023   0.023    0.114    0.114
    99  impcntr ~~ ipeqopt  4.448  0.025   0.025    0.055    0.055
    102 ipeqopt ~~ ipudrst  8.411  0.068   0.068    0.102    0.102
    104 ipudrst ~~  impenv 16.227 -0.114  -0.114   -0.183   -0.183

Of these 21, only three values are the same (to the second/third decimal) in both `R-lavaan` and MPlus outputs, and these are the ON/BY STATEMENTS in MPlus. Of the 7 that are in common in both `R` and MPlus, only one `IPUDRST  ON IMPENV`                 has the same value (again, to the second/third decimal), the other six `TRACO ON IPEQOPT, TRACO ON IMPENV, ATTI ON IMPENV, IMPTRAD ON IMSMETN, IMPTRAD  ON IPUDRST, IMPTRAD  ON IMPENV` are completely off. The remaining are not even in common.

Any suggestion on what could be actually be wrong in my code? Can the two algorithms estimating modification indexes be so different in `R` and in `MPlus` to give such output? What else?

Thanks

Manolo

*Update 11 August 2018*

I encountered the same problem when testing for invariance. When I tested for invariances in `R`, I got the same results as in `MPlus`. In both `R` and `MPlus`, the modification indexes for configural and metric invariance are under 10, so they are not reported. However, for scalar invariance, I have ten modification indexes in `MPlus` that are above the value of ten, while in `R` I do not have any modification index.

Terrence Jorgensen

unread,
Aug 14, 2018, 6:37:23 AM8/14/18
to lavaan
One important difference is the estimator: in `MPlus` is `ML`, in `R` is `FIML`.

You are confusing the estimator with the method for handling missing data.  And you mean `lavaan`, which is a specific software package that is written for the R environment.  There are other SEM packages for R, including `sem`, `OpenMx`, and `nlsem`.

I first started with `ML` in both, but in `R` I got all estimates different from `MPlus`. I guess this is due to the different way `R` and `MPlus` handle `NAs`.

Mplus uses FIML to handle missing values (for continuous data only) by default, whereas in lavaan you need to set missing = "fiml"

`MPlus` has many more indexes than `R`

I think lavaan checks whether a latent variable is exogenous or endogenous, and it only calculates modification indices that would make sense within the context of the fitted model (i.e., it will not suggest a factor should be regressed on another variable if it is exogenous).  Mplus suggests lots of extra parameters that make no sense in a CFA, like regressing a factor on an indicator of another factor.

and the values are significantly different... Can the two algorithms estimating modification indexes be so different in `R` and in `MPlus` to give such output?

The algorithm for calculating Lagrange multipliers (modification indices are a special case of a 1-df statistics) should be the same, but the optimizers are different.  Mplus uses their own, and lavaan uses ones available in R (e.g., ?nlminb).  Lagrange multipliers are calculated from the first and second derivatives of the parameter estimates, so they can differ to the degree that the different optimizers find different solutions.

I encountered the same problem when testing for invariance. When I tested for invariances in `R`, I got the same results as in `MPlus`. In both `R` and `MPlus`, the modification indexes for configural and metric invariance are under 10, so they are not reported. However, for scalar invariance, I have ten modification indexes in `MPlus` that are above the value of ten, while in `R` I do not have any modification index.

Mplus provides a 1-df modification index for each equality-constrained parameter, which are collectively redundant because a single equality constraint on 2 parameters only results in 1 df.  lavaan does not provide modification indices for any estimated parameters, only for fixed parameters.  But the lavTestScore() function provides a way to test whether equality constraints should be released.


Terrence D. Jorgensen
Postdoctoral Researcher, Methods and Statistics
Research Institute for Child Development and Education, the University of Amsterdam

Reply all
Reply to author
Forward
0 new messages