Is the use of DWLS estimator for not normal indicators producing overfitting?
150 views
Skip to first unread message
Fran
Aug 23, 2019, 5:52:33 PM8/23/19
to lavaan
Hi all,
I have a question that is, maybe, more a statistical question rather than lavaan-related but I would need the opinion from those who are familiar with lavaan output and functions.
I have a sem model with non-normal distributed indicators measured with Likert-scale (7 points - agreement). I have 4 LVs and 1 observed (N=287, data collection still in progress). Also, the indicators of the outcome variable (treated as latent in the sem model) are particularly skewed (I expected that from previous literature). To my knowledge, it's common to analyse ordinal variables as if they were continuous but only if they are normally distributed. As variables in my models are not normally distributed, I specified (before running the sem model) that the variables have to be ordered:
Then, I run the sem model and Lavaan automatically switched to a diagonally weighted estimator (DWLS). All my hypotheses have been confirmed (regressions' pvalues between 0.000 and 0.026), cfi is really high (0.987). When I worked with MLM/MLR estimators, with different datasets, honestly, it never happened that everything went so smoothly. So I am concerned that I am not able to read the sem output correctly and I would like to ask you:
My doubt is that the sample size is still too small. However, I am going to show these preliminary data soon, in the next days, and I would like to understand if I can trust these results.
Thank you very much,
Fran
I have a question that is, maybe, more a statistical question rather than lavaan-related but I would need the opinion from those who are familiar with lavaan output and functions.
I have a sem model with non-normal distributed indicators measured with Likert-scale (7 points - agreement). I have 4 LVs and 1 observed (N=287, data collection still in progress). Also, the indicators of the outcome variable (treated as latent in the sem model) are particularly skewed (I expected that from previous literature). To my knowledge, it's common to analyse ordinal variables as if they were continuous but only if they are normally distributed. As variables in my models are not normally distributed, I specified (before running the sem model) that the variables have to be ordered:
df %<>% mutate_at(.funs = list(~ordered(.)), .vars = vars(out_1, out_2, ...))Then, I run the sem model and Lavaan automatically switched to a diagonally weighted estimator (DWLS). All my hypotheses have been confirmed (regressions' pvalues between 0.000 and 0.026), cfi is really high (0.987). When I worked with MLM/MLR estimators, with different datasets, honestly, it never happened that everything went so smoothly. So I am concerned that I am not able to read the sem output correctly and I would like to ask you:
1) from the output below, it there something (some indices) indicating
that the model is doing something wrong (e.g, overfitting?);
My doubt is that the sample size is still too small. However, I am going to show these preliminary data soon, in the next days, and I would like to understand if I can trust these results.
> summary(fit, fit.measures= TRUE, ci= TRUE, standardized = TRUE) lavaan 0.6-4 ended normally after 45 iterations
Optimization method NLMINB Number of free parameters 118
Number of observations 287
Estimator DWLS Robust Model Fit Test Statistic 631.682 466.955 Degrees of freedom 114 114 P-value (Chi-square) 0.000 0.000 Scaling correction factor 1.580 Shift parameter 67.282 for simple second-order correction (Mplus variant)
Model test baseline model:
Minimum Function Test Statistic 69433.935 25525.694 Degrees of freedom 120 120 P-value 0.000 0.000
User model versus baseline model:
Comparative Fit Index (CFI) 0.993 0.986 Tucker-Lewis Index (TLI) 0.992 0.985
Robust Comparative Fit Index (CFI) NA Robust Tucker-Lewis Index (TLI) NA
Root Mean Square Error of Approximation:
RMSEA 0.126 0.104 90 Percent Confidence Interval 0.117 0.136 0.094 0.114 P-value RMSEA <= 0.05 0.000 0.000
Robust RMSEA NA 90 Percent Confidence Interval NA NA
Standardized Root Mean Square Residual:
SRMR 0.107 0.107
Parameter Estimates:
Information Expected Information saturated (h1) model Unstructured Standard Errors Robust.sem
Latent Variables: Estimate Std.Err z-value P(>|z|) ci.lower ci.upper Std.lv Std.all frn =~ frn_1 0.963 0.005 175.060 0.000 0.952 0.973 1.024 0.967 frn_2 0.932 0.008 113.337 0.000 0.916 0.949 0.992 0.940 frn_3 0.908 0.010 87.003 0.000 0.888 0.929 0.966 0.918 frn_4 0.902 0.012 75.140 0.000 0.878 0.925 0.959 0.912 frn_5 0.823 0.018 44.801 0.000 0.787 0.859 0.876 0.839 mpos =~ mpos1 0.645 0.034 19.125 0.000 0.579 0.711 0.762 0.706 mpos2 0.904 0.018 49.988 0.000 0.869 0.940 1.069 0.929 mpos3 0.927 0.019 49.320 0.000 0.890 0.964 1.095 0.946 mneg =~ mneg1 0.580 0.037 15.729 0.000 0.507 0.652 0.586 0.584 mneg2 0.838 0.031 27.192 0.000 0.777 0.898 0.847 0.841 mneg3 0.855 0.033 25.542 0.000 0.790 0.921 0.865 0.858 out =~ out_1 0.815 0.032 25.869 0.000 0.753 0.877 1.044 0.976 out_2 0.828 0.030 28.034 0.000 0.770 0.886 1.061 0.989 out_3 0.729 0.028 25.898 0.000 0.674 0.784 0.934 0.884 out_4 0.769 0.028 27.606 0.000 0.715 0.824 0.985 0.927 out_5 0.583 0.029 19.836 0.000 0.525 0.640 0.747 0.720
Regressions: Estimate Std.Err z-value P(>|z|) ci.lower ci.upper Std.lv Std.all out ~ frn 0.210 0.072 2.922 0.003 0.069 0.351 0.174 0.174 mpos 0.560 0.067 8.402 0.000 0.429 0.690 0.516 0.516 mneg -0.257 0.070 -3.653 0.000 -0.394 -0.119 -0.203 -0.203 frn ~ nomy 0.244 0.041 5.967 0.000 0.164 0.324 0.229 0.341 mpos ~ nomy 0.422 0.043 9.737 0.000 0.337 0.507 0.358 0.532 mneg ~ nomy -0.101 0.042 -2.381 0.017 -0.183 -0.018 -0.099 -0.148
Intercepts: Estimate Std.Err z-value P(>|z|) ci.lower ci.upper Std.lv Std.all .frn_1 0.000 0.000 0.000 0.000 0.000 .frn_2 0.000 0.000 0.000 0.000 0.000 .frn_3 0.000 0.000 0.000 0.000 0.000 .frn_4 0.000 0.000 0.000 0.000 0.000 .frn_5 0.000 0.000 0.000 0.000 0.000 .mpos1 0.000 0.000 0.000 0.000 0.000 .mpos2 0.000 0.000 0.000 0.000 0.000 .mpos3 0.000 0.000 0.000 0.000 0.000 .mneg1 0.000 0.000 0.000 0.000 0.000 .mneg2 0.000 0.000 0.000 0.000 0.000 .mneg3 0.000 0.000 0.000 0.000 0.000 .out_1 0.000 0.000 0.000 0.000 0.000 .out_2 0.000 0.000 0.000 0.000 0.000 .out_3 0.000 0.000 0.000 0.000 0.000 .out_4 0.000 0.000 0.000 0.000 0.000 .out_5 0.000 0.000 0.000 0.000 0.000 .frn 0.000 0.000 0.000 0.000 0.000 .mpos 0.000 0.000 0.000 0.000 0.000 .mneg 0.000 0.000 0.000 0.000 0.000 .out 0.000 0.000 0.000 0.000 0.000
Thresholds: Estimate Std.Err z-value P(>|z|) ci.lower ci.upper Std.lv Std.all frn_1|t1 -0.246 0.211 -1.165 0.244 -0.660 0.168 -0.246 -0.232 frn_1|t2 0.495 0.204 2.431 0.015 0.096 0.895 0.495 0.468 frn_1|t3 0.860 0.205 4.195 0.000 0.458 1.262 0.860 0.812 frn_1|t4 1.571 0.212 7.413 0.000 1.156 1.986 1.571 1.483 frn_1|t5 2.008 0.222 9.040 0.000 1.572 2.443 2.008 1.895 frn_1|t6 2.776 0.245 11.311 0.000 2.295 3.258 2.776 2.621 frn_2|t1 -0.250 0.204 -1.226 0.220 -0.649 0.149 -0.250 -0.236 frn_2|t2 0.461 0.199 2.313 0.021 0.070 0.851 0.461 0.436 frn_2|t3 0.909 0.201 4.530 0.000 0.516 1.302 0.909 0.861 frn_2|t4 1.679 0.209 8.029 0.000 1.270 2.089 1.679 1.591 frn_2|t5 2.228 0.220 10.147 0.000 1.798 2.659 2.228 2.111 frn_2|t6 2.658 0.239 11.121 0.000 2.190 3.126 2.658 2.518 frn_3|t1 -0.253 0.210 -1.206 0.228 -0.666 0.159 -0.253 -0.241 frn_3|t2 0.331 0.203 1.632 0.103 -0.066 0.728 0.331 0.314 frn_3|t3 0.777 0.202 3.847 0.000 0.381 1.173 0.777 0.738 frn_3|t4 1.358 0.206 6.577 0.000 0.953 1.762 1.358 1.289 frn_3|t5 1.833 0.211 8.695 0.000 1.419 2.246 1.833 1.740 frn_3|t6 2.460 0.231 10.659 0.000 2.007 2.912 2.460 2.336 frn_4|t1 -0.263 0.203 -1.293 0.196 -0.661 0.136 -0.263 -0.250 frn_4|t2 0.395 0.195 2.030 0.042 0.014 0.777 0.395 0.376 frn_4|t3 0.800 0.193 4.135 0.000 0.421 1.178 0.800 0.760 frn_4|t4 1.395 0.198 7.057 0.000 1.008 1.783 1.395 1.326 frn_4|t5 1.817 0.202 9.015 0.000 1.422 2.212 1.817 1.726 frn_4|t6 2.333 0.207 11.287 0.000 1.928 2.738 2.333 2.217 frn_5|t1 -0.474 0.217 -2.190 0.029 -0.899 -0.050 -0.474 -0.454 frn_5|t2 0.010 0.202 0.049 0.961 -0.386 0.406 0.010 0.009 frn_5|t3 0.465 0.199 2.332 0.020 0.074 0.855 0.465 0.445 frn_5|t4 1.395 0.205 6.806 0.000 0.993 1.797 1.395 1.336 frn_5|t5 1.830 0.212 8.647 0.000 1.415 2.245 1.830 1.753 frn_5|t6 2.371 0.222 10.700 0.000 1.936 2.805 2.371 2.271 mpos1|t1 -0.815 0.251 -3.253 0.001 -1.307 -0.324 -0.815 -0.756 mpos1|t2 -0.313 0.204 -1.538 0.124 -0.713 0.086 -0.313 -0.290 mpos1|t3 0.292 0.181 1.613 0.107 -0.063 0.648 0.292 0.271 mpos1|t4 0.895 0.175 5.122 0.000 0.553 1.238 0.895 0.829 mpos1|t5 1.661 0.175 9.479 0.000 1.318 2.005 1.661 1.539 mpos1|t6 2.621 0.183 14.343 0.000 2.263 2.979 2.621 2.429 mpos2|t1 -0.892 0.246 -3.629 0.000 -1.375 -0.410 -0.892 -0.776 mpos2|t2 -0.085 0.183 -0.464 0.643 -0.444 0.274 -0.085 -0.074 mpos2|t3 0.464 0.167 2.787 0.005 0.138 0.790 0.464 0.403 mpos2|t4 1.178 0.163 7.215 0.000 0.858 1.498 1.178 1.024 mpos2|t5 1.958 0.164 11.971 0.000 1.638 2.279 1.958 1.702 mpos2|t6 2.740 0.169 16.172 0.000 2.408 3.073 2.740 2.382 mpos3|t1 -0.532 0.226 -2.347 0.019 -0.975 -0.088 -0.532 -0.459 mpos3|t2 0.068 0.183 0.372 0.710 -0.290 0.427 0.068 0.059 mpos3|t3 0.653 0.171 3.828 0.000 0.319 0.987 0.653 0.564 mpos3|t4 1.392 0.167 8.313 0.000 1.064 1.720 1.392 1.202 mpos3|t5 1.960 0.170 11.560 0.000 1.628 2.293 1.960 1.693 mpos3|t6 2.863 0.166 17.235 0.000 2.538 3.189 2.863 2.473 mneg1|t1 -0.749 0.198 -3.791 0.000 -1.136 -0.362 -0.749 -0.746 mneg1|t2 -0.191 0.191 -1.001 0.317 -0.565 0.183 -0.191 -0.190 mneg1|t3 0.261 0.189 1.383 0.167 -0.109 0.631 0.261 0.260 mneg1|t4 0.845 0.192 4.401 0.000 0.468 1.221 0.845 0.841 mneg1|t5 1.623 0.211 7.684 0.000 1.209 2.037 1.623 1.617 mneg1|t6 2.170 0.237 9.166 0.000 1.706 2.634 2.170 2.162 mneg2|t1 -1.825 0.222 -8.228 0.000 -2.260 -1.391 -1.825 -1.811 mneg2|t2 -1.123 0.196 -5.728 0.000 -1.508 -0.739 -1.123 -1.114 mneg2|t3 -0.669 0.192 -3.488 0.000 -1.046 -0.293 -0.669 -0.664 mneg2|t4 -0.061 0.193 -0.315 0.753 -0.440 0.318 -0.061 -0.060 mneg2|t5 0.785 0.210 3.731 0.000 0.373 1.198 0.785 0.779 mneg2|t6 1.560 0.276 5.659 0.000 1.020 2.101 1.560 1.548 mneg3|t1 -1.374 0.202 -6.786 0.000 -1.770 -0.977 -1.374 -1.362 mneg3|t2 -0.765 0.192 -3.979 0.000 -1.142 -0.388 -0.765 -0.759 mneg3|t3 -0.300 0.190 -1.577 0.115 -0.673 0.073 -0.300 -0.298 mneg3|t4 0.284 0.195 1.458 0.145 -0.098 0.667 0.284 0.282 mneg3|t5 1.004 0.219 4.586 0.000 0.575 1.433 1.004 0.996 mneg3|t6 1.532 0.253 6.055 0.000 1.036 2.027 1.532 1.519 out_1|t1 -1.455 0.332 -4.390 0.000 -2.105 -0.806 -1.455 -1.360 out_1|t2 -0.924 0.236 -3.912 0.000 -1.386 -0.461 -0.924 -0.863 out_1|t3 -0.353 0.208 -1.698 0.089 -0.761 0.054 -0.353 -0.330 out_1|t4 0.144 0.212 0.678 0.498 -0.272 0.559 0.144 0.134 out_1|t5 0.619 0.217 2.850 0.004 0.193 1.045 0.619 0.579 out_1|t6 1.214 0.226 5.364 0.000 0.770 1.658 1.214 1.135 out_2|t1 -0.931 0.256 -3.635 0.000 -1.434 -0.429 -0.931 -0.869 out_2|t2 -0.858 0.248 -3.467 0.001 -1.344 -0.373 -0.858 -0.801 out_2|t3 -0.319 0.207 -1.540 0.124 -0.724 0.087 -0.319 -0.297 out_2|t4 0.057 0.200 0.285 0.776 -0.334 0.448 0.057 0.053 out_2|t5 0.632 0.203 3.116 0.002 0.235 1.030 0.632 0.590 out_2|t6 1.268 0.209 6.081 0.000 0.859 1.677 1.268 1.183 out_3|t1 -0.974 0.255 -3.814 0.000 -1.475 -0.474 -0.974 -0.922 out_3|t2 -0.759 0.233 -3.261 0.001 -1.216 -0.303 -0.759 -0.719 out_3|t3 -0.304 0.207 -1.466 0.143 -0.710 0.102 -0.304 -0.288 out_3|t4 0.251 0.202 1.239 0.215 -0.146 0.647 0.251 0.237 out_3|t5 0.700 0.207 3.378 0.001 0.294 1.107 0.700 0.663 out_3|t6 1.122 0.211 5.307 0.000 0.708 1.536 1.122 1.062 out_4|t1 -1.274 0.273 -4.669 0.000 -1.809 -0.739 -1.274 -1.199 out_4|t2 -1.017 0.235 -4.330 0.000 -1.477 -0.557 -1.017 -0.957 out_4|t3 -0.524 0.201 -2.603 0.009 -0.918 -0.129 -0.524 -0.493 out_4|t4 -0.119 0.198 -0.598 0.550 -0.507 0.270 -0.119 -0.112 out_4|t5 0.333 0.196 1.695 0.090 -0.052 0.718 0.333 0.313 out_4|t6 0.929 0.201 4.624 0.000 0.535 1.323 0.929 0.875 out_5|t1 -1.639 0.399 -4.109 0.000 -2.421 -0.857 -1.639 -1.581 out_5|t2 -1.238 0.345 -3.588 0.000 -1.914 -0.562 -1.238 -1.194 out_5|t3 -0.821 0.319 -2.572 0.010 -1.446 -0.195 -0.821 -0.792 out_5|t4 -0.673 0.308 -2.189 0.029 -1.277 -0.070 -0.673 -0.650 out_5|t5 -0.492 0.298 -1.653 0.098 -1.076 0.091 -0.492 -0.475 out_5|t6 -0.151 0.289 -0.523 0.601 -0.718 0.416 -0.151 -0.146
Variances: Estimate Std.Err z-value P(>|z|) ci.lower ci.upper Std.lv Std.all .frn_1 0.073 0.073 0.073 0.073 0.065 .frn_2 0.130 0.130 0.130 0.130 0.117 .frn_3 0.175 0.175 0.175 0.175 0.158 .frn_4 0.187 0.187 0.187 0.187 0.169 .frn_5 0.322 0.322 0.322 0.322 0.296 .mpos1 0.584 0.584 0.584 0.584 0.501 .mpos2 0.182 0.182 0.182 0.182 0.137 .mpos3 0.140 0.140 0.140 0.140 0.105 .mneg1 0.664 0.664 0.664 0.664 0.659 .mneg2 0.298 0.298 0.298 0.298 0.293 .mneg3 0.269 0.269 0.269 0.269 0.264 .out_1 0.055 0.055 0.055 0.055 0.048 .out_2 0.024 0.024 0.024 0.024 0.021 .out_3 0.243 0.243 0.243 0.243 0.218 .out_4 0.158 0.158 0.158 0.158 0.140 .out_5 0.517 0.517 0.517 0.517 0.481 .frn 1.000 1.000 1.000 0.884 0.884 .mpos 1.000 1.000 1.000 0.717 0.717 .mneg 1.000 1.000 1.000 0.978 0.978 .out 1.000 1.000 1.000 0.609 0.609
Scales y*: Estimate Std.Err z-value P(>|z|) ci.lower ci.upper Std.lv Std.all frn_1 1.000 1.000 1.000 1.000 1.000 frn_2 1.000 1.000 1.000 1.000 1.000 frn_3 1.000 1.000 1.000 1.000 1.000 frn_4 1.000 1.000 1.000 1.000 1.000 frn_5 1.000 1.000 1.000 1.000 1.000 mpos1 1.000 1.000 1.000 1.000 1.000 mpos2 1.000 1.000 1.000 1.000 1.000 mpos3 1.000 1.000 1.000 1.000 1.000 mneg1 1.000 1.000 1.000 1.000 1.000 mneg2 1.000 1.000 1.000 1.000 1.000 mneg3 1.000 1.000 1.000 1.000 1.000 out_1 1.000 1.000 1.000 1.000 1.000 out_2 1.000 1.000 1.000 1.000 1.000 out_3 1.000 1.000 1.000 1.000 1.000 out_4 1.000 1.000 1.000 1.000 1.000 out_5 1.000 1.000 1.000 1.000 1.000Thank you very much,
Fran
Terrence Jorgensen
Aug 25, 2019, 8:28:42 AM8/25/19
to lavaan
1) from the output below, it there something (some indices) indicating that the model is doing something wrong (e.g, overfitting?)
Overfitting is caused by including too many effects to explain every little nuance in your sample, not by changing estimators.
My doubt is that the sample size is still too small.
It is absolutely too small to trust the chi-squared test. Anything less than 500 has been shown to yield inflated errors. But I'm guessing from the other information that your model still fits poorly. CFI may be high, but that just means your model does a lot better than the independence model. RMSEA is showing you that the average amount of misfit per df is still high, and the SRMR says the average correlation residual is high. The latter implies that many individuals residuals are probably not ignorable.
2) what are the necessary checks in lavaan that I should carry out to see if the model is good?
Check lavResiduals(fit) to see which relationships your model fails to reproduce well.
Terrence D. Jorgensen
Assistant Professor, Methods and Statistics
Research Institute for Child Development and Education, the University of Amsterdam
UvA web page: http://www.uva.nl/profile/t.d.jorgensen
Reply all
Reply to author
Forward
Message has been deleted
0 new messages