DWLS or MLM

[Roman]

Hi everyone,

I have a second order CFA with predictors from a Likert scale. I predict 11 first order factors, which then predict 2 second order factors (5/6). In total I have 251 complete observations.

I noticed that I get significantly better results when I use the DWLS estimator as opposed to the ML or MLM. As far as I understand this might have to do with the distribution of the predictor data, which are not always normal distributed. I also thought I could justify using the DWLS estimator as the predictors might want to be treated as ordinal variables, since they are on a 7 point scale. 

The differences in the fit indices is extreme:

MLM predictor:

Chi^2 = ~3*DF

CFI = 0.721

TLI = 0.698

RMSEA = 0.087

SRMR = 0.094

AIC = 29872

DWLS predictors:

Chi^2 = ~2*DF

CFI = 0.930

TLI = 0.924

RMSEA = 0.064

SRMR = 0.091

AIC = ???

When I fit the model with DWLS however, I get several warnings which I don't understand:

Warning messages:
1: In lavSampleStatsFromData(Data = lavaanData, missing = lavaanOptions$missing,  :
  lavaan WARNING: number of observations (251) too small to compute Gamma

2: In estimateVCOV(lavaanModel, samplestats = lavaanSampleStats, options = lavaanOptions,  :
  lavaan WARNING: could not compute standard errors!

3: In computeTestStatistic(lavaanModel, partable = lavaanParTable,  :
  lavaan WARNING: could not compute scaled test statistic

Questions:

1) Is it ok to use the DWLS predictor in my case?

2) Do I have to worry about the warnings?

3) I have a 1st order (ref2) factor whose predictors have very very low estimates. The estimate of this first order factor on the second order factors is subsequently extremely high. I have a feeling something is not right there. I tried to set starting values for the predictors of this factor, but that doesn't seem to change anything. I will paste the example below.

                   Estimate  Std.err  Z-value  P(>|z|)   Std.lv  Std.all
Latent variables:
  ex2 =~
    Q9c               0.707                               1.058    0.664
    Q9s               0.591                               0.885    0.604
    Q9z               0.814                               1.218    0.866
    Q9al              0.583                               0.872    0.540
  ref2 =~
    Q9e               0.004                               0.782    0.559
    Q9o               0.005                               0.838    0.619
    Q9ai              0.005                               0.847    0.601

.....

  SOFT =~
    ex2               1.113                               0.744    0.744
    ref2            182.443                               1.000    1.000
    inp2              1.044                               0.722    0.722
    led2              1.101                               0.740    0.740
    pre2              3.276                               0.956    0.956
    ire2              1.382                               0.810    0.810

Probably all the questions are based on the same problem. If anyone could help I would appreciate that.

Thanks

Roman

[yrosseel]

On 11/21/2013 06:18 PM, Roman wrote:
> Hi everyone,
>
> I have a second order CFA with predictors from a Likert scale. I predict
> 11 first order factors, which then predict 2 second order factors (5/6).
> In total I have 251 complete observations.

Hi Roman,

This is a large model, but your sample size is rather small (N=251). If
you consider your observed variables as ordinal, and you have on average
3 indicators per latent variable, you already have 245 free parameters.
Many more if you have more indicators per latent variable.

> I noticed that I get significantly better results when I use the DWLS
> estimator as opposed to the ML or MLM.

Hm. Since you have a 7 point scale, you have either rather extreme
skewed/kurtotic data, or (more likely in this case), the DWLS estimator
fails because there is simply not enough data to get reliable estimates.

> When I fit the model with DWLS however, I get several warnings which I

> Warning messages:
> 1: In lavSampleStatsFromData(Data = lavaanData, missing = lavaanOptions$missing, :
> lavaan WARNING: number of observations (251) too small to compute Gamma

Ah. This one is serious: it means that the estimation of 'Gamma' (the
asymptotic variance matrix of the sample statistics, or the inverse of
'W' in DWLS) is most likely unstable, due to the sample size being too
small...

All following warnings are caused by this too.

> Questions:
>
> 1) Is it ok to use the DWLS predictor in my case?

I'm afraid not. I would use MLM or MLR instead.

> 2) Do I have to worry about the warnings?

Yes.

> 3) I have a 1st order (ref2) factor whose predictors have very very low
> estimates. The estimate of this first order factor on the second order
> factors is subsequently extremely high.

It would seem that the indicators of 'ref2' have little in common...
What about a small model with only this factor?

Hope this helps,

Yves.

[Roman]

Hi Yves, 

thank you for your response. That is really helpful!

I looked at the R-Square of a simple model with the three predictors of the latent variable ref2. I reckon they are rather weak:

R-Square of ref2 predictors:

    Q9e               0.240

    Q9o               0.489

    Q9ai              0.323

Estimates of first order factor "ref2":

                   Estimate  Std.err  Z-value  P(>|z|)   Std.lv  Std.all

  ref2 =~

    Q9e               0.015    0.014    1.065    0.287    0.741    0.531

    Q9o               0.017    0.015    1.147    0.251    0.876    0.648

    Q9ai              0.016    0.014    1.142    0.253    0.818    0.581

Estimates of second order factor "SOFT":

                   Estimate  Std.err  Z-value  P(>|z|)   Std.lv  Std.all

 SOFT =~

    ex2               1.158    0.277    4.183    0.000    0.757    0.757

    ref2             50.295   45.334   1.109    0.267    1.000    1.000

    inp2              1.092    0.260    4.209    0.000    0.738    0.738

    led2              1.065    0.205    5.190    0.000    0.729    0.729

    pre2              2.970    1.325    2.242    0.025    0.948    0.948

    ire2               1.278    0.297    4.298    0.000    0.788    0.788

However, I have noticed something else. When I print a path diagram of the model, the estimates look ok!?

semPaths(secondOrder.fit, "std", layout="tree", vsize.man=3, vsize.lat=6, edge.label.cex=0.75, residuals=TRUE, residSize=0.2, onefile=TRUE, width=4, height=8, rotation=4, titles=TRUE, exoCov = TRUE, intercepts = FALSE, curvePivot = TRUE, style = "lisrel")

See attached.

Do I need to do anything with ref2 or am I just using the values from the Std.all column to predict the variables (I need in the next step)?

Also, this is probably a more general question, is there a best practice how to exclude predictors for a latent variable? A R-Square cut-off value under consideration of the number of predictors per latent variable, or so?

Thanks

Roman

[yrosseel]

On 11/21/2013 08:43 PM, Roman wrote:
> the latent variable ref2. I reckon they are rather weak:
>

> *Estimates of first order factor "ref2":*
> / Estimate Std.err Z-value P(>|z|) Std.lv Std.all/

> ref2 =~
> Q9e 0.015 0.014 1.065 0.287 0.741 0.531
> Q9o 0.017 0.015 1.147 0.251 0.876 0.648
> Q9ai 0.016 0.014 1.142 0.253 0.818 0.581

Indeed. All factor loadings should be (highly) significant. It would
seem these three indicators have not enough in common to create a
factor, hence the bizarre factor loading (> 50) if you use 'ref2' as an
indicator for a second order factor.

> However, I have noticed something else. When I print a path diagram of
> the model, the estimates look ok!?
>
> semPaths(secondOrder.fit, "std", layout="tree", vsize.man=3,
> vsize.lat=6, edge.label.cex=0.75, residuals=TRUE, residSize=0.2,
> onefile=TRUE, width=4, height=8, rotation=4, titles=TRUE, exoCov = TRUE,
> intercepts = FALSE, curvePivot = TRUE, style = "lisrel")

I don't see what is wrong here. semPaths plots the standardized
values... (see Std.all column).

> Do I need to do anything with ref2

I think you need to either drop this factor altogether, or find (much)
better indicators for it.

> Also, this is probably a more general question, is there a best practice
> how to exclude predictors for a latent variable?

Hm. For this, you better consult SEM/CFA literature.

Yves.