Why RMSEA is too high even when CFI and TLI are showing good modelfit

[Pavneet Kaur]

I am working on a scale examining how teachers use representations in the classroom while teaching factors and decimals. I have fit a two factor cfa model with seven indicators for each factor. Since, I used the same items to get information for both fractions and decimals, I have correlated their error terms. The cfa fit indices for CFI and TLI shows good model fit, but RMSEA is not confirming the same. 

chi-sq=249.636

df=69

p=0.0000

cfi=.97

tli=.96

rmsea=.11

srmr=.085

I have also checked that:

1.       All the factor loading for both the factors are in reasonable range as std.lv ranges between 0.51-0.926

2.       The correlation between the factors is 0.527

3. The data is collected on a 4-point scale and the bar graphs have shown reasonable variability. 

4.     I tried to check mod-indices, but it shows me error message as:

In lav_start_check_cov(lavpartable = lavpartable, start = START) :

  lavaan WARNING: starting values imply a correlation larger than 1;

                   variables involved are:  F1   D1

  Can anyone please suggest me what can I do to improve the RMSEA for the model?

Attached file has all the necessary information.

Thanks in advance,  

[Chesnut, Ryan]

Hi Pavneet,

RMSEA and CFI can be out of sync with each other, with one suggesting acceptable fit and the other suggesting unacceptable fit. This paper might be helpful for that:

https://pubmed.ncbi.nlm.nih.gov/27014948/

As for the warning, it is telling you that some of your residual correlations exceed one, which is not permissible. Based upon the output you provided, it looks like this is true for F1/D1, F3/D3, F4/D4, F5/D5, and F6/D6 (look at the std.all column). Based on prior posts to this group (https://groups.google.com/g/lavaan/c/LE8w2-V_Ib8/m/Jt4aEAOEAgAJ), you could look at the 95% CI around those estimates to see if sampling error could be the cause of the out-of-bound values. If the CI does not contain values less than 1, there is a good chance the model is mis-specified.

Best,

Ryan 

---

Ryan P. Chesnut, PhD

Assistant Research Professor

Clearinghouse for Military Family Readiness

The Pennsylvania State University

402 Marion Place

University Park, PA 16802

Phone: 814-865-9637

Email: rpc...@psu.edu

Website: www.militaryfamilies.psu.edu

---

From: lav...@googlegroups.com <lav...@googlegroups.com> on behalf of Pavneet Kaur <pkbh...@iu.edu>
Sent: Wednesday, August 12, 2020 12:07 PM
To: lavaan <lav...@googlegroups.com>
Subject: Why RMSEA is too high even when CFI and TLI are showing good modelfit

 

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/2b97c45b-f786-4a4f-be79-8aef076ac72en%40googlegroups.com.

[Pavneet Bharaj]

Thanks a lot Dr. Ryan. I really appreciate your response to my query. 

I have worked on the 95% CI around those estimates and found that the CI does contain values less than 1 (attached sheet).

r-command used-

parameterEstimates(fit.model16, ci = TRUE, level = 0.95)

Can you please guide me where I should investigate next? 

Thanks in advance. 

On Wed, Aug 12, 2020 at 12:22 PM Chesnut, Ryan <rpc...@psu.edu> wrote:

This message was sent from a non-IU address. Please exercise caution when clicking links or opening attachments from external sources.

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/BL0PR02MB4497B9140FD3B86C2D987F19BB420%40BL0PR02MB4497.namprd02.prod.outlook.com.

--

Pavneet Kaur Bharaj

Doctoral Candidate

Indiana University Bloomington

pkbh...@iu.edu

[Chesnut, Ryan]

Hi Pavneet,

Based on the attached output, it looks like you ran CIs on the unstandardized estimates. You want to do it for the standardized estimates since the issue involves correlations greater than 1. There are a couple of ways to do that in lavaan. You could use the standardizedSolution command. In your case, this would be standardizedSolution(fit.model16, ci = TRUE, level = 0.95). You can also generate bootstrapped CIs with bootstrapLavaan, which I think is preferable based on this paper:

https://journals.sagepub.com/doi/10.1177/0049124112442138

I believe this code should provide you with the bootstrapped CIs:

boot <- bootstrapLavaan(fit.model16, type = "nonparametric", R = 100, FUN=function(x) standardizedSolution(x)$est.std)

apply(boot, 2, quantile, probs = c(.025, .975))

You can adjust the number of bootstrap draws by changing the value of R.

I must confess, I am not an expert in Heywood cases. It is my understanding that if the bootstrapped 95% CIs for your residual correlations contain values less than 1 (and your fit estimates are acceptable), then you have confidence to conclude that the inadmissible estimates are the result of sampling error. In other words, the population value of these correlations is less than 1. If that is the case, it is my understanding that you can proceed with the model as is making sure to note the warning message and the additional tests you performed in any write-ups you do. Conversely, if the CIs do not contain values less than 1, then the culprit is likely model mis-specification. It might be worth checking to see what the observed correlations are between your seven f items and your seven d items. If these items have high correlations, that would suggest multicollinearity, which means it would be statistically redundant to include both sets of items. You could also remove the residual correlations from your model and look to see what happens to the correlation between the f and d factors. If this correlation is high (greater than .85 according to Brown [2006]), the two factors lack discriminant validity and you might be better off with just one factor.

Best,

Ryan

Brown, T. A. (2006). Confirmatory factor analysis for applied research. New York: The Guilford Press.

---

Ryan P. Chesnut, PhD

Assistant Research Professor

Clearinghouse for Military Family Readiness

The Pennsylvania State University

402 Marion Place

University Park, PA 16802

Phone: 814-865-9637

Email: rpc...@psu.edu

Website: www.militaryfamilies.psu.edu

---

From: lav...@googlegroups.com <lav...@googlegroups.com> on behalf of Pavneet Bharaj <pkbh...@iu.edu>
Sent: Wednesday, August 12, 2020 1:12 PM
To: lav...@googlegroups.com <lav...@googlegroups.com>
Subject: Re: [External] Re: Why RMSEA is too high even when CFI and TLI are showing good modelfit

 

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/CAME-pw%3DgFTAjbdsNxLqakXudsHFS9XCsixX5_ztGH5zd3COJWQ%40mail.gmail.com.

[Pavneet Bharaj]

Dr.Ryan,

You were correct, as I ran the standardizedSolution(fit.model16, ci = TRUE, level = 0.95), I found CI's to have value >1 .

I think I need to reconsider the model specification once again, and go back to the observed correlation to see the behavior of items.

I also fit the models individually for seven "f" items and seven "d" items, which showed good model-fit. 

I  will post here again, in case if I have any further queries. 

Thanks for your help. 

To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/BL0PR02MB449796AF0D2B1DD527566070BB420%40BL0PR02MB4497.namprd02.prod.outlook.com.