Mediations: Discrepancies in p-values of Indirect Effects using Lavaan and Mediation package

139 views
Skip to first unread message

krmu...@gmail.com

unread,
Aug 27, 2020, 6:45:26 PM8/27/20
to lavaan
Hi there, 

I have run some simple mediation models with a binary IV and with continuous covariate, mediator, and DV. Prior to learning about the lavaan package, I ran the model with the mediation package. 

I've compared the results between both packages, and the results are nearly identical (e.g., same coefficient estimates), but the p-values for the indirect effect differ, even though the coefficient estimates are the same and the p-values for the other effects are the same. 

I'm using the bootstrapping technique in both packages, and have tried running the code with varying randomization seeds as well as number of simulations. Regardless, the same pattern is persisting. Below is the code I'm using and results. Has anyone encountered a similar situation? Is there an additional specifier I need to be adding to calculated the indirect effect accurately in lavaan? 


library(lavaan)

sem_model = '
att_neg ~ a*group + chi_beh_control 
avs_rpm_epi ~ c*group + chi_beh_control + b*att_neg
 

direct := c
 
#indirect effect
indirect := a*b
 
#total effect
total := c + (a*b)
'

set.seed(567)
model_sem2 = sem(sem_model, data=nona_data, se="bootstrap", test="bootstrap", bootstrap=1000) 

Screen Shot 2020-08-27 at 5.39.17 PM.pngScreen Shot 2020-08-27 at 5.39.32 PM.png

The code used in the Mediation package & output is below:

med_fit_avs_rpm_epi_na <- lm(att_neg ~ group + chi_beh_control, data = nona_data)
out_fit_avs_rpm_epi_na <- lm(avs_rpm_epi ~ group + chi_beh_control + att_neg, data = nona_data) 
set.seed(567)
med_out_avs_rpm_epi_na <- mediate(med_fit_avs_rpm_epi_na, out_fit_avs_rpm_epi_na, boot = TRUE, treat = "group", mediator = "att_neg", sims = 1000)
summary(med_out_avs_rpm_epi_na)
Screen Shot 2020-08-27 at 5.42.27 PM.png
the ACME, ADE, and total effect estimates are the same as from the lavaan package, but the p-value differs significantly. 
(ACME = average causal mediation effects/indirect effects)

Thank you!
Message has been deleted

Patrick (Malone Quantitative)

unread,
Aug 27, 2020, 6:55:04 PM8/27/20
to lav...@googlegroups.com
What happens if you ramp the bootstrap draws up to, say, 5000? 1000 really isn't a lot for a percentile bootstrap.

--
You received this message because you are subscribed to the Google Groups "lavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to lavaan+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/lavaan/06ca7ca9-3d89-48d7-92b0-ac5b3bdbb979n%40googlegroups.com.


--
Patrick S. Malone, Ph.D., Malone Quantitative
NEW Service Models: http://malonequantitative.com

He/Him/His

krmu...@gmail.com

unread,
Aug 27, 2020, 8:18:17 PM8/27/20
to lavaan
Thanks for the quick reply Patrick. Even when I set the bootstrap draws to 5000 in both packages, the same pattern exists..

Rakoon P

unread,
Aug 27, 2020, 11:13:38 PM8/27/20
to lavaan
If you use  test = "bootstrap"  instead of se = "bootstrap"  , do you still get the same results?

krmu...@gmail.com

unread,
Aug 28, 2020, 12:28:49 AM8/28/20
to lavaan
Interesting! The lavaan results are different when I only use test = "bootstrap" and remove se = "bootstrap." I tried this out with a couple different seed randomizations and bootstraps. Those results match up with the results from the mediation package. I guess now I'm not quite sure whether it's more correct to include se or not.. 

Rakoon P

unread,
Aug 28, 2020, 12:40:52 AM8/28/20
to lavaan
Not sure about your situation, but in my area I believe reviewers mainly want to see that the bootstrapped SE do not contain zero and the p-values of indirect effects do not get reported.

Patrick (Malone Quantitative)

unread,
Aug 28, 2020, 8:11:01 AM8/28/20
to lav...@googlegroups.com
Ah, I didn't catch se="bootstrap" that Rakoon spotted. Yes, that does not return the percentile bootstrap--it just computes the SE without parametric assumptions . The confidence intervals returned are symmetric, which is not going to be the most apt for indirect effects.

And I agree with Rakoon that it's unusual for journals to request p-values with percentile bootstrap CIs.

krmu...@gmail.com

unread,
Aug 28, 2020, 10:33:53 AM8/28/20
to lavaan
Got it, thank you both that's really helpful. I don't see the CIs in my output, so would I need to run separate code like this to calculate the CIs: parameterestimates(fsem1, boot.ci.type = "bca.simple", standardized = TRUE) ? this code is from https://nmmichalak.github.io/nicholas_michalak/blog_entries/2018/nrg01/nrg01.html?

Patrick (Malone Quantitative)

unread,
Aug 28, 2020, 10:43:15 AM8/28/20
to lav...@googlegroups.com
bca.simple (bias-corrected accelerated) is not identical to the percentile bootstrap, but you're on the right track!

car...@web.de

unread,
Aug 28, 2020, 10:54:09 AM8/28/20
to lav...@googlegroups.com
Just a small side note. Acceleration is not part of the procedure: https://groups.google.com/g/lavaan/c/FpF1TwBNIj8



--
Diese Nachricht wurde von meinem Android Mobiltelefon mit WEB.DE Mail gesendet.
Am 28.08.20, 16:43 schrieb "Patrick (Malone Quantitative)" <mal...@malonequantitative.com>:

Patrick (Malone Quantitative)

unread,
Aug 28, 2020, 11:00:11 AM8/28/20
to lav...@googlegroups.com
Thanks for the note. In broader contexts, I usually see the "a" in "bca" as abbreviation for accelerated.

krmu...@gmail.com

unread,
Aug 28, 2020, 1:15:40 PM8/28/20
to lavaan
Ah, thank you! I just read that in the lavaan reference manual as well. 

car...@web.de

unread,
Aug 28, 2020, 4:54:20 PM8/28/20
to lav...@googlegroups.com
You can use the boot package if acceleration is relevant for you. @Pat: true. I also find "bca.simple" a bit confusing. "bc" might be a bit clearer attribute value.
Am 28.08.20, 19:15 schrieb "krmu...@gmail.com" <krmu...@gmail.com>:
Reply all
Reply to author
Forward
0 new messages