Groups keyboard shortcuts have been updated
Dismiss
See shortcuts

Disagreement in bootstrapping p-value and confidence interval (SEM)

275 views
Skip to first unread message

Shay Yao

unread,
Dec 11, 2019, 1:04:00 PM12/11/19
to lavaan
Hi everyone! I'm new to the group. I need some help with my SEM output. Here're my codes:

fit <- lavaan::sem(model = mod_serial_medi, data = data, se = "boot", bootstrap = 10000,likelihood = "Wishart")
summary(fit, fit.measures=TRUE,ci=TRUE,rsquare = TRUE)
sink("all hypo_new model_unbiased covariance 10000 boots.txt")
summary(fit,fit.measures = TRUE,ci=TRUE, rsquare = TRUE)
sink()


And here is the output of one defined parameter (indirect effect of a moderated serial mediation):

                                     Estimate  Std.Err  z-value  P(>|z|) ci.lower ci.upper
       gp_nd_1_mns__0   -0.058    0.039   -1.502    0.133   -0.149   -0.002


As you can see in the output, the p-value indicates the result is nonsignificant but the CI indicates significance. How did this happen? Which result should I report? And how do I fix it in my codes? 


I'd really appreciate if you can help me with this. I tried everything on my own but couldn't figure out why. If you need to see the whole output, let me know. Thank you so much for your help!

Terrence Jorgensen

unread,
Dec 15, 2019, 8:33:37 AM12/15/19
to lavaan
the p-value indicates the result is nonsignificant but the CI indicates significance. How did this happen?

The p value is calculated assuming a normal sampling distribution.  The CI is based on 2.5 and 97.5 percentiles of the empirical bootstrap distribution.

Which result should I report?

The CI does not require an assumption of normality, which is clearly violated if the CI and p value do not agree.  So the CI is more trustworthy.

Terrence D. Jorgensen
Assistant Professor, Methods and Statistics
Research Institute for Child Development and Education, the University of Amsterdam
 

Christian Arnold

unread,
Jan 23, 2025, 7:56:26 AMJan 23
to lavaan
In case others are struggling with this issue, I've written a quick and dirty script for percentile and BC intervals and p-values. The reference can be found here. Note: I have omitted any form of error handling. The functions are largely untested. At best, it's a prototype. But maybe it will be useful to someone.

Best

Christian


ci.p <- list(
  "perc" = function(t0 = NULL, t, level = 0.95) {
    M <- length(t[t < 0])
    B <- length(t)
    p <- 2 * min(M / B, 1 - M / B)
    ci <- quantile(t, c(1 - level, 1 + level) / 2, type = 6)
    return(round(c(ci, p = p), 3))
  },
  "bc" = function(t0, t, level = 0.95) {
    M <- length(t[t < 0])
    B <- length(t)
    z0 <- qnorm(length(t[t < t0]) / length(t))
    p <- 2 * if(t0 > 0) pnorm(-2 * z0 + qnorm(M / B)) else 1 - pnorm(-2 * z0 + qnorm(M / B))
    ci <- quantile(t, pnorm(2 * z0 + qnorm(c(1 - level, 1 + level) / 2)), type = 6)
    return(round(c(ci, p = p), 3))
  }
)

# Example
library(lavaan)

pop.model <- "
M ~ 0.4 * X
Y ~ 0.4 * M + 0.2 * X
"

set.seed(123)
data <- simulateData(pop.model, sample.nobs = 50)
fit <- sem(c("M~a*X", "Y~b*M+X", "ind:=a*b"), data, se = "bootstrap")

# Percentile CI. CI and p disagree
parameterEstimates(fit, boot.ci.type = "perc")

# Extract bootstraped coefs (quick and dirty - using bootstrapLavaan would be better)
boot <- data.frame(fit@boot$coef)
colnames(boot) <- names(coef(fit))

# CI and p do not disagree
ci.p[["perc"]](t = boot$a * boot$b)

# BC-CI. CI and p disagree
parameterEstimates(fit, boot.ci.type = "bca.simple")

# CI and p do not disagree
ci.p[["bc"]](t0 = coef(fit, type = "user")["ind"], t = boot$a * boot$b)
Reply all
Reply to author
Forward
0 new messages