Correction of p-values of path coefficients in MG-SEM

240 views

Skip to first unread message

Anja

unread,

Aug 3, 2021, 12:55:15 PM8/3/21

to lavaan

Dear lavaan experts,

My question below is not strictly related to the lavaan package, but more to adequate analyses and reporting of the results. I still hope that you could help me with my problem.

One of the reviewers of our paper wrote: »A lot of statistical hypothesis testing and many adjustable parameters but no correction for multiple comparison artifacts – seems to be a major weakness in the study«. He does not have any comments on the MG-SEM (with three groups) analyses reported (steps described below).

I have already read the older posts on adjustment of p-values which confirmed my impression that corrections of p-values are rarely reported with SEM. And if calculated/reported, the family of tests (p-values) to be adjusted is chosen somehow arbitrary.

The basic scheme of my performed analyses:

Evaluation of the measurement model (cfa).
Establishment of measurement invariance.
Analyses of SEM model:

a) For each path I perform a test where path coefficients were constraint to be equal across groups. If the constrained model fits as well as the unconstrained model, I concluded that equality across groups is plausible for that path.

For those paths, one one p-value is reported.

b) If the constrained model fits worse than the unconstrained model, then the paths were not constrained to be equal in the final model. The individual tests on each parameter were performed and three regression coefficients and the corresponding p-values (3 groups) were reported.
c) The results of the final SEM model were reported where some paths are constrained to be equal (a)) across groups, while other (b)) not.

Now my questions are:

1.) Would it be ok to calculate the adjustable p-values for the set of path coefficients?

I think to take into account all (different) p-values. More precisely, if the path was equal across groups, I will take only one p-value of that path and if path coefficients differ across groups, I would take three different p-values. Would that be ok?

The snapshot of the code:

PE <- parameterEstimates(modelSEM_final)

pRegCoef_differenet<-PE[c(23:33,126:128, 134,227:229,235), ] ### to use only ‘different’ p-values of the path coefficients

pRegCoef_differenet$p.fdr <- p.adjust(pRegCoef_differenet$pvalue, method = "fdr") ### to control the false discovery rate

2.) Could you provide any published paper (with MG_SEM) where adjusted p-values are reported? (I tried to search on WoS, GoogleSchoolar, but was not successful.)

Thank you in advance for all your valuable answers and comments.

Best regards, Anja

Terrence Jorgensen

unread,

Aug 16, 2021, 5:52:50 AM8/16/21

to lavaan

1.) Would it be ok to calculate the adjustable p-values for the set of path coefficients?

Sure, it is always up to the researcher how to control Type I error rates and what is considered a family of tests.

2.) Could you provide any published paper (with MG_SEM) where adjusted p-values are reported?

It is not common practice in regression or SEM to consider effects of different predictors on an outcome as a family of tests, the way it is common in (M)AN(C)OVA to follow up a significant omnibus test with multiple tests of subsets of the omnibus H0. Doing so for the same predictor on different outcomes is somewhat more common in exploratory clinical trials (e.g., looking at the effect of a drug on a variety of potential outcomes). But precedent doesn't dictate what you can(not) or should (not) do. Just be transparent about how you decide to control Type I errors across the set(s) of tests, and provide some justification as to why a certain set belongs to a single "family" of tests (e.g., related null hypotheses).

Terrence D. Jorgensen

Assistant Professor, Methods and Statistics

Research Institute for Child Development and Education, the University of Amsterdam