absence of evidence using Blavaan / BSEM?

[Carolien T.]

Hi all!

I was wondering whether it makes sense to use bsem (as in blavaan) to provide an estimate of the absence of evidence (evidence for H0)?

I am running a path model in which the paths are insignificant using SEMinR and the latent constructs seem quite convincingly unrelated. However, using these frequentist analyses, I cannot really say anything about H0. Normally I would chose a bayes factor type of approach but I read that in BSEM relying on Bayes Factors is not a very good idea? Any ideas on which direction I could think of? I am familiar with Bayesian basics mostly from the Bayes Factor viewpoint, but no expert in this field. Thanks :)

I used fairly simple models with two latent variables with each 3 indicator variables and in a second step a standard mediation model  (X -> Y via M) with each of the latent variables having 3 indicators as well. 

Best,

Carolien

[Ed Merkle]

Carolien,

You could potentially use Bayes factors with blavaan. The "margloglik" entry of fitMeasures() provides a Laplace approximation of the marginal log-likelihood, which is involved in calculation of the log-BF. Other approximations are possible, but blavaan does not automatically supply them.

There are at least two objections to using the Bayes factor here. The first is that some researchers are generally opposed to the Bayes factor for all models, because it is sensitive to choice of prior distribution. Those researchers would tell you not to use the BF, regardless of whether your model is in blavaan or elsewhere. Second, for some specific blavaan models, it is difficult to evaluate the prior distribution. This makes it difficult to accurately compute or approximate the BF. blavaan will give you a "margloglik" value of NA for those models. To read more about difficulties evaluating the prior, see this paper: https://doi.org/10.5964/meth.11167

Ed

[Carolien T.]

Hi Ed,

Thanks a lot for your quick and helpful response! I do acknowledge the general disadvantages of BFs and that they may create even larger problems in complex models like in SEM... I was planning on running sensitivity analyses with several priors to see the impact of the priors in my models. But now I think of it, I am not even sure which model to compare it against for getting this Bayes Factor (I am particularly interested in the paths), so maybe this is just a step too far. 

Could you say something about the evidence for no effect based on the posterior distribution? I.e., if its really peaked around 0 then its probably a true null effect whereas if its very wide of course the evidence is less..?

Thanks again!

Carolien

Op woensdag 2 april 2025 om 17:31:44 UTC+2 schreef Ed Merkle:

[Mauricio Garnier-Villarreal]

Hi Carolien

I am one of those researchers that Ed mentioned. I think the Bayes factor has to be use carefully, and dont recommend it in general

Schad, D. J., Nicenboim, B., Bürkner, P.-C., Betancourt, M., & Vasishth, S. (2022). Workflow techniques for the robust use of bayes factors. Psychological Methods. https://doi.org/10.1037/met0000472

Tendeiro, J. N., & Kiers, H. A. L. (2019). A review of issues about null hypothesis Bayesian testing. Psychological Methods, 24(6), 774–795. https://doi.org/10.1037/met0000221

Now, about your question on how to test for evidence for H0, I would suggest a couple of options

1- Can use the p-direction and ROPE for the paths of interest (https://ecmerkle.github.io/blavaan/articles/probability_direction.html ), and for example state that about the proportion of the posterior that is "small effect size"

2- Could use a strong prior close to 0 e.g., N(0, .1) , and compare the unrestricted model with this restricted one with LOO/WAIC (https://ecmerkle.github.io/blavaan/articles/model_comparison.html ) , and if the restricted model has similar predictive accuracy, would be evidence for the path being close to 0

3- Could use fix the path to 0 , and compare the unrestricted model with this restricted one with LOO/WAIC (https://ecmerkle.github.io/blavaan/articles/model_comparison.html ) , and if the restricted model has similar predictive accuracy, would be evidence for the path being close to 0

Could use p-direction or ROPE with either 2 and 3.

And these would be ways to weight information in favor of the path been equal or very close to 0

Hope this is helpful

[Carolien T.]

Thanks a lot for the sources and additional information on how to approach this further. Great to see that there are good alternatives to using dubious bayes factors for this type of question. I'll dig into it further with your suggestions, thanks again! :)

Op donderdag 3 april 2025 om 15:47:49 UTC+2 schreef mauga...@gmail.com: