Issue with blavCompare

169 views
Skip to first unread message

Pedro Ribeiro

unread,
Oct 4, 2023, 11:09:13 PM10/4/23
to blavaan
Hi all,

Thanks everyone for the excellent R package! I'm currently working on version blavaan_0.5-2. I'm trying to run the command blavCompare using the example provided in the manual:

library(lavaan)
hsm1 <- ' visual =~ x1 + x2 + x3 + x4
textual =~ x4 + x5 + x6
speed =~ x7 + x8 + x9 '
fit1 <- bcfa(hsm1, data=HolzingerSwineford1939)
hsm2 <- ' visual =~ x1 + x2 + x3
textual =~ x4 + x5 + x6 + x7
speed =~ x7 + x8 + x9 '
fit2 <- bcfa(hsm2, data=HolzingerSwineford1939)
blavCompare(fit1, fit2)


However, I'm receiving the following error:

Error in apply(diffs, 2, sum) : dim(X) must have a positive length
I tried to run the command on two distinct machines but to no avail, I'm receiving the same error. Is this a bug? Otherwise, any insights regarding how to fix this?

Thanks once again for everything!

Cheers,
Pedro

Mauricio Garnier-Villarreal

unread,
Oct 5, 2023, 8:39:11 AM10/5/23
to blavaan
Pedro

Yeah, this is abug, but easy to fix. These calculations need you to include meanstructure=T, then it runs well.

library(lavaan)
hsm1 <- ' visual =~ x1 + x2 + x3 + x4
textual =~ x4 + x5 + x6
speed =~ x7 + x8 + x9 '
fit1 <- bcfa(hsm1, data=HolzingerSwineford1939, meanstructure=T)

hsm2 <- ' visual =~ x1 + x2 + x3
textual =~ x4 + x5 + x6 + x7
speed =~ x7 + x8 + x9 '
fit2 <- bcfa(hsm2, data=HolzingerSwineford1939, meanstructure=T)
blavCompare(fit1, fit2)


Ed, should we have this as a default, or how to prevent this error?

take care

Pedro Ribeiro

unread,
Oct 6, 2023, 1:39:31 AM10/6/23
to blavaan
Hi Mauricio,

Thanks so much, it solved the problem for me! Thanks once again!

Cheers,
Pedro

Ed Merkle

unread,
Oct 6, 2023, 12:23:26 PM10/6/23
to Pedro Ribeiro, blavaan
Thanks for the report, and this should now be fixed on github. (so that, in future versions, you do not need meanstructure=TRUE)

Ed
--
You received this message because you are subscribed to the Google Groups "blavaan" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blavaan+u...@googlegroups.com.

Roy Levy

unread,
Nov 8, 2023, 4:20:45 PM11/8/23
to blavaan
Hi all,

To continue with this thread, I've noticed the blavCompare no longer yields values for the Bayes factor in examples when it had previously. Picking up on the example Pedro started in this thread and revised code sent by Mauricio, I ran the following code

library(lavaan)
library(blavaan)

hsm1 <- ' visual =~ x1 + x2 + x3 + x4
textual =~ x4 + x5 + x6
speed =~ x7 + x8 + x9 '
fit1 <- bcfa(hsm1, data=HolzingerSwineford1939, meanstructure=T)
hsm2 <- ' visual =~ x1 + x2 + x3
textual =~ x4 + x5 + x6 + x7
speed =~ x7 + x8 + x9 '
fit2 <- bcfa(hsm2, data=HolzingerSwineford1939, meanstructure=T)
blavCompare(fit1, fit2)

This returns values for most of the results (WAIC and LOO), but returns NA for the approximation to the log-Bayes factor. Can you advise?

Thanks blavaan folks for the terrific package, and attentiveness to user issues, 
Roy

Mauricio Garnier-Villarreal

unread,
Nov 9, 2023, 6:27:19 AM11/9/23
to blavaan
Hi Roy

I think with the updates in blavaan the marginal LL is not calcuates automatically anymore. Not sure if Ed plans to add it again

But for now, we could do a few edits and calculate the BF with the bridgesampling package. By telling blavaan to save some of the underlying parameters

I have doubt about making a tutorial about this, because I persoanlly dont like the BF much. But I understand people like it. Also, we could switch the blavaan recommendation from the approximate log-BF to the bridgesampling method



library(blavaan)
library(lavaan)
library(bridgesampling)

origpars <- c("Lambda_y_free", "B_free", "Theta_sd_free", "Theta_r_free",
              "Psi_sd_free", "Psi_r_mat", "Psi_r_free", "Nu_free",
              "Alpha_free", "Tau_ufree",
              "z_aug",
              "Lambda_y_free_c","B_free_c", "Theta_sd_free_c",
              "Theta_r_free_c",
              "Psi_sd_free_c", "Psi_r_mat_c", "Psi_r_free_c", "Nu_free_c",
              "Alpha_free_c")


hsm1 <- ' visual =~ x1 + x2 + x3 + x4
textual =~ x4 + x5 + x6
speed =~ x7 + x8 + x9 '
fit1 <- bcfa(hsm1, data=HolzingerSwineford1939,
             meanstructure=T,
             mcmcextra = list(monitor = origpars))



hsm2 <- ' visual =~ x1 + x2 + x3
textual =~ x4 + x5 + x6 + x7
speed =~ x7 + x8 + x9 '
fit2 <- bcfa(hsm2, data=HolzingerSwineford1939,
             meanstructure=T,
             mcmcextra = list(monitor = origpars))


blavCompare(fit1, fit2)


bs1 <- bridge_sampler(fit1@external$mcmcout)
bs2 <- bridge_sampler(fit2@external$mcmcout)

bf(bs1, bs2)


Ed Merkle

unread,
Nov 9, 2023, 12:13:35 PM11/9/23
to Roy Levy, blavaan
Roy, Mauricio,

I did not intend to turn off marginal log-likelihood completely, and I believe the CRAN version of blavaan has a bug that makes these NA. The issue should be fixed on github, and it should make its way to CRAN in the coming weeks.

Marginal log-likelihood is turned off for some models, including 

- those where model covariance matrices are neither block diagonal nor unrestricted

- models with ordinal variables

- two-level models

In these cases, it can be tough to evaluate the prior and/or likelihood.

The marginal likelihoods that are supplied use a Laplace approximation, and I am not convinced that the approximation works well. More work could be done to evaluate these metrics as applied to SEM.

Ed

Roy Levy

unread,
Nov 9, 2023, 1:39:23 PM11/9/23
to blavaan

Hi Mauricio and Ed,

 

Thanks for attending to this. I’ll look to download the version of blavaan from github shortly. I’ll add that I too have concerns about Bayes factors, some of which it seems each of you share. Nevertheless, I have found Bayes factors, at least the concept of them, to be a useful entre to Bayesian methods for some who come from a (frequentist) hypothesis testing mindset.

 

Thanks,

Roy

Ed Merkle

unread,
Nov 9, 2023, 1:52:06 PM11/9/23
to Roy Levy, blavaan
I agree that the concept and interpretation of Bayes factors is nice, and I am not as worried about sensitivity to priors as some others are. My main concerns are computational: the Bayes factor is difficult to compute in general, and there are many ways to make a mistake without realizing it. We described some SEM-specific computational difficulties in the "opaque prior distributions" paper:


Best,
Ed

Mauricio Garnier-Villarreal

unread,
Nov 10, 2023, 4:58:23 AM11/10/23
to blavaan
Hi both

I agree that there are many ways that the BF can be misscalulctaed, and I think large models like SEM are easier to make these mistakes. And thye would reuqire more effort to properly calibrate its calculation. Following the " Workflow techniques for the robust use of bayes factors" paper https://doi.org/10.1037/met0000472

Would it be worthed to make it easier to use the bridgesampling approach from blavaan?

take care

Ed Merkle

unread,
Nov 13, 2023, 8:34:20 PM11/13/23
to Mauricio Garnier-Villarreal, blavaan
I think the bridgesampling thing could be interesting. But I see it as a somewhat difficult project because, to do it well, it will require carefully working through the issues with prior distributions and indeterminacies.

Ed

Reply all
Reply to author
Forward
0 new messages