Drastic differences in individual-level model fit btwn hybrid vs. VAR

[Lauren Forrest]

Hello,

My name is Lauren Forrest. I’m running into some issues with GIMME that I’d love to get your take on. Description of issues are below. 

Context: I’m using gimme v10.0. I have EMA data that includes n = 79 participants with >= 60 observations. All items have non-zero variance for all participants. Data are structured and being read in correctly.

Issue: I’ve discovered that model fit differs substantially when using VAR = TRUE vs. hybrid = TRUE. Specifically, when running a model with VAR = TRUE, model fit is poor for almost all participants—i.e., there are no (or almost no) participants who have excellent fit on at least 2 fit indices. However, when running that same model but with hybrid = TRUE, model fit is good for most participants. This issue has presented itself in three different network structures distributed among two different EMA datasets (i.e., one dataset has been used for only one network, the other dataset has been used for two different networks). 

For context, the distribution of directed and undirected contemporaneous paths is quite imbalanced. For example, in one dataset with 10 endogenous items + 2 exogenous items, there were 435 directed contemporaneous paths (5 at a subgroup level, 435 at an individual level) and only 30 undirected contemporaneous paths (all at individual level).

Although it seems that allowing for directed contemporaneous paths is making or breaking good model fit, I was still curious to see if I could identify the source of bad fit when VAR = TRUE. I’ve looked into missingness, whether variances among items were too disparate, collinearity, including fewer items in the network, and using different stopping criteria. None were driving or affecting fit. 

Question 1: Does anything seem off to you about this difference in fit between VAR vs. hybrid? Are you surprised by this difference? Relatedly, are you surprised that so many contemporaneous paths are directed vs. undirected?

Another discovery and potential bug (but maybe this is actually a user error): Just for fun, I tried running hybrid = TRUE while constraining directed contemporaneous paths to 0 to see if decent fitting models could still be obtained but without any directed contemporaneous paths in the models. Although I was able to get the model to run with constrained paths, the model gave me an error of “Error in rbind(deparse.level, …): numbers of columns of arguments do not match.” This error resulted in crucial parts of the output being missing, including indivPathEstimates, similarityMatrix, and summaryFit.

I tried running models with multiple iterations of constrained paths (constraining only directed contemporaneous paths that were estimated originally with hybrid = TRUE, constraining all possible directed contemporaneous paths, constraining only a single directed contemporaneous path) and all resulted in the same error. I’ve tried looking into the source code to try to fix this myself but I couldn’t find the location of this error or determine how to fix it. 

Question 2: Do you have advice for how to obtain all relevant output when including a path constraint?

Thank you for any thoughts you can share. I know that maintaining software is done entirely on the side so I really appreciate the time you take to help everyone with their issues!

Lauren

[Katie Gates]

Hi Lauren, 

This is an interesting outcome.

Response for Q1: 
We expect hybrid gimme to help figure out which type of path should be picked. It uses improvement in model fit, in part, to decide this. So there is some evidence here that directed paths improve model fit better here . 

Your case is a bit more exaggerated and I'm guessing it's because of the exogenous variables. I think your intuition is right -  this seems more drastic than you'd expect. 

The reason might be because correlations among errors are not allowed for the exogenous paths. This means that the only way they can have paths when VAR = TRUE is if it's lagged, and nothing explains variability in the exogenous variable. This will sink overall fit index values. 

I wonder if you run the analyses without the exogenous variables if you see this discrepancy between VAR and hybrid fits. I'd be curious to know! 

Response for Q2: 

This sounds like a bug. Would you be able to send the code you used and the data (even just a few people) to gi...@unc.edu? 

Best,

Katie