Individual-level path significance

[Maddie Kushner]

Hello! 

Looking at the individual path estimates spreadsheet from my GIMME output, I see that many files/participants have at least a few paths with p-val > .05. My understanding was that non-significant paths at the individual-level are pruned from the results, but these still show up on the plot and spreadsheet. 

In one of Katie’s posts on a previous question she said that “When looking at the individual-level plots, a path indicates significance, as long as it's not a group-level path.” Many of the paths showing up are not significant group-level paths, though. 

Wondering if these paths should be interpreted as significant or non-significant?

Thanks in advance!

[Maddie Kushner]

Hello!  

Looking at the individual path estimates spreadsheet from my GIMME output, I see that many files/participants have at least a few paths with p-vals > .05. My understanding was that non-significant paths at the individual-level are pruned from the results, but these still show up on the plot and spreadsheet. 

In one of Katie’s posts on a previous question she said that “When looking at the individual-level plots, a path indicates significance, as long as it's not a group-level path.” Many of the paths showing up with p > .05 are not significant group-level paths.

Wondering if these paths be interpreted as significant or non-significant?

Thanks in advance!

Maddie

[Katie Gates]

Hi Maddie, 

Good question. That's strange. Am I understanding correctly that you are finding that individual-level paths are not significant? Or, is it that you see some non-significant group-level paths across individuals? 

Remember that group level paths might not be significant for everyone - we just require 75% by default. So we'd expect up to 25% might be non-significant. 

Would you be able to please share the output? Either as an R object or a the "indivPathEstimates.csv" and summaryFit.csv files. It would be great if you could highlight for me the participants and path estimates that are causing worry. 

Best,

Katie

--
You received this message because you are subscribed to the Google Groups "gimme-r" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gimme-r+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/gimme-r/38c10fbf-5b3c-4452-8d7e-35c5b99a1b51n%40googlegroups.com.

[Katie Gates]

Oh that's totally fine, I was able to find them easily!

We've never seen this before. Initially, it looked like the problem mostly occurred for those who have "last known convergence", which means that these peoples' models didn't ever reach "good fit" (according to fit indices) without entering into a model that doesn't converge. In that case, we roll back to the last known converging model and end there. It seems that we don't prune these (usually fringe) cases. That's something for us to fix. 

The odd thing is that it's happening with people who's models do converge. So I'm not sure why pruning isn't taking place - my guess would be that these went through a less-travelled part of the code where they had a path that was pruned, and then if the model fit is not good we add paths that would be significant. I am not sure if pruning happens again after that. 

As for an immediate solution. On the next push to CRAN (due soon) we provide the syntax for individual-level models so that people can manipulate them as they see fit (such as removing non-significant paths). 

Hope this helps, and thank for the info. 

Katie

On Wed, Apr 10, 2024 at 12:45 PM Maddie Kushner <maddie.l...@gmail.com> wrote:

Oh shoot, I just realized the highlights went away when I saved it as a csv file. Here's a .xlsx of the individual path estimates sheet with the highlights. 

On Wed, Apr 10, 2024 at 12:39 PM Maddie Kushner <maddie.l...@gmail.com> wrote:

Hi Katie,

Yes to your first question – I am finding insignificant paths on the individual level which are not indicated as significant group-level paths in the output. I have attached the requested materials here and highlighted the paths I'm confused about. I also attached the summary paths plot. 

If you look at the summary paths plot, it seems that only the autoregressive paths are significant. However, if you look at the individual-level paths on the spreadsheet, there are paths other than the autoregressive paths which have p > .05, which I am not sure how to interpret.

Let me know if you need anything else!

Thanks,

Maddie

[Maddie Kushner]

Oh shoot, I just realized the highlights went away when I saved it as a csv file. Here's a .xlsx of the individual path estimates sheet with the highlights. 

On Wed, Apr 10, 2024 at 12:39 PM Maddie Kushner <maddie.l...@gmail.com> wrote:

Hi Katie,

Yes to your first question – I am finding insignificant paths on the individual level which are not indicated as significant group-level paths in the output. I have attached the requested materials here and highlighted the paths I'm confused about. I also attached the summary paths plot. 

If you look at the summary paths plot, it seems that only the autoregressive paths are significant. However, if you look at the individual-level paths on the spreadsheet, there are paths other than the autoregressive paths which have p > .05, which I am not sure how to interpret.

Let me know if you need anything else!

Thanks,

Maddie

On Wed, Apr 10, 2024 at 9:15 AM Katie Gates <katie...@gmail.com> wrote:

[Maddie Kushner]

Hi Katie,

Yes to your first question – I am finding insignificant paths on the individual level which are not indicated as significant group-level paths in the output. I have attached the requested materials here and highlighted the paths I'm confused about. I also attached the summary paths plot. 

If you look at the summary paths plot, it seems that only the autoregressive paths are significant. However, if you look at the individual-level paths on the spreadsheet, there are paths other than the autoregressive paths which have p > .05, which I am not sure how to interpret.

Let me know if you need anything else!

Thanks,

Maddie

On Wed, Apr 10, 2024 at 9:15 AM Katie Gates <katie...@gmail.com> wrote:

[Matt Mattoni]

Hi Katie, Maddie, and all,

To follow up on this, I am also getting similarly strange findings at the individual level using fMRI data. One subject had a group-level path with an extremely high beta and SE. I suppose this is likely a consequence of this subject not wanting this group-level path (p = .8), but being forced to have it:

For this same subject, there is also a non-significant path at the individual level. It also has extremely high beta and SE (for reference, most path betas are typically between -1 and 1). This path also involves one of the same nodes, so I'm thinking that is somehow related.

I'm not sure if fit is the issue:

All I can think to do is exclude this participant, but definitely interested if there are other ideas. 

I don't think I have a question here, but just wanted to add some additional info for this issue coming up. Let me know if any info or files would be helpful

Best,

Matt

To view this discussion on the web visit https://groups.google.com/d/msgid/gimme-r/CAGqVQSRDoy9skmNB1Bbzj%3Dn75qpt817hOWX2wEdUcCu%3D1E2oeA%40mail.gmail.com.

[Katie Gates]

Interesting. Would you be able to send the data and code to my email so I can replicate/solve it? Seems like this isn't just a fringe case if you two are both experiencing it. 

Looking into this might not happen quickly since we're in a busy time of the semester with no funding for RAs to work on this. In the meantime I'd probably remove this person since something is going on with their models. If you have time, I'd look at things that seem odd about the data (high collinearity? does Ins.R have low variance?). 

Another immediate fix might be running with standardized = TRUE in the arguments (if you haven't already).

Katie

[Noa Van Zwieten]

Hi all, 

I'm encountering similar issues. When using CS-GIMME with 'standardized = TRUE' in the arguments, I'm observing non-significant individual-level paths, with the majority of these models converging normally. I was wondering what the best solution could be for now when interpreting the results.Should I simply filter out these non-significant paths, or would the forthcoming CRAN version of GIMME, where these paths can be removed from the syntax, yield different results?

Thank you!

Best,

Noa

Op maandag 15 april 2024 om 18:37:44 UTC+2 schreef Katie Gates:

[Katie Gates]

Hey all, 

In the examples I've seen from this thread, it seems that this only happens in cases with the tag "last known convergence" in the "status" column of the SummaryFit.csv file. 

I found the spot where non-significance can happen. When a model doesn't converge for an individual, the algorithm rolls back one path and estimates that model, providing it as the final model. Pruning is not done at this point. 

One rationale for doing it this way is that if we prune again, we likely are getting further from a good fitting model. 

So, once we push the new version an option would be to re-estimate these models with the non-significant paths removed. Or, keep them, with the caveat that they were included in the model to improve fit but were not significant for ## of the individuals. Then perhaps do sensitivity analysis if results change with and without these people or with and without these paths included. All of this is totally up to you, there are no standards on this. 

hope this helps, 

Katie