Persistent Model Issues

[Morgan Beatty]

Hi Everyone, 

My team have been working on conducting some analyses with the hBayesDM toolbox for some time now, and we've been experiencing a combination of issues in doing so. The main error is a persistent 'pareto-k values are too high' warning message, which has been followed by some technological issues with our hardware that created an obstacle for running larger models with posterior predictive checks. Despite hardware improvements and numerous model iterations exploring a handful of argument alterations (increased iterations and warmups, adjusting nthin values, increasing 'adapt_delta' input, etc.), the pareto k value warning has persisted. Without being satisfied that the pareto-k values are mostly in the 'good' (<0.7) category, we can't move forward with analyses.

We have explored the behavioral data itself to ensure that participants were performing the task as expected, but we have also explored model set-up and arguments for the function, and we can't pinpoint whether the issue is in the data itself or in our modelling approach. I'm wondering if there's anyone who might be willing to help us, even to rule out an issue that you've previously encountered? I've attached our two-stage decision making task data file (for ts_par6() function application) , which contains several hundred participants and trials. I hope this isn't a burden to request, but any assistance would be incredibly appreciated - we have 4 or 5 similar datasets that are ready for analysis, so resolving the present conundrum would have great effects on our productivity with this package. 

[Eunhwi Lee]

Hello, and sorry for the delayed response.

Thanks for sharing the details. The high pareto-k warnings indicate that LOO approximation is unreliable, but doesn't necessarily mean that MCMC chains failed to converge (e.g., Rhat value are all okay). The issue might be due to some reasons, such as 1) some participants with atypical or very short data (I can see that in your dataset, some of participants have less than 100 (e.g., 18) trials), 2) a model-data mismatch, or 3) weak priors.

To deal with these possible reasons, fitting a subset of participants and inspecting participant-level Pareto-k values is a good way to check whether a small number of subjects are driving the issue. If that’s the case, you can temporarily exclude them for diagnostics or re-fit them with a simpler model. Or, it may also help to run simpler models (e.g., ts_par4) or to regularize the priors slightly, since ts_par6 can be hard to identify with certain behavioral patterns.

Hope this helps, and please feel free to reach out if needed. 

Best,

Eunhwi

2025년 8월 14일 목요일 오후 3시 35분 32초 UTC+2에 morgb...@gmail.com님이 작성: