Best Ways to Check Parameter Fitting Code?

[Dominique Hughes]

Hi everyone,

I want to fit parameters for my model by comparing simulated and empirical fMRI data (through FC matrices). I have already written my code based on methods described in the papers I've read and parameter fitting suggestions I have seen through this group, but I would like to be able to check that my code is accurate before starting the fitting process on my own data. My first idea was that I could try parameter fitting for a participant that has already been fit/has known correct parameters (I am using the Reduced Wong Wang with coupled excitatory and inhibitory populations), but I'm not sure if that type of information is shared or available to me.

Does anyone have any other ideas or recommendations on how they checked the accuracy of their parameter fitting code?  

Best,

Dominique Hughes

[meysam....@gmail.com]

Hi Dominique,

In order to perform a sanity check on the estimation process, one can make a set of the ground truth for parameters, make the inference, and then see how close is the recovered parameters to the  ground truth! eg. using linear regression on the truth versus estimated.
Using Bayesian setting, this can be quantified by using posterior shrinkage/z-score (see Eqs 11, 12 here).
Before this, one needs to verify the convergence of algorithms (MCMC using the metrics in appendix of this paper).
Using simulation-based inference, we can use posterior rank (see this paper); To assess posterior calibration, we can use simulation-based calibration  (see more details here). The posterior predictive check (i.e., generating data from the model using the parameters drawn from the estimated posterior and then comparing them with the observed data) can validate the reliability of inference process, if it correctly fits the the (feature of) observation.
See main ref here.

Best,

Meysam