step-out procedure failed error

[Shengjie Xu]

Hi all , 

I have a dataset which includes 3 within-subject factors, context, trial type and block type, each has 2 levels. I want to calculate drift rate in each condition. Based on a previous post, this issue may come from empty inputs. Then I checked the data for each subject in each condition, and I found 4 out of 52 subjects have only one response (I coded 1 as correct, and 0 as incorrect. That's to say, these four subjects are always correct there) in some conditions. I tried to remove them from fitting, however the same problem still exists. May I ask what caused this error and how should I fix it? I put the error message here:

 modelddm.sample(5000,burn=1000,dbname='../01_data/data_main/traces.db', db='pickle')

  File "C:\Users\shengjie\Anaconda35\lib\site-packages\kabuki\hierarchical.py", line 651, in sample

    self.mc.sample(*args, **kwargs)

  File "C:\Users\shengjie\Anaconda35\lib\site-packages\pymc\MCMC.py", line 279, in sample

    Sampler.sample(self, iter, length, verbose)

  File "C:\Users\shengjie\Anaconda35\lib\site-packages\pymc\Model.py", line 251, in sample

    self._loop()

  File "C:\Users\shengjie\Anaconda35\lib\site-packages\pymc\MCMC.py", line 315, in _loop

    step_method.step()

  File "C:\Users\shengjie\Anaconda35\lib\site-packages\kabuki\step_methods.py", line 383, in step

    assert iter < self.maxiter, "Step-out procedure failed"

AssertionError: Step-out procedure failed

Any suggestion or thought is much appreciated! 

Best,

Shengjie

[Andrew]

Hi,

Did you ever find a solution to this problem? I'm currently running a series of models (of differing complexity) on HPC and have found that some of my more complicated models error out similarly

Traceback (most recent call last):
  File "/home/s4577680/scratch/DR-RisSen-02/02_Scripts/Exp1HDDM.py", line 118, in <module>
    m = custom(train_sets[ModData], ModList[ModType], name= 'Exp1_'+f"{str(ModName[ModType])}"+'_'+f"{str(int(sys.argv[1])+1)}",
  File "/home/s4577680/scratch/DR-RisSen-02/02_Scripts/Exp1HDDM.py", line 93, in custom
    model.sample(samples, burn, dbname=f'/home/s4577680/scratch/DR-RisSen-02/03_Derivatives/Models/{str(name)}.db', db='pickle')
  File "/home/s4577680/Software/DR-RisSen-02-Env/BehHddm/lib/python3.8/site-packages/kabuki/hierarchical.py", line 659, in sample
    self.mc.sample(*args, **kwargs)
  File "/home/s4577680/Software/DR-RisSen-02-Env/BehHddm/lib/python3.8/site-packages/pymc/MCMC.py", line 286, in sample
    Sampler.sample(self, iter, length, verbose)
  File "/home/s4577680/Software/DR-RisSen-02-Env/BehHddm/lib/python3.8/site-packages/pymc/Model.py", line 245, in sample
    self._loop()
  File "/home/s4577680/Software/DR-RisSen-02-Env/BehHddm/lib/python3.8/site-packages/pymc/MCMC.py", line 322, in _loop
    step_method.step()
  File "/home/s4577680/Software/DR-RisSen-02-Env/BehHddm/lib/python3.8/site-packages/kabuki/step_methods.py", line 397, in step

    assert iter < self.maxiter, "Step-out procedure failed"
AssertionError: Step-out procedure failed

My "custom" function works on my other models but for completeness here it is:

def custom(data, model, name, samples, burn):

        model = hddm.HDDMRegressor(data,

                            model,

                            include=['z'],

                            group_only_regressors=False,

                            keep_regressor_trace=True)

        model.find_starting_values() #Find a good place to start

        #start the MCMC process

        model.sample(samples, burn, dbname=f'/home/s4577680/scratch/DR-RisSen-02/03_Derivatives/Models/{str(name)}.db', db='pickle')

I'm specifying 5000 samples with 500 burn-in. 

As for my data, there are no NaNs or participants who have only correct or incorrect responses, with 99% of my rt data >0.2 seconds. Covered by this post.

I was wondering if this is an issue with convergence due to model complexity as the code seems to run fine on almost all of my other specifications (and previously worked on my local PC albeit with a much lower sample rate of 500 and burn-in of 50 just to test if it worked -- which it did)

Any advice would be much appreciated!

Cheers, Andrew

​

[Leslie Held]

Hi Andrew,

Coincidentally, Shengjie (original post) and I solved a similar issue this morning. I got this error because I had ceiling effects in my accuracies (lots of participants with no or little incorrect responses per cell). I then excluded those with less than 5 observations per factor level combination for correct and incorrect responses and it worked. I also tried a less conservative criterion (removing only those with less than two observations) and still got the error. Hope this helps!

Cheers,

Leslie

[Andrew]

Hi Leslie, 

How fortunate! Thanks for the tip, I've looked into it and I do seem to have a couple of observations that meet that criteria which might be the critical issue since the models I'm having issues with include interaction terms. I'll try this out and see what I can come up with. Thanks!

Cheers, Andrew

--
You received this message because you are subscribed to a topic in the Google Groups "hddm-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/hddm-users/JFSfd5V4m1E/unsubscribe.
To unsubscribe from this group and all its topics, send an email to hddm-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/hddm-users/6630199c-78bf-445e-8ca3-8a237169b4c2n%40googlegroups.com.