AttributeError: 'MaskedArray' object has no attribute 'folded'

[Laura Méndez Cuéllar]

Dear Ryan, 

I am running dadi for hypothesis testing with 6 2D models ("no_mig", "sym_mig", "no_mig_size", "sym_mig_size", "anc_sym_mig_size", "sec_contact_sym_mig_size") on 8 different species with ddRAD sequencing data. All the species have been cleaned the same way and I made the folded sfs files with easySFS from vcf files which I am then feeding to dadi (I am using Portik's pipeline). 

Six of my species work perfectly fine, but two of them keep failing with the same error:

Traceback (most recent call last):
  File "6dadiModels_Run_2D_Rs.py", line 206, in <module>
    Optimize_Functions.Optimize_Routine(fs, pts, prefix, "sym_mig_size", Models_2D.sym_mig_size, rounds, 7, fs_folded=fs_folded,
  File "/gpfs1/work/mendez/DemoHis/dadi/running_dadi_2D/Optimize_Functions.py", line 267, in Optimize_Routine
    params_opt = dadi.Inference.optimize_log_fmin(params_perturbed, fs, func_exec, pts,
  File "/home/mendez/.local/lib/python3.8/site-packages/dadi/Inference.py", line 741, in optimize_log_fmin
    outputs = scipy.optimize.fmin(_object_func_log, numpy.log(p0), args = args,
  File "/home/mendez/.local/lib/python3.8/site-packages/scipy/optimize/_optimize.py", line 695, in fmin
    res = _minimize_neldermead(func, x0, args, callback=callback, **opts)
  File "/home/mendez/.local/lib/python3.8/site-packages/scipy/optimize/_optimize.py", line 845, in _minimize_neldermead
    fsim[k] = func(sim[k])
  File "/home/mendez/.local/lib/python3.8/site-packages/scipy/optimize/_optimize.py", line 569, in function_wrapper
    fx = function(np.copy(x), *(wrapper_args + args))
  File "/home/mendez/.local/lib/python3.8/site-packages/dadi/Inference.py", line 78, in _object_func_log
    return _object_func(numpy.exp(log_params), *args, **kwargs)
  File "/home/mendez/.local/lib/python3.8/site-packages/dadi/Inference.py", line 54, in _object_func
    result = ll_multinom(sfs, data)
  File "/home/mendez/.local/lib/python3.8/site-packages/dadi/Inference.py", line 563, in ll_multinom
    ll_arr = ll_multinom_per_bin(model, data)
  File "/home/mendez/.local/lib/python3.8/site-packages/dadi/Inference.py", line 547, in ll_multinom_per_bin
    return ll_per_bin(theta_opt*model, data)
  File "/home/mendez/.local/lib/python3.8/site-packages/dadi/Inference.py", line 484, in ll_per_bin
    if hasattr(data, 'folded') and data.folded and not model.folded:
AttributeError: 'MaskedArray' object has no attribute 'folded'

From the output, I can see that Species1 (Bm) fails during the optimization of the model "sec_contact_sym_mig_size" and Species2 (Rs) fails during the optimization of model "sym_mig_size". Bm has been projected down to 18,16 (1154, 962 segregating sites, respectively) and Rs to 12,10 (5733, 5613). I have read other messages in the group talking about a similar error, but haven't found an answer to fix my problem... I have tried to increase/decrease the pts but get the same error. I am attaching the error files for the 2 species, the python scripts for running dadi, the sfs files and the python script with the written-out models (Models_2D.py). I would really appreciate some tips on how to make these species run for dadi!

Thank you so much and all the best, 

Laura

[Ryan Gutenkunst]

Hello Laura,

I’m not sure what exactly is happening, since I don’t have the Portik pipeline handy to debug myself. But looking at your output files, you’re running into this error when the numerical integration underlying dadi is really struggling, with almost the entire frequency spectrum being masked for poor calculation results. This may be triggering some corner-case in which a Spectrum object is incorrectly constructed.

As to how to fix this, I’m not sure. Have you evaluated the performance of simpler models? It’s possible you may just be overfitting these particular data, causing the parameters to evaporate to ridiculous values in the optimization of the more complex models. If you were running dadi vanilla, I’d ask you to do a verbose optimization, so we could follow the path more closely.

Best,
Ryan

> --
> You received this message because you are subscribed to the Google Groups "dadi-user" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to dadi-user+...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/dadi-user/777c0d31-061e-4961-848e-c8608d1dc13en%40googlegroups.com.
> <6dadiModels_Run_2D_Bm.py><Models_2D.py><error_Bm.txt><6dadiModels_Run_2D_Rs.py><Rs_South-Rs_North.fs><Bm_South-Bm_North.fs><error_Rs.txt>

[RAYIS T R]

Hi Laura and Ryan,

I recently started running dadi using the same pipeline, but I keep encountering the error mentioned in this thread during some of my runs. Were you able to resolve this issue? If so, please let me know. 

Thank you in advance!

Rayis