MACS2.1.1 apparently breaks with large numbers of reads: [ValueError: cannot resize this array: it does not own its data]

[Ian]

I have been using 2.1.1.20160309 with hiseq2500 runs of about 20 million read pairs per sample with no problem.  We have just started getting reads from a hiseq4000 that had ~100 million read pairs per sample.

The default callpeak analysis ran OK, but when I used --broad the analysis failed to complete:

Read info:

INFO  @ Mon, 08 Aug 2016 11:39:26: #1 fragment size = 58 

INFO  @ Mon, 08 Aug 2016 11:39:26: #1  total fragments in treatment: 93683162 

INFO  @ Mon, 08 Aug 2016 11:39:26: #1 user defined the maximum fragments... 

INFO  @ Mon, 08 Aug 2016 11:39:26: #1 filter out redundant fragments by allowing at most 1 identical fragment(s) 

INFO  @ Mon, 08 Aug 2016 11:41:39: #1  fragments after filtering in treatment: 68560539 

INFO  @ Mon, 08 Aug 2016 11:41:39: #1  Redundant rate of treatment: 0.27 

INFO  @ Mon, 08 Aug 2016 11:41:39: #1  total fragments in control: 91471879 

INFO  @ Mon, 08 Aug 2016 11:41:39: #1 user defined the maximum fragments... 

INFO  @ Mon, 08 Aug 2016 11:41:39: #1 filter out redundant fragments by allowing at most 1 identical fragment(s) 

INFO  @ Mon, 08 Aug 2016 11:43:49: #1  fragments after filtering in control: 60129668 

INFO  @ Mon, 08 Aug 2016 11:43:49: #1  Redundant rate of control: 0.34 

Error:

INFO  @ Mon, 08 Aug 2016 11:49:48: #3 Call peaks for each chromosome... 

Traceback (most recent call last):

  File "/opt/gridware/depots/4baff5c5/el7/pkg/apps/macs/2.1.1.20160309/gcc-4.8.5+numpy-1.9.2/bin/macs2", line 4, in <module>

    __import__('pkg_resources').run_script('MACS2==2.1.1.20160309', 'macs2')

  File "/opt/gridware/depots/4baff5c5/el7/pkg/apps/setuptools/15.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-15.1-py2.7.egg/pkg_resources/__init__.py", line 723, in run_script

  File "/opt/gridware/depots/4baff5c5/el7/pkg/apps/setuptools/15.1/python-2.7.8/python/lib/python2.7/site-packages/setuptools-15.1-py2.7.egg/pkg_resources/__init__.py", line 1643, in run_script

  File "/opt/gridware/depots/4baff5c5/el7/pkg/apps/macs/2.1.1.20160309/gcc-4.8.5+numpy-1.9.2/python/lib/python2.7/site-packages/MACS2-2.1.1.20160309-py2.7-linux-x86_64.egg/EGG-INFO/scripts/macs2", line 617, in <module>

    

  File "/opt/gridware/depots/4baff5c5/el7/pkg/apps/macs/2.1.1.20160309/gcc-4.8.5+numpy-1.9.2/python/lib/python2.7/site-packages/MACS2-2.1.1.20160309-py2.7-linux-x86_64.egg/EGG-INFO/scripts/macs2", line 57, in main

    

  File "/opt/gridware/depots/4baff5c5/el7/pkg/apps/macs/2.1.1.20160309/gcc-4.8.5+numpy-1.9.2/python/lib/python2.7/site-packages/MACS2-2.1.1.20160309-py2.7-linux-x86_64.egg/MACS2/callpeak_cmd.py", line 264, in run

  File "MACS2/PeakDetect.pyx", line 105, in MACS2.PeakDetect.PeakDetect.call_peaks (MACS2/PeakDetect.c:1632)

  File "MACS2/PeakDetect.pyx", line 259, in MACS2.PeakDetect.PeakDetect.__call_peaks_w_control (MACS2/PeakDetect.c:3359)

  File "MACS2/IO/CallPeakUnit.pyx", line 1431, in MACS2.IO.CallPeakUnit.CallerFromAlignments.call_broadpeaks (MACS2/IO/CallPeakUnit.c:19498)

  File "MACS2/IO/CallPeakUnit.pyx", line 1488, in MACS2.IO.CallPeakUnit.CallerFromAlignments.call_broadpeaks (MACS2/IO/CallPeakUnit.c:18756)

  File "MACS2/IO/CallPeakUnit.pyx", line 1567, in MACS2.IO.CallPeakUnit.CallerFromAlignments.__chrom_call_broadpeak_using_certain_criteria (MACS2/IO/CallPeakUnit.c:19625)

  File "MACS2/IO/CallPeakUnit.pyx", line 475, in MACS2.IO.CallPeakUnit.CallerFromAlignments.__pileup_treat_ctrl_a_chromosome (MACS2/IO/CallPeakUnit.c:6861)

ValueError: cannot resize this array: it does not own its data

I then randomly extracted and remapped 30 million read pairs from the original fastq; the analysis completed. 

Read info:

INFO  @ Tue, 09 Aug 2016 13:41:35: #1 fragment size = 152 

INFO  @ Tue, 09 Aug 2016 13:41:35: #1  total fragments in treatment: 22628802 

INFO  @ Tue, 09 Aug 2016 13:41:35: #1 user defined the maximum fragments... 

INFO  @ Tue, 09 Aug 2016 13:41:35: #1 filter out redundant fragments by allowing at most 1 identical fragment(s) 

INFO  @ Tue, 09 Aug 2016 13:42:18: #1  fragments after filtering in treatment: 20787363 

INFO  @ Tue, 09 Aug 2016 13:42:18: #1  Redundant rate of treatment: 0.08 

INFO  @ Tue, 09 Aug 2016 13:42:18: #1  total fragments in control: 21414429 

INFO  @ Tue, 09 Aug 2016 13:42:18: #1 user defined the maximum fragments... 

INFO  @ Tue, 09 Aug 2016 13:42:18: #1 filter out redundant fragments by allowing at most 1 identical fragment(s) 

INFO  @ Tue, 09 Aug 2016 13:42:59: #1  fragments after filtering in control: 19093763 

Also, the fragment size was smaller and redundancy higher in the full set (58bp) compared to the subset (152bp).

Thanks,

Ian

[Ian]

Version 2.1.0.20140616 was able to run with the full dataset using --broad.