dtriple error in "danpos3" TypeError: 'float' object cannot be interpreted as an integer

[Eric Wright]

I just upgraded to the Python3 compatible version of DANPOS. Had success with Danpos2.2.2, but can't get python3 version working

My system is running Ubuntu 20.04 and the following software versions:

Python 3.8.2

R 4.0.0

RPY2 3.3.4

numpy 1.19.0

samtools 1.17

htslib 1.17

Upon running the command consisting of

python3 danpos.py dtriple --out path/to/output/folder /path/to/data.BAM:/path/to/control.BAM

I get the following error

parsing from Bam file /mnt/Storage-2/raw-data/MNase-seq/aligned/BAM-files/q30+reads-only/TCDD/TCDD.q30.bam ...
Traceback (most recent call last):
  File "danpos.py", line 1366, in <module>
    if sys.argv[1]=='dpos':runDANPOS(command='dpos')
  File "danpos.py", line 397, in runDANPOS
    danpos(\
  File "/home/eric/Programs/DANPOS3-master/functions.py", line 63, in danpos
    groups[groupname]=loadinput(groups[groupname],fs=fs,cut=cut,save=False,wgfmt=wgfmt,\
  File "/home/eric/Programs/DANPOS3-master/functions.py", line 822, in loadinput
    rd=reads(infile,step=step,paired=paired,cut=cut,format='bam')
  File "/home/eric/Programs/DANPOS3-master/reads.py", line 70, in __init__
    elif file!="" and format=='bam':self.loadBam(file=file,step=self.step,cut=cut)
  File "/home/eric/Programs/DANPOS3-master/reads.py", line 653, in loadBam
    self.data[chr]['+'].resize(sizes[chr],refcheck=0)
TypeError: 'float' object cannot be interpreted as an integer

Based on line 653 in reads.py I'm pretty sure this is an issue with Numpy. Or could it be an issue with my BAM files?

[C Schu]

There are a number of lines that perform floating point division with integer arguments where Python2 would have performed integer division (e.g. lines 301, 303, but there are more). I would bet that numpy.array.resize takes integers and not floats as arguments, which would cause the problem. For now, maybe just patch it yourself with doing integer division in the respective lines ( use // instead of / ). I don't have time for testing this, but given the error and what seems to have caused it, this should be an appropriate fix. Maybe @sklasfeld can amend her port?

However, there are other division instructions involving the midpoint calculation that convert the result back into an integer present in the code (e.g. line 576: mid=int(np.floor((col[3]+col[7]+col[9])/(2*step))) ), which makes me think that all those previous integer divisions might need to be modified as well.

Hope that helps.

Christian

I assume that 

[Sammy Klasfeld]

This is correct. I tried to patch these lines that worked for Python2 and not Python3, but may have missed some. Feel free to pull the repository and see if it works now. Otherwise, let me know and I can add more modifications. 

Thanks for the feedback,

Sammy

[Eric Wright]

Thanks for the quick response.

I got further, though while parsing reads, I got the following error,

Traceback (most recent call last):
  File "danpos.py", line 1366, in <module>
    if sys.argv[1]=='dpos':runDANPOS(command='dpos')
  File "danpos.py", line 397, in runDANPOS
    danpos(\
  File "/home/eric/Programs/DANPOS3-master/functions.py", line 63, in danpos
    groups[groupname]=loadinput(groups[groupname],fs=fs,cut=cut,save=False,wgfmt=wgfmt,\
  File "/home/eric/Programs/DANPOS3-master/functions.py", line 822, in loadinput
    rd=reads(infile,step=step,paired=paired,cut=cut,format='bam')
  File "/home/eric/Programs/DANPOS3-master/reads.py", line 70, in __init__
    elif file!="" and format=='bam':self.loadBam(file=file,step=self.step,cut=cut)

  File "/home/eric/Programs/DANPOS3-master/reads.py", line 661, in loadBam
    if cut>0:self.rvClonal(cut=cut)
TypeError: '>' not supported between instances of 'str' and 'int'

any additional help is much appreciated

Thanks

[Sammy Klasfeld]

I updated the code again. Sorry about the bugs. Let me know if it works okay now.

Thanks,

Sammy

[Eric Wright]

Thanks for looking into this again. I got further this time, still had an error in what I believe was the last step of the dtriple analysis.

Traceback (most recent call last):

  File "danpos.py", line 1369, in <module>
    elif sys.argv[1]=='dtriple':runDANPOS(command='dtriple')
  File "danpos.py", line 307, in runDANPOS
    danpos(\
  File "/home/eric/Programs/DANPOS3-master/functions.py", line 357, in danpos
    smtgroups[groupname]=pooledgroups[groupname].callPositions(os.path.join(opath,'pooled',groupname+addname+"positions.xls"),width=width,distance=distance,edge=0,fill_gap=fill_gap,fill_value=fill_value,pcut=pheight,height=height,calculate_P_value=0,poscal=1,regions=peaks,rd=rd)
  File "/home/eric/Programs/DANPOS3-master/wig.py", line 381, in callPositions
    smts.merge(wg=twig,distance=distance)
  File "/home/eric/Programs/DANPOS3-master/summits.py", line 142, in merge
    nps[ni],nvs[ni]=(ps[-2]+ps[-1])/2,wg.data[cr][((ps[-2]+ps[-1])/2)/wg.step] ###### added by Kaifu Chen Jul 10,2012 ######
IndexError: only integers, slices (`:`), ellipsis (`...`), numpy.newaxis (`None`) and integer or boolean arrays are valid indices
R[write to console]: Warning messages:

R[write to console]: 1:
R[write to console]: In ppois(q, avg, lower.tail = FALSE, log = TRUE) :
R[write to console]:  NaNs produced

R[write to console]: 2:
R[write to console]: In ppois(q, avg, lower.tail = FALSE, log = TRUE) :
R[write to console]:  NaNs produced

I was able to successfully run dtriple with these BAM files previously with DANPOS2.2.2, when I saw it was updated to work with Python3 I was really excited to no longer have to maintain multiple versions of Python and associated dependencies on my computer. Overall, it's getting closer it seems. Unfortunately I have zero experience in writing or editing programs so I can't really be of help other than test driving it, so thank you for your continued help on this.

[Sammy Klasfeld]

I fixed that bug. If you get another error do you mind sending me part of the files you are using? You can just give me one chromosome using `samtools view [file.bam] [chromosome_1]` so that the data is small enough to easily share (and doesn't give away important data info). It doesn't need to be labeled or anything. Otherwise, if you don't feel comfortable I can try my hardest to fix the bugs as they come along.

Thanks for letting me know how it goes,

Sammy

[Eric Wright]

Got what looks like another numpy issue.

Traceback (most recent call last):
  File "danpos.py", line 1369, in <module>
    elif sys.argv[1]=='dtriple':runDANPOS(command='dtriple')
  File "danpos.py", line 307, in runDANPOS
    danpos(\
  File "/home/eric/Programs/DANPOS3-master/functions.py", line 357, in danpos
    smtgroups[groupname]=pooledgroups[groupname].callPositions(os.path.join(opath,'pooled',groupname+addname+"positions.xls"),width=width,distance=distance,edge=0,fill_gap=fill_gap,fill_value=fill_value,pcut=pheight,height=height,calculate_P_value=0,poscal=1,regions=peaks,rd=rd)
  File "/home/eric/Programs/DANPOS3-master/wig.py", line 381, in callPositions
    smts.merge(wg=twig,distance=distance)

  File "/home/eric/Programs/DANPOS3-master/summits.py", line 143, in merge
    nps[ni](ps[-2]+ps[-1])/2
TypeError: 'numpy.int64' object is not callable

R[write to console]: Warning messages:

R[write to console]: 1:
R[write to console]: In ppois(q, avg, lower.tail = FALSE, log = TRUE) :
R[write to console]:  NaNs produced

R[write to console]: 2:
R[write to console]: In ppois(q, avg, lower.tail = FALSE, log = TRUE) :
R[write to console]:  NaNs produced

I'll try to get you a small chunk of my BAM files to test.

Thanks!

[C Schu]

Eric, as a quick patch, just change line 143 in summits.py to:

 nps[ni] = (ps[-2]+ps[-1])/2

[Eric Wright]

Hi C Schu,

Thanks for the quick patch. Looks good so far. I implemented that in conjunction with Sammy Klasfeld's idea of using just a single chromosome. Testing with my full samples was taking ~4 hours just to find out if there's a syntax issue.

Was able to run just half a chromosome in about 6 minutes. Looks pretty good now.

total time elapsed: 0.10416666666666667 hours 0.0 minutes 0.0 seconds

job done, cheers!

R[write to console]: Warning messages:

R[write to console]: 1:
R[write to console]: In ppois(q, avg, lower.tail = FALSE, log = TRUE) :
R[write to console]:  NaNs produced

R[write to console]: 2:
R[write to console]: In ppois(q, avg, lower.tail = FALSE, log = TRUE) :
R[write to console]:  NaNs produced

I got this warning message at the very end. Not sure if this is actually an error? Perhaps it's just an effect of me running 1 sample of treatment vs 1 sample of control?

[C Schu]

This seems to be an R issue: https://stat.ethz.ch/R-manual/R-devel/library/stats/html/Poisson.html

Whether this has any influence on your results, I have no idea.

[Eric Wright]

I'm actually running into an issue now where my whole system is locking up, totally unresponsive. Had everything logged in the terminal with 'python3 -m trace --trace danpos.py dtriple', the last line in the terminal when it locked up was

    --- modulename:_rinterface_capi, funcname: _TYPEOF

 _rinterface_capi(218):    return robj.sxpinfo.type

Seems to be an R issue? Or more specifically how RPY2 interfaces with R? Not totally sure

I'm curious what version of RPY2 you were using Sammy?

[Sammy Klasfeld]

I use rpy2 version 3.3.3

[Eric Wright]

I rolled my system back from Ubuntu 20.04 to 18.04 and it doesn't totally lock up when running DANPOS anymore, I have no idea what the issue was. Now running Python 3.6.9, Numpy 1.19, rpy2 3.3.5, R 4.0.2, samtools 1.7.

Running my full size samples however, I got the following error

34989477 70 18201306
34989477 80 18201306
34989477 90 18201306
34989477 100 18201306

Traceback (most recent call last):
  File "danpos.py", line 1369, in <module>
    elif sys.argv[1]=='dtriple':runDANPOS(command='dtriple')

  File "danpos.py", line 333, in runDANPOS
    pcfer=0)#args.pcfer)#0)
  File "/home/eric/Programs/DANPOS3-master/functions.py", line 427, in danpos
    fuzFDRlist=fuzFDR(cwig=pooledgroups[groupnames[1]],twig=pooledgroups[groupnames[0]],simu=fdrsimu,rd=rd,regions=peaks)
  File "/home/eric/Programs/DANPOS3-master/functions.py", line 1178, in fuzFDR
    vec[id]=tempv[0]
IndexError: index 100000 is out of bounds for axis 0 with size 100000