bamCoverage v1.6 error

[David Valle]

Dear Deeptools team:

We are having some trouble running bamCoverage (v1.6.0-640). Do you have any suggestion?

This is the command:

bamCoverage --outFileFormat=bigwig --ignoreDuplicates --binSize=10 --numberOfProcessors=max/2 --extendReads=147 --normalizeUsingRPKM -b SKMel147-H3K27ac-l70n2m20k1.bam -o /Seq-5/Enhancers/Analysis/Enc-Nov-2015/bigwig/bigwig-new/SKMel147-H3K27ac-N.bw

Here is the error:

ignoreDuplicates: True

numberOfProcessors: 4

maxPairedFragmentLength: 588

samFlag_exclude: None

verbose: False

samFlag_include: None

region: None

save_data: False

out_file_for_raw_data: None

binLength: 10

numberOfSamples: None

stepSize: 10

defaultFragmentLength: 147

bedFile: None

center_read: False

chrsToSkip: []

bamFilesList: ['SKMel147-H3K4me1-l70n2m20k1.bam']

smoothLength: 0

minMappingQuality: None

zerosToNans: False

ignoreDuplicates: True

numberOfProcessors: 4

maxPairedFragmentLength: 588

samFlag_exclude: None

verbose: False

samFlag_include: None

region: None

save_data: False

out_file_for_raw_data: None

binLength: 10

numberOfSamples: None

stepSize: 10

defaultFragmentLength: 147

bedFile: None

center_read: False

chrsToSkip: []

bamFilesList: ['SKMel147-H3K27ac-l70n2m20k1.bam']

smoothLength: 0

minMappingQuality: None

zerosToNans: False

Traceback (most recent call last):

  File "/usr/local/bin/bamCoverage", line 7, in <module>

    main()

  File "/usr/local/lib/python2.7/dist-packages/deeptools/bamCoverage.py", line 218, in main

    format=args.outFileFormat, smooth_length=args.smoothLength)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 155, in run

    chrom_names_and_size, bedgraph_file, out_file_name, True)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 317, in bedGraphToBigWig

    assert(bw is not None)

AssertionError

Traceback (most recent call last):

  File "/usr/local/bin/bamCoverage", line 7, in <module>

    main()

  File "/usr/local/lib/python2.7/dist-packages/deeptools/bamCoverage.py", line 218, in main

    format=args.outFileFormat, smooth_length=args.smoothLength)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 155, in run

    chrom_names_and_size, bedgraph_file, out_file_name, True)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 317, in bedGraphToBigWig

    assert(bw is not None)

AssertionError

--

David Valle García

Exchange PhD Student
Emily Bernstein Lab
Oncological Sciences Department
Icahn School of Medicine at Mount Sinai

[Devon Ryan]

Hi David,

Does "/Seq-5/Enhancers/Analysis/Enc-Nov-2015/bigwig/bigwig-new" exist
and do you have write permissions?

As an aside, you might want to upgrade to the most recent version,
there have been some notable performance increases and added features!

Best,
Devon
--
Devon Ryan, Ph.D.
Email: dpr...@dpryan.com
Data Manager/Bioinformatician
Max Planck Institute of Immunobiology and Epigenetics
Stübeweg 51
79108 Freiburg
Germany

> --
> You received this message because you are subscribed to the Google Groups
> "deepTools" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to deeptools+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Fidel Ramirez]

Can you test that you have write permissions to /Seq-5/Enhancers/Analysis/Enc-Nov-2015/bigwig/bigwig-new/SKMel147-H3K27ac-N.bw

Can you test that your bam file is not empty? (samtools view SKMel147-H3K27ac-l70n2m20k1.bam) 

-fidel

On Tue, Mar 22, 2016 at 11:12 AM, David Valle <dva...@email.ifc.unam.mx> wrote:

--

Fidel Ramirez

[Hasson, Dan]

Hi,

Thank you for your reply. Taking the opportunity to mention we are very pleased with deeptools. It is very helpful in our analysis!!

In regards to bamCoverage:

-The bigwig-new directory, does exist.

-Writing permissions, I do have them. Also, when running the bamCoverage, the program was creating a bed graph file which I guess confirm my permissions (not as expected by the way since I have -outFileFormat=bigwig, and the file had bw.bg extension)

-The bam file is not empty. I previously used it for other analysis and actually for running some other tools with deeptools.

Please let me know what you think.

Thank you so much in advance.

dan.

[Fidel Ramirez]

Hi,

Then the problem could be the conversion from bedgraph to bigwig. In the 1.6 release we were using UCSC tools bedGraphToBigWig to do the conversion. So please check that you have this program installed and that it is in your PATH. 

In the newer version of deepTools2 we use our own library to produce bigwig files.

-fidel

--

Fidel Ramirez

[Hasson, Dan]

Thank you. The version we work with is 2.2.2. Based on the error message it is indeed seem that something is happening at the bed graph to bigwig transition (see below).

File "/usr/local/lib/python2.7/dist-packages/deeptools/bamCoverage.py", line 218, in main

    format=args.outFileFormat, smooth_length=args.smoothLength)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 155, in run

    chrom_names_and_size, bedgraph_file, out_file_name, True)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 317, in bedGraphToBigWig

    assert(bw is not None)

Is there a newer version that I should install?

Thank you,

dan.

On Mar 22, 2016, at 12:16 PM, Fidel Ramirez <fidel....@gmail.com> wrote:

bedgraph to bigwig

[Hasson, Dan]

Sorry. We had a confusion in regards to the versions. I will try again and let you know.

Thank you.

dan.

On Mar 22, 2016, at 12:16 PM, Fidel Ramirez <fidel....@gmail.com> wrote:

[Hasson, Dan]

Hi again,

Ran the program with the 2.2 version. Still the same error message (see below):

Bedtobig is installed and running as well.

Thank you,

dan.

binLength: 10

blackListFileName: None

minMappingQuality: None

zerosToNans: False

Traceback (most recent call last):

  File "/usr/local/bin/bamCoverage", line 7, in <module>

    main()

  File "/usr/local/lib/python2.7/dist-packages/deeptools/bamCoverage.py", line 251, in main

    format=args.outFileFormat, smoothLength=args.smoothLength)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 158, in run

    chrom_names_and_size, bedgraph_file, out_file_name, True)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 320, in bedGraphToBigWig

    assert(bw is not None)

AssertionError

On Mar 22, 2016, at 12:16 PM, Fidel Ramirez <fidel....@gmail.com> wrote:

[Devon Ryan]

Hi Dan,

The only way that should be possible across these versions is if you lack write permissions, the partition is full, or something like that. Try a different directory.

Devon

--
You received this message because you are subscribed to the Google Groups "deepTools" group.
To unsubscribe from this group and stop receiving emails from it, send an email to deeptools+...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

-- 
Devon Ryan, PhD
Bioinformatician / Data manager
Bioinformatics Core Facility
Max Planck Institute for Immunobiology and Epigenetics
Email: dpry...@gmail.com

[Hasson, Dan]

Thank you for your replay. Please have a loo below:

4th line you can see the permissions for the bigwig-new directory. Then you can see the bw.bg file created by bamCoverage in the same directory.

Below that is the space available. You can see that bigwig-new is a directory within Seq-5 still has 400GB available.

Lastly, you can see the bg file which is not empty which means the bam file is not empty. 

I am wondering why do I still get the bedtobig error. You mentioned that the newer version doesn’t call this function any longer. 

Also, we did run the program in another directory using other files. We did get the same error message. Please see below. However, in this case we also see something about not enough space for sorting.

david@hafiz:/seq-3/ATRX/Bowtie/K562$ bamCoverage --outFileFormat=bigwig --ignoreDuplicates --binSize=10 --numberOfProcessors=max/2 --extendReads=147 --normalizeUsingRPKM -b hg19_K562_H3k9me3-Broad.bam -o ./hg19_K562_H3k9me3-Broad.bw
verbose: False
out_file_for_raw_data: None
numberOfSamples: None
bedFile: None
bamFilesList: ['hg19_K562_H3k9me3-Broad.bam']
ignoreDuplicates: True
numberOfProcessors: 4
samFlag_exclude: None
save_data: False
blackList: None
stepSize: 10
smoothLength: None
center_read: False
defaultFragmentLength: 147
chrsToSkip: []
region: None
maxPairedFragmentLength: 588
samFlag_include: None

binLength: 10
blackListFileName: None
minMappingQuality: None
zerosToNans: False

sort: write failed: standard output: No space left on device
sort: write error

Traceback (most recent call last):
  File "/usr/local/bin/bamCoverage", line 7, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/deeptools/bamCoverage.py", line 251, in main
    format=args.outFileFormat, smoothLength=args.smoothLength)
  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 158, in run
    chrom_names_and_size, bedgraph_file, out_file_name, True)
  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 320, in bedGraphToBigWig
    assert(bw is not None)
AssertionError

Please advice!!

Thanks so much and sorry for the inconvenience.

David and Dan.

[Devon Ryan]

Hi Dan,

It's possible that having only 2.4 gigs left in / is the problem. Under the hood, deepTools is writing a bunch of plain text bedGraph files in /tmp. These then need to be sorted and merged in order to then make a bigWig file. You might modify the deepTools config file and set the "tmp_dir" elsewhere. That might solve the problem.

Devon

[Hasson, Dan]

Hi,

We made the change you suggested and rerun the program. Please see the error message below:

blackListFileName: None

minMappingQuality: None

zerosToNans: False

sort: write failed: /tmp/sortaHStMz: No space left on device

Traceback (most recent call last):

  File "/usr/local/bin/bamCoverage", line 7, in <module>

    main()

  File "/usr/local/lib/python2.7/dist-packages/deeptools/bamCoverage.py", line 251, in main

    format=args.outFileFormat, smoothLength=args.smoothLength)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 158, in run

    chrom_names_and_size, bedgraph_file, out_file_name, True)

  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 320, in bedGraphToBigWig

    assert(bw is not None)

AssertionError

It seems as if it does not recognize the change we did (the sorting is now at Seq-5/tmp/)?!

Thanks,

dan.

On Mar 22, 2016, at 4:32 PM, Devon Ryan <dpry...@gmail.com> wrote:

Hi Dan,

It's possible that having only 2.4 gigs left in / is the problem. Under the hood, deepTools is writing a bunch of plain text bedGraph files in /tmp. These then need to be sorted and merged in order to then make a bigWig file. You might modify the deepTools config file and set the "tmp_dir" elsewhere. That might solve the problem.

Devon

On 03/22/2016 09:23 PM, Hasson, Dan wrote:

Thank you for your replay. Please have a loo below:

4th line you can see the permissions for the bigwig-new directory. Then you can see the bw.bg file created by bamCoverage in the same directory.

Below that is the space available. You can see that bigwig-new is a directory within Seq-5 still has 400GB available.

Lastly, you can see the bg file which is not empty which means the bam file is not empty. 

I am wondering why do I still get the bedtobig error. You mentioned that the newer version doesn’t call this function any longer. 

<Mail Attachment.png>

[Fidel Ramirez]

Hi,

How did you change the path for the temp dir?

The way to do this is by modifying the deeptools.cfg file that is located on the installation directory. In case you are not sure where such file is, the following lines can help you on that:

Enter the python shell:

$ python

Then in the python shell type

import deeptools

print deeptools.__file__

In this path (remove the __init__.py) you can find the file `config/deeptools.cfg`

Best,

Fidel

--

Fidel Ramirez

[Devon Ryan]

Out of curiosity, what version of pyBigWig do you have installed?
Someone just posted a similar issue on github that was due to having
an old version installed. I suspect that that's not the root cause in
this case, but just in case...

Devon
--
Devon Ryan, Ph.D.
Email: dpr...@dpryan.com
Data Manager/Bioinformatician
Max Planck Institute of Immunobiology and Epigenetics
Stübeweg 51
79108 Freiburg
Germany

[David Valle]

Dear Ryan and Fidel:

Yes, I changed the deeptools.cfg file. Actually, while running the command we checked the new tmp folder (/Seq-5/tmp with ~500Gb of space) and saw that deeptools was writing files to it correctly (a bunch of _deeptools_xxxx.bg files). Because we were getting the same sort issue I figured deeptools may be calling the system's sort command so I set up the environment variable $TMPDIR to /Seq-5/tmp. Now the command is generating a sorted bedgraph and the sort error disappeared but I keep getting the Assertion error:

Traceback (most recent call last):
  File "/usr/local/bin/bamCoverage", line 7, in <module>
    main()
  File "/usr/local/lib/python2.7/dist-packages/deeptools/bamCoverage.py", line 251, in main
    format=args.outFileFormat, smoothLength=args.smoothLength)
  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 158, in run
    chrom_names_and_size, bedgraph_file, out_file_name, True)
  File "/usr/local/lib/python2.7/dist-packages/deeptools/writeBedGraph.py", line 320, in bedGraphToBigWig
    assert(bw is not None)
AssertionError

You mentioned that this new version of deeptools uses its own bg2bw program but I see that the error is coming from something called "bedGraphToBigWig". Is that a function from deeptools or is it looking for the UCSC binary? (anyway we have all the UCSC binaries installed and available).

Our pyBigWig version is 1.0.8 (it's the latest version according to pip).

Because we already hav the bg we could just convert it to a bw, but do you know if the values on that file are already normalized? (we asked for RPKM normalization in the command).

Thanks,

David and Dan

[Devon Ryan]

Hi Dan and David,

Pip is apparently lying to you, please upgrade to pyBigWig version 0.2.7 and this error should go away. I'm not sure why pypi is occasionally giving people a non-existent version.

Devon

[Hasson, Dan]

Hi,

Finally we were able to get it to work and generate the normalized (RPKM) bigwig files.

Thank you all for your support.

Lovely weekend,