Bamqc is hanging, and pdf is damaged

[AT]

I am running the following command:

nohup qualimap_v2.1.2/qualimap bamqc -bam ../tophat_A04/accepted_hits.bam -gd HUMAN -ip -outdir A04/ -outfile A04.def.qualimap -sd &

This seems to be running ok, and I get the following output as it runs:

Java memory size is set to 1200M

Launching application...

QualiMap v.2.1.2

Built on 2015-09-23 14:22

Selected tool: bamqc

Available memory (Mb): 33

Max memory (Mb): 1118

Thu Nov 12 12:37:57 EST 2015            WARNING output folder already exists

Starting bam qc....

Loading sam header...

Loading locator...

Loading reference...

Number of windows: 400, effective number of windows: 483

Chunk of reads size: 1000

Number of threads: 96

Processed 50 out of 483 windows...

Processed 100 out of 483 windows...

Processed 150 out of 483 windows...

Processed 200 out of 483 windows...

Processed 250 out of 483 windows...

Processed 300 out of 483 windows...

Processed 350 out of 483 windows...

Processed 400 out of 483 windows...

Processed 450 out of 483 windows...

Total processed windows:483

Number of reads: 70445308

Number of valid reads: 2425398

Number of duplicated reads: 0

Number of correct strand reads:0

Number of reads with

Inside of regions...

Num mapped reads: 70445308

Num mapped first of pair: 36133389

Num mapped second of pair: 34275363

Num singletons: 27487234

Time taken to analyze reads: 181

Computing descriptors...

numberOfMappedBases: 275132705

referenceSize: 3101804741

numberOfSequencedBases: 144737865

numberOfAs: 42482798

Computing per chromosome statistics...

Computing histograms...

Overall analysis time: 182

end of bam qc

Computing report...

But, the program continues to show as running for a long time, and when I try to open the *pdf file, it complains that the PDF file is damaged.

Any help will be great.

Thanks,

Arti

 

[Konstantin Okonechnikov]

Hi,

thanks a lot for the report!

Did you try launch the latest version (v2.1.3) ?   Which OS and which Java version are applied? 

Does this issue occur only a specific file or on some others as well?

It would be great if you could share the problematic BAM file, so I can check it in detail.

--

  Konstantin

--
You received this message because you are subscribed to the Google Groups "QualiMap" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qualimap+u...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Daniel Sobral]

I also have that problem, usually with samples that have very little data or the reference is incorrect (so probably no alignments).
Qualimap seems to hang eternally for no reason at all...

Daniel

[Konstantin Okonechnikov]

Hi Daniel,

Thanks for info! 

Is it possible that you could share privately the datasets (along with applied commands) where Qualimap got stuck?

Then I'll be able to check the issue in detail. I got already some reports about hang, but I can not reproduce this issue.

--

  Konstantin

[Kim Warren]

I have also been encountering this problem, with the pdf creation at the end of bamqc hanging indefinitely. My input bam file is 6.5GB, though, so it's tricky to find a reasonable way to send it to you. 

The command I'm using from the command line: ../qualimap_v2.1.3/qualimap bamqc -bam Rhph_mix_BWA_Sorted.bam -outformat pdf -outfile Rhph_mix_BWA.pdf-bash-4.1

OS: Scientific Linux release 6.2 (Carbon)

Java version: 1.6.0_37

Output: 

Java memory size is set to 1200M

Launching application...

Picked up _JAVA_OPTIONS: -Xmx2G

QualiMap v.2.1.3

Built on 2015-10-29 09:46

Selected tool: bamqc

Available memory (Mb): 32

Max memory (Mb): 1908

Mon Dec 21 12:12:57 GMT 2015            WARNING Output folder already exists, the results will be saved there

Starting bam qc....

Loading sam header...

Loading locator...

Loading reference...

Number of windows: 400, effective number of windows: 89144

Chunk of reads size: 1000

Number of threads: 48

Processed 50 out of 89144 windows...

Processed 100 out of 89144 windows...

[Continues in increments of 50 for quite a while]

Processed 89050 out of 89144 windows...

Processed 89100 out of 89144 windows...

Total processed windows:89144

Number of reads: 71864022

Number of valid reads: 67787216

Number of duplicated reads: 0

Number of correct strand reads:0

Number of reads with

Inside of regions...

Num mapped reads: 67787216

Num mapped first of pair: 34332766

Num mapped second of pair: 33454450

Num singletons: 3760441

Time taken to analyze reads: 1991

Computing descriptors...

numberOfMappedBases: 9265630233

referenceSize: 2356967742

numberOfSequencedBases: 9209220250

numberOfAs: 2732062175

Computing per chromosome statistics...

Computing histograms...

Overall analysis time: 2051

[Arumilli, Meharji]

Dear all,

I too have trouble with bamqc. Below is the command and error:

$/qualimap_v2.1.3/qualimap bamqc -bam TMD8_recalibrated.bam -gff hg19_coverage_summary.bed -outformat PDF -c --java-mem-size=12G
Java memory size is set to 12G
Launching application...

QualiMap v.2.1.3
Built on 2015-10-29 09:46

Selected tool: bamqc

Available memory (Mb): 33
Max memory (Mb): 11453

Starting bam qc....
Loading sam header...
Loading locator...
Loading reference...

Number of windows: 400, effective number of windows: 424

Chunk of reads size: 1000

Number of threads: 16
Initializing regions from hg19_coverage_summary.bed.....
Found 8260 regions
Filling region references...
Failed to run bamqc
java.lang.NumberFormatException: For input string: ""
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Integer.parseInt(Integer.java:504)
    at java.lang.Integer.parseInt(Integer.java:527)
    at org.bioinfo.ngs.qc.qualimap.common.GenomicFeatureStreamReader$2.parseFeatureRecord(GenomicFeatureStreamReader.java:75)
    at org.bioinfo.ngs.qc.qualimap.common.GenomicFeatureStreamReader.readNextRecord(GenomicFeatureStreamReader.java:186)
    at org.bioinfo.ngs.qc.qualimap.process.BamStatsAnalysis.loadSelectedRegions(BamStatsAnalysis.java:919)
    at org.bioinfo.ngs.qc.qualimap.process.BamStatsAnalysis.run(BamStatsAnalysis.java:260)
    at org.bioinfo.ngs.qc.qualimap.main.BamQcTool.execute(BamQcTool.java:213)
    at org.bioinfo.ngs.qc.qualimap.main.NgsSmartTool.run(NgsSmartTool.java:187)
    at org.bioinfo.ngs.qc.qualimap.main.NgsSmartMain.main(NgsSmartMain.java:111)


Could someone offer help to fix this.

Br
Mehar

[Konstantin Okonechnikov]

Hi!

Thanks a lot for the reports! 

Actually, the first issue (report from Kim Warren) occurs most likely due to PDF format output. The problem is that there are too many covering windows and PDF output got stuck (there was a similar report previously: https://bitbucket.org/kokonech/qualimap/issues/30/slow-pdf-format-output ). Fixing this problem currently is possible by performing output in HTML format.  Would be great if you could check this. 

The second issue (report from Meharji  Arumilli)  is completely different. This is a problem with annotation BED file. How was it generated? Could you perhaps share it, so I can check?

--

  Konstantin

[Laura Dijkhuizen]

Hello all,

I have run in to very similar problems as AT and Daniel. Both with pdf and with html output, bamqc hangs at 'Computing report' for at least two hours. Thus changing to HTML output was, in my situation, not a suitable fix. Also, this issue persisted for various combinations of reference genomes and sequencing data. 

Laura

Ubuntu 15.10  kernel 4.2.0-16

R 3.2.3

Java versiom 1.7.0_91 (7u91-2.6.3-0ubuntu0.15.10.1)

qualimap 2.1.3

[Konstantin Okonechnikov]

Hi Laura,

thanks a lot for the report! 

The best way to figure out the issue is to share the data (BAM file, annotations), exact command that was applied and full log, so I can take look on it in detail. No data from Arti or Daniel was available and I could not reproduce the bug.

--

  Konstantin

[Laura Dijkhuizen]

Hi Konstantin, 

thanks for your rapid reply. I will gladly elaborate on my situation, the sequencing data however is not mine to share. Alternatively, I could perhaps process some illumina sequencing of your choosing and share results thereof. 

In all cases, I have mapped +/- 10 million reads (100 to 150bp PE) to various references using bowtie2 as such.

bowtie2 -x $dir/refgenomes/$rf/$rf.btindex \

                                -1 $reads/$s/*subset*.mreads_R1.fq \

                                -2 $reads/$s/*subset*.mreads_R2.fq \

                                --minins $minins        \

                                --maxins $maxins        \

                                --threads $CPU  \

                                --very-sensitive | \

                                        samtools view -@ 6 -bS -F 0x040  - | \

                                        samtools sort -@ 6 -m 4G - -f $rf.$s.bam

I then run qualimap bamqc on all bam files in a specific folder

../scripts/qualimap/qualimap bamqc -bam ./bowtie_results/$f -outdir ./bowtie_results/$name

As qualimap starts, all seems normal. However, at the 'computing report...' stage, the programme hangs for hours using one CPU core for a 100%.

Java memory size is set to 1200M

Launching application...

QualiMap v.2.1.3

Built on 2015-10-29 09:46

Selected tool: bamqc

Available memory (Mb): 33

Max memory (Mb): 1118

Starting bam qc....

Loading sam header...

Loading locator...

Loading reference...

Number of windows: 400, effective number of windows: 400

Chunk of reads size: 1000

Number of threads: 12

Processed 50 out of 400 windows...

Processed 100 out of 400 windows...

Processed 150 out of 400 windows...

Processed 200 out of 400 windows...

Processed 250 out of 400 windows...

Processed 300 out of 400 windows...

Processed 350 out of 400 windows...

Processed 400 out of 400 windows...

Total processed windows:400

Number of reads: 10000000

Number of valid reads: 299

Number of duplicated reads: 0

Number of correct strand reads:0

Number of reads with

Inside of regions...

Num mapped reads: 299

Num mapped first of pair: 0

Num mapped second of pair: 299

Num singletons: 291

Time taken to analyze reads: 18

Computing descriptors...

numberOfMappedBases: 32840

referenceSize: 4641652

numberOfSequencedBases: 32814

numberOfAs: 6678

Computing per chromosome statistics...

Computing histograms...

Overall analysis time: 18

end of bam qc

Computing report...

html pages, figures and a txt file are actually produced by qualimap. The only figures missing are the finel two in the html output: 'insert size across reference' and 'insert size histogram'

I encounter this error for various combinations of sequencing files and reference genomes. If I can be of any further assistance, please let me know. What logs specifically are you interested in?

kind regards, 

Laura

[Konstantin Okonechnikov]

Hi Laura,

thanks a lot for the detailed report, I have several questions. 

From the log I see an interesting detail: only 299 reads from 10 000 000 are marked as mapped. Moreover, all of them are the second of a pair (which might introduce a problem in computing insert size plots). 

Are there any insert size statistics reported in the Summary?

Could you perhaps extract only a subsample with mapped reads from the BAM file (and make it smaller, i.e 10 or 20 alignments) and try Qualimap once again?

This SAMtools command might be helpful to extract only mapped reads:

samtools view -h -F 0x4 file.bam

If the issue happens for the small subsample as well, so maybe you can report how at least one alignment record looks like. Or maybe a small subsample can be shared privately. This was actually performed several times previously and bugs were fixed fast. 

Just in case, my e-mail address: k.okonechnikov[at]gmail[dot]com

--

  Konstantin

[Laura Dijkhuizen]

Hi Konstantin,

you indirectly found a typo in my script, explaining why only the second pairs are represented. ( samtools view -F 0x040 vs samtools view -F 0x04 ) I am running the whole thing again, and both read pairs map now. However, the bug remains, qualimap gets stuck at the same point. (so no solution there) Indeed 299 is a small proportion, this specific reference is my negative control. I tried with some of my own data, which I will e-mail you shortly.

FYI, indeed 299 reads is a small proportion. The reference genome is the negative control so I expected it to be low. 

Kind regards, Laura

[Konstantin Okonechnikov]

Hi Laura,

I received the BAM files and checking them, I'll reply to your message.

--

  Konstantin

[VAst]

Hey Konstantin,

I'm having the same issue that qualimap got stuck and doesn't create the insert size histograms.

I would like to know if you could solve this issue?

I'm using qualimap v.2.2.1, Java 1.8.0_212

Cheers,

Volker

Am Donnerstag, 14. Januar 2016 14:52:25 UTC+1 schrieb Konstantin Okonechnikov:

Hi Laura,

I received the BAM files and checking them, I'll reply to your message.

--

  Konstantin

To unsubscribe from this group and stop receiving emails from it, send an email to qual...@googlegroups.com.

[Konstantin Okonechnikov]

Hi Volker,

Did you try the latest build version 2.2.2-dev? Some of similar issues were fixed during development. Just in case, here's the link: https://bitbucket.org/kokonech/qualimap/downloads/?tab=downloads 

Best regards, 

  Konstantin

пн, 24 июн. 2019 г., 9:22 VAst <volke...@gmx.de>:

To view this discussion on the web visit https://groups.google.com/d/msgid/qualimap/1319704b-398b-444c-9ecb-388fb6a9c555%40googlegroups.com.