sambamba-sort error: index exceeds array of length

[Maxine Liu]

Hi, Pj

I'm dealing with very big aligned BAM files (1.3T per file), and I use sambamba sort to sorting them. To clarify, the chromosome length of the research organism is more than 2^29 bases, so a bai&tbi index is not fit. The reason why I mention this situation is I think that's probably the reason for the error msg.

The conmand I use is:

sambamba sort -m 185G -t 44 -p --tmpdir=/home/maxine91/scratch/pbmm2/cy201704 pbaln.cy201704.bam

When sambamba starting to merge sorted chunks, the error appeared:

sambamba-sort: index [32770] exceeds array of length 32770

How can I avoid this error? Thank you!

Maxine

[Nick Rozinsky]

Forwarding.

This issue may be relevant

https://github.com/biod/sambamba/issues/284

---------- Forwarded message ---------
From: Maxine Liu <maxin...@gmail.com>
Date: Thu, Jul 21, 2022, 19:56
Subject: Re: [sambamba-discussion] sambamba-sort error: index exceeds array of length
To: Nick Rozinsky <nick.r...@gmail.com>

Hi, Nick

Thank you so much for the response.

I can't tell how long it takes to reach merging stage. The total process (reach the error msg) with 44 cores and 185G RAM takes about 2.5 hours. I run sambamba again without progressbar. This time, it takes 5 hours to reach the exactly identical error msg with 32cores and 125G. I can't run sambamba through  GDB because of technical problem. 

I notice ".sorted.bam.bai" is created as index for the ".sorted.bam". "The BAI index format can handle individual chromosomes up to 512 Mbp (2^29 bases) in length. If your input file might contain reads mapped to positions greater than that, you will need to use a CSI index." I think that is likely the reason.

Maxine

Nick Rozinsky <nick.r...@gmail.com> 于2022年7月20日周三 19:10写道：

Hey Maxine,

Do you run it with progressbar enabled? Try without it. Also maybe you can launch it through gdb: https://stackoverflow.com/a/6121299

And get a stacktrace with backtrace command in gdb.

How long does it take to reach merging stage?

--
You received this message because you are subscribed to the Google Groups "sambamba-discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sambamba-discus...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/sambamba-discussion/c3a8371c-1d94-45a9-8cca-68f4faa65533n%40googlegroups.com.

[Nick Rozinsky]

There is probably no support for csi index. However I think that index may be not necessary for sorting. Bam writer seemingly creates index by default. You can disable the index creation but that would require rebuilding sambamba.

https://github.com/biod/sambamba/blob/c795656721b3608ffe7765b6ab98502426d14131/BioD/bio/std/hts/bam/writer.d#L109

https://github.com/biod/sambamba/blob/c795656721b3608ffe7765b6ab98502426d14131/sambamba/sort.d#L394

On Thu, Jul 21, 2022, 22:15 Maxine Liu <maxin...@gmail.com> wrote:

If I understand correctly, the function that supports CSI has not been added in sambamba yet. Am I right?  

Nick Rozinsky <nick.r...@gmail.com> 于2022年7月21日周四 13:18写道：