Please upgrade to new version of bwa as bwa64 provided with DeconSeq fails for some genomes

[Matthew Oldach]

Currently the newer version of bwa does not for Deconseq as it doesn't produce all the required files:

"Indeed, in my database folder I only have 5 files instead of 8, despite the fact the indexing was complete without errors. Apparently it is due to the fact that the version of bwa I use (0.6.2) handles forward and reverse reads differently than the previous versions (< 0.6), thus creating 3 files less then before. "

I've noticed that bwa64 version provided with DeconSeq fails for some genomes. For example, it fails on the following genome from NCBI RefSeq Genomes ASM150690v1 with the error:

Estimate size of input data for status report (this might take a while for large files)
        done
Parse and process input data
^M    status: 0 %^M   status: 1 %^M   status: 2 %^M   status: 3 %^M   status: 4 %^M   status: 5 %^$
Check for duplicates
^M    status: 1 %^M   status: 2 %^M   done
Write results to output file(s)
^M    status: 0 %^M   status: 1 %^M   done
Clean up empty files
        done
Input and filter stats:
        Input sequences: 1
        Input bases: 1,635,130
        Input mean length: 1635130.00
        Good sequences: 1 (100.00%)
        Good bases: 1,635,130
        Good mean length: 1635130.00
        Bad sequences: 0 (0.00%)
        Sequences filtered by specified parameters:
        none
[bwa_index] Pack FASTA... 0.02 sec
[bwa_index] Reverse the packed sequence... 0.02 sec
[bwa_index] Construct BWT for the packed sequence...
BWTIncConstruct() : Not enough memory allocated!


However, when you use version bwa-0.7.15:


Estimate size of input data for status report (this might take a while for large files)
        done
Parse and process input data
^M    status: 0 %^M   status: 1 %^M   status: 2 %^M   status: 3 %^M   status: 4 %^M   status: 5 %^$
Check for duplicates
^M    status: 1 %^M   status: 2 %^M   done
Write results to output file(s)
^M    status: 0 %^M   status: 1 %^M   done
Clean up empty files
        done
Input and filter stats:
        Input sequences: 1
        Input bases: 1,635,130
        Input mean length: 1635130.00
        Good sequences: 1 (100.00%)
        Good bases: 1,635,130
        Good mean length: 1635130.00
        Bad sequences: 0 (0.00%)
        Sequences filtered by specified parameters:
        none
[bwa_index] Pack FASTA... 0.02 sec
[bwa_index] Construct BWT for the packed sequence...
[BWTIncCreate] textLength=3270260, availableWord=4154776
[bwa_index] 1.08 seconds elapse.
[bwa_index] Update BWT... 0.01 sec
[bwa_index] Pack forward-only FASTA... 0.01 sec
[bwa_index] Construct SA from BWT and Occ... 0.24 sec
[main] Version: 0.7.15-r1140
[main] CMD: /export/common/programs/bwa-0.7.15 index -p GCF_001506905.1_ASM150690v1_genomic -a bwtsw$
[main] Real time: 1.620 sec; CPU: 1.367 sec

[Matthew Oldach]

I followed NCBI’s instructions on the FAQ (#1) for downloading multiple genome assemblies using the following query to download 27,260 genomes from RefSeq:

Search: all[filter] AND latest[filter] AND "complete genome"[filter] NOT "Homo sapiens"[Organism]

The frustrating thing was that BWA64 fails and only partially completes all of the index files but by searching for those that did not complete files (i.e. those missing .bwt extension) I found that only 13,661 of the 27,260 (~50%) genomes from NCBI RefSeq can be successfully indexed using this older version of BWA64 which is required by DeconSeq.

The errors for the other 13,126 genomes are either:

1) BWTIncConstruct(): Not enough memory allocated
2) BWTIncSetBuildSizeandTextAdder(): Not enough space allocated to continue construction