[bwt_restore_sa] Failed to allocate 16473698552 bytes

[gmoore777]

Using the Docker set up, I ran an analysis (with 4 cpus) which failed after 12 minutes.

I ran this a 2nd time (with 2 cpus) with the same failure.

It fails during this step:    [Contigs Taxonomy Classification]

This is the error found in ./EDGE_output/.../AssemblyBasedAnalysis/Taxonomy/log.txt and <ProjectName>.log

COMMAND FAILED: bwa mem -a -M -t 2 /home/edge/edge_dev/database/bwa_index/NCBI-Bacteria-Virus.fna temp22998/<ProjectName>_contigs.fa > temp22998/<ProjectName>_contigs.fa.sam 2>> <ProjectName>.log

  [bwt_restore_sa] Failed to allocate 16473698552 bytes at bwt.c line 436: Cannot allocate memory

My Linux machine has 32 GB of memory.

I ran this exact same analysis, previously and successfully, on a different but similar Linux machine that had Edge installed the non-Docker way.

I think the problem has to do with the size of the bwa database file of NCBI-Bacteria-Virus.fna as that is the only difference, I think,

between both Edge installations, other than one is a Docker set up, and the other one is not.

On the non-Docker machine, I used the light version of BWA and the size of NCBI-Bacteria-Virus.fna is approximately 10 GB.

On the Docker machine, that file is 33 GB.

Questions:

1.)

Where in the Edge code does it ask for 16 GB of memory?

2.)

Is this memory allocation calculated on the fly, based on the size of that BWA file?

3.)

Is there a way to tell Edge or BWA to use less memory?

4.)

The man page for bwa at: http://bio-bwa.sourceforge.net/bwa.shtml does not list any memory options,

but it does claim that bwa only uses 5 GB of memory ( which must not be true).

      Memory Requirement  With bwtsw algorithm, 5GB memory is required for indexing the complete human genome sequences.

      For short reads, the aln command uses ~3.2GB memory and the sampe command uses ~5.4GB.

5.)

I do see in the edge installation notes for both Docker and non-Docker:

## For machine with < 32Gb memory, we suggest to use the smaller BWA index (~14Gb) and contains the databases for bwa taxonomic identification pipeline

Is this note specifically for the problem that I encountered?
If yes, is there a way to tell BWA to use less memory? 
Is there another way around this memory problem other than building out a new machine with 64 GB of memory or
installing the smaller BWA index?

Thank you
Guy

[Lo, Chien-Chi]

The memory usage of the bwa run is depends on the bwa index files. I think it will load the index into memory. So, if your machine has hard time to run bwa with the bigger reference sequence database, please use the smaller index file.

 

  # For machine with < 32Gb memory, we suggest to use the smaller BWA index (~14Gb) and contains the databases for bwa taxonomic identification pipeline

  wget -c https://edge-dl.lanl.gov/EDGE/dev/edge_dev_bwa_mini_index.tgz

--
You received this message because you are subscribed to the Google Groups "edge-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to edge-users+...@googlegroups.com.
To post to this group, send email to edge-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/edge-users/0d0e480b-6225-485c-a4df-78b986b2b34f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[gmoore777]

Thank you for your reply.

Since my problem, I kept the large BWA index files, but I up-sized my Linux machine from 32 GB to 64 GB of memory and ran the Edge analysis on my amplicon sequence.

It ran successfully in an expected time of 33 minutes.

I watched the Docker container consume up to 56 GB memory during the BWA step.

I then ran an Edge analysis on a metagenomic sequence, successfully, in an expected time of 8 hours.

That gobbled up 61.47 GB of memory during the BWA step. (just under my memory threshold)

Note to self:

Next, I will downsize my Linux machine to 32 GB, and will use the smaller BWA index files, then compare the difference in memory usage, time of analysis, and resultant findings in the heatmaps and tree maps.

To post to this group, send email to edge...@googlegroups.com.