SOAPdenovo: Problem of sharing memory in cluster

183 views

Skip to first unread message

lamz1...@163.com

unread,

Jun 9, 2016, 8:33:09 AM6/9/16

to BGI-SOAP

Hi, dear all!

I had used SOAPdenovo2 to assemble genome. There are four libraries with 120G data respectively, and about 160X (the genome is about 3G). In the step of pregraph, it didn't report error, but when I ran the step of contig, it failed immediately. I think the reason is running out of memory (there are only 500G memory available). Then I turned to cluster machine (three big nodes, and each node only have 500G memory) and tried to use memory of these nodes (I have little knowledge about cluster). I installed openmpi, and used qsub to submit the job with demo data, but it seemed it just run the SOAPdenovo several times and there are some problems according to the log. Also, the result is different from when it didn't run in cluster. Did SOAPdenovo support running in cluster, how can I make the nodes share memory, or how can I finished my assembly? By the way, although I run the "hello" script successfully, I didn't know whether I am right in setting up openmpi.

Any suggestion would be grateful!

Following is what had I done:

1. Run the step of pregraph:

soapPath="/tools/SOAPdenovo2/SOAPdenovo-63mer"

$soapPath pregraph -s config_file -R -p 45 -o graph_prefix > pregraph.log 2> pregraph.err

2. Run the step of contig:

$soapPath contig -g graph_prefix -R -p 45 1>contig.log 2>contig.err

3. The error in contig step:

cat contig.err

Version 2.04: released on July 13th, 2012

Compile Apr 27 2016 15:50:02

********************

Contig

********************

Parameters: contig -g graph_prefix -R -p 45

There are 561510855 kmer(s) in vertex file.

There are 1856621603 edge(s) in edge file.

Ran out of memory while applying -32782782016bytes

There may be errors as follows:

1) Not enough memory.

2) The ARRAY may be overrode.

3) The wild pointers.

4. Install openmpi and submit the job

1) The script of job:

cat mpi.qsub:

#!/bin/bash

#$ -cwd

#$ -j y

#$ -S /bin/bash

export PATH=/tools/openmpi/bin:$PATH

export LD_LIBRARY_PATH=/tools/openmpi/lib:$LD_LIBRARY_PATH

soapPath="/tools/SOAPdenovo2/SOAPdenovo-63mer"

workPath="/NGS"

outputPath="assembly/soap/demo"

/tools/openmpi/bin/mpirun $soapPath all -s $workPath/$outputPath/config_file -K 23 -R -F -p 60 -V -o $workPath/$outputPath/graph_prefix > $workPath/$outputPath/ass.log 2> $workPath/$outputPath/ass.err

2) Submit the job:

qsub -pe orte 60 mpi.qsub

3) The log in ass.err

cat ass.err | grep "Pregraph" | wc -l

less ass.err:

Version 2.04: released on July 13th, 2012

Compile Apr 27 2016 15:50:02

********************

Pregraph

********************

Parameters: pregraph -s /NGS/assembly/soap/demo/config_file -K 23 -p 16 -R -o /NGS/assembly/soap/demo/graph_prefix

In /NGS/assembly/soap/demo/config_file, 1 lib(s), maximum read length 35, maximum name length 256.

Version 2.04: released on July 13th, 2012

Compile Apr 27 2016 15:50:02

********************

Pregraph

********************

and so on

cat ass.log:

--------------------------------------------------------------------------

WARNING: A process refused to die despite all the efforts!

This process may still be running and/or consuming resources.

Host: smp03

PID: 75035

--------------------------------------------------------------------------

mpirun noticed that process rank 58 with PID 0 on node c0214.local exited on signal 11 (Segmentation fault).