Whole genome assembly using abyss-pe mpi

[Malabady]

Hi,

I have been trying to use abyss-pe to assembly a whole Eukaryota genome (2.13G) using different types of data (PE, SE, and MP). I running the assembly on our computer cluster using the following shell script. I have tried so many times and got various error messages. Here are the error messages from my last trial:

$ more abyss10.q.err
mpUndetermined_R1.pe.fastq.gz': discarded 545687 reads shorter than 64 bases mpUndetermined_R1.pe.fastq.gz': discarded 568 reads containing non-ACGT characters
mpUndetermined_R2.pe.fastq.gz': discarded 466839 reads shorter than 64 bases mpUndetermined_R2.pe.fastq.gz': discarded 457 reads containing non-ACGT characters
mp9k_R1.pe.fastq.gz': discarded 29934147 reads shorter than 64 bases mp9k_R1.pe.fastq.gz': discarded 380 reads containing non-ACGT characters
mp9k_R2.pe.fastq.gz': discarded 29726082 reads shorter than 64 bases mp9k_R2.pe.fastq.gz': discarded 24510 reads containing non-ACGT characters
mp6k_R2.pe.fastq.gz': discarded 32082292 reads shorter than 64 bases mp6k_R2.pe.fastq.gz': discarded 27539 reads containing non-ACGT characters
mp6k_R1.pe.fastq.gz': discarded 32031088 reads shorter than 64 bases mp6k_R1.pe.fastq.gz': discarded 422 reads containing non-ACGT characters
n83:2.0.Exhausted 1048576 MQ irecv request descriptors, which usually indicates a user program error or insufficient request descriptors (PSM_MQ_RECVREQS_MAX=1048576)

$ tail abyss10.q.out
4: Reading mp6k150_R1.pe.fastq.gz'…

5: Readingmp6k150_R2.pe.fastq.gz'...
2: Reading K_R1_001_val_1.fq.gz'…

3: ReadingK_R2_001_val_2.fq.gz'...
1: Reading `D_R2_001_val_2.fq.gz’…

0: Reading `D_R1_001_val_1.fq.gz'...

Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted.

If you have any suggestions about this error and how to solve it, I'll really appreciate them.

Thanks,
Magdy

--------------------------------Shell Script----------------------------------------------------------------------------
#!/bin/bash

#PBS -N 10_abyss_mpi
#PBS -q batch
#PBS -l nodes=8:ppn=1:AMD
#PBS -l walltime=480:00:00
#PBS -l mem=800gb

CONTEXTS=$(/usr/local/bin/set_contexts.sh $PBS_NUM_PPN)
if [[ "$?" -eq "0" ]] ; then
export PSM_SHAREDCONTEXTS_MAX=$CONTEXTS
fi

module load anaconda/2.2.0 openmpi/1.8.3/gcc/4.4.7 boost/1.56.0/gcc447 sparsehash/2.0.2 abyss/1.9.0

echo
echo "Job ID: $PBS_JOBID"
echo "Queue: $PBS_QUEUE"
echo "Cores: $PBS_NP"
echo "Nodes: $(cat $PBS_NODEFILE | sort -u | tr '\n' ' ')"
echo "mpirun: $(which mpirun)"
echo "$d"
echo

cd /lustre1/malabady/Ab10/abyss_assembly1

time /usr/local/apps/abyss/latest/bin/abyss-pe np=8 k=64 name=Ab10_10 -d \
lib='pe75_1 pe75_2 pe75_3 pe150_1 pe150_2 pe150_3 pe150_4 pe150_5 pe150_6' \
mp='mp6k75 mp9k75 mpUnde75 mp6k150 mp9k150 mpUnde150' \
pe75_1='mp6k_R1.pe.fastq.gz mp6k_R2.pe.fastq.gz' \
pe75_2='mp9k_R1.pe.fastq.gz mp9k_R2.pe.fastq.gz' \
pe75_3='mpUndetermined_R1.pe.fastq.gz mpUndetermined_R2.pe.fastq.gz' \
pe150_1='P_R2_001_val_2.fq.gz P_R1_001_val_1.fq.gz' \
pe150_2='D_R1_001_val_1.fq.gz D_R2_001_val_2.fq.gz' \
pe150_3='K_R1_001_val_1.fq.gz K_R2_001_val_2.fq.gz' \
pe150_4='mp6k150_R1.pe.fastq.gz mp6k150_R2.pe.fastq.gz' \
pe150_5='mp9k150_R1.pe.fastq.gz mp9k150_R2.pe.fastq.gz' \
pe150_6='mpUndetermined150_R1.pe.fastq.gz mpUndetermined150_R2.pe.fastq.gz' \
mp6k75='mp6k_R1.mp.fastq.gz mp6k_R2.mp.fastq.gz' \
mp9k75='mp9k_R1.mp.fastq.gz mp9k_R2.mp.fastq.gz' \
mpUnde75='mpUndetermined_R1.mp.fastq.gz mpUndetermined_R2.mp.fastq.gz' \
mp6k150='mp6k150_R1.mp.fastq.gz mp6k150_R2.mp.fastq.gz' \
mp9k150='mp9k150_R1.mp.fastq.gz mp9k150_R2.mp.fastq.gz' \
mpUnde150='mpUndetermined150_R1.mp.fastq.gz mpUndetermined150_R2.mp.fastq.gz' \
se='mp6k_R1.unknown.fastq.gz mp6k_R2.unknown.fastq.gz mp6k.se.fastq.gz mp9k_R1.unknown.fastq.gz mp9k_R2.unknown.fastq.gz mpUndetermined_R1.unknown.fastq.gz
mpUndetermined_R2.unknown.fastq.gz mpUndetermined.se.fastq.gz All_R2_unpaired_2.fq.gz mp6k150_R1.unknown.fastq.gz mp6k150_R2.unknown.fastq.gz mp6k150.se.fastq.gzmpUndetermined150_R1.unknown.fastq.gz mp9k.se.fastq.gz All_R1_unpaired_1.fq.gz mpUndetermined150_R2.unknown.fastq.gz mpUndetermined150.se.fastq.gz' 1>abyss10.q.out 2>abyss10.q.err

[Ben Vandervalk]

Hi Magdy,

This line seems to be the source of the problem:

```

n83:2.0.Exhausted 1048576 MQ irecv request descriptors, which usually indicates a user program error or insufficient request descriptors (PSM_MQ_RECVREQS_MAX=1048576)

```

It sounds like this is an OpenMPI bug and not something specific to ABySS.  It sounds like a buffer overrun.  When I googled the error message, I came across this old thread: http://www.open-mpi.org/community/lists/users/2010/03/12274.php, but it doesn't offer any concrete suggestions for fixes or workarounds.

It is possible that you might be able to work around the issue by adjusting the OpenMPI parameters -- however, when looking through `ompi_info --all` I was unable to find anything that seemed relevant.  Your best chance at success might be to try older/newer versions of OpenMPI or a different MPI library altogether (e.g. MPICH).

Sorry I couldn't be more helpful,

- Ben