Segmentation fault

[Owen Francis]

I am getting a segmentation fault when I try to map 1127 reads (fasta
format) to 2 genomes. The strange part is that it works if I map 11278
reads where the first 1127 reads are the same as in the smaller job.
The sequence files look identical right down to the hidden characters
so I don't think it's a formatting problem. Is there any other reason
for this error? The text from the error file is included below.

-- Owen Francis

Hashing the genome.
[0/1] gen_size=0, my_start=0, my_end=0
Size of genome: 2255894
[0/-] Converting to Vector...
[0/-] Trying to create hash of size 16777216
[0/-] Finished create hash.
[0/-] Stats: Total hashes is 2255608, Longest hash is 62,
shortest is 1,
and average is 0.134445
[0/-] Trying to create a new genome with a size of
2255896...Success!
[0/-] Trying to malloc 281987 elements for positions
array...Success!
[0/-] Finished Vector Conversion

Time to hash: 1 seconds
Matching 1 file(s):

[-/0] Matching 1127 sequences of: /fslhome/ofrancis/specsim/
test_01X.fa
Reads per processor: 1024
[m6-23-4:32714] *** Process received signal ***
[m6-23-4:32714] Signal: Segmentation fault (11)
[m6-23-4:32714] Signal code: (128)
[m6-23-4:32714] Failing at address: (nil)
[m6-23-4:32714] [ 0] /lib64/libpthread.so.0 [0x311680e7c0]
[m6-23-4:32714] [ 1] fsl_groups/fslg_genome/software/gnumap_MPI/bin/
gnumap(_Z19parallel_thread_runPv+0x175) [0x4196f5]
[m6-23-4:32714] [ 2] /lib64/libpthread.so.0 [0x31168064a7]
[m6-23-4:32714] [ 3] /lib64/libc.so.6(clone+0x6d) [0x31160d3c2d]
[m6-23-4:32714] *** End of error message ***
--------------------------------------------------------------------------
mpiexec noticed that process rank 0 with PID 32714 on node
m6-23-4.local exited on signal 11 (Segmentation fault).
--------------------------------------------------------------------------

[Steve Piccolo]

I was getting this error, too (yesterday). I found that when I removed the --MPI_largemem flag, it no longer gave me the error. So there may be a bug that's associated with that flag.

-Steve

[Owen Francis]

The genomes are only 1.1MB so I didn't use the --MPI_largmem flag. The only difference that I can see between the job that works and the job that throws an error is the size of the sequence file.

[Nathan Clement]

Owen,

I'm not aware of any bugs that would cause this problem—nor the problem Steve is describing below.  If you could send me the genome and/or sequence files that are causing the problem, that would be great.  You can just put them in /bluescr/nclement/DropBox/owen/ (make sure you do chmod a+r /bluescr/nclement/DropBox/owen/* on them after you've done this) and I'll look into it.

Nathan

[Nathan Clement]

Owen,

I spent some time over the holiday making speed adjustments, and must not have updated the public version of GNUMAP on marylou.  I've made those changes and it looks like both of your scripts run just fine.  

I'm releasing a new version (Version 2.2.3) with these updates.  Keep me updated with other errors you might find.

Nathan

On Jan 6, 2011, at 4:50 PM, Owen Francis wrote:

[wei zhou]

Nathan,
I ran the gnumap on supercomputer today, and still have met the
problem of "Segmentation fault". I've sent my submission file,
sequence file, genome file and the error file to /bluescr/nclement/
DropBox/wei/. Could you please take a look at it?

Thanks,
Wei

On Jan 7, 5:05 pm, Nathan Clement <nathanlclem...@gmail.com> wrote:
> Owen,
>
> I spent some time over the holiday making speed adjustments, and must not have updated the public version of GNUMAP on marylou.  I've made those changes and it looks like both of your scripts run just fine.  
>
> I'm releasing a new version (Version 2.2.3) with these updates.  Keep me updated with other errors you might find.
>
> Nathan
>
> On Jan 6, 2011, at 4:50 PM, Owen Francis wrote:
>
>
>
> > The genomes are only 1.1MB so I didn't use the --MPI_largmem flag. The only difference that I can see between the job that works and the job that throws an error is the size of the sequence file.
>

> > On Thu, Jan 6, 2011 at 2:38 PM, Steve Piccolo <steve.picc...@gmail.com> wrote:
> > I was getting this error, too (yesterday). I found that when I removed the --MPI_largemem flag, it no longer gave me the error. So there may be a bug that's associated with that flag.
>
> > -Steve
>

[Nathan Clement]

Wei,

I looked at your submission file and can find one error quickly that would present a problem. When you define
nproc=???
it's a fairly large number (I changed it before I realized that might be the problem).

This number should be identical to the number of nodes you request on the -l line. If not, MPI is trying to run all the jobs (let's say 360) on only 20 nodes. This will definitely have errors if you run any sort of job with large memory requests and might also be the cause of your current issue.

I changed the numbers (set to nproc=20) and ran it myself and it completed successfully. If it doesn't work for you still after you make this change, I can look into it again.

Nathan

[wei zhou]

Nathan,

You are right. I changed the nproc to be the number of nodes, and it
worked for all of my jobs, except for one of them. The same
segmentation fault occurred. Then I changed the CPU from cpu2800MHz to
Intel, and it worked this time. So I'm guessing probably there is
something with the CPU that I ran my job with, besides the wrong nproc
numbers.

Thanks for all your help.
Wei