Trouble running RAxML-MPI on cluster.

[Mike]

Greetings raxml group!

I have successfully compiled raxmlHPC-MPI-SSE3 on our head node without issue. However, when I try to submit a raxml job to the queue I get some errors I do not understand. Below is the batch script I used:

#### BEGIN SUBMISSION ####

# Walltime Limit: hh:nn:ss 

#PBS -l walltime=48:00:00 

# Node Specification:

#PBS -l ncpus=4 -l nodes=1

# Queue 

#PBS -q long 

# supress email notification

#PBS -m n

# Job Name:

#PBS -N RAxML_test 

# Keep output

#PBS -k oe

echo ------------------------------------------------------

echo PBS: qsub is running on $PBS_O_HOST

echo PBS: originating queue is $PBS_O_QUEUE

echo PBS: executing queue is $PBS_QUEUE

echo PBS: working directory is $PBS_O_WORKDIR

echo PBS: execution mode is $PBS_ENVIRONMENT

echo PBS: job identifier is $PBS_JOBID

echo PBS: job name is $PBS_JOBNAME

echo PBS: node file is $PBS_NODEFILE

echo PBS: current home directory is $PBS_O_HOME

echo PBS: PATH = $PBS_O_PATH

echo ------------------------------------------------------

cd $PBS_O_WORKDIR

mpiexec -n 4 raxmlHPC-MPI-SSE3 -f d -m PROTGAMMALG -N 100 -o Escherichia_coli_str__K_12_substr__MG1655_chromosome_NC_000913 -s /home/mike/raxml_test/Concatenated_AA_alignment.phy -n /home/mike/raxml_test/Concatenated_AA_alignment -q /home/mike/raxml_test/AA_raxml_parts.txt

#### END SUBMISSION ####

The error file that the cluster outputs reads:

#### BEGIN ERROR FILE ####

raxmlHPC-MPI-SSE3: axml.c:4188: analyzeRunId: Assertion `0' failed.

[node041:18522] *** Process received signal ***

[node041:18522] Signal: Aborted (6)

[node041:18522] Signal code:  (-6)

raxmlHPC-MPI-SSE3: axml.c:4188: analyzeRunId: Assertion `0' failed.

[node041:18521] *** Process received signal ***

[node041:18521] Signal: Aborted (6)

[node041:18521] Signal code:  (-6)

[node041:18521] [ 0] /lib64/libpthread.so.0 [0x35ad40eb10]

[node041:18521] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x35acc30265]

[node041:18521] [ 2] /lib64/libc.so.6(abort+0x110) [0x35acc31d10]

[node041:18521] [ 3] /lib64/libc.so.6(__assert_fail+0xf6) [0x35acc296e6]

[node041:18521] [ 4] raxmlHPC-MPI-SSE3(main+0x1852) [0x415dc2]

[node041:18521] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) [0x35acc1d994]

[node041:18521] [ 6] raxmlHPC-MPI-SSE3 [0x404339]

[node041:18521] *** End of error message ***

[node041:18522] [ 0] /lib64/libpthread.so.0 [0x35ad40eb10]

[node041:18522] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x35acc30265]

[node041:18522] [ 2] /lib64/libc.so.6(abort+0x110) [0x35acc31d10]

[node041:18522] [ 3] /lib64/libc.so.6(__assert_fail+0xf6) [0x35acc296e6]

[node041:18522] [ 4] raxmlHPC-MPI-SSE3(main+0x1852) [0x415dc2]

[node041:18522] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) [0x35acc1d994]

[node041:18522] [ 6] raxmlHPC-MPI-SSE3 [0x404339]

[node041:18522] *** End of error message ***

raxmlHPC-MPI-SSE3: axml.c:4188: analyzeRunId: Assertion `0' failed.

[node041:18524] *** Process received signal ***

[node041:18524] Signal: Aborted (6)

[node041:18524] Signal code:  (-6)

[node041:18524] [ 0] /lib64/libpthread.so.0 [0x35ad40eb10]

[node041:18524] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x35acc30265]

[node041:18524] [ 2] /lib64/libc.so.6(abort+0x110) [0x35acc31d10]

[node041:18524] [ 3] /lib64/libc.so.6(__assert_fail+0xf6) [0x35acc296e6]

[node041:18524] [ 4] raxmlHPC-MPI-SSE3(main+0x1852) [0x415dc2]

[node041:18524] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) [0x35acc1d994]

[node041:18524] [ 6] raxmlHPC-MPI-SSE3 [0x404339]

[node041:18524] *** End of error message ***

--------------------------------------------------------------------------

mpiexec noticed that process rank 1 with PID 18522 on node node041 exited on signal 6 (Aborted).

--------------------------------------------------------------------------

raxmlHPC-MPI-SSE3: axml.c:4188: analyzeRunId: Assertion `0' failed.

[node041:18523] *** Process received signal ***

[node041:18523] Signal: Aborted (6)

[node041:18523] Signal code:  (-6)

[node041:18523] [ 0] /lib64/libpthread.so.0 [0x35ad40eb10]

[node041:18523] [ 1] /lib64/libc.so.6(gsignal+0x35) [0x35acc30265]

[node041:18523] [ 2] /lib64/libc.so.6(abort+0x110) [0x35acc31d10]

[node041:18523] [ 3] /lib64/libc.so.6(__assert_fail+0xf6) [0x35acc296e6]

[node041:18523] [ 4] raxmlHPC-MPI-SSE3(main+0x1852) [0x415dc2]

[node041:18523] [ 5] /lib64/libc.so.6(__libc_start_main+0xf4) [0x35acc1d994]

[node041:18523] [ 6] raxmlHPC-MPI-SSE3 [0x404339]

[node041:18523] *** End of error message ***

#### END ERROR FILE ####

I am currently discussing this with our cluster admin, currently neither of us can determine what is causing the issue. I get the same error regardless if I use either the RAxML v7.2.8 or the github version. Any ideas on what the issue may be?

-Thanks for any help in advance.

-Mike

[Alexandros Stamatakis]

Hi Mike,

The run name passed via -n just needs to be a name and not a directory
structure, something like "-n myFirstMPIJob",

the function analyzeRunId() that checks the format of that strings exits
with an error because the string you pass contains "/"

Alexis

--
Alexandros (Alexis) Stamatakis

Research Group Leader, Heidelberg Institute for Theoretical Studies
Full Professor, Dept. of Informatics, Karlsruhe Institute of Technology
Adjunct Professor, Dept. of Ecology and Evolutionary Biology, University
of Arizona at Tucson

www.exelixis-lab.org

[Mike]

Hi Alexis,

That was it! It's all running fine now. :-)

-Thanks!

-Mike

[Érica Souza]

Hi

I have the same problem and when I substituted the -n for a name, the message was:

This is RAxML MPI Process Number: 0

The file /dados/software/standard-RAxML-8.2.11/RAxML_info.Mfirst RAxML wants to open for writing or appending can not be opened [mode: ab], exiting ...

I search for an answer but only what can find it was something about the permission to the folder.

Thanks

[Alexey Kozlov]

Hi Erica,

that's right, you probably don't have write permission for the folder where RAxML is installed
(/dados/software/standard-RAxML-8.2.11).

So just tell RAxML to save output files in the folder you have access to (e.g. under your home folder), you can do it
with the "-w" option, e.g.

-w /home/erica/raxml_output

Hope this helps,
Alexey

> www.exelixis-lab.org <http://www.exelixis-lab.org>
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Érica Souza]

Dear Alexey

It works \o/

Thank you very much

All the best

Sent with Free Email Tracker by cloudHQ

To unsubscribe from this group and stop receiving emails from it, send an email to raxml+unsubscribe@googlegroups.com <mailto:raxml+unsubscribe@googlegroups.com>.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "raxml" group.

To unsubscribe from this group and stop receiving emails from it, send an email to raxml+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--

Érica Souza

Bel. Ciências Biológicas - Biodiversidade e Conservação  (Universidade Federal do Amazonas - UFAM)

Me. Genética, Conservação e Biologia Evolutiva (Instituto Nacional de Pesquisas da Amazônia - INPA)

Doutoranda pelo programa de Genética, Evolução, Microbiologia e Imunologia (Universidade Estadual de Campinas - UNICAMP)

CV: http://lattes.cnpq.br/9530933208734813

[Alexey Kozlov]

great, you're welcome :)

On 17.05.2018 17:25, Érica Souza wrote:
> Dear Alexey
>
> It works \o/
>
> Thank you very much
>
> All the best
>
>

> Sent with Free Email Tracker by cloudHQ <https://www.cloudhq.net/install_mail_tracker?source=signature&referral=593826>
>
> 2018-05-17 11:13 GMT-03:00 Alexey Kozlov <alexei...@gmail.com <mailto:alexei...@gmail.com>>:

> www.exelixis-lab.org <http://www.exelixis-lab.org> <http://www.exelixis-lab.org>

>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> raxml+un...@googlegroups.com <mailto:raxml%2Bunsu...@googlegroups.com>
> <mailto:raxml+un...@googlegroups.com <mailto:raxml%2Bunsu...@googlegroups.com>>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.

>
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> raxml+un...@googlegroups.com <mailto:raxml%2Bunsu...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
>
>
>
> --
> *Érica Souza*
> *Bel. Ciências Biológicas - Biodiversidade e Conservação  (Universidade Federal do Amazonas - UFAM)*
> *Me. Genética, Conservação e Biologia Evolutiva (Instituto Nacional de Pesquisas da Amazônia - INPA)*
> *Doutoranda pelo programa de Genética, Evolução, Microbiologia e Imunologia (Universidade Estadual de Campinas - UNICAMP)**
> *
> *CV:http://lattes.cnpq.br/9530933208734813/
> /*
> */
> /*

>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.

> To unsubscribe from this group and stop receiving emails from it, send an email to raxml+un...@googlegroups.com
> <mailto:raxml+un...@googlegroups.com>.

[Carolina Piña Páez]

Hello everyone!

I have a similar error running RAxML-MPI on the Univerity cluster:

mpiexec has exited due to process rank 9 with PID 57204 on

node symbiosis.cgrb.oregonstate.local exiting without calling "finalize". This may

have caused other processes in the application to be

terminated by signals sent by mpiexec (as reported here).

Here is the full command: mpiexec -n 10 raxmlHPC-MPI -N 50 -n myMLJob -s gt.phy -m MULTICAT -f a -x 12345 -p 12345

Not sure if this is a infrastructural or a user error. Any help will be greatly appreciated.

Best,

Caro

[Alexey Kozlov]

Hi Caro,

it would help if you can post RAxML_info file and console output for this run.

Best,
Alexey

> <http://www.exelixis-lab.org>> <http://www.exelixis-lab.org <http://www.exelixis-lab.org>>

> >
> > --
> > You received this message because you are subscribed to the Google Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> > raxml+un...@googlegroups.com <mailto:raxml%2Bunsu...@googlegroups.com>
> > <mailto:raxml+un...@googlegroups.com <mailto:raxml%2Bunsu...@googlegroups.com>>.
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>>.
> >
> >
> > --
> > You received this message because you are subscribed to the Google Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> > raxml+un...@googlegroups.com <mailto:raxml%2Bunsu...@googlegroups.com>.
> > For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>>.
> >
> >
> >
> >
> > --
> > *Érica Souza*
> > *Bel. Ciências Biológicas - Biodiversidade e Conservação  (Universidade Federal do Amazonas -
> UFAM)*
> > *Me. Genética, Conservação e Biologia Evolutiva (Instituto Nacional de Pesquisas da Amazônia
> - INPA)*
> > *Doutoranda pelo programa de Genética, Evolução, Microbiologia e Imunologia (Universidade
> Estadual de Campinas - UNICAMP)**
> > *

> > *CV:http://lattes.cnpq.br/9530933208734813/ <http://lattes.cnpq.br/9530933208734813/>

> > /*
> > */
> > /*
> >
> > --
> > You received this message because you are subscribed to the Google Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com
> > <mailto:raxml+un...@googlegroups.com>.

> > For more options, visit https://groups.google.com/d/optout <https://groups.google.com/d/optout>.
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to

> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/raxml/a978bf1c-1728-4bd9-a4c0-f419566c50ean%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/a978bf1c-1728-4bd9-a4c0-f419566c50ean%40googlegroups.com?utm_medium=email&utm_source=footer>.

[Carolina Piña Páez]

Hi Alexey, 

Thanks for the response. This run did not generate a RaxML_info file. Here is the output information:

snp_tree_6.o9884263 

This is RAxML MPI Process Number: 9

This is RAxML MPI Process Number: 5

This is RAxML MPI Process Number: 7

This is RAxML MPI Process Number: 8

This is RAxML MPI Process Number: 2

This is RAxML MPI Process Number: 6

Multi State Error, characters must be used in the order they are available, i.e.

0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V

You are using the following characters:

0 1 2 N

  Finished at:            Mon Aug 9 14:39:07 PDT 2021

Thanks for your help.

Best,

Caro

[Alexey Kozlov]

Hi Carolina,

you should replace "N" with "-" (gap) in your alignment. See error message below:

Multi State Error, characters must be used in the order they are available, i.e.
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, C, D, E, F, G, H, I, J, K, L, M, N, O, P, Q, R, S, T, U, V
You are using the following characters:
0 1 2 N

Best,
Alexey

> <https://groups.google.com/d/optout> <https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>>.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "raxml" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to
> > raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> > To view this discussion on the web visit
> >
> https://groups.google.com/d/msgid/raxml/a978bf1c-1728-4bd9-a4c0-f419566c50ean%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/a978bf1c-1728-4bd9-a4c0-f419566c50ean%40googlegroups.com>
> >

> <https://groups.google.com/d/msgid/raxml/a978bf1c-1728-4bd9-a4c0-f419566c50ean%40googlegroups.com?utm_medium=email&utm_source=footer
> <https://groups.google.com/d/msgid/raxml/a978bf1c-1728-4bd9-a4c0-f419566c50ean%40googlegroups.com?utm_medium=email&utm_source=footer>>.

>
>
> --
> You received this message because you are subscribed to the Google Groups "raxml" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to
> raxml+un...@googlegroups.com <mailto:raxml+un...@googlegroups.com>.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/raxml/2a6a89bd-14e9-4937-9303-0abf14694453n%40googlegroups.com
> <https://groups.google.com/d/msgid/raxml/2a6a89bd-14e9-4937-9303-0abf14694453n%40googlegroups.com?utm_medium=email&utm_source=footer>.