Groups

User Level Fault Mitigation

Contact owners and managers

1–30 of 151

Welcome to the User Level Fault Mitigation (ULFM) mailing list. This is a forum for discussion regarding an extension to the MPI standard to allow faults to be exposed at the user level, and provide the applications developers with useful means of tolerating the exposed faults.

To join this group without creating a Google account, send an email to ulfm+subscribe@googlegroups.com.

0 selected

5/4/24

stuck in mpi_comm_shrink

Hello, I encountered several situations where the mpi program get stuck in either MPI_Comm_shrink or

unread,

stuck in mpi_comm_shrink

Hello, I encountered several situations where the mpi program get stuck in either MPI_Comm_shrink or

5/4/24

Kawin Nimsaila, George Bosilca3

11/28/23

Question on ULFM OpenMPI 5.0.0 for node failure

Dear George, Thank you for the reply. I also try running kill_node program from https://github.com/

unread,

Question on ULFM OpenMPI 5.0.0 for node failure

Dear George, Thank you for the reply. I also try running kill_node program from https://github.com/

11/28/23

Rodrigo Coacci, George Bosilca5

2/13/23

ULFM on ompi/main getting stuck on MPI_FInalize()

Thanks for the pointers. I'll try the 5.0 branch and see if I can make that work. BTW, this is

unread,

ULFM on ompi/main getting stuck on MPI_FInalize()

Thanks for the pointers. I'll try the 5.0 branch and see if I can make that work. BTW, this is

2/13/23

Sarthak Joshi, Aurelien Bouteiller4

11/2/22

Transport retry count exceeded error

I have tested this using openucx version 1.9.0, 1.11.0 and 1.13.1 (latest release) and gotten this

unread,

Transport retry count exceeded error

I have tested this using openucx version 1.9.0, 1.11.0 and 1.13.1 (latest release) and gotten this

11/2/22

Howard Pritchard, … George Bosilca4

8/22/22

api folder in openmpi-tests-public/ulfm-testing question

The only thing I can say from these log files is that there seems to be a regression on PRTE, where a

unread,

api folder in openmpi-tests-public/ulfm-testing question

The only thing I can say from these log files is that there seems to be a regression on PRTE, where a

8/22/22

郭小天, George Bosilca2

8/2/22

rankfile is deprecated

Xiaotian, Thanks for the bug report. It turns out this is an issue in PRRTE, sometime in the recent

unread,

rankfile is deprecated

Xiaotian, Thanks for the bug report. It turns out this is an issue in PRRTE, sometime in the recent

8/2/22

Vahid Jafari, George Bosilca2

4/25/22

openmpi crashes, hangs or does not finalize despite the MPI error handler

Vahid, OMPI 4.x did not include ULFM, so there was no support for resilience. The OMPI 5.0 does

unread,

openmpi crashes, hangs or does not finalize despite the MPI error handler

Vahid, OMPI 4.x did not include ULFM, so there was no support for resilience. The OMPI 5.0 does

4/25/22

Ayşhe Mihaylova, George Bosilca4

4/25/22

configure error when building, syntax error near unexpected token `)'

Loay, This is good news. Thanks for the report, it could have been an issue with our software, but I

unread,

configure error when building, syntax error near unexpected token `)'

Loay, This is good news. Thanks for the report, it could have been an issue with our software, but I

4/25/22

Daniel Torres, … Amina Guermouche12

4/12/22

Process failure not caught on OMPI 5.1.0a1

Here's the ouput : $ git show Merge: dfb9ca0bb5 4f3257d7b6 Author: Tommy Janjusic <janjust@

unread,

Process failure not caught on OMPI 5.1.0a1

Here's the ouput : $ git show Merge: dfb9ca0bb5 4f3257d7b6 Author: Tommy Janjusic <janjust@

4/12/22

Sarthak Joshi, … George Bosilca4

1/11/22

Shrink for inter-communicators in OpenMPI and other implementations

Good catch. You are right, the return should always be related to the group of processes you can

unread,

Shrink for inter-communicators in OpenMPI and other implementations

Good catch. You are right, the return should always be related to the group of processes you can

1/11/22

Edson Camargo, … Aurelien Bouteiller4

11/4/21

openmpi-5.0.0rc2 does not finalize

This version is free of the mpiexec issue https://github.com/open-mpi/ompi/tree/

unread,

openmpi-5.0.0rc2 does not finalize

This version is free of the mpiexec issue https://github.com/open-mpi/ompi/tree/

11/4/21

Daniel Torres, … George Bosilca6

9/29/21

Error produced by grid topology?

Daniel, I tried with the current master and some of our examples using comm_spawn and things seem to

unread,

Error produced by grid topology?

Daniel, I tried with the current master and some of our examples using comm_spawn and things seem to

9/29/21

Martin Küttler, George Bosilca5

8/19/21

Simulate failed ranks

This sure looks easy, but I think it matches my intentions. I'm contemplating details, but so far

unread,

Simulate failed ranks

This sure looks easy, but I think it matches my intentions. I'm contemplating details, but so far

8/19/21

Charel Mercatoris, Aurelien Bouteiller2

2/15/21

MPI_Comm_shrink failure in combination with MPI_Comm_Alltoall

Charel, This is a known bug in the version you are using. Good news is that we have just merged ULFM

unread,

MPI_Comm_shrink failure in combination with MPI_Comm_Alltoall

Charel, This is a known bug in the version you are using. Good news is that we have just merged ULFM

2/15/21

Daniel Torres, Aurelien Bouteiller2

2/10/21

ULFM most recent version for high performance?

Hello Daniel, Good news is that the ulfm-expanded branch got merged into Open MPI master. I now

unread,

ULFM most recent version for high performance?

Hello Daniel, Good news is that the ulfm-expanded branch got merged into Open MPI master. I now

2/10/21

Polykarpos Thomadakis2

11/19/20

MPI_Finalize hangs

Also when run outside of slurm on a single node of 40 cores it works correctly. On Thursday, November

unread,

MPI_Finalize hangs

Also when run outside of slurm on a single node of 40 cores it works correctly. On Thursday, November

11/19/20

Daniel Torres, … George Bosilca12

10/29/20

Error re-spawning processes in 2 or more nodes

Daniel, It's awesome you have a working version. You can move forward with what you have or you

unread,

Error re-spawning processes in 2 or more nodes

Daniel, It's awesome you have a working version. You can move forward with what you have or you

10/29/20

Luke Smith, Aurelien Bouteiller3

5/20/20

Doubt about mpi_comm_spawn

Thank you so much for the answer Aurelien! I understood this limitation in differents MPI_COMM_WORLDs

unread,

Doubt about mpi_comm_spawn

Thank you so much for the answer Aurelien! I understood this limitation in differents MPI_COMM_WORLDs

5/20/20

Lucas Baptista De Moraes, George Bosilca2

3/5/20

Doubt about failure detection

For scalability reasons, detected failures are only reported on processes that have ongoing

unread,

Doubt about failure detection

For scalability reasons, detected failures are only reported on processes that have ongoing

3/5/20

Martin Küttler, George Bosilca6

1/29/20

Detection of failed nodes

On Wed, Jan 29, 2020 at 5:30 AM Martin Küttler <martin....@gmail.com> wrote: George, I

unread,

Detection of failed nodes

On Wed, Jan 29, 2020 at 5:30 AM Martin Küttler <martin....@gmail.com> wrote: George, I

1/29/20

Lukas, … George Bosilca6

1/28/20

Communicator being split after rank failure

Lukas, I was able to reproduce your findings, but unfortunately I don't yet have a clear picture

unread,

Communicator being split after rank failure

Lukas, I was able to reproduce your findings, but unfortunately I don't yet have a clear picture

1/28/20

Lucas Baptista De Moraes, George Bosilca2

1/21/20

Doubt about MPI_Comm_shrink

Lucas, First: MPIX_Comm_shrink is a collective function it must therefore be called by all alive

unread,

Doubt about MPI_Comm_shrink

Lucas, First: MPIX_Comm_shrink is a collective function it must therefore be called by all alive

1/21/20

Lukas, … Aurelien Bouteiller5

1/16/20

Failure detection mechanism -- infinito loop detected as failure

As an addendum, we have documented the most important UFLM runtime flags in this blog post: https://

unread,

Failure detection mechanism -- infinito loop detected as failure

As an addendum, we have documented the most important UFLM runtime flags in this blog post: https://

1/16/20

Atis Degro, … Zhong, Dong4

1/10/20

processes stuck in mpix_comm_shrink

Hi, Atis I tried to create your problem, I didn't see the same behavior you have. If you could

unread,

processes stuck in mpix_comm_shrink

Hi, Atis I tried to create your problem, I didn't see the same behavior you have. If you could

1/10/20

Pedro Henrique Rosso, Aurelien Bouteiller5

9/4/19

ULFM on Singularity containers

Hi Aurelien. Thanks for the help, now I can run singularity containers with the flag --enable-

unread,

ULFM on Singularity containers

Hi Aurelien. Thanks for the help, now I can run singularity containers with the flag --enable-

9/4/19

Supun Kamburugamuve, George Bosilca4

8/27/19

Is Java binding supported

The Java bindings are only covering the MPI API, so it has little to do with the ULFM internals, and

unread,

Is Java binding supported

The Java bindings are only covering the MPI API, so it has little to do with the ULFM internals, and

8/27/19

Aurelien Bouteiller

8/14/19

ULFM 2.1 over Open MPI 4.0.1: Release Candidate 1

Dear ULFM users, The ULFM team is proud to announce that we have produced a version of ULFM based

unread,

ULFM 2.1 over Open MPI 4.0.1: Release Candidate 1

Dear ULFM users, The ULFM team is proud to announce that we have produced a version of ULFM based

8/14/19

Atis Degro, George Bosilca2

8/5/19

simulating node failure

Atis, Your version is now almost 2 years old, I am not entirely sure of my answer, but at that time I

unread,

simulating node failure

Atis, Your version is now almost 2 years old, I am not entirely sure of my answer, but at that time I

8/5/19

Atis Degro, George Bosilca5

6/14/19

process killed while in mpix_comm_agree

George, Thank you! The suggested configure command worked. We just had to slightly modify it to: ./

unread,

process killed while in mpix_comm_agree

George, Thank you! The suggested configure command worked. We just had to slightly modify it to: ./

6/14/19

damodar sahasrabudhe, George Bosilca3

3/30/19

crashes during repeated concurrent failures

Damodar, I can't replicate your issue with the current master, on a IB cluster. Can you update to

unread,

crashes during repeated concurrent failures

Damodar, I can't replicate your issue with the current master, on a IB cluster. Can you update to

3/30/19

Search

Clear search

Close search

Google apps

Main menu