CRITICAL PMIx race condition vulnerability affecting Slurm (CVE-2023-41915)

[Shane Filus]

CI Operators:

A CRITICAL rated vulnerability concerning PMIx has been discovered [1]. The vulnerability affects all sites using Slurm built with PMIx support. This issue is tracked as CVE-2023-41915 [2,3] with a CVSS v3 score of 8.1.

Impact:

A filesystem race condition could enable a malicious user to obtain ownership of an arbitrary file when parts of the PMIx library are called by a process running as uid 0. This may happen under the default configuration of certain workload managers, including Slurm [4].

Affected Software: 

• OpenPMIx PMIx before 4.2.6 and 5.0.x before 5.0.1

• Note that *all* versions prior to PMIx 4.2.6 are vulnerable, but some older PMIx versions are no longer supported and will not be patched.

To check if your version of Slurm was built with PMIx support, issue the following command:

$ srun --mpi=list

Typical output will be something like

MPI plugin types are...

        cray_shasta

        none

        pmi2

        pmix

specific pmix plugin versions available: pmix_v3,pmix_v4

This will tell you whether Slurm is built with PMIx support and which version of PMIx.

If the command returns no pmix option, your Slurm installation is unaffected by this vulnerability.

Recommendation:

Upgrade PMIx to the fixed releases v4.2.6 [5] or v5.0.1 [6].

If Slurm upgrade isn't an option, you can disable PMIx support by removing the mpi_pmix*.so libraries on the compute nodes and adjusting the MpiDefault setting in your Slurm configuration.

You can also patch your current PMIx version by replacing the chown function with lchown in the source and rebuilding the PMIx rpm. After installing the patched PMIx, this command should return no result:

objdump -t /usr/lib64/libpmi*.so* | grep chown

After the installation, slurmd on the compute nodes needs to be restarted.

References:

[1] https://github.com/advisories/GHSA-m8fg-c37h-w29q 

[2] https://nvd.nist.gov/vuln/detail/CVE-2023-41915

[3] https://access.redhat.com/security/cve/CVE-2023-41915

[4] https://github.com/openpmix/openpmix/pull/3150

[5] https://github.com/openpmix/openpmix/releases/tag/v4.2.6

[6] https://github.com/openpmix/openpmix/releases/tag/v5.0.1 

How Trusted CI can help:

The potential impact of any vulnerability, and therefore the appropriate response, depends in part on operational conditions that are unique to each cyberinfrastructure deployment. Trusted CI cannot provide a one-size-fits-all severity rating and response recommendation for all NSF cyberinfrastructure. Please contact us (https://trustedci.org/help/) if you need assistance with assessing the potential impact of this vulnerability in your environment and/or you have additional information about this issue that should be shared with the community.