Core: WorkerDied handler throws NotImplementedException, masking root cause of "CUDA-capable device(s) is/are busy or unavailable" — reproducible across multiple nodes/GPUs/GPU counts

[Dylan Noone]

Hi all,

Running into a reproducible crash in MCore (Warp v2.0.0dev39) and looking for advice.

Setup: 4x A10 GPUs, CUDA 12.8/12.4, PyTorch/LibTorch 2.12.1 (cuda12.9 build), RHEL/Rocky. ~23,000 particles, box 160.

Command:

MCore --population m/22586_2_25_post_image_warp.population --devicelist 1 2 3 --iter 0

Issue: Refinement completes fine, but it crashes during the "reconstructing and filtering" step with:

CUDA error: CUDA-capable device(s) is/are busy or unavailable

This then triggers a second, unhandled System.NotImplementedException in MCore.WorkerDied(), which crashes the whole program and hides the real root cause.

Troubleshooting so far:

• Ruled out a single bad GPU — reproduces across different nodes/GPU counts, even after excluding a GPU that threw an Xid 31 fault. I have tried on A10's and ls40's.
• GPU monitoring shows the "lead" GPU spiking to ~10GB+ memory while others idle during reconstruction — looks like all reconstruction work funnels onto one device, possibly exceeding VRAM or destabilizing its CUDA context.
• Tested with/without --cpu_memory — same crash either way.
• Confirmed torch/CUDA environment is sane and all GPUs are visible.
• Can't fully rule out host RAM pressure in one instance, but crash reproduces on other nodes regardless.

Has anyone hit this or have ideas on the underlying CUDA trigger or a workaround? Happy to share full stack traces, dmesg logs, or GPU/disk monitoring plots.

Full details/logs: https://github.com/warpem/warp/issues/495

Thanks!

[Dylan Noone]

Hey all,

Problem was my clusters gpus were on exclusive mode.

Changing them to compute mode helped - talk to your HPC administrator as this requires use of sudo:

sudo nvidia-smi -c 0

My now working script:

#!/bin/sh

#SBATCH --partition="bubeck"
#SBATCH --job-name=M
#SBATCH --output=M-%j.out
#SBATCH --error=M-%j.err
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=90
#SBATCH --time=10:00:00
#SBATCH --gres=gpu:4
#SBATCH --mem=0

#Load cuda
module purge

# Initialize conda
source /bbsrc/home/dnoone/soft/conda/etc/profile.d/conda.sh

# --- Activate environment
conda activate warp_06_2026

#GPU needs to be in compute mode -- Dylan has permission you probably dont
sudo nvidia-smi -c 0
echo "star"
echo "$(nvidia-smi -q -d COMPUTE)"

#check GPU

echo $CUDA_VISIBLE_DEVICES
nvidia-smi -L
python -c "import torch; print(torch.__version__, torch.version.cuda, torch.cuda.is_available())"

# --- Put programs below - WarpTools, MTools, MCore, ...
MCore \
--population m/22586_2_25_post_image_warp.population \
--refine_imagewarp 3x3 \
--cpu_memory \
--perdevice_refine 1 \
--perdevice_postprocess 1 \
--devicelist 0 1 2 3 \
--refine_particles

#Clean
conda deactivate
module purge
sudo nvidia-smi -c EXCLUSIVE_PROCESS
echo "end"
echo "$(nvidia-smi -q -d COMPUTE)"