segmentation violation error at low timestep, but GPU memory not at full utilisation
70 views
Skip to first unread message
Thomas Rialan
Nov 18, 2024, 8:19:21 AM11/18/24
to mcx-users
Hi Dr. Fang,
Great fan of MCX, thank you for your work. For the most part I've been able to get mcxlab to work well, but I've spent the last 3-4 days trying to get my new simulations to work with the correct number of timesteps without segmentation violation errors (code works fine for fewer timesteps).
My problem: the GPU has 40GB of memory, my simulation uses at most 20GB when tstep=5e-12 (as I expect from calculations provided), RAM is also plentiful, but I get a segmentation violation error. If tstep=1e-11, peak GPU usage is 10GB/40GB and there is no error. I expect MCX needs a few spare GB for various functions, but still GPU memory doesn't seem to be the issue.
What do you think might be causing the segmentation violation?
I am currently running with the latest binaries (18 november 2024) from this page: https://mcx.space/nightly/linux64/.
I provide the full code, output log, error log below:
My script:
disp('Starting MATLAB script execution...');add_paths();disp('Hello friends');mcxlab('gpuinfo')clear cfg cfgs;%cfg.gpuid='11111111'; % use 8 GPUs togethercfg.gpuid=1;cfg.autopilot = 1;cfg.seed = hex2dec('623F9A9E');cfg.srctype='gaussian';cfg.unitinmm = 1;cfg.bc = 'rrrrrr'; % Reflective on all faces except the top (+z) facecfg.nphoton=1e8; cfg.tstart=0;cfg.tend=1e-9; % Total simulation timecfg.tstep=5e-12; % Time step (5e-12)cfg.issrcfrom0 = 1; % Positions are from (0,0,0)cfg.isspecular = 0; %reflection on air-surface boundary (source outside so 0)cfg.isreflect = 0;% cfg.workload = [12.5, 12.5, 12.5, 12.5, 12.5, 12.5, 12.5, 12.5];% cfg.workload = [10, 10];disp('base config set, loading data');
and add_paths is:
function add_paths() % Function to add necessary toolbox paths %toolbox_root = 'C:\Users\Lenovo\AppData\Local\MathWorks\MATLAB\R2024b\'; toolbox_root = '/u/trialan'; addpath(fullfile(toolbox_root, 'mcxlab')); addpath(fullfile(toolbox_root, 'mcx')); addpath(fullfile(toolbox_root, 'jsonlab')); addpath(fullfile(toolbox_root, 'zmat')); addpath(fullfile(toolbox_root, 'mcx', 'utils'));end
If I set cfg.tstep=1e-11 instead of cfg.tstep=5e-12. then there is no error and everything works (in this case, peak GPU memory usage is about 10GB, which seems odd).
My expectation is that the memory usage should be ~ 192*256^2*4*2*200 bytes, or around 20GB (following your calculation here: https://groups.google.com/g/mcx-users/c/w_YL7M6G-e8/m/NybKPx3jAAAJ). And indeed tracking memory with nvidia-smi in the background indicates peak usage at around 20GB.
Unfortunately I cannot share the data I am running this on as it is medical data I'm not allowed to share, sorry about that.
The full error is this:
Running "module reset". Resetting modules to system default. The following $MODULEPATH directories have been removed: NoneCurrently Loaded Modules: 1) gcc/11.4.0 3) cuda/11.8.0 5) slurm-env/0.1 7) matlab/2024a 2) openmpi/4.1.6 4) cue-login-env/1.0 6) default-s11 -------------------------------------------------------------------------------- Segmentation violation detected at 2024-11-18 05:53:07 -0600--------------------------------------------------------------------------------Configuration: Crash Decoding : Disabled - No sandbox or build area path Crash Mode : continue (default) Default Encoding : UTF-8 Deployed : false GNU C Library : 2.28 stable Graphics Driver : Uninitialized software Graphics card 1 : 0x10de ( 0x10de ) 0x2235 Version 550.90.7.0 (0-0-0) Graphics card 2 : 0x10de ( 0x10de ) 0x2235 Version 550.90.7.0 (0-0-0) Graphics card 3 : 0x10de ( 0x10de ) 0x2235 Version 550.90.7.0 (0-0-0) Graphics card 4 : 0x102b ( 0x102b ) 0x538 Version 0.0.0.0 (0-0-0) Graphics card 5 : 0x10de ( 0x10de ) 0x2235 Version 550.90.7.0 (0-0-0) Java Version : Java 1.8.0_202-b08 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode MATLAB Architecture : glnxa64 MATLAB Entitlement ID : 7087517 MATLAB Root : /sw/external/matlab/2024a MATLAB Version : 24.1.0.2653294 (R2024a) Update 5 OpenGL : software Operating System : "Red Hat Enterprise Linux release 8.8 (Ootpa)"
The output log is:
Job started on node: gpub038 at Mon Nov 18 05:52:21 CST 2024Starting MATLAB script execution...Hello friends============================= GPU Information ================================Device 1 of 1: NVIDIA A40Compute Capability: 8.6Global Memory: 47608692736 BConstant Memory: 65536 BShared Memory: 49152 BRegisters: 65536Clock Speed: 1.74 GHzNumber of SMs: 84Number of Cores: 5376Auto-thread: 344064Auto-block: 64ans = struct with fields: name: 'NVIDIA A40' id: 1 devcount: 1 major: 8 minor: 6
Great fan of MCX, thank you for your work. For the most part I've been able to get mcxlab to work well, but I've spent the last 3-4 days trying to get my new simulations to work with the correct number of timesteps without segmentation violation errors (code works fine for fewer timesteps).
My problem: the GPU has 40GB of memory, my simulation uses at most 20GB when tstep=5e-12 (as I expect from calculations provided), RAM is also plentiful, but I get a segmentation violation error. If tstep=1e-11, peak GPU usage is 10GB/40GB and there is no error. I expect MCX needs a few spare GB for various functions, but still GPU memory doesn't seem to be the issue.
What do you think might be causing the segmentation violation?
I am currently running with the latest binaries (18 november 2024) from this page: https://mcx.space/nightly/linux64/.
I provide the full code, output log, error log below:
My script:
disp('Starting MATLAB script execution...');add_paths();disp('Hello friends');mcxlab('gpuinfo')clear cfg cfgs;%cfg.gpuid='11111111'; % use 8 GPUs togethercfg.gpuid=1;cfg.autopilot = 1;cfg.seed = hex2dec('623F9A9E');cfg.srctype='gaussian';cfg.unitinmm = 1;cfg.bc = 'rrrrrr'; % Reflective on all faces except the top (+z) facecfg.nphoton=1e8; cfg.tstart=0;cfg.tend=1e-9; % Total simulation timecfg.tstep=5e-12; % Time step (5e-12)cfg.issrcfrom0 = 1; % Positions are from (0,0,0)cfg.isspecular = 0; %reflection on air-surface boundary (source outside so 0)cfg.isreflect = 0;% cfg.workload = [12.5, 12.5, 12.5, 12.5, 12.5, 12.5, 12.5, 12.5];% cfg.workload = [10, 10];disp('base config set, loading data');
and add_paths is:
function add_paths() % Function to add necessary toolbox paths %toolbox_root = 'C:\Users\Lenovo\AppData\Local\MathWorks\MATLAB\R2024b\'; toolbox_root = '/u/trialan'; addpath(fullfile(toolbox_root, 'mcxlab')); addpath(fullfile(toolbox_root, 'mcx')); addpath(fullfile(toolbox_root, 'jsonlab')); addpath(fullfile(toolbox_root, 'zmat')); addpath(fullfile(toolbox_root, 'mcx', 'utils'));end
If I set cfg.tstep=1e-11 instead of cfg.tstep=5e-12. then there is no error and everything works (in this case, peak GPU memory usage is about 10GB, which seems odd).
My expectation is that the memory usage should be ~ 192*256^2*4*2*200 bytes, or around 20GB (following your calculation here: https://groups.google.com/g/mcx-users/c/w_YL7M6G-e8/m/NybKPx3jAAAJ). And indeed tracking memory with nvidia-smi in the background indicates peak usage at around 20GB.
Unfortunately I cannot share the data I am running this on as it is medical data I'm not allowed to share, sorry about that.
The full error is this:
Running "module reset". Resetting modules to system default. The following $MODULEPATH directories have been removed: NoneCurrently Loaded Modules: 1) gcc/11.4.0 3) cuda/11.8.0 5) slurm-env/0.1 7) matlab/2024a 2) openmpi/4.1.6 4) cue-login-env/1.0 6) default-s11 -------------------------------------------------------------------------------- Segmentation violation detected at 2024-11-18 05:53:07 -0600--------------------------------------------------------------------------------Configuration: Crash Decoding : Disabled - No sandbox or build area path Crash Mode : continue (default) Default Encoding : UTF-8 Deployed : false GNU C Library : 2.28 stable Graphics Driver : Uninitialized software Graphics card 1 : 0x10de ( 0x10de ) 0x2235 Version 550.90.7.0 (0-0-0) Graphics card 2 : 0x10de ( 0x10de ) 0x2235 Version 550.90.7.0 (0-0-0) Graphics card 3 : 0x10de ( 0x10de ) 0x2235 Version 550.90.7.0 (0-0-0) Graphics card 4 : 0x102b ( 0x102b ) 0x538 Version 0.0.0.0 (0-0-0) Graphics card 5 : 0x10de ( 0x10de ) 0x2235 Version 550.90.7.0 (0-0-0) Java Version : Java 1.8.0_202-b08 with Oracle Corporation Java HotSpot(TM) 64-Bit Server VM mixed mode MATLAB Architecture : glnxa64 MATLAB Entitlement ID : 7087517 MATLAB Root : /sw/external/matlab/2024a MATLAB Version : 24.1.0.2653294 (R2024a) Update 5 OpenGL : software Operating System : "Red Hat Enterprise Linux release 8.8 (Ootpa)"
The output log is:
Job started on node: gpub038 at Mon Nov 18 05:52:21 CST 2024Starting MATLAB script execution...Hello friends============================= GPU Information ================================Device 1 of 1: NVIDIA A40Compute Capability: 8.6Global Memory: 47608692736 BConstant Memory: 65536 BShared Memory: 49152 BRegisters: 65536Clock Speed: 1.74 GHzNumber of SMs: 84Number of Cores: 5376Auto-thread: 344064Auto-block: 64ans = struct with fields: name: 'NVIDIA A40' id: 1 devcount: 1 major: 8 minor: 6
If it helps, I have access to large numbers of GPUs through the cluster I use, but I don't get the impression this is the root cause.
Again, great package, and thanks for a super helpful google group!
Regards,
Thomas Rialan
Regards,
Thomas Rialan
Fang, Qianqian
Nov 18, 2024, 11:22:46 PM11/18/24
to mcx-users
Hi Thomas,
thanks for reporting this.
I took a look at the issue you reported, and was able to reproduce the crash on a A100-40GB GPU. Yes, the GPU memory was only 50% utilized (10GB Nx*Ny*Nz*Nt*sizeof(float), *2 = 20GB due to the use of double-buffer to reduce error shown in
https://github.com/fangq/mcx/issues/41). What was causing the crash were due to two reasons
-
The output data buffer length variable in various units is an "int" type, which can not have a large value more than 2^31-1
-
Our previous released mcxlab .mex file built on Linux was compiled using an old version of MATLAB, which uses 32bit int for array dimensions (mwSize); this was changed to uint64 at some point.
Because of both issues, the data buffer prepared to receive the GPU output did not have the correct length.
I created an issue at
https://github.com/fangq/mcx/issues/235, and was able to commit a fix - please checkout the github action CI build and see if it has been resolved.
On a related note - I have to emphasize while it is feasible to create many time gates or using high resolution voxelated domains, you have to be aware that MC is ultimately a game of stochastic noise; increasing output data buffer voxel size while keeping
the total simulated photon count the same will just result in noisy data - it is just a trade-off.
we have prepared some general best practices guide in the below document. of course, our use case may be different.
Qianqian
From: mcx-...@googlegroups.com <mcx-...@googlegroups.com> on behalf of Thomas Rialan <thomas...@gmail.com>
Sent: Monday, November 18, 2024 7:17 AM
To: mcx-users <mcx-...@googlegroups.com>
Subject: [mcx-users] segmentation violation error at low timestep, but GPU memory not at full utilisation
Sent: Monday, November 18, 2024 7:17 AM
To: mcx-users <mcx-...@googlegroups.com>
Subject: [mcx-users] segmentation violation error at low timestep, but GPU memory not at full utilisation
--
You received this message because you are subscribed to the Google Groups "mcx-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mcx-users+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/mcx-users/0f4a95f4-5d82-4919-9b8a-d7e1bd963e4bn%40googlegroups.com.
You received this message because you are subscribed to the Google Groups "mcx-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mcx-users+...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/mcx-users/0f4a95f4-5d82-4919-9b8a-d7e1bd963e4bn%40googlegroups.com.
Thomas Rialan
Nov 19, 2024, 4:17:19 AM11/19/24
to mcx-users
Notebook not found
There was an error loading this notebook. Ensure that the file is accessible and try again.
https://github.com/COTILab/MCX24Workshop/blob/master/Training/MCX2024_1C_mcx_command_line.ipynbMy collaborator was also not able to open it so I think it's not specific to my account.
"On a related note - I have to emphasize while it is feasible to create many time gates or using high resolution voxelated domains, you have to be aware that MC is ultimately a game of stochastic noise; increasing output data buffer voxel size while keeping the total simulated photon count the same will just result in noisy data - it is just a trade-off."
Do you mean by this that I should consider a much larger photon count? This is also something I've thought about, I haven't yet done the calculations for establishing an acceptable SNR.
Many thanks,
Thomas Rialan
Fang, Qianqian
Nov 19, 2024, 12:08:52 PM11/19/24
to mcx-users
Hi Thomas, please try this alternative link
>
Do you mean by this that I should consider a much larger photon count?
This is also something I've thought about, I haven't yet done the calculations for establishing an acceptable SNR.
In the end, it is the number of photons that determines the overall MC output SNR - splitting the results into many time gates or many voxels won't change the overall SNR, trading higher spatial/temporal resolution with noisier per-voxel data.
Hope this makes sense.
If you want to see other ways to speed up mcx simulations, please check out my replies in another thread
Qianqian
From: mcx-...@googlegroups.com <mcx-...@googlegroups.com> on behalf of Thomas Rialan <thomas...@gmail.com>
Sent: Tuesday, November 19, 2024 4:17 AM
To: mcx-users <mcx-...@googlegroups.com>
Subject: Re: [mcx-users] segmentation violation error at low timestep, but GPU memory not at full utilisation
To: mcx-users <mcx-...@googlegroups.com>
Subject: Re: [mcx-users] segmentation violation error at low timestep, but GPU memory not at full utilisation
To view this discussion visit
https://groups.google.com/d/msgid/mcx-users/8e6f44df-8f1a-43e3-a421-23de6221c6c7n%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages