Memory allocation

[Demian Augusto Vera]

Hi, Dr. Fang. Hope you and all the community have a great 2022.

As you had recommended me some weeks ago, for simulating multisource light propagation in brain tissue, the best (and only, I think) way to do it is to make a for loop and change the source position each iteration. That's what I'm doing, but I find some problems in MMCLAB (nightly builld). After some iterarions (about 40) Matlab throws the next error:

MMCLAB ERROR (4): Memory object allocation failure in unit mmc_cl_host.c:479
Error from thread (0): Memory object allocation failure
Error using mmc
MMCLAB Terminated due to exception!
Error in mmclab (line 400)
    [varargout{1:mmcout}]=mmc(cfg);
Error in Abide_50041 (line 283)
            [~, detpt{j} ~,seeds{j}] = mmclab(cfg, 'havel'); 

I've never had this problem with MCXLAB, even when I iterate in 225 source positions. 

It seems like the GPU memory is filling up, so each 10 or 20 iterations I clear it "manually" with reset(gpuDevice(1)). 
GPU info:

mcxlabcl('gpuinfo')
Platform [0] Name NVIDIA CUDA
============ GPU device ID 0 [1 of 1]: NVIDIA GeForce RTX 3050 Ti Laptop GPU  ============
 Device 1 of 1:                NVIDIA GeForce RTX 3050 Ti Laptop GPU
 Compute units   :        20 core(s)
 Global memory   :        0 B
 Local memory    :        49152 B
 Constant memory :        65536 B
 Clock speed     :        1485 MHz
 Compute Capacity:        8.6
 Stream Processor:        1280
 Vendor name    :        NVIDIA
 Auto-thread    :        81920
 Auto-block     :        64
Platform [1] Name Intel(R) OpenCL HD Graphics
============ GPU device ID 1 [1 of 1]: Intel(R) UHD Graphics  ============
 Device 2 of 1:                Intel(R) UHD Graphics
 Compute units   :        16 core(s)
 Global memory   :        -1810554880 B
 Local memory    :        65536 B
 Constant memory :        -905277440 B
 Clock speed     :        1450 MHz
 Vendor name    :        IntelGPU
 Auto-thread    :        1024
 Auto-block     :        64

Another issue is that I have to reset Matlab and run the code again. If not, MMCLAB get stacked. 

I would greatly appreciate if you could help me. Thanks!

Best regards, 

Demián Vera

[Josef Probst]

Hello Dr. Fang,

I experience the same issue. (mmclab with octave)

I ran nvidia-smi before and after a simulation.

Although the simulation was already over, nvidia-smi said that octave now uses ~115MiB more Memory.

After some iterations mmc will either stall or terminate with an exception.

Is there an equivalent to reset(gpuDevice(1)) in octave?

Thanks for all your help.

Best regards,

Josef

[Qianqian Fang]

sorry Josef and Demian, I received no email notifications for either of your messages, google group seems to have some issues in the past few months.

I believe this issue is related to an OpenCL driver-side memory leakage that I have found previously, please see 

https://groups.google.com/g/mcx-users/c/f31KBsmj5aw/m/H_fabg3zAwAJ

and stackoverflow questions I posted at

https://stackoverflow.com/questions/61163373/how-to-force-nvidia-opencl-to-release-gpu-context-to-avoid-memory-leak

https://stackoverflow.com/questions/61091039/opencl-clcreatecontextfromtype-function-results-in-memory-leaks

if you had run mmclab/mcxlabcl on your nvidia hardware, the memory keeps on increasing in a loop, and perhaps had caused the gpu failure.

one workaround is to call each mmclab/mcxlabcl call in a separate session, using something like

matlab -nojvm -nodesktop -r "addpath('/path/to/mcxlabcl'); run_my_mcxlabcl_cmd(srcpos1, srcdir1, ...); exit" 

matlab -nojvm -nodesktop -r "raddpath('/path/to/mcxlabcl'); un_my_mcxlabcl_cmd(srcpos2, srcdir2, ...); exit" 

...

or

octave-cli --eval "addpath('/path/to/mcxlabcl'); run_my_mcxlabcl_cmd(srcpos1, srcdir1, ...); exit" 

octave-cli --eval "addpath('/path/to/mcxlabcl'); run_my_mcxlabcl_cmd(srcpos2, srcdir2, ...); exit" 

...

where replacing the path to mcxlabcl/mmclab's path, and srcpos/srcdir to each of your source positions/directions. It is a bit verbose, but should work.

I am planning to add multi-source support to mcx/mmc, so that one can simulate more than 1 src in one kernel call. Will let you know when this feature is ready.

Qianqian