Out of memory issue with modified maxdetphoton

[Paloma Casteleiro Costa]

Dr. Fang, 

I am attempting to run a simple simulation. I have two NVIDIA GeForce RTX 3090 Ti (24Gb vRAM each). I set gpuid to '11' to use borh GPUs. 

Everything runs correctly, except if I try to increase the maxdetphoton variable I always get an out of memory error. Here is my command display from the simulation:

mcx.nphoton=5e+09;
mcx.maxdetphoton=1e+07;
mcx.seed=126986;
mcx.tstart=0;
mcx.tend=1e-09;
mcx.tstep=1e-09;
mcx.isrefint=1;
mcx.isreflect=1;
WARNING: redundant field 'savedetflag'
mcx.autopilot=1;
mcx.gpuid='11';
mcx.unitinmm=0.002;
mcx.medianum=104;
mcx.srcpos=[100 500 1];
mcx.srctype='cone';
mcx.srcparam1=[0.523599 0 0 0];
mcx.srcdir=[0 0 1 0];
mcx.detnum=1;
mcx.issaveexit=1;
mcx.issaveref=1;
mcx.session='Angle45';
mcx.debuglevel='4';
mcx.dim=[1000 1000 500];
mcx.mediabyte=1;
###############################################################################
#                      Monte Carlo eXtreme (MCX) -- CUDA                      #
#          Copyright (c) 2009-2018 Qianqian Fang <q.fang at neu.edu>          #
#                             http://mcx.space/                               #
#                                                                             #
# Computational Optics & Translational Imaging (COTI) Lab- http://fanglab.org #
#            Department of Bioengineering, Northeastern University            #
###############################################################################
#    The MCX Project is funded by the NIH/NIGMS under grant R01-GM114365      #
###############################################################################
$Rev::91e0de $ Last $Date::2018-08-20 00:34:32 -04$ by $Author::Qianqian Fang $
###############################################################################
- variant name: [Fermi] compiled for GPU Capability [100] with CUDA [7050]
- compiled with: RNG [xorshift128+] with Seed Length [4]
- this version CAN save photons at the detectors


GPU=1 (NVIDIA GeForce RTX 3090 Ti) threadph=19376 extra=30976 np=-1794967296 nthread=129024 maxgate=1 repetition=1
initializing streams ... init complete : 152 ms
requesting 21312 bytes of shared memory
lauching MCX simulation for time window [0.00e+00ns 1.00e+00ns] ...
simulation run# 1 ...
Progress: [==============================================================] 100%
kernel complete:   1263698 ms
retrieving fields ... MCXLAB ERROR -2 in unit mcx_core.cu:1999: out of memory
Error from thread (0): out of memory
C++ Error: MCXLAB Terminated due to an exception!

Is there any way around this to allow for more detphotons in my simulation? According to the command display, mcx is only requesting ~20Mb of shared memory. Am I interpreting this wrong?

Thank you for your time! 

Paloma

[Qianqian Fang]

hi Paloma,

from your printed log, you are using a version of mcxlab that is really old. please update to the latest version,

https://mcx.space/nightly/github/

and rerun your simulation again. we've made many bug fixes since 2018. if the error happens again, please follow up and share your full script so I can reproduce the issue.

Qianqian

--
You received this message because you are subscribed to the Google Groups "mcx-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to mcx-users+...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/mcx-users/d2fbc55a-33f9-4789-a219-cda162b532ban%40googlegroups.com.

[Paloma Casteleiro Costa]

Dr. Fang, 

The error remains with the new version (log attached below). Additionally, i have found a bug (i think) with the new version's 'cone' illumination: the srcparam1, which should only require one input for the NA of illumination, asks for multiple. I have populated the srcparam1 with dummy 0s in locations 2 and 3 so that I can run the simulation. (here is the error log - C++ Error: the 'srcparam1' field must have 3-4 columns)

Another observation I have made is that if I run the same simulation in the old (2018) and new (2024) version of the code, the detected photons distribution is significantly different Attached is two images of the angular distribution of detected photons in the exact same configuration, one run with the old version and the other with the new one. Both use the cone illumination. 

Log from out of memory problem with maxdetphoton: 

mcx.nphoton=5e+09;
mcx.maxdetphoton=5e+07;
mcx.seed=950222;

mcx.tstart=0;
mcx.tend=1e-09;
mcx.tstep=1e-09;
mcx.isrefint=1;
mcx.isreflect=1;

mcx.savedetflag=53;
mcx.autopilot=1;
mcx.gpuid='11';
mcx.unitinmm=0.2;
mcx.dim=[75 75 38];
mcx.mediabyte=1;
mcx.medianum=4;
mcx.srcpos=[12.5 37.5 1 1];
mcx.srctype='cone';
mcx.srcparam1=[0.523599 0.523599 0.523599 0];
mcx.srcdir=[0.707107 0 0.707107 0];

mcx.detnum=1;
mcx.issaveexit=1;
mcx.issaveref=1;
mcx.session='Angle45';

mcx.debuglevel=4;

###############################################################################
#                      Monte Carlo eXtreme (MCX) -- CUDA                      #

#          Copyright (c) 2009-2024 Qianqian Fang <q.fang at neu.edu>          #
#                https://mcx.space/  &  https://neurojson.io/                 #

#                                                                             #
# Computational Optics & Translational Imaging (COTI) Lab- http://fanglab.org #

#   Department of Bioengineering, Northeastern University, Boston, MA, USA    #

###############################################################################
#    The MCX Project is funded by the NIH/NIGMS under grant R01-GM114365      #
###############################################################################

#  Open-source codes and reusable scientific data are essential for research, #
# MCX proudly developed human-readable JSON-based data formats for easy reuse.#
#                                                                             #
#Please visit our free scientific data sharing portal at https://neurojson.io/#
# and consider sharing your public datasets in standardized JSON/JData format #
###############################################################################
$Rev::1e6c40$v2024.2 $Date::2024-03-04 23:18:30 -05$ by $Author::Qianqian Fang$
###############################################################################
- code name: [Heroic Hadron] compiled by nvcc [9.2] for CUDA-arch [350] on [Mar  5 2024]

- compiled with: RNG [xorshift128+] with Seed Length [4]

GPU=1 (NVIDIA GeForce RTX 3090 Ti) threadph=7266 extra=30976 np=-1794967296 nthread=344064 maxgate=1 repetition=1
initializing streams ... init complete : 0 ms
requesting 3072 bytes of shared memory
launching MCX simulation for time window [0.00e+00ns 1.00e+00ns] ...

simulation run# 1 ...
Progress: [==============================================================] 100%

kernel complete:   40683 ms
retrieving fields ... MCXLAB ERROR -2 in unit mcx_core.cu:3417: out of memory

Error from thread (0): out of memory
C++ Error: MCXLAB Terminated due to an exception!

Elapsed time is 40.900893 seconds.

[Qianqian Fang]

On 3/6/24 14:20, Paloma Casteleiro Costa wrote:

Dr. Fang, 

The error remains with the new version (log attached below). Additionally, i have found a bug (i think) with the new version's 'cone' illumination: the srcparam1, which should only require one input for the NA of illumination, asks for multiple. I have populated the srcparam1 with dummy 0s in locations 2 and 3 so that I can run the simulation. (here is the error log - C++ Error: the 'srcparam1' field must have 3-4 columns)

this restriction was introduced when I added the multi-src support end of last year, but I understand that the older relaxed input is more convenient. so I committed the below change to revert this check back to the relaxed form

https://github.com/fangq/mcx/commit/3c7ac2a5d66f2817288c8e200bd22699ff81bc43

give another 10 min for the github action script to complete, you should be able to download the latest binary from

https://mcx.space/nightly/github/

please also be aware that if one defined Param1/Param2 from the JSON input, you must give 3-4 numbers.

Another observation I have made is that if I run the same simulation in the old (2018) and new (2024) version of the code, the detected photons distribution is significantly different Attached is two images of the angular distribution of detected photons in the exact same configuration, one run with the old version and the other with the new one. Both use the cone illumination.

in this case, please share your mcxlab script so I can tell which part of the memory exceeds the limit.

To view this discussion on the web visit https://groups.google.com/d/msgid/mcx-users/3ffc4f07-866a-49f0-b923-405d57eaeba1n%40googlegroups.com.