In the SDRTrunk Directory, open the SDRTrunk App Log and look for the RAM usage in the brackets. This is the Java memory allocation. See if it gets close to the threshold. Regardless of the RAM in your system, edit the SDRTrunk.bat file and add Java parameters to allocate more RAM.
"-Xms4g" will allocate 4GB of RAM.