C++ Engine Array Size Limits?
Ryan
James Tursa
> I am using a custom C++ application that uses engOpen() and then put several mxArrays into the engine. At a certain array size, just above 50,000,000 doubles, the engPutVariable fails with a return of 1. It is as if the engine won't allow a single array to be transferred larger than a certain size. Perhaps this is a stack size issue for the matlab enginer? Any ideas?
That's 400MB per mxArray. How many mxArray variables are you working with? How much RAM do you have? What is the largest contiguous block of memory that your have available? Etc. Etc. Note that the engine will have a separate copy of the variable, so this doubles the amount of memory required for each copy. Unless you are freeing memory on the C++ side as you go this can double everything up.
James Tursa
Ryan
I misstated the amount, it is actually 5.6 million doubles, so only 45mb or so. I will look into contiguous memmoy, however I am able to go to the command window right after the call fails and create an array 10 times that size.
James Tursa
> "James Tursa"
>
> I misstated the amount, it is actually 5.6 million doubles, so only 45mb or so. I will look into contiguous memmoy, however I am able to go to the command window right after the call fails and create an array 10 times that size.
Some observations from some quick tests on a WinXP system:
Largest contiguous blocks (feature memstats):
938MB
851MB
238MB
etc.
I opened a C program and created a 425MB array and put it into the engine successfully. However, when I tried a 426MB array it failed. So, interestingly, 425MB x 2 is just under the 851MB block available, and 426MB x 2 is just over the 851MB block available. My preliminary conclusion is that there *is* a hidden limitation with the COM interface here and an extra copy in the same block is needed in the background to do the copy. I will try to look at the doc to see if there are any engine/COM limitations listed there.
Interestingly, the failure takes several seconds ... same amount of time as the success. Like the size isn't checked ahead of time but is tried and only fails when the copy finally exceeds the limit.
Also interestingly, I ran another test where a C program created a 600MB array and a 426MB array. While the C program was up and running with these arrays allocated, I opened another MATLAB session and created two new variables in the workspace of 600MB and 426MB size, and it worked. Given the largest blocks available according to feature memstats, this shouldn't have worked, but it did. Also at that time, feature memstats from the engine window did not agree with feature memstats from the normal MATLAB window, the engine was still showing the large blocks being available, whereas the normal MATLAB was showing reduced blocks by the 600MB and 426MB amounts. I don't know how this all works yet ...
James Tursa
Rune Allnor
> I am using a custom C++ application that uses engOpen() and then put several mxArrays into the engine. At a certain array size, just above 50,000,000 doubles, the engPutVariable fails with a return of 1. It is as if the engine won't allow a single array to be transferred larger than a certain size. Perhaps this is a stack size issue for the matlab enginer? Any ideas?
Well, a variable that contains 50,000,000 doubles needs at least
400,000,000 mytes of memory. In a 32 bit OS the memory limit is
2 GBytes, which is enough to hold at most 5 such variables.
On a single-core system the 2GB will also accomodate the matlab
executable, the OS as well as all the non-related executables
which run while you do your computations.
So you have hit the RAM ceiling on your computer. Restructure the
data flow so that you only have the data you need at any one time
loaded into RAM. Then you can improve run-times by adding small
data buffers.
Rune
Ryan
It is interesting how much extra memory is required to transfer variables to the Matlab engine.
Thank you again.
Ryan
Rune Allnor <all...@tele.ntnu.no> wrote in message <d5f10a19-a443-48b0...@w32g2000hsf.googlegroups.com>...
> On 3 Okt, 23:37, "Ryan " <ryan.tint...@ngc.com> wrote:
> > I am using a custom C++ application that uses engOpen() and then put seve=
> ral mxArrays into the engine. =A0At a certain array size, just above 50,000=
> ,000 doubles, the engPutVariable fails with a return of 1. =A0It is as if t=
> he engine won't allow a single array to be transferred larger than a certai=
> n size. =A0Perhaps this is a stack size issue for the matlab enginer? =A0An=