Casadi consumes ever-increasing memory:

[Dave Goel]

Casadi consumes ever-increasing memory:

My program defines a Function that does a lot of computation on some params and then returns a DM(1,1). 

It then computes double(Function(params)) millions of times. The double is not stored anywhere, and is promptly added to an exponential average.

Thus, I know I am not creating any large memory objects. Yet, as it runs, it consumes more and more system memory, till the entire machine is eventually out-of-memory. 

If a casadi DM is never assigned to a variable, and never used except being immediately converted to full, does casadi not automatically discard its memory usage? 

[Joris Gillis]

Hi,

What you describe is a memory leak. Our unittests check for memory leaks, so this comes as surprise. Can you give an example?

Best,
Joris

[Dave Goel]

Joris,

Thanks for replying so fast! Here it is. It is a very simple function that takes a scalar as input, and returns a 10k X 10k version of that. Octave then averages it back up to a scalar and adds it to an accumulator. (Octave re-uses the same memory object; I have used done far more humongous things in octave, and I think the problem is not at octave's end.).

As this example runs, it creates and discards an anonymous large DM.  Octave assigns it to a variable via full, but re-uses the same variable. But, my memory usage shoots up, incrementally, from 5G to 8G to ... 13G (ii=12) to ... 43G (ii=48) 

input=MX.sym('input',1,1);

b = repmat(input, 1, 10123);  ## 10k elements. 

c=b'; 

largedm=c * b; ## Has 100M elements! 

Ftest = Function('testf', {input}, {largedm});

accum = 0;

for ii=1:9000;

  real_input = randn(1,1);

  largeoutput = full(Ftest(real_input));

  accum += sum(sum(largeoutput))/numel(largeoutput);

  printf("ii: %d, result: %d\n", ii, accum);

endfor

--
Sent from CasADi's user forum at http://forum.casadi.org.
---
You received this message because you are subscribed to a topic in the Google Groups "CasADi" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/casadi-users/x4tk9hKjiAE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to casadi-users+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/casadi-users/3670027c-5d46-4078-964c-0438d5a83f2a%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Dave Goel]

I should have mentioned: 

octave:1> version

ans = 4.0.3

with the latest casadi-octave-v3.1.1.tar.gz

[Dave Goel]

Hi Joris, 

Here's an even more straightforward example of memory leak:

octave:2> a=randn(10123);

octave:3> memorymy 

Process 11923 consumes RES memory (estimated): 1109.180 M

ans = 

octave:4> b=DM(a);

octave:5> memorymy 

Process 11923 consumes RES memory (estimated): 2314.548 M

ans = 

octave:6> b=DM(a);

octave:7> memorymy 

Process 11923 consumes RES memory (estimated): 3115.136 M

ans = 

octave:8> b=DM(a);

octave:9> memorymy 

Process 11923 consumes RES memory (estimated): 3915.724 M

ans = 

octave:10> b='';

octave:11> memorymy 

Process 11923 consumes RES memory (estimated): 3915.724 M

ans = 

octave:12> b=DM(a);

octave:13> memorymy 

Process 11923 consumes RES memory (estimated): 4716.312 M

On Saturday, April 8, 2017 at 12:58:22 PM UTC-4, Dave Goel wrote:

I should have mentioned: 

octave:1> version

ans = 4.0.3

with the latest casadi-octave-v3.1.1.tar.gz

[Dave Goel]

Ah, I saw .delete and .clear. The latter doesn't help, but the former seems to stem the leak:

octave:5> a=randn(10123);

octave:6> b=DM(a);

memorymy

octave:7> memorymy

Process 17268 consumes RES memory (estimated): 2314.496 M

ans = 

octave:8> b.delete;

octave:9> memorymy 

Process 17268 consumes RES memory (estimated): 1113.572 M

ans = 

octave:10> b=DM(a);

memorymy

octave:11> memorymy

Process 17268 consumes RES memory (estimated): 2314.496 M

ans = 

octave:12> b.delete;

octave:13> memorymy 

Process 17268 consumes RES memory (estimated): 1113.572 M

ans = 

octave:14> b=DM(a);

octave:15> memorymy 

Process 17268 consumes RES memory (estimated): 2314.496 M

[Dave Goel]

fyi, This memory leak continues to exist in casadi 3.4.4 (octave 4.2)

Hi Joris, 

[dee...@gmail.com]

Hi

FYI, this memory leak continues to exist in casadi  3.5.5_for_octave_6.1.0 on linux.

I provided very simple examples above (in 2017) to see this bug in action.

[dee...@gmail.com]

This example shows that the memory leak continues to happen even if we carefully delete all created variables:

input=MX.sym('input',1,1);
repeatedinput = repmat(input, 10000,10000);  
Ftest = Function('testf', {input}, {input+repeatedinput});  

for ii=1:9000;
  real_input = randn(1,1);

  largeoutputDM = Ftest(real_input);
  largeoutputDM.delete;
  memorymy; ## print current memory footprint here.
endfor

Notice the largeoutputDM.delete within the for loop.

Yet, in every for loop, the memory consumed increases by 0.8G.