Running out of memory

[othman moumni abdou]

Hi Johan,
I am running out of memory for my problem when creating constraints.
I have vectors/matrix that have a size of 43800 values (43800 time step).

The model crash at the first constraints created (Numeric value|       Equality constraint 43800x1):

Error using sdpvar/eq (line 18)
Error using eye
Out of memory. Type HELP MEMORY for your options.


Is there any way to handle this out of memory issue or is the problem too big for YALMIP (preallocation maybe, but I don't know how to do this on constraints)?

Thank you

ps: In attached the YALMIP code of the model, the variable p_max controls the length of my decision variables

[Johan Löfberg]

> bigM = 1E30;

http://www.youtube.com/watch?v=nuu2iNisoQc

The correct term is not big-M but as-small-as-possible-but-sufficiently-large. You have to use a realistic bound to get a sensible model

The memory issues you talk about is fixed by replacing eye with speye in the file sdpvar/cleandoublefactors.m

[othman moumni abdou]

Yes, indeed, my big-M was too big. I forgot to change it to a reasonable value.

Thanks for the fix, it seems to work much better.

Othman

[Johan Löfberg]

BTW, this pice runs horribly slow. No reason to use a double-for loop and touching data back and forth

r=1;
for p=1:pas_reseau:p_max-pas_reseau
    delta_p(r) = 0;
    for k=p:p+pas_reseau
        delta_p(r) = delta_p(r) + p_reseaupos(k) - p_souscrite;
    end
    delta_p(r) = delta_p(r)/ pas_reseau;
    r =r+1;
end

You're simply doing (and this can be simplified further with a cumsum, but the gain is not that large)

r=1;
for p=1:pas_reseau:p_max-pas_reseau
    delta_p(r) = (sum(p_reseaupos(p:p+pas_reseau))-(pas_reseau+1)*p_souscrite)/pas_reseau;  
    r =r+1;
end

However, I don't see why you define delta_p first, and then kill the elements with new stuff. The best approach (if not vectorizing with cumsum etc) would be

delta_p = [];
for p=1:pas_reseau:p_max-pas_reseau
    delta_p = [delta_p;(sum(p_reseaupos(p:p+pas_reseau))-(pas_reseau+1)*p_souscrite)/pas_reseau];  
end

(and then add some trailing crap, since you expect this vector to have a certain length, although you only set some part of it)

[othman moumni abdou]

That's exactly what I was changing right now. I just changed it to this:

r=1;

for p=1:pas_reseau:p_max-pas_reseau
    delta_p(r) = (sum(p_reseaupos(p:p+pas_reseau))-(pas_reseau+1)*p_souscrite)/pas_reseau;  
    r =r+1;
end

About defining delta_p, isn't it better to define the sdpvar to allocate necessary memory in the beginning of the code? Because with your last approach, the vector change size every iteration.

Vectorization will be the best solution, I will look into it deeper in the future days.

Thank you for your help

[Johan Löfberg]

and the same goes for other things, like this (takes 50secs for p_max= 5000!)

for p=1:p_max-1

    constraints = [constraints,ea_stock(p+1,:)==-pa_stock(p,:)*delta_t_sim_en_heure+ ea_stock(p,:)*(1-coef_stockadch)'];         

end

which simply is (0s)

ea_stock((2:p_max),:) == -pa_stock((1:p_max-1),:)*delta_t_sim_en_heure+ ea_stock((1:p_max-1),:)*(1-coef_stockadch)'

[Johan Löfberg]

No, it does not help to preallocate. The only things available in the object then is the sparse matrix representing a basis for a unit vector. When you start changing elements inside that, the matrices change, and manipulating elements in sparse matrices is expensive. Cheaper to concatentate given matrices (obviously, from empirical evidence)

[othman moumni abdou]

Ok, duly noted.

Thanks again