“out of memory” problem with MISOCP

[di zhang]

Dear Johan,

I encountered with an "out of memory" problem solving a MISOCP problem , I tried to change sdpsettings as IBM website https://www.ibm.com/support/knowledgecenter  says, but the problem is the same, could you give me some advice?

Thanks in advance!

the error information is as follows:

sol = 
      yalmiptime: NaN
      solvertime: NaN
            info: 'Unknown problem in solver (Turn on 'debug' in sdpsettings) (Error using diag
Out of memory.  Type HELP MEMORY for your options)'
         problem: 9
    solveroutput: []

I tried to change these settings,

opts.cplex.emphasis.memory=1;
opts.cplex.mip.strategy.file=4;
opts.cplex.workmem=4096;
opts.cplex.mip.limits.treememory=10240;
opts.cplex.mip.strategy.variableselect=3;

by the way, there are some 3-D variables in my codes,like P_IJ=sdpvar(33,33,Times,'full');

and I solved the problem when I set the Times=8 within  tens of seconds, but i encountered this problem when i simply changed Times=16.

[Johan Löfberg]

I guess the problem simply grows to large for cplex during the branching process. No simple fix

[Mark L. Stone]

Have you tried limiting the number of threads CPLEX can use?
http://www-01.ibm.com/support/docview.wss?uid=swg21653811
http://www-01.ibm.com/support/docview.wss?uid=swg21399933
https://www.ibm.com/developerworks/community/forums/html/topic?id=4dc902e9-38df-4a7a-954b-a173f23176f1&ps=25

[di zhang]

Thanks for your reply!

 I tried these other solvers, like gurobi or mosek, but their results are different from the cplex result, and cpelx gets "good" result and these results seems unreasonable...

by the way, though there are some 3-D variables in my codes, most elements in the variable matrix remins 0, could it be possiable to define some kind of "sparse" sdpvar variable?

[di zhang]

Thank you for your advice!

I just read these Q&As, and I put the successfully solved results below, so you mean limiting the parrallel mode?

Tried aggregator 3 times.
MIQCP Presolve eliminated 134007 rows and 115780 columns.
MIQCP Presolve modified 3392 coefficients.
Aggregator did 3808 substitutions.
Reduced MIQCP has 3825 rows, 3204 columns, and 12048 nonzeros.
Reduced MIQCP has 356 binaries, 0 generals, 0 SOSs, and 0 indicators.
Reduced MIQCP has 592 quadratic constraints.
Presolve time = 0.28 sec. (160.68 ticks)
Probing fixed 0 vars, tightened 2692 bounds.
Probing time = 0.02 sec. (15.70 ticks)
Cover probing fixed 0 vars, tightened 1280 bounds.
Clique table members: 224.
MIP emphasis: balance optimality and feasibility.
MIP search method: dynamic search.
Parallel mode: deterministic, using up to 8 threads.
Root relaxation solution time = 0.08 sec. (78.28 ticks)

[Mark L. Stone]

Yes, I mean limiting the number of threads in parallel mode.

[di zhang]

Dear Mark and Johan, I'm very sad to say it's still the same condition. I've limitted the number of threads to 2, but the problem remained.

According to the results displayed below, did it mean the out of memory problem occur even before the optimization began? I mean could I make some progress in the initialization?

As I described before, there are some 3-D variables in my codes, most elements in the variable matrix remins 0, could it be possiable to define some kind of "sparse" sdpvar variable?

Hope for your thoughts!

>> IEEE33_MISOCP_case4

sol = 
      yalmiptime: NaN
      solvertime: NaN
            info: 'Unknown problem in solver (Turn on 'debug' in sdpsettings) (Error using diag
Out of memory.  Type HELP MEMORY for your options)'
         problem: 9
    solveroutput: []

在 2018年1月21日星期日 UTC+8下午9:48:06，Mark L. Stone写道：

[di zhang]

here are the memory-command results of the out of memory problem.

1. Maximum possible array:            17082 MB (1.791e+10 bytes) *
2. Memory available for all arrays:   17082 MB (1.791e+10 bytes) *
3. Memory used by MATLAB:              2043 MB (2.143e+09 bytes)
4. Physical Memory (RAM):             16294 MB (1.709e+10 bytes)
5.   
6. *  Limited by System Memory (physical + swap file) available.

[Johan Löfberg]

As we see display from cplex presolve, it iappears the crash occurs in cplex

Everything is done using sparse data in YALMIP, and as I said, all numerics have obviously already been compiled and sent to cplex. This is a memory issue deep inside cplex, and unless you can tweak various solution options in cplex to avoid this, you will simply have to accept that the problem leads to an excessively large search tree and runs out of memory

[di zhang]

Thanks for your explanation! 

As you said, what are these options that I can try to tweak?

Could you give me some advice on how to reduce the search tree? Besides the code optimization, could it be possible for other solver to avoid the problem?

[Johan Löfberg]

I don't have any direct tip on options to try.

I would try another solver to start with (gurobi/mosek)

[di zhang]

Appreciate for your every quick reply! They did help a lot! I'll try another solver and see what I can get.

[di zhang]

Hi Johan

     It's the same 'memory' problem, and on the cplex forum, I was told to export the model in '.sav' for further examination. During the process of the model creation, it seems that the out of memory problem occured even before the model was sent to cplex, cuz the 

error information was about the function 'diag' in the  "append_normalized_socp" file, and the '.sav' file wasn't created even if I wanted to export the model. The debug information and memory occupation of all variables is as follows. I've noticed that the constraints 

vector 'C' occupied about 1500Mb with more than 300,000 constraints, could it be the reason of the problem? Or the 'diag' function problem? Thanks in advance!

Error using diag
Out of memory. Type HELP MEMORY for your options)'

 

Error in append_normalized_socp (line 35)
            aux = diag(sparse(ind,1,vals,nTotal,1));
Error in  yalmip2cplex (line 21)
[F_struc,K,c,H,ub,lb,x0,Qi,Li,ri] = append_normalized_socp(F_struc,K,c,H,ub,lb,x0);
Error in  call_cplexibm_qcmiqp (line 17)
[model,nonlinearremain] = yalmip2cplex(interfacedata);
Error in  solvesdp (line 350)
    eval(['output = ' solver.call '(interfacedata);']);
Error in  optimize (line 31)
[varargout{1:nargout}] = solvesdp(varargin{:});
Error in  IEEE33_MISOCP_case4_test (line 166)
optimize(C,Obj,opts)

the memory occupation of variables:

>> whos

  Name                       Size                    Bytes  Class       Attributes

  AG                        33x33                     8712  double                

  Branch                    37x6                      1776  double                

  Branch_b                  33x33                     8712  double                

  Branch_fromnum            37x1                       296  double                

  Branch_g                  33x33                     8712  double                

  Branch_num                37x1                       296  double                

  Branch_r                   1x37                      296  double                

  Branch_tonum              37x1                       296  double                

  Branch_x                   1x37                      296  double                

  C                     316881x1                1622149554  lmi                   

  DZ_IJ                     33x33x24                838912  ndsdpvar              

  E_IJ                      33x33x24                838912  ndsdpvar              

  IB                         1x1                         8  double                

  II_IJ                     33x33x24                838912  ndsdpvar              

  I_max                      1x1                         8  double                

  Loss                       1x24                     3320  sdpvar                

  Max_State_change           1x1                         8  double                

  Node                      33x9                      2376  double                

  Node_num                   1x1                         8  double                

  Obj                        1x1                      4088  sdpvar                

  P_DG                       1x24                     3320  sdpvar                

  P_G                       33x24                    27896  sdpvar                

  P_Gmax                    33x1                       264  double                

  P_Gmin                    33x1                       264  double                

  P_IJ                      33x33x24                838912  ndsdpvar              

  P_Load                    33x1                       264  double                

  P_Lt                      33x24                     6336  double                

  Q_G                       33x24                    27896  sdpvar                

  Q_Gmax                    33x1                       264  double                

  Q_Gmin                    33x1                       264  double                

  Q_IJ                      33x33x24                838912  ndsdpvar              

  Q_Load                    33x1                       264  double                

  Q_Lt                      33x24                     6336  double                

  SB                         1x1                         8  double                

  T                         33x33x24                838912  ndsdpvar              

  T_ini                      1x1                         8  double                

  Times                      1x1                         8  double                

  U                         33x24                    27896  sdpvar                

  U_l                       33x33x24                838912  ndsdpvar              

  U_ltrans                  33x33x24                838912  ndsdpvar              

  VB                         1x1                         8  double                

  V_L                       24x1                       192  double                

  V_PV                      24x1                       192  double                

  V_W                       24x1                       192  double                

  V_max                      1x1                         8  double                

  V_min                      1x1                         8  double                

  Variation                 24x7                      1344  double                

  W                         33x33x24                838912  ndsdpvar              

  Z_IJ                      33x33x24                838912  ndsdpvar              

  i                          1x1                         8  double                

  j                          1x1                         8  double                

  opts                       1x1                    244592  struct                

  remote_set                16x2                       256  double                

  remote_set_t               8x2                       128  double                

  t                          1x1                         8  double 

[Johan Löfberg]

YALMIP normalizes the SOCP model which introduces a bunch of new variables in the SOCP models. Hence, as I see it now, there is not much you can do. YALMIP always puts in a normalized format, and when doing so, it passes the memory limit of your machine (thus you have a giant problem)