Out of space error: Does MOSEK predetermines allocation size?

[Carlos de Gois]

When solving some fairly large SDPs in a sizeable cluster (1.5TB RAM), MOSEK terminates with error 1051: Out of space.

It may be the case that the problem is indeed too large. But, on the other hand, when I watch resource usage while it runs, it never goes past 1% of memory usage.

Thus, my question is: does MOSEK estimate the allocation size it needs, and terminates immediately if such amount of memory is not available?

Here are some logs of the problem (from JuMP.jl and MOSEK):

┌ Info: Available memory (MiB):
└   Sys.free_memory() / 2 ^ 20 = 1.53332351953125e6
┌ Info: Problem was setup.
│   problem =
│    A JuMP Model
│    Maximization problem with:
│    Variables: 1119745
│    Objective function type: VariableRef
│    `VariableRef`-in-`MathOptInterface.GreaterThan{Float64}`: 1 constraint
│    `VariableRef`-in-`MathOptInterface.LessThan{Float64}`: 1 constraint
│    `Vector{VariableRef}`-in-`ComplexOptInterface.HermitianPositiveSemidefiniteConeTriangle`: 6 constraints
│    `GenericAffExpr{ComplexF64, VariableRef}`-in-`MathOptInterface.EqualTo{ComplexF64}`: 186732 constraints
│    Model mode: AUTOMATIC
│    CachingOptimizer state: EMPTY_OPTIMIZER
│    Solver name: Mosek
└    Names registered in the model: vis

Problem
  Name                   :                
  Objective sense        : max            
  Type                   : CONIC (conic optimization problem)
  Constraints            : 1495368        
  Cones                  : 0              
  Scalar variables       : 1              
  Matrix variables       : 6              
  Integer variables      : 0              

Optimizer started.
Presolve started.
Linear dependency checker started.
Linear dependency checker terminated.
Eliminator started.
Freed constraints in eliminator : 0
Eliminator terminated.
Eliminator - tries                  : 1                 time                   : 0.00            
Lin. dep.  - tries                  : 1                 time                   : 0.21            
Lin. dep.  - number                 : 0              
Presolve terminated. Time: 0.37    
MOSEK error 1051: Out of space.
Optimizer terminated. Time: 190.16 

[Erling D. Andersen]

It is most likely an allocation of a large block that fails during the optimizer setup that fails.

Given the large number of constraints and few matrix variables that is likely to happen.

Since then at least one matrix variable appears in many constraints and then you need a lot of storage.

If your original problem was on LMI form, then sometimes it is better to feed in the dual problem of that.

Cannot say whether that is the case for you based on the information provided.

PS.   Only slow optimizers do many small allocations since this is computationally expensive.