Gurobi breaks with non-zero return code (137) when solving MIP problem

[Cord Wayne]

Hi everybody,

I have an (abstract) MIP model which is solved about 50 times with a different parametrization which works perfectly within less then an hour for all but 4 problems even with MIPGap changed to 0.01.

As an example one of these problems breaks with non-zero return code (137) solving after a runtime of about 15 hours:

17:07:13-INFO-Solving optimization problem.
Changed value of parameter MIPGap to 0.01
   Prev: 0.0001  Min: 0.0  Max: 1e+100  Default: 0.0001
Optimize a model with 551881 rows, 464280 columns and 1410286 nonzeros
Variable types: 324120 continuous, 140160 integer (140160 binary)
Coefficient statistics:
  Matrix range     [1e-01, 2e+02]
  Objective range  [5e-01, 1e+08]
  Bounds range     [1e+00, 2e+03]
  RHS range        [1e+00, 8e+02]
Found heuristic solution: objective 1.19028e+12
Presolve removed 341738 rows and 289150 columns
Presolve time: 3.10s
Presolved: 210143 rows, 175130 columns, 595394 nonzeros
Variable types: 96327 continuous, 78803 integer (78803 binary)

Root simplex log...

Iteration    Objective       Primal Inf.    Dual Inf.      Time
   80011   -1.3957438e+07   9.092965e+05   0.000000e+00      5s
  129552   -6.3247837e+06   7.784634e+03   0.000000e+00     10s
  130154   -6.3204527e+06   0.000000e+00   0.000000e+00     10s

Root relaxation: objective -6.320453e+06, 130154 iterations, 7.15 seconds

    Nodes    |    Current Node    |     Objective Bounds      |     Work
 Expl Unexpl |  Obj  Depth IntInf | Incumbent    BestBd   Gap | It/Node Time

     0     0 -6320452.7    0 6624 1.1903e+12 -6320452.7   100%     -   12s
H    0     0                    6.820762e+10 -6320452.7   100%     -   16s
     0     0 -6198874.6    0 8271 6.8208e+10 -6198874.6   100%     -   21s
     0     0 -6189241.1    0 9555 6.8208e+10 -6189241.1   100%     -   27s
     0     0 -6022543.0    0 10259 6.8208e+10 -6022543.0   100%     -   37s
     0     0 -6020067.6    0 13075 6.8208e+10 -6020067.6   100%     -   47s
     0     0 -5972198.4    0 14036 6.8208e+10 -5972198.4   100%     -   61s
     0     0 -5969625.3    0 14251 6.8208e+10 -5969625.3   100%     -   75s
     0     0 -5941885.6    0 14381 6.8208e+10 -5941885.6   100%     -   92s
     0     0 -5939337.3    0 14311 6.8208e+10 -5939337.3   100%     -  112s
     0     0 -5922388.4    0 13895 6.8208e+10 -5922388.4   100%     -  130s
     0     0 -5919927.1    0 13881 6.8208e+10 -5919927.1   100%     -  134s
     0     0 -5913197.8    0 13832 6.8208e+10 -5913197.8   100%     -  137s
     0     0 -5911655.1    0 13845 6.8208e+10 -5911655.1   100%     -  139s
     0     0 -5908583.1    0 13784 6.8208e+10 -5908583.1   100%     -  142s
     0     0 -5908148.2    0 13782 6.8208e+10 -5908148.2   100%     -  144s
     0     0 -5907071.2    0 13672 6.8208e+10 -5907071.2   100%     -  146s
     0     0 -5906883.0    0 13646 6.8208e+10 -5906883.0   100%     -  148s
     0     0 -5906292.8    0 13667 6.8208e+10 -5906292.8   100%     -  151s

[...]

 351517 302156 -5450550.7  351 6618 -5315027.9 -5844554.3  10.0%  22.7 54174s
 351659 302316 -5443537.9  373 6450 -5315027.9 -5844554.3  10.0%  22.7 54194s
 351747 302401 -5435859.1  382 6364 -5315027.9 -5844554.3  10.0%  22.7 54215s
 351888 302536 -5426279.3  399 6321 -5315027.9 -5844554.3  10.0%  22.7 54235s
 352032 302677 -5421305.7  412 6154 -5315027.9 -5844554.3  10.0%  22.7 54255s
H352183 302428                    -5315308.486 -5844554.3  10.0%  22.7 54279s
H352184 302402                    -5315342.426 -5844554.3  10.0%  22.7 54279s
H352186 302399                    -5315347.020 -5844554.3  10.0%  22.7 54279s
H352187 302301                    -5315417.558 -5844554.3  10.0%  22.7 54280s
H352195 302321                    -5315443.599 -5844554.3  10.0%  22.7 54280s
 352211 302413 -5404065.3  433 5993 -5315443.6 -5844554.3  10.0%  22.7 54309s
 352284 302462 -5389830.8  432 6071 -5315443.6 -5844554.3  10.0%  22.7 54315s
 352364 302489 -5396908.4  447 5932 -5315443.6 -5844554.3  10.0%  22.7 54343s
 352514 302702 -5389496.3  460 5803 -5315443.6 -5844554.3  10.0%  22.7 54362s
Killed
ERROR: Solver (gurobi) returned non-zero return code (137)
08:14:07-ERROR-Solver (gurobi) returned non-zero return code (137)
ERROR: See the solver log above for diagnostic information.
08:14:07-ERROR-See the solver log above for diagnostic information.
Traceback (most recent call last):
  File "app.py", line 144, in <module>
    cli()
  File "/home/user/.anaconda3/lib/python3.6/site-packages/click/core.py", line 722, in __call__
    return self.main(*args, **kwargs)
  File "/home/user/.anaconda3/lib/python3.6/site-packages/click/core.py", line 697, in main
    rv = self.invoke(ctx)
  File "/home/user/.anaconda3/lib/python3.6/site-packages/click/core.py", line 895, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/home/user/.anaconda3/lib/python3.6/site-packages/click/core.py", line 535, in invoke
    return callback(*args, **kwargs)
  File "app.py", line 106, in cli
    cmdline_options={'mipgap': 0.01})
  File "/home/user/Projekte/asom/asom/shared.py", line 78, in solve_optimization_model
    shortcut = self.optimization_model.solve(**kwargs)
  File "/home/user/Projekte/oemof/oemof/solph/models.py", line 168, in solve
    results = opt.solve(self, **solve_kwargs)
  File "/home/user/.anaconda3/lib/python3.6/site-packages/pyomo/opt/base/solvers.py", line 626, in solve
    "Solver (%s) did not exit normally" % self.name)
pyutilib.common._exceptions.ApplicationError: Solver (gurobi) did not exit normally

My OS is Ubuntu 16.04 x64 and I am using the Pyomo solver interface. The weird thing is that the incumbent and best bound are still not very close which -if it was the other way round- could be solved context-wise by using MIPGapAbs.

Any hints how I can solve this except for playing arround with MIPGap and MIPGapAbs?

Thanks in advance!
Cord

[Tobias Achterberg]

What does the error code 137 mean in Pyomo? This is not a Gurobi error code.

My guess is that you are running out of memory. The model is not tiny, and the tree is
also relatively large. You could try to set the "NodefileStart" parameter in order to use
the hard drive to help storing search tree nodes, see
http://www.gurobi.com/documentation/8.0/refman/nodefilestart.html#parameter:NodefileStart

Best regards,

Tobias

[Cord Wayne]

Hi Tobias,

thanks for your answer. I'll try my luck with your suggestion and post my progress here!

Best
Cord

[Cord Wayne]

Hi Tobias,

I have now tried your recommendation directly using gurobi_cl and the memory scarcity seems to be the reason for the breaking model.

If I solve my model using

$gurobi_cl nodefilestart=0.5 mipgap=0.01 problem.lp

it does not break and keeps on solving.

Nevertheless, the gap does not reduce quickly and is still at about 9% after a runtime of 26 hours on my i7 quad core machine with 16 GB RAM.

I have read about the Parameter Tuning Tool. Would you recommend to apply this to e.g. 20 of the other problems to achieve a runtime boost? Or do you have any recommendations to boost the MIP runtime except for reformulating the problem (which would break my whole pre- and postprocessing process)?

Cheers

Cord

Am Mittwoch, 25. April 2018 09:56:33 UTC+2 schrieb Tobias Achterberg:

[Tobias Achterberg]

The tuning tool is always worth a try. But you should give it a set of models to tune for,
and those models should ideally be easier to solve while still having similar
characteristics as your hard model.

You can also post a log file of your hard model solve. Maybe we can see something to
provide some hints on what to try next.

Tobias

[Cord Wayne]

Hi Tobias,

Am Montag, 30. April 2018 20:02:24 UTC+2 schrieb Tobias Achterberg:

The tuning tool is always worth a try. But you should give it a set of models to tune for,
and those models should ideally be easier to solve while still having similar
characteristics as your hard model.

I will try to provide some models and report my progress here.

You can also post a log file of your hard model solve. Maybe we can see something to
provide some hints on what to try next.

Tobias

You can see the log file here: https://www.dropbox.com/s/sqhl6xubi3uuglc/gurobi.log?dl=0

The solver has quit after 28 hours of solving time and I am guessing that the harddisk memory limit was exceeded due to the log file. If you have  a look at the file this seems to be plausible because the mipgap hardly changes over hours.

Best
Cord

[Cord Wayne]

Hi Tobias,

I have played arround now with the tuning tool and a set of smaller models (time horizon reduced to a fourth) of the original model and it already provided me some improved parameter sets.

Would you recommend to optimize for an optimal solution or for the MIPGap e.g. by using

grbtune TimeLimit=100 my_problem

?

Cheers
Cord

Am Montag, 30. April 2018 20:02:24 UTC+2 schrieb Tobias Achterberg:

[Tobias Achterberg]

Yes, error 10019 is (as the log file says) that there was an issue with reading or writing
node files. I guess that the hard drive was full.

Whether you tune for optimality or for a small MIPGap after a given time limit depends on
what you want for your application. For example, if the process where the optimization is
embedded is required to find some solution within 20 minutes, then you would use a time
limit of 20 minutes and tune for a small MIPGap in this time limit. If you just want to
solve your model as fast as possible, you would not specify a time limit.


Regards,

Tobias

[Cord Wayne]

Thanks for clarification! My goal is a mixture of both criteria because I want to solve a model as fast as possible until a provided MIPGap (e.g. 0.01).
If I understand your explanation right, I would then invoke the tuning tool with this parameter fixed as for example:

grbtune MIPGap=0.01 TuneTimeLimit=7200 my_problem1 my_problem2 my_problem3

Right? And do you think that the reduced problem size of one fourth is appropriate for the training problems?

Thanks,
Cord

Regards,

Tobias

[Tobias Achterberg]

Yes, the parameters to grbtune look right. Regarding the problem size, there is no general
answer. You have to play with it a little bit. The problem should be easy enough so that
the tuning tool has a chance to try various parameter combinations in a reasonable amount
of time. On the other hand, it should be realistic enough so that the results are
applicable to your original model.

Tobias

[Cord Wayne]

Hi Tobias,

thanks for your explanation. I have played arround a bit yesterday and indeed the tuning tool already helped me to solve some problems in reasonable time which had not been possible before.

Cheers
Cord