Out of memory error

[hadar....@mail.huji.ac.il]

Dear All,

I'm trying to run a very big MIP model on matlab (p-median) with 10,000 nodes. Due to the size of the martices I came up to 300,010,000 non zeros. 

When I try to run the model I get an out of memory error with gurobi. 

I tried to set the nodefilestart to 0.5, but that doesn't help.

My thread count is 1.

Is there anything else to do?

I have a i-7 4770 3.4GHz processor with 32GB RAM and a 64-bit operating system.

How much more RAM do I need in order to solve this?

Thank you,

Hadar

[Tobias Achterberg]

At which point during the solving process do you get the out of memory error?
Already while solving the initial LP relaxation or only after having processed a
certain number of MIP search tree nodes? Can you send a log file?

Regards,

Tobias

[hadar....@mail.huji.ac.il]

model = 

        A: [100010001x100010000 double]

      obj: [1x100010000 double]

      rhs: [1x100010001 double]

    sense: '=====================================================================================================...'

params = 

       outputflag: 1

          LogFile: 'logfile'

       resultfile: 'mip1.lp'

    NodefileStart: 0.5000

         presolve: 0

Optimize a model with 100010001 rows, 100010000 columns and 300010000 nonzeros

Presolve removed 0 rows and 0 columns (presolve time = 908s) ...

Variable types: 0 continuous, 100010000 integer (100010000 binary)

Explored 0 nodes (0 simplex iterations) in 1722.02 seconds

Thread count was 1 (of 8 available processors)

Solve interrupted (error code 10001)

Best objective -, best bound -, gap -

Error using gurobi

Gurobi error 10001: Out of memory

Error in PmedianGurobigeneral (line 39)

result = gurobi(model, params);

[hadar....@mail.huji.ac.il]

And I'm adding the one with the presolve:

model = 

        A: [100010001x100010000 double]

      obj: [1x100010000 double]

      rhs: [1x100010001 double]

    sense: '=====================================================================================================...'

Optimize a model with 100010001 rows, 100010000 columns and 300010000 nonzeros

Presolve removed 0 rows and 0 columns (presolve time = 537s) ...

Presolve removed 0 rows and 0 columns (presolve time = 1162s) ...

Presolve removed 0 rows and 0 columns (presolve time = 1165s) ...

Presolve removed 0 rows and 0 columns (presolve time = 1170s) ...

Presolve removed 0 rows and 0 columns (presolve time = 1175s) ...

Presolve removed 0 rows and 0 columns (presolve time = 1180s) ...

Presolve removed 0 rows and 0 columns (presolve time = 1185s) ...

Presolve removed 0 rows and 0 columns (presolve time = 1190s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3795s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3882s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3885s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3890s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3895s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3900s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3905s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3910s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3915s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3920s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3925s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3930s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3935s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3940s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3946s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3952s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3957s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3963s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3965s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3971s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3975s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3980s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3986s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3990s) ...

Presolve removed 0 rows and 0 columns (presolve time = 3996s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4001s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4005s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4012s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4017s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4021s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4027s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4030s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4035s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4040s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4045s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4050s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4055s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4060s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4065s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4070s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4075s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4080s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4085s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4090s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4095s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4100s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4105s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4110s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4115s) ...

Presolve removed 0 rows and 0 columns (presolve time = 4120s) ...

Presolve time: 4680.42s

Explored 0 nodes (0 simplex iterations) in 4999.83 seconds

Thread count was 1 (of 8 available processors)

Best objective -, best bound -, gap -

Error using gurobi

Gurobi error 10001: Out of memory

Error in PmedianGurobigeneral (line 37)

result = gurobi(model, params);

בתאריך יום שני, 22 ביוני 2015 בשעה 14:40:58 UTC+3, מאת Tobias Achterberg:

[cw]

thats a pretty large model. i think that is what motivated column generation. seems just too big.

--

---
You received this message because you are subscribed to the Google Groups "Gurobi Optimization" group.
To unsubscribe from this group and stop receiving emails from it, send an email to gurobi+un...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Carl Wilburn
-------------------------------------------------------
Cell Number:            901-240-5684
-------------------------------------------------------
Operations Research: The Science of Better: http://www.scienceofbetter.org

[hadar....@mail.huji.ac.il]

Yeah it's a big one indeed. 

But, does anybody know if I increase the RAM will it run? How much more RAM do I need?

בתאריך יום שלישי, 23 ביוני 2015 בשעה 19:46:18 UTC+3, מאת carl w:

[Tobias Achterberg]

I agree, this is a huge model. You need a machine with lots of memory to even be
able to start solving it (your logs seem to indicate that as soon as the LP
solving process starts you go out of memory, which could mean that there is not
enough memory to factorize the matrix). But even with enough memory it is not
clear whether you will be able to solve the model within a reasonable time frame.

Tobias

[Tobias Achterberg]

If you can put the model somewhere for me to download, I could give it a try on
one of our machines, which has 384 GB.

Tobias

Am 2015-06-24 um 08:36 schrieb hadar....@mail.huji.ac.il:
> Yeah it's a big one indeed.
> But, does anybody know if I increase the RAM will it run? How much more RAM do I
> need?
>
>
> בתאריך יום שלישי, 23 ביוני 2015 בשעה 19:46:18 UTC+3, מאת carl w:
>
> thats a pretty large model. i think that is what motivated column
> generation. seems just too big.
>

> On Tue, Jun 23, 2015 at 3:25 AM, <hadar....@mail.huji.ac.il <javascript:>>

> an email to gurobi+un...@googlegroups.com <javascript:>.

> For more options, visit https://groups.google.com/d/optout

> <https://groups.google.com/d/optout>.

>
>
>
>
> --
> Carl Wilburn
> -------------------------------------------------------
> Cell Number: 901-240-5684
> -------------------------------------------------------
> Operations Research: The Science of Better: http://www.scienceofbetter.org
>

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "Gurobi Optimization" group.
> To unsubscribe from this group and stop receiving emails from it, send an email

> to gurobi+un...@googlegroups.com <mailto:gurobi+un...@googlegroups.com>.

> For more options, visit https://groups.google.com/d/optout.

--

-----------------------------------------------------------------
Dr. Tobias Achterberg
Senior Software Developer
Gurobi Optimization
achte...@gurobi.com

-----------------------------------------------------------------
Sitz der Gesellschaft: Bad Homburg v.d.H.
Registergericht: Bad Homburg v.d.H., HRB 12607
Geschäftsführer: Robert E. Bixby, Dirk Zechiel

[Paloma Auladell]

Good Morning Dr. Achterberg

I'm getting the gurobi error "out of memory" just before initializing the optimization. It's a huge LP  with at least 10 variable matrixes of 35000x26. The problem it's not difficult to solve but it deals with a lot of data. I'm using a i7-4500U @1.8Ghz processor with 8GB of RAM. Could I send you my python and excel data files to you, as you offered to another user some time ago?

[Silke Horn]

Hi,

Could you share a log file?

- Silke

> To unsubscribe from this group and stop receiving emails from it, send an email to gurobi+un...@googlegroups.com.

[Paloma Auladell]

Here's the log file

[Silke Horn]

Have you tried to solve it with Method=1? The simplex methods need less memory than the concurrent algorithm.

- Silke

> <gurobi.log>

[Jamie Noss]

I am battling with a similar issue which consumes over 500GB (LP problem using barrier). From my experience, the biggest issue is the presolve, as it consumes >4x the base memory usage. Turning the presolve off reduces the mem spike, however, it then takes too long to run. I have also found that it helps to call the presolve explicitly. Using the python api, gurobipy.Model.presolve() returns a new model, thus an entire copy doubling the mem usage. 

# create model
m = gurobipy.Model()

# add vars, obj, constraints etc
# blah blah

# presolve and replace model with presolved instance
m = m.presolve()
# collect old model's mem
gc.collect()

# turn presolve off so that it doesn't run twice
m.Params.Presolve = 0

m.optimize()

Here's the mem usage of a much smaller problem.

[user34]

That's a really nice graph; how did you generate it?

[Jamie Noss]

memory_profiler (mprof) and matplotlib, though mprof has it's own plotter.

[Jamie Noss]

Scrap my code suggestion above! Model.presolve().optimize() is NOT equivilant to Model.optimize(). See Tobias' comment here https://groups.google.com/d/msg/gurobi/CukempCh3uE/fmUDd472GQAJ

On Wednesday, January 23, 2019 at 3:42:38 PM UTC-5, Jamie Noss wrote: