Parallel formulation of constraints and memory usage

[Cristián Serpell]

Hi all,

Some years ago I asked regarding this, and now we are looking into it again.

We formulate very big concrete models (sometimes many millions of variables) using Pyomo programatically. We see that the time spent formulating the model is often bigger than the time used to solve it using different solvers. Also, the memory usage of python during the formulation step is huge, in the order of many tens of GB, in comparison with the solver memory usage of the same model, usually just a few GB.

This is starting to be the bottle neck for us for big problems. So my questions are:

- Has anyone managed to parallelize constraint formulation? That probably would increase the memory usage even further, but time would drop.

- Do you have any ideas on how to drastically reduce memory usage? Probably this is produced by the fact that Pyomo creates several python structures to do stuff, but maybe there is a way to work with "light" structures, or a point within pyomo where we can work to improve its memory usage.

Any help is appreciated,

Cristián

[Gabriel Hackebeil]

We formulate very big concrete models (sometimes many millions of variables) using Pyomo programatically. We see that the time spent formulating the model is often bigger than the time used to solve it using different solvers. Also, the memory usage of python during the formulation step is huge, in the order of many tens of GB, in comparison with the solver memory usage of the same model, usually just a few GB.

IMO, the low hanging fruit is:

  (1) Use PyPy (faster and reduced memory usage): We’re very close to fully supporting PyPy at this point. Just a few more kinks to work out in the expression generation system.

  (2) Develop support for more homogenous vector components in Pyomo: One could express large blocks of constraints using a lightweight vector variable combined with a sparse constraint matrix (numpy?) and a rhs vector.

  (3) Make it easier for users to create their own classes of things like variables that are more lightweight, and thus provide less flexibility, but still work for their limited, super large-scale applications. I think there is a lot of room to expand our concrete modeling interface to support this, but we’re struggling to agree on how keep this in sync with the abstract modeling approaches.

- Do you have any ideas on how to drastically reduce memory usage? Probably this is produced by the fact that Pyomo creates several python structures to do stuff, but maybe there is a way to work with "light" structures, or a point within pyomo where we can work to improve its memory usage.

Historically, I think our design philosophy has placed more weight on providing easy-to-use and flexible modeling components that we can use to implement new and interesting optimization strategies, but we still make some attempts at making sure things stay lightweight where possible (e.g., by using __slots__).

I can direct you to some prototype components related to points (2) and (3), but it’s hard to say how much they will help without more details about your large-scale test instances. In general, it would be interesting to know the following:

 - What proportion of the total memory is used by variables? constraints (expressions)? data (Set, Param, other data)?

 - What proportion of the total time is spent creating the variables? creating the constraints? converting the model to the solver input format (e.g., inside the LP writer)?

Some of this might be difficult to determine, but it would be very useful to us.

Gabe

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Cristián Serpell]

Thanks for your insightful answer.

We will try to collect answers for your questions, in order to know which are the real sources of our memory/time usage.

Cristián

[Andres Collart]

Hi Cristian,

I've had similar issues before, and given the problem I found that using lazy constraints I was able to reduce the formulation time drastically while only slightly increasing solution time. Just a thought, won't work for all problems though.

Cheers,

Andres

[Cristián Serpell]

Thanks Andres.

What do you mean with lazy constraints in this case?

Cristián

[Andres Collart]

Hi Cristian,

Instead of generating all the constraints you can only generate some basic ones, therefore reducing your model creation time significantly. Once you have an interim solution you can check if any of your other constraints are broken and add them in as necessary to re-solve. 

I've done this with RCPSP formulations such that most constraints are included in the base model, but resource violations are checked on interim solutions and the associated constraints added as needed. Not all solvers will support lazy constraints though (I was using Gurobi locally- cloud version doesn't support lazy constraints). Let me know and I can look into providing the code privately.

-Andres

[Watson, Jean-Paul]

The first thing I would do in your model is add the “report_timing=True” (I think that’s the option – let me know if it doesn’t work) to the model create_instance() call – assuming you are using AbstractModel. 

This should yield a display of how long it took to construct each model component. I wouldn’t call this profiling (which will tell you more about what is slow in Pyomo internals), but it should still help. 

It could also help with identifying which constraint to (temporarily) relax, as Andres and others are recommending.

jpw

--