Time required to create and solve model instance

[arde...@ncsu.edu]

Hello everybody,

I am sending this message to tell you about a problem we are facing here in the CCEE department at NCSU and also to ask about ways to improve the time required to build a model through Pyomo.

We have a model right now that has the following size:

Variables                   :  319113  [Nneg: 316725,  Free: 2388]
Objective nonzeros   :    2596
Linear constraints     :  341368  [Less: 213961,  Greater: 2808,  Equal: 124599]
 Nonzeros                : 1453637
 RHS nonzeros        :    3237

The time required for reading and building a model using the current version of Pyomo is about 300 seconds and the time for Cplex to solve it is about 20 seconds.

The above example is for a deterministic version of our model, however, we want to solve stochastic versions with number of stages varying from 2 to 6 for a specific number of branches (scenarios). When we try to solve an instance of the model with 6 stages and 2 branches per stage (using runef) the computational time grows significantly, and after two days running in an extreme powerful machine the model was not finalized yet for Cplex to start (it reached only in the building model portion), and we decided to stop the run. We also tested an instance of such model with two stages and two branches for the uncertain stage. For this later instance the time required was about 3 hours, where, probably less than 5% of the time was spent by Cplex. The model building is taking most of the time there as well.

I am sending this inquiry to ask if these shares of building/solving times were to be expected with Pyomo (for a problem of this size) and also to see if you would have any advices about how to reduce the time for building instances (deterministic and stochastic).

Thank you very much!

Anderson Rodrigo de Queiroz

[Gabriel Hackebeil]

Hi Anderson,

There is usually a lot you can do to speed up the construction time of a Pyomo model after it is first written. If you are starting from an AbstractModel formulation, you can add the "--output-instance-construction-time” option to the runef command to see detailed output about the time taken to construct each component on your model.

If you see output like "Cloning detected! (clone counters: …” under the output for a constraint where a lot of time is being spent, it tells you that you need to focus on rewriting that constraint expression in such a way that Pyomo doesn’t have to build and throw away a lot of intermediate expressions. We can tell you how to do this once you identify some spots like this.

A more detailed profile can be obtained by adding something like “--profile 1000” to the runef command. It is usually best to pipe all output to a file when this option is used (e.g., runef … --profile 1000 > out). If you want to send this output to me directly (off of the forum), I don't might taking a quick look to see if I can identify anything obvious.

I can also point you at some more bleeding-edge developments that speed up Pyomo (e.g., alternate API’s), but we shouldn’t go there until checking for more low-hanging fruit.

Finally, note that the runef command isn’t really meant to scale. The decomposition algorithms in PySP (like runph) are designed to allow you to distribute scenario instances over many compute nodes and do work in parallel, but everything in runef is in serial. I could envision a tool that allows us to merge separate scenario LP files into a final LP file with the additional non-anticipativity constraints (which would allow one to parallelize the model building part of runef for certain solvers), but something like this isn’t really a priority for the developers at the moment.

Gabe

P.S. We are also in the final stages of adding support for the PyPy interpreter, which would possibly allow your code to run 6 or so times faster for free.

--
You received this message because you are subscribed to the Google Groups "Pyomo Forum" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pyomo-forum...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[arde...@ncsu.edu]

Hi Gabe,

Thank you very much for your message! I will try to follow your suggestions and I will let you know about what is happening.

Best,

Anderson