Multiprocessing

[Matteo Nicoli]

Hello everyone, I was wondering if there was an option in Temoa algorithm to enable multiprocessing simulations (to take full advantage of system resources), or if you ever thought about introducing it (if possible). Regards,
Matteo Nicoli

[Cameron Wade]

Hi Matteo,

Although I'm not a member of the Temoa team, I have been using the tool for some time. Here are my thoughts around multiprocessing:

• A single model run consists of several stages: 1) Reading the data files, 2) Creating the Temoa model instance, 3) Solving the instance, 4) Calculating reporting variables and formatting the results.
• For large model runs, the majority of the time is spent in the third stage, i.e. solving the model. In this stage, Pyomo sends the model instance to a solver such as CPLEX, Gurobi, glpk or COIN-OR CBC. Some of these solvers, such as CPLEX, use several threads/cores to solve the problem. With CPLEX, the solver simultaneously tries different solution algorithms on different cores until it determines a preferred algorithm, e.g. barrier. However, some of these solvers, e.g. CBC, do not make good use of multithreading. Although you can specify the number of threads as a CBC parameter, it has been my experience that it is ineffectual for CBC and the linear programs that Temoa creates. It would seem CBC and glpk are inherently serial in their approach to solving LPs. The takeaway is that some solvers support using multiple threads, but that is independent of Temoa. For more information on, you can consult the Pyomo documentation.
• Although solving the model consumes the most time, for large models there is considerable time taken in steps 2) and 4) above. From my understanding, these steps -- creating the Temoa instance and calculating and reporting the output -- could likely benefit from multiprocessing or further optimization. The code is essentially formulating the millions of variable definitions and constraint equations for the model instance, which I believe could be done in parallel.
  
• I haven't had the time to dig into it, but it seems like the folks behind PyPSA have looked into speeding up the model instantiation process. See these resources. Perhaps Temoa could benefit from a similar approach -- I'm not sure.
  

Thanks for bringing this to our attention... Looking into this in more detail could be a great project.

Cheers,

Cam

On Tue, Jan 25, 2022 at 12:57 PM Matteo Nicoli tramite Temoa Project <temoa-...@googlegroups.com> wrote:

Hello everyone, I was wondering if there was an option in Temoa algorithm to enable multiprocessing simulations (to take full advantage of system resources), or if you ever thought about introducing it (if possible). Regards,
Matteo Nicoli

--
You received this message because you are subscribed to the Google Groups "Temoa Project" group.
To unsubscribe from this group and stop receiving emails from it, send an email to temoa-projec...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/temoa-project/f7038ac9-66e9-423d-82df-83cacd4c16b8n%40googlegroups.com.

--

Cameron Wade

Principal, Sutubra Research

p: 902-292-4478