Debug for very slow run

[Xi Xi]

Hi all,

I built a SWITCH model for Kenya using 2.0.7 and I noticed that it takes very long to run. Without the storage module, it take about 22 hours, and with storage, it takes 3+ days and I terminated the run. I have 47 load zones, 14 technologies, 24h/day * 1day/year * 12 years. It doesn't seem like it should've taken this long to run. Can you give me some ideas to debug where I might be having issues?

Thank you!

Best,

Xi Xi

[Matthias Fripp]

Hi Xi Xi,

One of the biggest issues is which solver and settings you're using. We've generally found that the commercial solvers (Gurobi, CPLEX, Mosek, Cardinal) run 10-100+ times faster than the open solvers (glpk, cbc) although HiGHS is closing that gap a bit. We usually find the barrier algorithm is fastest, but with few cores you may find simplex or dual simplex is faster. 

Is the long wait time in Python/Switch (before you start seeing text from the solver itself) or in the solver?

You might also check how much RAM you have -- that is a medium size model and I would expect it to need about 8 GB in the Python stages and around 2 GB in the solver stage. If you have much less than that, it can slow work down as it swaps to disk. 

Another common option is to reduce the problem size. I've found solver time is roughly cubic with respect to the number of timepoints or load zones in the model. So reducing those can help a lot. I would generally aggregate to larger zones (10-25 total?) and run with more sample days per year (12+) and fewer sample years (4-6 total). If that is still too big, I would switch the timepoints to every 2 or 3 hours instead of hourly. That size of model is usually tractable with the commercial solvers with 16 GB RAM or so. But with the open solvers you generally need a much smaller model (generally only toy models will work with GLPK or CBC, but HiGHS may solve a medium size model if you are patient). 

I hope this helps. Let me know if you have more questions.

Matthias​

---

From: 'Xi Xi' via Switch Model <switch...@googlegroups.com>
Sent: Friday, September 20, 2024 4:27:38 PM
To: Switch Model <switch...@googlegroups.com>
Subject: [EXTERNAL] [switch-model] Debug for very slow run

 

--
You received this message because you are subscribed to the Google Groups "Switch Model" group.
To unsubscribe from this group and stop receiving emails from it, send an email to switch-model...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/switch-model/11aeadc4-bb50-42f1-88b6-0b716c7d6fa3n%40googlegroups.com.

[Matthias Fripp]

One other thing I forgot to mention -- for these size models, it's generally best to completely avoid integer variables if possible. Any integer variables push the solver onto a much slower solution path. And I would only expect to solve a heavily integer constrained model for something like 1 zone x 12 timepoints/day x 13 days/period x 5 periods. 

You can usually see messages about integer variables in the solver log, described as "binary", "integer" or "general". They are introduced if you use the discrete_build or discrete_commit modules or if you specify a minimum unit size (gen_min_build_capacity in gen_info.csv) for any generation projects. If you've ruled these out and are still getting integer variables, you could search for Binary, Integer or NonNegativeInteger in the modules you are using (e.g. open switch_model in VS Code and use the multi-file search). 

Matthias

Matthias​

---

From: switch...@googlegroups.com <switch...@googlegroups.com> on behalf of Matthias Fripp <mfr...@edf.org>
Sent: Saturday, September 21, 2024 9:11:55 AM
To: switch...@googlegroups.com <switch...@googlegroups.com>
Subject: Re: [EXTERNAL] [switch-model] Debug for very slow run

 

To view this discussion on the web visit https://groups.google.com/d/msgid/switch-model/PH0PR15MB4152FA55244911661E950705D96D2%40PH0PR15MB4152.namprd15.prod.outlook.com.