Linopy - boosting linear programming in python
Fabian Hofmann
Dear openmodders,
it's a pleasure to announce a new package to boost Linear
Programming in Python. It is called Linopy (open at
https://github.com/PyPSA/linopy) and is much faster than Pyomo
while keeping the memory usage significantly lower. Syntax and
performance are more comparable with the famous JuMP package in
Julia.
Background
In Python, the common tool for creating and solving optimization
problems is Pyomo, which works well for small problems. But when
it comes to larger problems, Pyomo takes a lot of time and memory
resources. This is mainly due to Pyomo's slow writing of LP files
and inefficient handling of references. Also, initializing models
and constraints is sometimes quite unintuitive. Therefore, we saw
some room for improvement.
Linopy is a fairly new Python package designed for fast and
memory-efficient problem solving. It is inspired by the PyPSA code
that we wrote to boost the optimization process
(https://github.com/PyPSA/PyPSA/blob/master/pypsa/linopf.py), but
it is a stand-alone package and works for abitrary Linear
Problems. Linopy uses xarray for flexible multi-dimensional
labeling and handling of variables and constraints. For large
problems, i.e. problems with more than a million variables, you
can expect:
* a speedup of times 4-7
* a memory reduction of roughly 50%
compared to Pyomo. The advantage grows if you use non-integer
labels for the variables, e.g. generator names or timestamps.
Also, problem formulation is easier and more intuitive. Linopy
works well with Numpy arrays, Pandas Series/DataFrames, and
especially with xarray.DataArrays/xarray.Datasets.
Compared to JuMP, the much-touted optimization framework in Julia,
Linopy definitely holds its own. It outperforms JuMP in memory
consumption, but is slower in translating to the solver. However,
Linopy does not need to be compiled firstly in order to run fast,
which makes life easier for developers.
Attached is a figure showing the benchmark for a simple
optimization problem (created with this workflow
https://github.com/PyPSA/linopy/tree/master/benchmark). Another
figure is attached. It profiles the memory usages for an
optimization of 37-node PyPSA network using Linopy and using
Pyomo. Compared to Pyomo, Linopy uses 60% of the memory and 20% of
the time.
The package lives at https://github.com/PyPSA/linopy where you
find further information. You can install it via
pip install linopy
The somehow improvable documentation can be found at
https://linopy.readthedocs.io/en/latest/index.html.
Things to Look Forward in Near Future
* In the next version Linopy will support to solve models on a
remote server via an SSH channel. The implementation will
automatically store the locally initialized model to a netcdf file
on the server, run the optizimation there and retrieve the
optimization results.
* An improved documentation.
* PyPSA will support the use Linopy in one of the new major
versions, in order to facilitate the assignment of custom
constraints.
Things Remaining on the ToDo List
* Support of piecewise linear constraints
* Support of quadratic terms
* Problem scaling methods
I am happy about any feedback and food for thought. I hope that
the package can provide a good service to the energy modelling
community.
Best,
Fabian Hofmann
-- Fabian Hofmann Postdoctoral Researcher Institute of Energy Technology Technische Universität Berlin