Error when CVXOPT is deployed with Python buildpack on Cloud Foundry

[ClubberVonTT]

I am wondering why cvxopt does not work in Linux environment (deployed on IBM Bluemix)? I am using the latest python buildpack (https://github.com/cloudfoundry/python-buildpack/releases) and Python 3.6.0. Is there some special buildpack that works for CVXOPT?

Thank you for your help!

This is the shell output of the error (fatal error: umfpack.h: No such file or directory)

2/14/2017 2:48:20 AM	OUT	STG	running install
2/14/2017 2:48:20 AM	OUT	STG	running build
2/14/2017 2:48:20 AM	OUT	STG	creating build
2/14/2017 2:48:20 AM	OUT	STG	creating build/lib.linux-x86_64-3.5/cvxopt
2/14/2017 2:48:20 AM	OUT	STG	copying src/python/cvxprog.py -> build/lib.linux-x86_64-3.5/cvxopt
2/14/2017 2:48:20 AM	OUT	STG	copying src/python/__init__.py -> build/lib.linux-x86_64-3.5/cvxopt
2/14/2017 2:48:20 AM	OUT	STG	copying src/python/_version.py -> build/lib.linux-x86_64-3.5/cvxopt
2/14/2017 2:48:20 AM	OUT	STG	copying src/python/printing.py -> build/lib.linux-x86_64-3.5/cvxopt
2/14/2017 2:48:20 AM	OUT	STG	UPDATING build/lib.linux-x86_64-3.5/cvxopt/_version.py
2/14/2017 2:48:20 AM	OUT	STG	set build/lib.linux-x86_64-3.5/cvxopt/_version.py to '1.1.9'
2/14/2017 2:48:20 AM	OUT	STG	running build_ext
2/14/2017 2:48:20 AM	OUT	STG	building 'base' extension
2/14/2017 2:48:20 AM	OUT	STG	creating build/temp.linux-x86_64-3.5/src
2/14/2017 2:48:20 AM	OUT	STG	gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python3.5m -c src/C/dense.c -o build/temp.linux-x86_64-3.5/src/C/dense.o
2/14/2017 2:48:20 AM	OUT	STG	gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -fPIC -I/app/.heroku/python/include/python3.5m -c src/C/sparse.c -o build/temp.linux-x86_64-3.5/src/C/sparse.o
2/14/2017 2:48:20 AM	OUT	STG	src/C/sparse.c: In function ‘sparse_concat’:
2/14/2017 2:48:20 AM	OUT	STG	src/C/sparse.c:368:30: warning: variable ‘blk_ncols’ set but not used [-Wunused-but-set-variable]
2/14/2017 2:48:20 AM	OUT	STG	int_t blk_nrows = 0, blk_ncols = 0;
2/14/2017 2:48:20 AM	OUT	STG	gcc -pthread -shared build/temp.linux-x86_64-3.5/src/C/base.o build/temp.linux-x86_64-3.5/src/C/dense.o build/temp.linux-x86_64-3.5/src/C/sparse.o -L/usr/lib -L/app/.heroku/vendor/lib -lm -llapack -lblas -lpython3.5m -o build/lib.linux-x86_64-3.5/cvxopt/base.cpython-35m-x86_64-linux-gnu.so
2/14/2017 2:48:20 AM	OUT	STG	gcc -pthread -shared build/temp.linux-x86_64-3.5/src/C/blas.o -L/usr/lib -L/app/.heroku/vendor/lib -lblas -lpython3.5m -o build/lib.linux-x86_64-3.5/cvxopt/blas.cpython-35m-x86_64-linux-gnu.so
2/14/2017 2:48:20 AM	OUT	STG	gcc -pthread -shared build/temp.linux-x86_64-3.5/src/C/lapack.o -L/usr/lib -L/app/.heroku/vendor/lib -llapack -lblas -lpython3.5m -o build/lib.linux-x86_64-3.5/cvxopt/lapack.cpython-35m-x86_64-linux-gnu.so
2/14/2017 2:48:20 AM	OUT	STG	src/C/umfpack.c:23:21: fatal error: umfpack.h: No such file or directory
2/14/2017 2:48:20 AM	OUT	STG	#include "umfpack.h"
2/14/2017 2:48:20 AM	OUT	STG	^
2/14/2017 2:48:20 AM	OUT	STG
2/14/2017 2:48:20 AM	OUT	STG	----------------------------------------
2/14/2017 2:48:21 AM	ERR	STG	Command "/app/.heroku/python/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-pgi7qybb/cvxopt/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-6vk81aol-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-pgi7qybb/cvxopt/
2/14/2017 2:48:21 AM	ERR	STG	Failed to compile droplet
2/14/2017 2:48:21 AM	ERR	STG	Staging failed: Exited with status 223
2/14/2017 2:48:21 AM	OUT	STG	Destroying container

[Dima Pasechnik]

It is hard to say what exactly goes wrong, but the log seems to indicate that there was no attempt to build 'blas' and 'lapack' extensions - there was only an attempt to link them.

Note that, however,  cvxopt assumes too many hardcoded locations, so for instance we at sagemath.org modify its setup.py quite a bit, in particular to support pkgconfig:

https://github.com/sagemath/sage/blob/master/build/pkgs/cvxopt/patches/setup.py.patch

Surely cvxopt works on lots of flavours of linux (e.g. sagemath has cvxopt 1.1.8 on several different linuxes), so it's most probably weirdness of your toolchain that hits here, and you need to do something along the lines of the patch linked above.

[Martin]

We have updated the installation instructions:

http://cvxopt.org/install/index.html

The SuiteSparse source is no longer included, so if you do not already have SuiteSparse on your system, you will need to download the source and let CVXOPT know where it is:

wget http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.5.3.tar.gz
tar -xf SuiteSparse-4.5.3.tar.gz
export CVXOPT_SUITESPARSE_SRC_DIR=$(pwd)/SuiteSparse

[Dima Pasechnik]

On Tuesday, February 14, 2017 at 8:04:59 AM UTC, Martin wrote:

We have updated the installation instructions:

http://cvxopt.org/install/index.html

The SuiteSparse source is no longer included, so if you do not already have SuiteSparse on your system, you will need to download the source and let CVXOPT know where it is: 

wget http://faculty.cse.tamu.edu/davis/SuiteSparse/SuiteSparse-4.5.3.tar.gz
tar -xf SuiteSparse-4.5.3.tar.gz
export CVXOPT_SUITESPARSE_SRC_DIR=$(pwd)/SuiteSparse

(needless to say, one needs to compile it)

How about SuiteSparce 4.5.4 --- does it work, too?

Did you try using its CUDA mode?

[Martin]

SuiteSparse 4.5.4 works too—we just haven't updated the documentation yet. I haven't tried the CUDA mode yet.

If you set CVXOPT_SUITESPARSE_SRC_DIR then the relevant parts of SuiteSparse will automatically be build when you build CVXOPT. Alternatively, if you already have a dynamic/shared SuiteSparse library, you can set CVXOPT_SUITESPARSE_INC_DIR and CVXOPT_SUITESPARSE_LIB_DIR (and CVXOPT_SUITESPARSE_SRC_DIR must not be set) if the default values do not work. 

[Dima Pasechnik]

On Tuesday, February 14, 2017 at 9:25:36 AM UTC, Martin wrote:

SuiteSparse 4.5.4 works too—we just haven't updated the documentation yet. I haven't tried the CUDA mode yet.

If you set CVXOPT_SUITESPARSE_SRC_DIR then the relevant parts of SuiteSparse will automatically be build when you build CVXOPT. Alternatively, if you already have a dynamic/shared SuiteSparse library, you can set CVXOPT_SUITESPARSE_INC_DIR and CVXOPT_SUITESPARSE_LIB_DIR (and CVXOPT_SUITESPARSE_SRC_DIR must not be set) if the default values do not work.  

Do you plan to start using pkgconfig, which is a modern pythonic way to discover installed libraries?

(editing setup.py or setting environment vars by hand ought to be a method of last resort)

This is what we do in

https://github.com/sagemath/sage/blob/master/build/pkgs/cvxopt/patches/setup.py.patch

[Martin]

That's an excellent suggestion. However, it is only useful when building CVXOPT, so I don't think it should be added as a dependency/requirement in setup.py. A possible solution would be to use it if it is available when building CVXOPT.

[Dima Pasechnik]

pkgconfg can be installed by pip, and in case it does not find the package, setup.py can fall back on pre-defined values. pkg-config itself is pretty standard nowadays, on most if not all platforms one cares about. In a typical setup pkg-config will be present already.