semidefinite programming using csdp -- python import errors
Robert Samal
Hi,
I'm trying to use library Csdp in Sage by means of Python numpy wrapper for CSDP called pycsdp (by B. Kern).
I was assuming that it'll be easy, as one can generally run python scripts from sage.
So I installed Csdp library (built against ATLAS library for linear algebra), and installed the above said wrapper -- which is done in a standard python way by
python setup.py install
-- and it works. But only as a "stand-alone python script".
So, next to get it running in sage: I installed it for the sage-python by
sage -python setup.py install.
Now when I run a python script that works in standalone python,
I get an import error -- undefined symbol.
Traceback (most recent call last):
File "example2.py", line 4, in <module>
from pycsdp import _csdp
ImportError: /data/local/sage-4.8-linux-32bit-ubuntu_10.04_lts-i686-Linux/local/lib/python2.6/site-packages/pycsdp/_csdp.so: undefined symbol: dnrm2_
I also found that the undefined symbol dnrm2_ is defined by the Csdp library, that I'd like to use. However, the setup.py script has the address of the library in it and it puts it where needed when installed by the "normal" python.
What should I do differently, to get this to work in sage? Could the ATLAS library I used to build Csdp be interfering with some standard sage libraries? (lapack etc.)
Is there some different way to call python programs from sage, that wouldn't need to go through this?
Any ideas are appreciated, thanks for reading this,
Robert Samal
P.S. I'm using
Sage Version 4.8, Release Date: 2012-01-20
with Debian/Wheezy
Jason Grout
>
> Hi,
>
> Sage by means of Python numpy wrapper for CSDP called pycsdp (by B.
> Kern)
Can you post the logs from doing sage -python setup.py instasll, as well
as explicit instructions (where to download, etc.) for how to duplicate
the error?
Thanks,
Jason
Robert Samal
Can you post the logs from doing sage -python setup.py instasll, as well
as explicit instructions (where to download, etc.) for how to duplicate
the error?
Anyway, here you go:
1) download ATLAS library --- I did use debian package
apt-get install libatlas-base-dev --install-suggests
but one can also download from the web-site http://math-atlas.sourceforge.net/
--> Now I discovered that sage has its own atlas libraries. Should I be trying
to build against them? Do I need to build sage from source to do this?
2) download http://www.coin-or.org/download/source/Csdp/Csdp-6.1.1.tgz
unzip and install using the instructions. This is straightforward process
make
make install
3) download http://ifatwww.et.uni-magdeburg.de/syst/about_us/people/kern/downloads/pycsdp-0.10.zip
unzip, edit the setup.py to point to the Csdp library and includes, then run
python setup.py install as a root to install "to the system" .
To install to sage I tried
sage -python setup.py install
but I believe now that already here is a problem (or the problem?):
the installation using python finds the ATLAS library
FOUND:
libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
language = f77
include_dirs = ['/usr/include/atlas']
while sage-python doesn't find it. Instead, it complains about missing directories
(full logs attached):
/data/local/sage-4.8-linux-32bit-ubuntu_10.04_lts-i686-Linux/local/lib/python2.6/site-packages/numpy/distutils/system_info.
UserWarning: Specified path /home/wstein/build/sage-4.8/local/lib is invalid.
warnings.warn('Specified path %s is invalid.' % d)
Is this a bug? What should I do to use my libraries instead of wstein's? :-)
Do I need to compile my sage, so that I can add stuff to it?
Thanks!
Robert
Dima Pasechnik
> On 3/22/12 8:58 PM, Robert Samal wrote:
>>
>> Hi,
>>
>> I'm trying to use library Csdp <https://projects.coin-or.org/Csdp/> in
>> Sage by means of Python numpy wrapper for CSDP called pycsdp (by B.
>> Kern)
>> <http://ifatwww.et.uni-magdeburg.de/syst/about_us/people/kern/index.shtml>.
>>
>> I was assuming that it'll be easy, as one can generally run python
>> scripts from sage.
By the way, if you just want a quick way to call a (slow) SDP solver in Sage,
you can use CVXOPT, which is a standard package in Sage.
(You need to disable Sage's preparser though)
Dima Pasechnik
> ------=_Part_772_24642055.1332490982625
> Content-Type: multipart/alternative;
> boundary="----=_Part_773_30231439.1332490982625"
>
> ------=_Part_773_30231439.1332490982625
> Content-Type: text/plain; charset=ISO-8859-1
>
>
>>
>> Can you post the logs from doing sage -python setup.py instasll, as well
>> as explicit instructions (where to download, etc.) for how to duplicate
>> the error?
>>
> Sure. It's rather long, but I suppose if it works it would be useful for
> other people, too. I was hoping someone could spot the problem just from
> the above description.
>
> Anyway, here you go:
>
> 1) download ATLAS library --- I did use debian package
> apt-get install libatlas-base-dev --install-suggests
> but one can also download from the web-site
> http://math-atlas.sourceforge.net/
>
> --> Now I discovered that sage has its own atlas libraries. Should I be
> trying
> to build against them?
yes, absolutely.
> Do I need to build sage from source to do this?
no, not really (but YMMV, as always :))
> 2) download http://www.coin-or.org/download/source/Csdp/Csdp-6.1.1.tgz
> unzip and install using the instructions. This is straightforward process
> make
> make install
>
> 3) download
> http://ifatwww.et.uni-magdeburg.de/syst/about_us/people/kern/downloads/pycsdp-0.10.zip
> unzip, edit the setup.py to point to the Csdp library and includes, then run
> python setup.py install as a root to install "to the system" .
>
> To install to sage I tried
> sage -python setup.py install
> but I believe now that already here is a problem (or the problem?):
> the installation using python finds the ATLAS library
> FOUND:
> libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
> library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
> language = f77
> include_dirs = ['/usr/include/atlas']
indeed, it's a problem. More than one, actually. To fix paths, you can
do the following: somewhere at the top
of setup.py, insert lines
import os
SAGE_LIB = os.environ['SAGE_LOCAL']+'/lib'
SAGE_INCLUDE = os.environ['SAGE_LOCAL']+'/include'
then replace
library_dirs = ['/usr/lib/atlas-base/atlas', '/usr/lib/atlas-base']
with
library_dirs = [SAGE_LIB+'/atlas-base/atlas', SAGE_LIB+'/atlas-base']
and
include_dirs = ['/usr/include/atlas']
with
include_dirs = [SAGE_INCLUDE+'/atlas']
But Sage does not have 'ptf77blas', 'ptcblas', it has 'blas',
'cblas' (i.e. non-multithreaded versions of them).
Perhaps replacing
libraries = ['lapack', 'ptf77blas', 'ptcblas', 'atlas']
with
libraries = ['lapack', 'blas', 'cblas', 'atlas']
would still work. You should also have just
library_dirs = [SAGE_LIB]
(not what I posted above).
An alternative would be to install these 'pt'*blas in Sage, doing
this in Sage shell (i.e. sage -sh), but I have no idea whether this will
work out of the box, if at all (multithreading is tricky...).
Best,
Dmitrii
Dima Pasechnik
> On 2012-03-23, Robert Samal <robert...@gmail.com> wrote:
>> ------=_Part_772_24642055.1332490982625
>> Content-Type: multipart/alternative;
>> boundary="----=_Part_773_30231439.1332490982625"
>>
>> ------=_Part_773_30231439.1332490982625
>> Content-Type: text/plain; charset=ISO-8859-1
>>
>>
>>>
>>> Can you post the logs from doing sage -python setup.py instasll, as well
>>> as explicit instructions (where to download, etc.) for how to duplicate
>>> the error?
I have had success with installing and running this extenstion on MacOSX
10.6.8, under Sage 4.8. (That is, I didn't even use Atlas - Sage uses
native MacOSX lapack/blas)
Instructions:
fire up Sage shell: sage -sh
1) Compile Csdp static lib in lib/ subdirectory of Csdp distibution and
copy it (i.e. libcsdp.a) to $SAGE_LOCAL/lib/
2) mkdir $SAGE_LOCAL/incluce/sdp/
3) Copy Csdp .h files in include to $SAGE_LOCAL/incluce/sdp/
4) in pycsdp, edit setup.py as follows:
add at the line 3:
import os
SAGE_LIB = os.environ['SAGE_LOCAL']+'/lib'
SAGE_INCLUDE = os.environ['SAGE_LOCAL']+'/include'
replace the lines
library_dir = ['/usr/local/numerics/lib']
includes = ['/usr/local/numerics/include/',
with
library_dir = [SAGE_LIB]
includes = [SAGE_INCLUDE+'/sdp',
5) run python setup.py install
6) Done! Now you can do python examples/example.py and see it running
(or you can start Sage proper and try doing
from pycsdp import _csdp, etc)
I imagine this should work on Linux too. No parallel/multithreaded blas,
though; one has to work on adding parallel Atlas libs to Sage in order
to fix this.
--------------------
We should definitely think of making this a Sage package, perhaps even a
standard one, as Csdp is really fast!
Dmitrii
Dima Pasechnik
>
> I have had success with installing and running this extenstion on MacOSX
> 10.6.8, under Sage 4.8. (That is, I didn't even use Atlas - Sage uses
> native MacOSX lapack/blas)
>
>
> Instructions:
> fire up Sage shell: sage -sh
> 1) Compile Csdp static lib in lib/ subdirectory of Csdp distibution and
> copy it (i.e. libcsdp.a) to $SAGE_LOCAL/lib/
> 2) mkdir $SAGE_LOCAL/incluce/sdp/
> 3) Copy Csdp .h files in include to $SAGE_LOCAL/incluce/sdp/
> 4) in pycsdp, edit setup.py as follows:
> add at the line 3:
> import os
> SAGE_LIB = os.environ['SAGE_LOCAL']+'/lib'
> SAGE_INCLUDE = os.environ['SAGE_LOCAL']+'/include'
>
> replace the lines
> library_dir = ['/usr/local/numerics/lib']
> includes = ['/usr/local/numerics/include/',
> with
> library_dir = [SAGE_LIB]
> includes = [SAGE_INCLUDE+'/sdp',
> 5) run python setup.py install
> 6) Done! Now you can do python examples/example.py and see it running
> (or you can start Sage proper and try doing
> from pycsdp import _csdp, etc)
>
>
> I imagine this should work on Linux too.
I just checked, and it does. (With a minor tweak, see above).
> No parallel/multithreaded blas,
> though; one has to work on adding parallel Atlas libs to Sage in order
> to fix this.
Actually, I am told on sage-devel (see a thread on multithreaded Atlas)
that Sage's Atlas blas/lapack are multi-threaded,
assuming one can build such libbraries on the system.
Best,
Dmitrii
Dima Pasechnik
> On 2012-03-23, Dima Pasechnik <dim...@gmail.com> wrote:
> [...]
>>
>> I have had success with installing and running this extenstion on MacOSX
>> 10.6.8, under Sage 4.8. (That is, I didn't even use Atlas - Sage uses
>> native MacOSX lapack/blas)
>>
>>
>> Instructions:
>> fire up Sage shell: sage -sh
>> 1) Compile Csdp static lib in lib/ subdirectory of Csdp distibution and
>> copy it (i.e. libcsdp.a) to $SAGE_LOCAL/lib/
> (on Linux, you will need to add -fPIC to CFLAGS in Makefile)
As far as I know, Sage is compatible with openmp.
Emil
stand-alone program, rather than using the Python interface (which
isn't a very good solution).
I was wondering whether using pycsdp is the right way to go though.
Wouldn't it be best to write a small Cython wrapper for the C library?
(I was planning at some point to do this myself.)
Emil
Dima Pasechnik
> I've also been using CSDP from Sage lately. I am calling the
> stand-alone program, rather than using the Python interface (which
> isn't a very good solution).
>
> I was wondering whether using pycsdp is the right way to go though.
> Wouldn't it be best to write a small Cython wrapper for the C library?
> (I was planning at some point to do this myself.)
well, if pycsdp works, then why you might want another interface?
For efficiency?
You still would have basically the same overhead of
converting Python matrices into C arrays needed by Csdp,
I think...
Indeed, if you'd start from scratch then Cython is easier to
use, but otherwise?
Dmitrii
>
> Emil
>
Emil
> On 2012-03-25, Emil <emi...@gmail.com> wrote:
>> I've also been using CSDP from Sage lately. I am calling the
>> stand-alone program, rather than using the Python interface (which
>> isn't a very good solution).
> why? Is it buggy?
Well, you can't stop/pause/save intermediate solutions. Changing
options means writing to a parameters file. Precision could be lost in
the double->string->double translations (?).
>> I was wondering whether using pycsdp is the right way to go though.
>> Wouldn't it be best to write a small Cython wrapper for the C library?
>> (I was planning at some point to do this myself.)
>
> well, if pycsdp works, then why you might want another interface?
I don't know, i was only asking a question! (I guess I looked at the
code and thought that it would be lot simpler in Cython, but as you
say it is already written in the way it is...)
Emil
Dima Pasechnik
> On 26 March 2012 17:37, Dima Pasechnik <dim...@gmail.com> wrote:
>> On 2012-03-25, Emil <emi...@gmail.com> wrote:
>>> I've also been using CSDP from Sage lately. I am calling the
>>> stand-alone program, rather than using the Python interface (which
>>> isn't a very good solution).
>> why? Is it buggy?
>
> Well, you can't stop/pause/save intermediate solutions.
A user interrupt would need to propagate all the way to Csdp and be
gracefully processed by it. Doing this right in a multithreaded
application is not easy.
> Changing
> options means writing to a parameters file.
parameters *file* ?
Well, this does not look nice, but perhaps it's again
a limitation of Csdp? Which would be easy to fix, of course.
> Precision could be lost in
> the double->string->double translations (?).
why would you think there is such a translation?
IMHO Python has bindings (PyFloat_FromDouble, etc) allowing one to go from its
floats to C floats/doubles
and back without resorting to this.
I imagine Cython uses these bindings too.
>>> I was wondering whether using pycsdp is the right way to go though.
>>> Wouldn't it be best to write a small Cython wrapper for the C library?
>>> (I was planning at some point to do this myself.)
>>
>> well, if pycsdp works, then why you might want another interface?
>
> I don't know, i was only asking a question! (I guess I looked at the
> code and thought that it would be lot simpler in Cython, but as you
> say it is already written in the way it is...)
I was just answering a qiestion :)
Dmitrii
>
> Emil
>
Emil
>> the double->string->double translations (?).
> why would you think there is such a translation?
I'm referring to using the stand-alone CSDP. You have to write your
matrices, and read the solution matrix, as a text file. pycsdp avoids
this, as would a Cython interface. Emil
Robert Samal
It works for me now. (Actually for quite some time, but now I can happily report, that
compared to cvxopt that I was using before, using csdp is about 10-times faster!)
I didn't know about sage -sh , that is a very useful trick (perhaps it should
be documented more, or may be it is and I just didn't read documentation recently).
I was trying to manually modify the makefile to get csdp to compile using
sage libraries, but I wasn't up to the task ...
> and on all platorms, add -fopenmp to CFLAGS.
> As far as I know, Sage is compatible with openmp.
-fopenmp doesn't work for me (everything compiles, but the code doesn't run), but as the
code seems to be using multithreading anyway, I don't mind.
Thanks again,
Robert
Dima Pasechnik
On Saturday, 24 March 2012 04:31:05 UTC+8, Dima Pasechnik wrote:
On 2012-03-23, Dima Pasechnik <> wrote: