[SciPy-User] Correctly compiling SciPy with UMFPACK support?
Ioan-Alexandru Lazar
I know that SciPy vanilla support for UMFPACK is going to be deprecated
due to license issues, so I hope I am not incorrectly directing my
message.
I have spent the last two days trying to build SciPy with UMFPACK.
Normally, I would simply use the packages available from my distribution;
however, I want SciPy to use a version of UMFPACK I am compiling myself
using AMD's ACML (because I need multithreaded BLAS support).
Unfortunately, there's something that I don't seem to be getting straight
because regardless of how I do it, I end up with an error message.
If I am using scikits.umfpack or scipy version 0.7.0, the error message I
eventually receive is:
Traceback (most recent call last):
File "/home/alexlz/Projects/2010/llssolve/scipytest.py", line 16, in
<module>
umf = scikits.umfpack.UmfpackContext("zi")
File
"/home/alexlz/.local/lib/python2.6/site-packages/scikits.umfpack-5.1.0-py2.6-linux-x86_64.egg/scikits/umfpack/umfpack.py",
line 278, in __init__
self.control = nm.zeros( (UMFPACK_CONTROL, ), dtype = nm.double )
NameError: global name 'UMFPACK_CONTROL' is not defined
If I am using scipy.sparse.linalg.dsolve.umfpack in scipy version 0.8.0,
it complains that I have not compiled it with UMFPACK support.
However, I am quite certain that I have done so -- __umfpack.so appears in
the module directory and in fact, if I replace it with the __umfpack.so
file that my distribution installed in its systemwide installation
directory, it works; however, that one is not linked against ACML's BLAS.
I am further baffled by this because building the UMFPACK module actually
works -- I have made sure it doesn't fail, as I've read around the mailing
list and noted that building it fails silently if it cannot be correctly
compiled, being an optional extension of SciPy.
I am compiling UMFPACK using the following options (snippets from
UFconfig.mk):
CFLAGS = -fPIC -m64 -O3 -fexceptions
-L/home/alexlz/libraries/acml/gfortran64_mp/lib
-I//home/alexlz/libraries/acml/gfortran64_mp/include
-L/home/alexlz/src/CBLAS/lib/LINUX -I/home/alexlz/src/CBLAS/src
# Fortran compiler (not normally required)
F77 = gfortran
F77FLAGS = -O
F77LIB =
# C and Fortran libraries
LIB = -lm -lgomp -lgfortran
BLAS = -lcblas -lacml_mp -lgfortran
(Note: cblas is required because ACML doesn't follow the Netlib calling
convention. Also, note that I'm not trying to use it to build the entire
SciPy package -- I'm only interested in having multithreaded BLAS with
UMFPACK)
And the relevant entries in my site.cfg are:
[amd]
library_dirs = /home/alexlz/libraries
include_dirs = /home/alexlz/includes
amd_libs = amd
#
[umfpack]
library_dirs = /home/alexlz/libraries
include_dirs = /home/alexlz/includes
umfpack_libs = umfpack
Judging from the behavior (replacing the wrapper-built __umfpack.so with
the one supplied by my systemwide distribution) I am assuming this is an
issue with the way I am building it, especially considering that the
__umfpack.so I'm building and the one in the systemwide installation have
radically different sizes (mine is about 1M). However, I haven't found
anything in the installation instruction -- I appear to be building it
correctly for all I know. It doesn't work regardless of whether I am using
the scikits.umfpack version or the one to be deprecated from scipy.
Do you have any hint as to what I am doing wrong? I am currently out of
ideas.
Thank you,
Alexandru Lazar,
LMN, Politehnica University of Bucharest
PS: If this is not an option, I would be grateful for any other suggestion
about how to use a version of UMFPACK compiled with a multithreaded BLAS.
Not necessarily the one from ACML (which is actually a bit complicated
because they're not following the netlib calling convention so I'm forced
to use CBLAS as well). If a readily-built binary is available, or there's
any other approach to this, I'd be happy to hear about it.
The background: I am trying to build my HPC application using Python, and
this is the only obstacle I am currently encountering. Unfortunately, the
very peculiar structure of the matrices I'm dealing with means that using
SuperLU is not an option, *but* a multithreaded BLAS library and UMFPACK
gives us a significant speedup on the 8-core nodes in our cluster.
_______________________________________________
SciPy-User mailing list
SciPy...@scipy.org
http://mail.scipy.org/mailman/listinfo/scipy-user
Dag Sverre Seljebotn
> Hello everyone,
>
> I know that SciPy vanilla support for UMFPACK is going to be deprecated
> due to license issues, so I hope I am not incorrectly directing my
> message.
>
> I have spent the last two days trying to build SciPy with UMFPACK.
> Normally, I would simply use the packages available from my distribution;
> however, I want SciPy to use a version of UMFPACK I am compiling myself
> using AMD's ACML (because I need multithreaded BLAS support).
>
> Unfortunately, there's something that I don't seem to be getting straight
> because regardless of how I do it, I end up with an error message.
>
> If I am using scikits.umfpack or scipy version 0.7.0, the error message I
> eventually receive is:
>
> Traceback (most recent call last):
> File "/home/alexlz/Projects/2010/llssolve/scipytest.py", line 16, in
> <module>
> umf = scikits.umfpack.UmfpackContext("zi")
> File
> "/home/alexlz/.local/lib/python2.6/site-packages/scikits.umfpack-5.1.0-py2.6-linux-x86_64.egg/scikits/umfpack/umfpack.py",
> line 278, in __init__
> self.control = nm.zeros( (UMFPACK_CONTROL, ), dtype = nm.double )
> NameError: global name 'UMFPACK_CONTROL' is not defined
>
>
> If I am using scipy.sparse.linalg.dsolve.umfpack in scipy version 0.8.0,
> it complains that I have not compiled it with UMFPACK support.
>
I just wanted to make you aware that the UMFPACK wrapper has a new home
over at scikits (scikits.appspot.com). Perhaps you'll have better luch
with that one...
Haven't tried myself though.
Dag Sverre
Ioan-Alexandru Lazar
As I mentioned in my initial message, I get the same error with the
UMFPACK wrapper in scikits. With my message being fairly long, this was
probably hidden somewhere non-obvious so I'll reiterate this here -- I get
exactly the same error message with the scikits UMFPACK wrapper.
I suspect it's something about how I build UMFPACK because if I replace
the __umfpack.so that gets built from source with the one supplied by
Fedora's scipy package, it works (although that's not linked with the BLAS
libraries I need).
Perhaps someone who has managed to compile the scikits wrapper from source
with a custom-built UMFPACK can shed some light on this? In the meantime,
I think I'll try to look for Fedora's Scipy and suitesparse or UMFPACK
SRPMs to see if I can find something interesting in there.
Thanks for your time!
Sincerely yours,
Alexandru Lazar
Jonathan Stickel
> Date: Wed, 11 Aug 2010 12:00:22 -0300
> From: Renato Fabbri<renato...@gmail.com>
> Subject: [SciPy-User] how to fit a given pdf
> To: Discussion of Numerical Python<numpy-di...@scipy.org>,
> scipy...@scipy.org
> Message-ID:
> <AANLkTinmmEbSo9_-ZmCXD...@mail.gmail.com>
> Content-Type: text/plain; charset="iso-8859-1"
>
> Dear All,
>
> help appreciated, thanks in advance.
>
> how do you fit a pdf you have with a given pdf (say gamma).
>
> with the file attached, you can go like:
>
> a=open("AC-010_ED-1m37F100P0.txt","rb")
> aa=a.read()
> aaa=aa[1:-1].split(",")
> data=[int(i) for i in aaa]
>
> if you do pylab.plot(data); pylab.show()
>
> The data is something like:
> ___|\___
>
> It is my pdf (probability density function).
>
> how can i find the right parameters to make that fit with a gamma?
>
> if i was looking for a normal pdf, for example, i would just find mean
> and std and ask for the pdf.
>
> i've been playing with scipy.stats.distributions.gamma but i have not
> reached anything.
>
> we can extend the discussion further, but this is a good starting point.
>
> any idea?
>
I am not familiar with the scipy.stats module, and so I do not know what
it can do for you. However, I would just generate a model gamma
distribution from the mean and variance, just as for a normal
distribution. The gamma distribution equation can be written as
p(x) = p0/(b^a*Gamma(a))*x^(a-1)*exp(-x/b)
assuming x starts at zero (x>=0)
(http://en.wikipedia.org/wiki/Gamma_distribution). Then the the
parameters a and b are related to the mean and variance by
a = mean^2/var
b = var/mean
(I did the algebra quickly just now, and so you might want to
double-check). p0 is the area under the distribution and may be simply
1 if your distribution is normalized.
Hope this helps.
Jonathan
josef...@gmail.com
> On 8/11/10 09:00 , scipy-use...@scipy.org wrote:
>> Date: Wed, 11 Aug 2010 12:00:22 -0300
>> From: Renato Fabbri<renato...@gmail.com>
>> Subject: [SciPy-User] how to fit a given pdf
>> To: Discussion of Numerical Python<numpy-di...@scipy.org>,
>> scipy...@scipy.org
>> Message-ID:
>> <AANLkTinmmEbSo9_-ZmCXD...@mail.gmail.com>
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>> Dear All,
>>
>> help appreciated, thanks in advance.
>>
>> how do you fit a pdf you have with a given pdf (say gamma).
>>
>> with the file attached, you can go like:
>>
>> a=open("AC-010_ED-1m37F100P0.txt","rb")
>> aa=a.read()
>> aaa=aa[1:-1].split(",")
>> data=[int(i) for i in aaa]
>>
>> if you do pylab.plot(data); pylab.show()
The first step is to try the fit method. If generic Maximum Likelihood
estimation works with your data, then it's good.
An example is at http://projects.scipy.org/scipy/ticket/832
If you want to fix the support, the lower bound then you could use the
link on the ticket, current scipy trunk or statsmodels, to estimate.
In some uncommitted changes to statsmodels, I'm preparing the standard
errors for the MLE estimates using the Hessian of the log-likelihood,
which I think will work in this case.
Since your data are all integers, is there a discrete analog to the
gamma distribution? I never checked this.
I hope that helps, keep asking if there are additional issues. I just
spend some time fighting with estimation for pareto and genpareto and
I would like to get this to work "out-of-the-box".
Josef
Benjamin Root
Note, those are the point estimators for gamma distribution, but it does not use maximum likeihood estimation (MLE). Using MLE, finding the estimator for alpha is tricky, requiring convergence. Probably best to use the fit module as Josef mentioned. However, there is a decent estimator for the estimator (I heard you liked estimation...)
a = 0.5 / (ln(mean(X)) - mean(ln(X)))
Note that if you have a zero value in your dataset this won't work. Also, the reference is not 100% clear if it is a base 10 log or not. I am pretty certain it is a natural log.
Ben Root
josef...@gmail.com
Travis also added a different estimator in
http://projects.scipy.org/scipy/browser/trunk/scipy/stats/distributions.py#L2867
I never looked specifically at gamma, only as example for generic MLE,
and have no idea what this estimator is.
Ben, do you have the reference or know what type of estimators your's is?
(I like references, the R package POT has 17 estimators for genpareto,
and I have 20 pdf files for pareto/genpareto/genextreme open)
Josef
>
> Ben Root
Benjamin Root
Josef,
The reference for that particular estimator is http://research.microsoft.com/en-us/um/people/minka/papers/minka-gamma.pdf
There is another estimator that I have used that came from a very good 'statistics for meteorologists' book, but I don't have that on me. I can get it to you tomorrow, though.
Ben Root
josef...@gmail.com
thanks, that also describes the current scipy trunk estimator.
Josef
>
> There is another estimator that I have used that came from a very good
> 'statistics for meteorologists' book, but I don't have that on me. I can
> get it to you tomorrow, though.
>
> Ben Root
>
Ioan-Alexandru Lazar
A while ago I posted a problem I had -- namely both the scipy and the
scikits UMFPACK interface failing to work when I was trying to compile
them by hand with UMFPACK also being compiled by hand. A couple of
friendly folks over here gave me a hand but even with their advice, I
wasn't able to do much.
I postponed it for a while (it wasn't really urgent -- now it is, but it's
also solved), so I though I'd post here just in case someone bumps against
the same issue.
There are two problems, one that is mentioned in most instructions and one
that seems not to be, namely:
1. When building, SciPy (and the Scikits umfpack kit) both assume that
UFconfig.h is also in the include directory -- so umfpack and amd's won't
suffice.
2. The umfpack interface module needs the shared umfpack and amd
libraries. However, SuiteSparse does *not* compile them as such; as many C
programs actually expect to end up being statically linked with
SuiteSparse's library, it's not such a good idea to tweak UFconfig.mk so
as to build them all as shared libraries. It's best to manually compile
AMD and UMFPACK as shared libraries (with gcc -shared -Wl and remembering
to link the math library and whatever blas you're using with umfpack) and
place them in the libraries folder defined in site.cfg, and add that path
to LD_LIBRARY_PATH.
I hope this helps. I hope I haven't made an ass of myself by not reading
the docs correctly -- in any case, I haven't seen it mentioned at
http://http://www.scipy.org/Installing_SciPy/Linux
--- Slightly less interesting section following ---
By the way -- I think I could have figured this out slightly faster; when
I looked at it again, I only needed a morning of quick hacking. The
culprit was actually here:
try: # Silence import error.
import _umfpack as _um
except:
_um = None
(
http://svn.scipy.org/svn/scikits/trunk/umfpack/scikits/umfpack/umfpack.py
, but this is how it currently looks in SciPy as well)
Unfortunately, the error message that appears when _um = None is simply
that SciPy was compiled with umfpack support in scipy, or that the symbol
UMFPACK_CONTROL is undefined. However, this can be misleading -- in my
particular case, the import error initially pointed to a symbol in cholmod
(!) not being defined, and after sume further digging, to umfpack.so
simply not being found.
The deadline for the paper I'm working on is in about three weeks and I'm
kind of entagled right now (you know how academics is...) but as soon as
I'm done with it, I'll try to provide a fix for this (a more particular
error message) and update the docs if anyone else thinks this is useful.
Since direct umfpack support is going to be deprecated in SciPy, I assume
I should do this in the scikits version, right?
Thank you all for your help and your time.
Alexandru Lazar
Numerical Modeling Laboratory, Politehnica University of Bucharest