umfpack error

[Juan Nunez-Iglesias]

Hi guys,

How do I build scipy with umfpack to silence this error:

Exception AttributeError: "'UmfpackContext' object has no attribute '_symbolic'" in <bound method UmfpackContext.__del__ of <scipy.sparse.linalg.dsolve.umfpack.umfpack.UmfpackContext object at 0x101b62050>> ignored

and speed up random_walker? I've just been doing a pip install scipy till now.

Juan.

[Josh Warner]

SciPy has a number of optional dependencies, which `pip` doesn't attempt to install. Largely because several of them are not directly Python based, and `pip` isn't a true package manager. Without them SciPy builds and runs, but certain pieces either run slower (in this case, parts of `scipy.sparse`) or don't exist at all.

Near the top of the very lengthy build and compile log for SciPy there is a brief report on which optional dependencies are present at time of build.

Presumably you don't have umfpack. It's also possible you do, but lack the development headers - but for the purposes of this argument, let's assume you don't. 

1. The first step is to get umfpack. Doing this varies across platforms, but if you're on linux check repos first. If not, look online for the best method; I've always gotten it this way.
   
2. Next step is to get the development headers for umfpack. Usually in Linux package managers (yum, apt, etc.) these have the same name with `'-dev'` appended. They're what SciPy needs to find and use umfpack.
   
3. Finally, you'll need to remove and rebuild SciPy with `$ pip uninstall scipy` followed by `$pip install scipy`. The new build should auto-detect umfpack and build appropriately.

Hopefully that helps. God help you if you're on Windows; I sure can't.

Josh

[Josh Warner]

Final note - umfpack enables mode='cg' in random_walker, which is better than brute force but not the fastest or most efficient mode. The multi-grid preconditioned conjugate gradient method is: mode='cg_mg'. This is enabled by pyamg, which is independent of umfpack and is considerably easier to install and use than the method I just described.

Come to think of it, we probably should suppress that warning if import pyamg works. It incorrectly implies the user doesn't have the fastest mode available.

On Thursday, October 3, 2013 1:21:43 AM UTC-5, Juan Nunez-Iglesias wrote:

[Juan Nunez-Iglesias]

Thanks Josh for the detailed response and discussion! I think you're absolutely right that it should be suppressed if pyamg is available. And it's probably worth getting that in before the release. Happy to tackle this.

Juan.

PS: I'm mostly on OSX, with some Linux on the side. =)

--

You received this message because you are subscribed to the Google Groups "scikit-image" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scikit-image...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Juan Nunez-Iglesias]

Just for future reference, and because it wasn't immediately obvious from googling: the umfpack headers in Ubuntu 13.04 are in the package libsuitesparse-dev.

But now the scipy build fails with some weird error. Hmmm.

building extension "scipy.sparse.linalg.dsolve.umfpack.__umfpack" sources

creating build/src.linux-x86_64-2.7/scipy/sparse/linalg/dsolve

creating build/src.linux-x86_64-2.7/scipy/sparse/linalg/dsolve/umfpack

  adding 'scipy/sparse/linalg/dsolve/umfpack/umfpack.i' to sources.

swig: scipy/sparse/linalg/dsolve/umfpack/umfpack.i

swig -python -I/usr/include/suitesparse -I/usr/include/suitesparse -I/usr/include/suitesparse -o build/src.linux-x86_64-2.7/scipy/sparse/linalg/dsolve/umfpack/_umfpack_wrap.c -outdir build/src.linux-x86_64-2.7/scipy/sparse/linalg/dsolve/umfpack scipy/sparse/linalg/dsolve/umfpack/umfpack.i

unable to execute swig: No such file or directory

error: command 'swig' failed with exit status 1

[Johannes Schönberger]

Are you sure you have SWIG installed?

[Juan Nunez-Iglesias]

On Fri, Oct 4, 2013 at 8:59 PM, Johannes Schönberger <js...@demuc.de> wrote:

Are you sure you have SWIG installed?

I wasn't and I didn't. =)

Bloody hell scipy takes a long time to build. =)

Juan.

[Ralf Gommers]

$ time python setup.py build_ext -i
real    12m6.843s
user    6m3.244s
sys     0m13.124s

$ time ../Bento/bentomaker build -i -j   # -j == parallel build (on 4 cores here)
real    1m46.466s
user    2m54.628s
sys     0m10.900s

Blame distutils (and start using Bento) :)

Ralf

[Juan Nunez-Iglesias]

Thanks! Although in this case I was using pip install *and* it was on an EC2 micro instance so I only have 1 CPU. =)

But it is noted and very useful going forward. =)

[Juan Nunez-Iglesias]

Ok so I'm having trouble actually suppressing this thing. The problem is that to detect whether umfpack is present we try to build the UmfpackContect() object:

try:
    from scipy.sparse.linalg.dsolve import umfpack
    UmfpackContext = umfpack.UmfpackContext()
except:
    UmfpackContext = None

The constructor raises an ImportError, which we duly catch (although that line should really specify the error type, but whatever). Unfortunately, the UmfpackContext class has a destructor (__del__) which then tries to delete some objects that were never created and raises an AttributeError — which, since we are already in a try: except: clause, gets ignored, hence the message to stderr.

Basically this is a bug in scipy I think?

The standard approach is to temporarily redirect stderr to /dev/null, and this works interactively:

In [1]: from scipy.sparse.linalg.dsolve import umfpack

In [2]: try:
    UC = umfpack.UmfpackContext()
except ImportError:
    print 'hello'
   ...:
hello
Exception AttributeError: "'UmfpackContext' object has no attribute '_symbolic'" in <bound method UmfpackContext.__del__ of <scipy.sparse.linalg.dsolve.umfpack.umfpack.UmfpackContext object at 0x103938250>> ignored

In [3]: import sys, os

In [4]: null = open(os.devnull, 'w')

In [5]: null.write('hello!')

In [6]: sys.stdout.write('hello!')
hello!
In [7]: sys.stderr.write('hello')
hello
In [8]: sys.stderr = null

In [9]: sys.stderr.write('hello')

In [10]: try:
    UC = umfpack.UmfpackContext()
except ImportError:
    print 'hello'
   ....:
hello

In [11]: sys.stderr = sys.__stderr__

In [12]: sys.stderr.write('hello')
hello

However, doing this in the code doesn't work, even when I try to flush the redirected stderr! (I'm not even sure that makes sense!)

sys.stderr = open(os.devnull, 'w')
try:
    from scipy.sparse.linalg.dsolve import umfpack
    UmfpackContext = umfpack.UmfpackContext()
except:
    UmfpackContext = None
finally:
    sys.stderr.flush()
    sys.stderr = sys.__stderr__

Thoughts/ideas?

[Josh Warner]

This would probably involve an API change, but instead of importing umfpack at all we could try `import pyamg` first. If pyamg is available, we wouldn't even attempt to import umfpack, instead using `'cg_mg'` instead of `'cg'`. This sidesteps stderr issues.

The approach only works because the results are equivalent, differing only in speed/memory. I don't know why we default to the worst brute force option, either. Seems like we could default to `None` and attempt a fallback scheme of pyamg -> umfpack -> brute force in order, with a different warning structure.

[Juan Nunez-Iglesias]

I'm in favor of the API change for 0.9, but otherwise we can always go the deprecation route, and:

- not change the order

- add deprecation warning iff user calls random_walker without explicitly specifying mode

- add the change in priority to the to-do list.

In the meantime, I did manage to suppress the error message that comes up on import, at least.

https://github.com/scikit-image/scikit-image/pull/759