CLAPACK source in NMatrix?

[Sameer Deshmukh]

Hello,

I was just going through the numpy source and found that it carries the CLAPACK source code with it, which frees it of any external dependencies.

Was there any specific reason why this wasn't done in NMatrix?

Regards,

Sameer Deshmukh

[John Woods]

Some CLAPACK functions are directly implemented in NMatrix, but mostly I wasn't aware that this is done in Python. It's C source code and not Fortran, right? If it's Fortran, it may be because there's no equivalent of f2py for Ruby.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sameer Deshmukh]

They've just put all the CLAPACK and CBLAS code in two files and it compiles alongwith numpy core. Its all C code, but if you set an ENV variable pointing to a FORTRAN implementation of LAPACK, numpy will bypass the compilation and use that instead.

Thanks for your inputs :)

[John Woods]

That would be an awesome summer of code project, if you're looking. You'd have to do some IP research and make sure you use CBLAS and CLAPACK versions with compatible licenses to ours (BSD 3-clause, which is similar to MIT).

[Sameer Deshmukh]

Yes I'm working on it.

[Sameer Deshmukh]

Hey,

We can have a gem called nmatrix-clapack which wraps over the CLAPACK from netlib. Alternatively, we can also use openBLAS for it. The only problem with openBLAS is that its LAPACK C interface is LAPACKE, which is a C wrapper over FORTRAN code. So if something goes wrong in the future, it would be hard to debug due to FORTRAN. It also requires a FORTRAN compiler.

The CBLAS implementation of openBLAS should be faster than that provided by CLAPACK because it tunes itself to the user's computer during compilation.

I think it would be better to include CLAPACK from netlib into the source. An option for using your own CLAPACK/CBLAS interface can be provided to users by way of ENV variables. User's can set this to an openBLAS binary-backed CBLAS file, and will just work because the interfaces of CBLAS with CLAPACK and CBLAS with openBLAS are exactly the same.

What do you guys think?

[John Woods]

Is openBLAS easier to install than ATLAS? I want to make sure we don't split efforts across multiple libraries when we really just need to support one or two — so let's pick a few things we want to support and focus on them.

Keep in mind that ATLAS is automatically tuned also.

--

[Sameer Deshmukh]

openBLAS is battle tested (runs on 7 of 10 of the world's fastest supercomputers) and it can compile with the CLAPACK (netlib) interface.

I'm thinking that including the source code of these libraries alongwith nmatrix would be the best approach. I checked the licenses and they are compliant.

[Sameer Deshmukh]

Ok I just did some fiddling around and it turns out that the openBLAS cblas.h interface works perfectly well with CLAPACK from netlib.

This is much faster than the cblas that ships with netlib CLAPACK. Upon running the test suite on my machine I found a difference of a whole second or more in a lot of test cases.

Therefore, I think the best way to proceed with the nmatrix-clapack gem would be to ship the openBLAS and CLAPACK source code with the gem, compile it as per the user's machine and build a Ruby interface for it. This will ensure speed and usability at the same time. Also the total source code does not go more than 20 MB so thats an issue either.

And John, to address your concern, ATLAS just implements the same interface that openBLAS does. Both these libraries can be thought of as backends to 'cblas.h'. Thus an interface built for ATLAS will work for openBLAS too.

[Carlos Agarie]

To install openBLAS on OSX, it should be as easy as (if you already have Homebrew):

$ brew tap homebrew/science

$ brew install openblas

And it should be good to go.

One point to note, Sameer, is that openBLAS is optimized beforehand (AFAIK), i.e. it is designed to be extremely fast in some hardware and "good enough" in others. ATLAS is tuned during installation, so it should adapt to your specific hardware by choosing good values for matrix block algorithms and things like that.

Did you turn off processor throttling when installing ATLAS? It was a bit faster on my laptop when I tested it against openblas...

I totally agree this is a great project for GSoC. Good job so far! 

-----

Carlos Agarie

Software Engineer

+55 11 97320-3878 | @carlos_agarie

[John Woods]

It is bound to be faster than netlib, which is the reference implementation (really old and just designed to get things done, not necessarily efficient).

I think if you want this to be part of your proposal, you should perhaps do a comparison in your application of all of the options you've considered. Make it clear why this one is the best option.

Cheers,

John

[Ondřej Čertík]

Hi John,

On Mon, Mar 9, 2015 at 12:31 PM, John Woods <john.o...@gmail.com> wrote:
> It is bound to be faster than netlib, which is the reference implementation
> (really old and just designed to get things done, not necessarily
> efficient).

Just a little correction that the reference Lapack implementation is
in Fortran, available from: http://www.netlib.org/lapack/, the latest
version being 3.5.0, from November 16, 2013, so they keep it updated.
The speed (if you compile in Release mode, and add the flags "-O3
-fPIC -march=native -ffast-math -funroll-loops" if you use gfortran)
is not bad at all, though OpenBlas is faster. In my experience, Lapack
is easy to compile on any architecture, while OpenBlas sometimes
requires tweaks, as an example I faced this problem:
https://github.com/xianyi/OpenBLAS/issues/179#issuecomment-39457766,
the fix was simple, but I wouldn't be able to figure it out on my own
without the help from the author.

Ultimately you should allow the user to link any blas and lapack he or
she wants (in addition to any default values that you might provide).
That's how it works in numpy.

Ondrej

[John Woods]

Ahh, fair point. Thanks for the correction.

>> email to sciruby-dev+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "SciRuby Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to sciruby-dev+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev+unsubscribe@googlegroups.com.

[Sameer Deshmukh]

This blog post (http://nicolas.limare.net/pro/notes/2014/10/31_cblas_clapack_lapacke/) suggests that CLAPACK is outdated and the most standardized way of interacting with LAPACK through a C interface is through LAPACKE (which is also compatible with MKL if need be in the future). It also says that ATLAS doesnt fully provide the LAPACK interface.

LAPACKE ships with the openBLAS source and is very easily available. Its also easy to ship the whole openBLAS code with the gem rather than putting CLAPACK and openBLAS components and then linking them later.

But my argument against this is that LAPACKE is still a FORTRAN interface, and something going wrong in the future might hamper debugging efforts considerably. Hence we should choose between openBLAS or ATLAS.

Oh and Carlos, I'm not talking about letting users install openBLAS with their package manager, I'm saying that we will ship openBLAS with the new gem and IT WILL COMPILE on the user's machine. If you'll see the README (https://github.com/xianyi/OpenBLAS#installation-from-source), they have a whole list of CPUs for which openBLAS will tailor itself to when compiled on their machine with 'make'.

I havn't compared ATLAS and openBLAS myself yet, but it looks promising.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

[Will Levine]

I was thinking about proposing a summer of code project on this, so I started to do some research. Wanted to share my thoughts before I go any further.

Reason I'm interested in this:
Installation currently seems like a stumbling block for nmatrix adoption because of weird library dependencies. Making this situation better makes nmatrix more useful. Although it's possible that my opinion (as an Ubuntu user) on what makes an easy installation may be different from people running OS X or whateverelse Linux, so I'd like to know what other people think.

The current situation (my view from Ubuntu 14.10):
Debian/Ubuntu has a useful tool for switching between different implementations of libraries (see http://www.scipy.org/scipylib/building/linux.html#debian-ubuntu ), so I thought I'd play around with it.

The libopenblas* packages provide one implementation of liblapack.{so|a} and libblas.{so|a}. The libatlas* packages provide another liblapack and libblas. The liblapack* packages provide yet another liblapack (reference implementation). The libblas* packages provide their own libblas.

nmatrix will build and run only with the liblapack from atlas (with the other two you run into an undefined symbol: clapack_cgetrf).

There's at least one additional complication: in addition to external dependency on liblapack, nmatrix also has an external dependency on libcblas, which is only provided by the atlas packages.

There may be other dependencies on atlas that I haven't figured out yet.

Optimal situation:
Work like numpy: work with any variety of liblapack.so and libblas.so. No external dependencies beyond these.

Non-optimal situation:
Building either atlas/openblas as a part of the nmatrix build process. I've tried to build these before. It's not always trivial.

Way to get to the optimal situation:
Not sure yet. I wanted to hear what other people had to say before thinking any more about this.

Additional question:
Is it worth it to make a version of nmatrix with no external dependencies on lapack, blas, etc.? What exactly would be the benefit of this?

[John Woods]

Hi Will,

Replies inline below.

On Fri, Mar 13, 2015 at 1:18 AM Will Levine <wle...@gmail.com> wrote:

I was thinking about proposing a summer of code project on this, so I started to do some research. Wanted to share my thoughts before I go any further.

Reason I'm interested in this:
Installation currently seems like a stumbling block for nmatrix adoption because of weird library dependencies. Making this situation better makes nmatrix more useful. Although it's possible that my opinion (as an Ubuntu user) on what makes an easy installation may be different from people running OS X or whateverelse Linux, so I'd like to know what other people think.

 

The current situation (my view from Ubuntu 14.10):
Debian/Ubuntu has a useful tool for switching between different implementations of libraries (see http://www.scipy.org/scipylib/building/linux.html#debian-ubuntu ), so I thought I'd play around with it.

The libopenblas* packages provide one implementation of liblapack.{so|a} and libblas.{so|a}. The libatlas* packages provide another liblapack and libblas. The liblapack* packages provide yet another liblapack (reference implementation). The libblas* packages provide their own libblas.

nmatrix will build and run only with the liblapack from atlas (with the other two you run into an undefined symbol: clapack_cgetrf).

There's at least one additional complication: in addition to external dependency on liblapack, nmatrix also has an external dependency on libcblas, which is only provided by the atlas packages.

There may be other dependencies on atlas that I haven't figured out yet.

It sounds like you've got a good understanding of the problem. Excellent!

 

Optimal situation:
Work like numpy: work with any variety of liblapack.so and libblas.so. No external dependencies beyond these.

Non-optimal situation:
Building either atlas/openblas as a part of the nmatrix build process. I've tried to build these before. It's not always trivial.

Way to get to the optimal situation:
Not sure yet. I wanted to hear what other people had to say before thinking any more about this.

Additional question:
Is it worth it to make a version of nmatrix with no external dependencies on lapack, blas, etc.? What exactly would be the benefit of this?

Have a look at how the distribution gem works. It has a fall-back Ruby option for nearly every available statistical distribution, but prefers to use C or JRuby libraries for these distributions when those are available.

I think the idea is that NMatrix should have simple, straightforward implementations of most functions on its own — but that the really heavy-duty algorithms should be accessed through API by supplying gems like nmatrix-atlas, for example.

Another possibility way down the road is that NMatrix becomes a spec rather than an actual gem — and we have "native" implementations of it in both Java and C which allows NMatrix to work even for those using JRuby. But that's probably a project for next year, unless someone is really interested in it.

Anyway, the benefit is that NMatrix becomes much less platform dependent than it is. When people have really heavy-duty number-crunching to do, they can do a little extra work to install the plugins for external, highly-optimized libraries.

Make sense?

John

[Will Levine]

John, thanks for your reply. Just to be clear, when you say

"the idea is that NMatrix should have simple, straightforward
implementations of most functions on its own"

do you mean pure Ruby implementations like in the distribution gem, or
C implementations, which already exist to some extent in nmatrix? I
think you mean the latter, but I just want to make sure. Either way,
the reason I'm somewhat skeptical of this idea is because if you make
nmatrix independent of LAPACK implementation (see next paragraph),
then installing the external dependencies becomes trivially easy. It
looks like OS X comes with liblapack pre-installed and on Linux
systems it's easy to install through whatever standard package
manager. Adding a second implementation of everything means additional
complexity, additional testing, additional maintenance. If this has
benefit of making nmatrix easier to use, then that's great and the
extra complexity is worth tolerating. But if the only benefit is
saving people from running 'sudo apt-get install liblapack3' then it
doesn't really seem worth it.

On a related note, it turns out that it only requires a few changes to
get nmatrix to build and run without ATLAS, though a few features are
disabled. As a test, I tried to reimplement one of the missing
functions (sgetri) using some functions copied from LAPACKE, as
suggested by Sameer. Things appear to be working. I think replacing
CLAPACK with LAPACKE is a good idea:
It has a fully compatible license
Removes dependency on ATLAS, without adding any additional external
dependencies beyond lapack and blas
Works with either the liblapack from ATLAS, OpenBLAS, or the reference
implementation (I tested these three, should also work with other
implementations as well). This means you can use your standard
OS-provided library or a special customized library without having to
change anything in nmatrix.
There are some remaining complications like the question of CBLAS,
which does not have a friendly license. But overall, I think it should
be not so hard to make nmatrix lapack-agnostic, which would help in
making it easier to install.

[John Woods]

I think there are certain core functionalities that people need access to that should be duplicated, when possible, in very simple ways. Matrix multiplication, for example, needs to work whether people have external packages or not. Determinants and inverses, too. Do we need linear solvers? No, probably not — those should require external libraries.

I'd definitely include your LAPACKE proof-of-concept in your application, along with an explanation of what you did to make it work. This is good research.

John

>> email to sciruby-dev+unsubscribe@googlegroups.com.

>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "SciRuby Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to sciruby-dev+unsubscribe@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev+unsubscribe@googlegroups.com.