Python interface to Elemental

[Jack Poulson]

Dear all,

A number of research groups have recently been asking about plans for a Python interface to Elemental, and so I thought I would ask the many Python experts on this list for their opinions on what the current and future best approaches are for exposing templated C++ libraries in Python.

The history of previous and current efforts and participants is a bit complicated, so here goes:

- About a year ago, Dag Sverre Seljebotn [1] began working on such an interface, elemental4py [2], using Cython [3], but he has (rightly so) been busy finishing up his PhD and so the project stalled about 11 months ago.

- About a month ago, I toyed around with creating a simple set of wrappers [4] using ctypes [5]. I chose to use ctypes because it was easy for me to hack together support for a few key data structures and functions in a short amount of time. Unfortunately it would be a significant amount of work to extend it to a full-fledged Python interface to Elemental.

- Recently Michael Grant [6] has picked up the gauntlet and begun working on a Python interface (currently via Cython). This is part of a collaboration with Emmanuel Candes [7] and Stephen Boyd [8] on distributed-memory optimization. Andy Terrel [9] has been providing Python advice and could perhaps say more about the relative benefits of ctypes/Cython/Numba [10].

- I had a meeting today with Anju Kambadur from TJ Watson about the functionality that his team would find useful for distributed optimization (Anju, can you say more here?). One of his concerns is that he would like to ensure that his Python-wrapped (via boost::python [11]) C++ library is compatible with whichever approach Elemental takes. This of course applies to any other Python-wrapped library. The other team members are Vikas Sindhwani and Yves Ineichen (CC'd).

- I have also been collaborating with several researchers at Georgia Tech: Richard Boyd, Barry Drake [12], and Haesun Park [13], who are very interested in using a Python interface to Elemental to perform large-scale convex optimization (especially non-negative matrix factorization).

- Aron Ahmadia [14] has also expressed interest in working on the Python interface.

Given the significant amount of interest in such an interface at the moment, the largest obstacle right now is to make sure that everyone is not working on their own separate implementations. It would thus be helpful to start by collectively deciding on an approach. As of now, at least one person has suggested each of the following:
- ctypes
- CFFI [15]
- Cython
- Numba

Andy, Aron, and Dag, do you have any input on this?

I had originally hoped that work on such an interface would have the byproduct of a full-fledged C interface, which could then easily result in full-fledged F90 and R interface, but it seems that many of the Python wrappers would not provide this.

Jack

[1]: http://folk.uio.no/dagss/
[2]: https://github.com/dagss/elemental4py
[3]: http://en.wikipedia.org/wiki/Cython
[4]: https://github.com/poulson/Elemental/tree/master/experimental/python
[5]: http://docs.python.org/3/library/ctypes.html
[6]: http://cvxr.com/bio/
[7]: http://www-stat.stanford.edu/~candes/
[8]: http://www.stanford.edu/~boyd/
[9]: http://andy.terrel.us/
[10]: https://github.com/numba/numba
[11]: http://www.boost.org/doc/libs/1_53_0/libs/python/doc/index.html
[12]: http://www.pe.gatech.edu/instructors/barry-drake
[13]: http://www.cc.gatech.edu/~hpark/
[14]: http://aron.ahmadia.net/
[15]: http://eli.thegreenplace.net/2013/03/09/python-ffi-with-ctypes-and-cffi/

[Michael Grant]

Hey folks,

Jack, thanks for starting this conversation.

As Jack indicated I've been working on a Cython-based interface. But in fact, today I investigated what it would take to do one in boost::python instead, and frankly it looks reasonably promising as well. I too have a keen interest in not duplicating other people's work, so if there is already a somewhat complete "direct" interface implemented in boost::python, I would be happy to suspend at least part of my work. Or if there is not yet an interface, but the expertise is there to get it done more quickly, I'm happy to jump over to boost::python. I do not want to be the sole and primary developer and maintainer of the Python interface in the long run, so I need to be flexible.

However, one of the things that I am keen to see is a Python-native interface. We can certainly expose all of the objects and methods of the C++ interface to Python via Cython or boost::python, and one could learn to be quite productive with that. But Elemental's structure is very C++-like, which is to say it's not very Python-like. It implements its own Matrix<T> class; and while theoretically that can be ported directly to Python, it seems a shame to do so when there is a perfectly serviceable matrix class in NumPy.

What about DistMatrix<T>, then? Well, what I would like to see a Python "distmatrix" class that wraps the Elemental DistMatrix class, but presents an interface that is as familiar to NumPy users as possible. I think this is entirely possible---and yet, it could be done in addition or on top of a Python interface. I can readily see a two-layered approach emerging: a low-level Python interface that mirrors the C++ API very closely; and an additional layer that re-creates DistMatrix in a Python-friendly way.

One thing I would like to point out is that the folks at Continuum Analytics (http://continuum.io) have expressed a very strong interest in seeing Elemental work smoothly with Python, and eventually with their Blaze and Numba numerics efforts (http://blaze.pydata.org, http://numba.pydata.org/). Continuum was founded by some pretty heavy NumPy/SciPy contributors, including Travis Oliphant. They are based here in Austin, where I live, and they have encouraged me to visit when I want and consult with them when I need.

Blaze and Numba seem a bit too preliminary to provide us with any direct benefits for this effort. But the expertise that Continuum brings to the table could be valuable int he long run. And I think that if we do a Python interface well, it will influence the development of Blaze and Numba tremendously, which in turn would provide considerable performance boosts for numeric software built around NumPy and Elemental.

As for the C interface: I think that is an admirable goal, and something that would still be worthwhile to pursue, but I do not think that Python in particular would benefit from it. I think that a direct connection to the C++ interface is the right approach, whether it be with Cython, boost::python, or Numba.

Thanks for the hard work, everyone, and I look forward to hearing everyone else's thoughts.

Michael

[Michael Grant]

I'm attaching the comments offered by Anju Kambadur of IBM here, since he is not a member of the elemental-dev list, so they will be archived. If you intend to offer your own comments, please join the Google Group first by clicking this link and signing up.

 

Jack and Michael --- Thanks for your comments. I would like to first point out that our interest in Elemental stems from a DARPA project that is required to be open-sourced; so, anything we do/discuss would benefit the community at-large. Here are my thoughts:
(1) Like Elemental, our library --- skylark --- is implemented using templated C++. We provide a comprehensive python interface using boost::python. We would like to have the ability to talk to the same Elemental data structures in *both* the python and the C++ layers seamlessly. To give you an example of what I mean by this, consider an operator skylark::foo_t that is templated on a MatrixType.
namespace skylark {
  // Fully generic implementation
  template <typename MatrixType>
  struct foo_t {
    typedef MatrixType matrix_type;
    void bar(const matrix_type& A) { ... }
  };
  // Specialization for elem::DistMatrix<double, VR, STAR>
  template <>
  struct foo_t <elem::DistMatrix<double, VR, STAR> > {
    typedef elem::DistMatrix<double, VR, STAR> matrix_type;
    void bar(const matrix_type& A) { ... }
  };
}

We now expose the specialization in python using using boost::python, which looks somewhat like this:
// add boiler-plate code
typedef skylark::foo_t<elem::DistMatrix<double, VR, STAR> > foo_DistMatrix_VR_STAR_t;
boost::python::class_<foo_DistMatrix_VR_STAR_t>("foo_DistMatrix_VR_STAR", init<...>())
                       .def ("bar", &foo_DistMatrix_VR_STAR_t::bar, ...);                      
Now, the skylark python type "foo_DistMatrix_VR_STAR" can be instantiated in python. To call "foo_DistMatrix_VR_STAR.bar()" in python, the user would have to create a "elem::DistMatrix_VR_STAR" *python* object and then pass object as the argument to "foo_DistMatrix_VR_STAR.bar()". However, for this scheme to work, the "elem::DistMatrix_VR_STAR" python interface should be type preserving. IMHO, this is not possible if the Elemental's python interface is facilitated by ctypes (we have tried it). I might be completely wrong about this; so please correct me if I am wrong. I think that researchers would benefit greatly from having access to Elemental's full functionality in a first-class manner. I suspect that we are not the only C++-based python library that wants to consume Elemental objects in python, then pass them through to our C++ backend and expect to have some type information. Although I am not advocating that everyone switch to using boost::python, I am hoping that any reasonable approach for proving a python interface to a C++ library will preserve types. In fact, we are open to suggestions ourselves.
(2) We decided to use boost::python for many reasons --- I'll try to remember a few here: (a) the boost community has an excellent track-record of providing high-quality implementations; of these, boost::python is one of the premier libraries, (b) as boost::python is completely based in C++, it will always support advanced generic programming paradigms; I have been made aware of shortcomings of python based approaches such as SWIG w.r.t handling templates (which scares me immensely), and (c) creating boost::python interfaces to skylark was *very* trivial; once we put down some boiler-plate code, it was a matter of mostly cutting and pasting and changing variable names here and there (especially if you can manage to write some basic preprocessor macros).
(3) Michael mentions a very important point about having a python-native interface. I agree with him in that a boost::python-based one-to-one mapping to python of a C++ library is *not* going to attract many python users. Our solution to this is exactly what he suggested in his email! (a) We have a boost::python layer that is excellent at supporting all the programming paradigms we use and provides a one-to-one mapping of the C++-library's functionality in python and (b) A thin *python* interface layer that sits on top of our boost::python layer that makes our library more consumable to python users. An additional benefit of this approach would be that the python interface is resistant to changes in the C++ backend. Also, typically, people like me who are developing (a) are *not* be experts in python, so as long as (b) is independent of (a), we're happy :-).

My personal knowledge of python is both bare minimal and very recently acquired. I would like to hear others' thoughts on how to go about doing this. Yves --- who is copied in this mail --- pointed out today that there *might* be a way to use boost::python with ctypes. I am pasting his email here:
"It seems the only way is to use boost python to import the ctypes module (boost::python::import("ctypes")) and then do all the type conversations. If I understand boost python correctly boost::python::to_python_converter does something similar. This is not a very elegant solution ...".
As Jack mentioned, we want to have a solution that not just works for Elemental + skylark, but potentially with any library that has the same structure as Elemental/skylark. I apologize if I repeated some of the things that Jack and Michael mentioned earlier in their mails.

[Andy R. Terrel]

[resending since elemental-dev got dropped from my original message]

Just to pipe in a bit.

The reason I told Jack to look at ctypes was three fold.

1) ctypes has a very easy to understand syntax and in my experience

has been easier to get people started on .

2) For ctypes you have to write a c-shim layer, but then that shim

layer can be reused for many other languages such as fortran, matlab,

julia, to name a few.

3) ctypes is part of the python builtin system, thus you don't require

extra dependencies.

I usually don't recommend boost:python unless your code is already

boost based.  It is a horrible dependency that gives software

maintainers nightmares.  In general scipy devs try to avoid C++

because of all the dependencies it requires on compile time options.

Anaconda can smooth that out for users, but the support for python and

compilers is pretty bad.  Another approach is to write c++ extensions

via PyCXX http://cxx.sourceforge.net/

Cython provides a nice intermediate language and can make the users

python code much faster.  It does not provide as much c++ support as

one might like but at least it gets folks in the mood of compiling

their python hooks.  Unfortunately last time I used cython you

basically had to read the source to use it effectively.

For the fenics project we basically do what Michael is suggesting,

except we use SWIG to wrap the c++ code.  It works out well there, but

often users don't know what is python and what is C++ wrappers.

So while there are lots of options to pursue, I still think having a

simple c-shim layer is appropriate, but perhaps if someone already

-- Andy

[Andy R. Terrel]

[resending since elemental-dev got dropped from my original message]

On Fri, Mar 29, 2013 at 8:55 AM, Michael C. Grant <...> wrote:
> Andy,
>
> Can you elaborate just a bit on the "nightmares" caused by boost::python? One thing I do notice is their strong recommendation to use Boost's build tool, bjam. I am not inclined to do so, either. But I did notice that others had successfully used CMake to build Boost Python libraries; for instance.
>         http://preney.ca/paul/archives/107

So my experience has been jaded over supporting boost for many year,
but as of the last three I've just given up on it.

Firstly, you have to have a cutting edge compiler which is often not
available on HPC systems.  Any library that has to have specific
instructions for compiling with different compilers is bound to have
trouble.  Second, version mismatch is hell with boost.  They change
many things from version to version and you have to be really careful
with the installation checks.  Finally, most people do not install
boost correctly, and you have deal with all the headaches of helping a
user debug their software build.  For example, python is usually not
turned on by default and I don't think they support building with
multiple versions of python (thus you will have to teach the user how
to sandbox the builds).

As far as depending on boost, I have seen it slow compile times for
libraries by ten fold.  And when something does go wrong, the debug
messages are long and opaque.  Of course my caveat is that I have
excised this library from my retenue as much as possible years ago.

>
> Are there other concerns we should be aware of?
>
> As for Cython, the biggest weakness in the C++ support that I'm seeing now is the lack of support for template functions. Template classes work, but just not undecorated template functions. That is remedied by defining each instantiation separately---tedious, perhaps, but once it's done, it's done.

Yes and once again you have an interface that is usable by other
languages.  I don't like to use Fortran or Matlab, but many of our
colleagues do.  Giving them a route to that is important for open
source.

[ANJU KAMBADUR]

I usually advise people to stay away from any large library they don't need, elemental currently has no dependency on boost...

This is a very good point. Skylark has bought into boost, but Elemental has no dependency (yet). The way I look at it, we have to add a dependency one way or the other. What would be really nice is if --- regardless of the choice of cython or boost::python --- our python and C++ layers could interact like I pointed out in my first email. While it would be nice to agree to a common solution, we shouldn't have to enforce any single solution. Of course, I am still partial to boost::python as I am apprehensive about cython supporting modern C++ features both as of today and moving forward :-). Any thoughts?


- Anju
--------------------------------------------
Prabhanjan Kambadur
Research Staff Member
Business Analytics and Mathematical Sciences
IBM TJ Watson Research Center
Room 30-229 A

Andy Ray Terrel ---03/29/2013 08:51:42 AM---Hey guys, At the very least can we include Jack on the non mailing list version of

From:	Andy Ray Terrel <andy....@gmail.com>
To:	ANJU KAMBADUR/Watson/IBM@IBMUS, Jack Poulson <jack.p...@gmail.com>
Cc:	Aron Ahmadia <aron.a...@gmail.com>, "barry...@gtri.gatech.edu" <barry...@gtri.gatech.edu>, Dag Sverre Seljebotn <d.s.se...@astro.uio.no>, "Michael C. Grant" <m...@cvxr.com>, "richar...@gtri.gatech.edu Boyd" <richar...@gtri.gatech.edu>, Travis Oliphant <tra...@continuum.io>, Vikas Sindhwani/Watson/IBM@IBMUS, Yves Ineichen1 <ZRL...@ch.ibm.com>
Date:	03/29/2013 08:51 AM
Subject:	Re: Python interface to Elemental

---

Hey guys,

At the very least can we include Jack on the non mailing list version of this thread.

-- Andy

On Fri, Mar 29, 2013 at 10:43 AM, ANJU KAMBADUR <pkam...@us.ibm.com> wrote:

Firstly, you have to have a cutting edge compiler which is often not available on HPC systems...

I appreciate your experience and I have been using it for roughly that same time period. But our results, admittedly subject, differ.  This is why I gave a big caveat there that things might have changed in the last few years. But for all these reasons below that you mention, I usually advise people to stay away from any large library they don't need, elemental currently has no dependency on boost.
 

I am a HPC practitioner and have been a regular user of boost for close to a decade now. The issue of having a "cutting edge compiler" is not completely accurate. Boost has a standard and simple way of configuring and compiling it's libraries; the ease of configuring and installing boost is --- to me --- not as hard as you mentioned. "bjam", boost's build tool has improved vastly recently and it is included with the boost distribution, so there is no need to build that as we once used to. For example, here are the instructions for boost installation that we have, which seems to work:

Boost (minimum 1.49): We use only a few libraries from boost. Currently, this list includes --- mpi, mpi_python, serialization, python,  and random. To build these libraries, please follow these instructions:
(a) Download (if possible the latest version of) Boost.
(b) Untar/Unzip everything in a directory (let's say 'boost-src').
(c) Go into 'boost-src'
   #cd boost-src
(d) Boost strap the system by running bootstrap.sh script, which creates a configuration file called 'project-config.jam'. Note that <INSTALL-HERE> should be replaced by the directory where you want to install boost!
   #/bootstrap.sh --with-libraries=python,random,mpi --prefix=<INSTALL-HERE>
(e) Add a line to 'project-config.jam' to ensure Boost.Mpi is built properly.
   #echo "using mpi ;" >> project-config.jam
(f) Install everything using the following command.
   #./b2 install

Regarding backward compatibility, this is not unique to Boost --- it would be fair to say that even our own libraries will not be backwards compatible :-). However, highly used boost libraries are mostly good about following standard procedures before dropping support for a particular feature. Also, it is completely OK to "freeze" a particular version of boost as our requirement.

As far as depending on boost, I have seen it slow compile times for libraries by ten fold ....

Template instantiation is current compilers is slow. The comment about error messages as well is valid, but that is not a result of using boost, but a result of using templates. Since both Elemental and skylark use templates, we have already bought into that cost. But, the benefits of using boost --- to me --- far out weight the drawbacks. Our core development is 100% in templated C++ --- our primary focus should be to ensure that we can export *every* feature in our C++ libraries to python users. It would be unfortunate if we had to give up certain programming options in C++ to ensure that SWIG or Cython can handle exposing our functionality in python. Since many of our projects have people who are great at C++ and people who are great at python --- the two-step approach that Michael pointed out seems to be a good way to build a first-class python interface.




- Anju
--------------------------------------------
Prabhanjan Kambadur
Research Staff Member
Business Analytics and Mathematical Sciences
IBM TJ Watson Research Center
Room 30-229 A

Andy Ray Terrel ---03/29/2013 07:16:05 AM---On Fri, Mar 29, 2013 at 8:55 AM, Michael C. Grant <m...@cvxr.com> wrote: > Andy,


From:	Andy Ray Terrel <andy....@gmail.com>
To:	"Michael C. Grant" <m...@cvxr.com>
Cc:	Vikas Sindhwani/Watson/IBM@IBMUS, ANJU KAMBADUR/Watson/IBM@IBMUS, Aron Ahmadia <aron.a...@gmail.com>, "barry...@gtri.gatech.edu" <barry...@gtri.gatech.edu>, Dag Sverre Seljebotn <d.s.se...@astro.uio.no>, "richar...@gtri.gatech.edu Boyd" <richar...@gtri.gatech.edu>, Yves Ineichen1 <ZRL...@ch.ibm.com>, Travis Oliphant <tra...@continuum.io>
Date:	03/29/2013 07:16 AM
Subject:	Re: Python interface to Elemental

---

-- Andy

>
> On Mar 29, 2013, at 8:42 AM, Andy Ray Terrel <andy....@gmail.com>
>  wrote:

http://cxx.sourceforge.net/

>> On Fri, Mar 29, 2013 at 8:11 AM, Michael C. Grant <m...@cvxr.com> wrote:
>>> Good idea. Andy Terrel is affiliated with Continuum, so he might already be
>>> aware ;) But I will let him know.
>>>
>>> On Mar 29, 2013, at 8:00 AM, "Vikas Sindhwani" <vsi...@us.ibm.com> wrote:
>>>
>>> Michael:
>>>
>>> Since Travis Oliphant (Continuum Analytics) is also working on XDATA, it
>>> would really make sense to alert them of this thread. I am sure they must
>>> have given these general issues a lot of thought and it would be great to
>>> see what they have to say, and where they intend to take numpy/scipy in
>>> terms of distributed computing and interfacing with C/C++ libraries.
>>>
>>> best,
>>> Vikas
>>>
>>> -----"Michael C. Grant" <m...@cvxr.com> wrote: -----
>>> To: ANJU KAMBADUR/Watson/IBM@IBMUS
>>> From: "Michael C. Grant" <m...@cvxr.com>
>>> Date: 03/29/2013 08:52AM
>>> Cc: Andy Terrel <andy....@gmail.com>, Aron Ahmadia
>>> <aron.a...@gmail.com>, "barry...@gtri.gatech.edu"
>>> <barry...@gtri.gatech.edu>, Dag Sverre Seljebotn
>>> <d.s.se...@astro.uio.no>, "richar...@gtri.gatech.edu Boyd"
>>> <richar...@gtri.gatech.edu>, Vikas Sindhwani/Watson/IBM@IBMUS, "Yves
>>> Ineichen1" <ZRL...@ch.ibm.com>
>>> Subject: Re: Python interface to Elemental
>>>
>>> Anju, Thanks for the comments. I don't think I'll respond yet because 1) I
>>> really have nothing to disagree with and 2) I want to hear from folks who
>>> might have experience with other C++/Python interfaces.
>>>
>>> For everyone: may I suggest that if you intend to offer comments, *please
>>> join the elemental-dev Google Group first*. That way, your reply will be
>>> distributed to the rest of the group and archived. Anju's post seems not to
>>> have been posted there, and I think that's the reason why.
>>>
>>> I purposefully left elemental-dev off of this email, so please DO NOT REPLY
>>> to this one to add your points about Elemental & Python. Thanks!
>>>
>>> Michael
>

[Andy Ray Terrel]

Poking around, I see Eigen gets around this dependency by having boost::python bindings outside the main branch

https://code.launchpad.net/~knarkles/pyeigen/

Petsc and Qt does the same thing with several different Python wrappers.

-- Andy

[Michael C. Grant]

I think there are a couple issues here that might be worth separating---or at least talking about separately.

POINT 1: What interfaces should *Elemental* support? Put another way, which interfaces should be contained within the main Elemental repository? At least initially, it seems to me that the right answer might be C++, C, and perhaps SWIG. This seems like the quickest and easiest-to-maintain way to provide multi-language support for Elemental. Given that Jack is likely to remain the primary developer for some time, I would say that limiting the scope of what he would be directly involved in is worthwhile.

I know that those of us who want a first-class Python interface to start with might not prefer SWIG. But C/C++/SWIG immediately covers not just Python but Fortran, Java, Scala, C#, R, Octave... (HPC in Tcl, anyone?) While I suspect that in many of these languages, there will be a benefit to having a higher-level layer that provides native language abstractions, at least it will *work*.

POINT 2: Building on Andy's last point. Does a decision like the above mean we can't build a boost::python or Cython interface to Elemental? Of course not! Before I began toying with Cython for Elemental, I looked at how petsc4py (http://code.google.com/p/petsc4py/) does it. It seems to be quite a comfortable arrangement to have a separate Python package.

So if some of us want work on a Python interface outside of Elemental, great! If it ends up working very well, it could eventually be merged into the Elemental tree alongside SWIG, in glorious redundancy. But in all likelihood, it would simply continue to flourish as its own project.

I think accepting this "fork" frees everyone up to to be flexible. Those of us who are ready to dive into a higher-level Python interface can do so without feeling like we're stepping on the toes of Jack or anyone else with interests in other language interfaces.

[ANJU KAMBADUR]

Hey Michael ---

You bring out excellent points; I think the purpose of this discussion is mostly to give Jack additional information so that he can take a decision (hopefully soon) about what he wants to support. Once Jack fixes what the *official* support for python bindings is going to be, we can either decide to contribute to that effort or decide to maintain a separate set of bindings outside the trunk. Given most of our time constraints and that Elemental is itself in flux, it makes sense to contribute to the *official* effort, which is my personal opinion.

Regarding SWIG:
There are several internet forums dealing with the "SWIG vs boost::python", "Cython vs boost::python", etc debates. There is no hard and fast rule about which one approach to pick. I've already stated the reasons why I think boost::python might be a better choice. I am willing to contribute a small "first-cut" prototype of the boost::python-based bindings for Elemental in the next few days if people are interested in seeing how this might look like. A similar effort by the SWIG-lovers might help Jack (and us) make up our minds.

Regarding making approaches independent:
(Repeated) There is another parallel issue at stake here. Let's say Elemental decides to use SWIG (noooooo :)); this might force budding libraries --- like ours --- to move over to SWIG as well. Now say that there is a rogue C++ library --- Satan --- that refuses to switch to SWIG and insists on using boost::python (or something else for that matter). But Satan is a very powerful library and it would be nice to not antagonize it. There should theoretically be a way for Satan's python bindings to interact freely with Elemental's python and C++ bindings.

- Anju
--------------------------------------------
Prabhanjan Kambadur
Research Staff Member
Business Analytics and Mathematical Sciences
IBM TJ Watson Research Center
Room 30-229 A

"Michael C. Grant" ---03/29/2013 10:01:03 AM---I think there are a couple issues here that might be worth separating---or at least talking about se

From:

"Michael C. Grant" <m...@cvxr.com>

To:	ANJU KAMBADUR/Watson/IBM@IBMUS
Cc:	Andy Ray Terrel <andy....@gmail.com>, Aron Ahmadia <aron.a...@gmail.com>, "barry...@gtri.gatech.edu" <barry...@gtri.gatech.edu>, Dag Sverre Seljebotn <d.s.se...@astro.uio.no>, Jack Poulson <jack.p...@gmail.com>, "richar...@gtri.gatech.edu Boyd" <richar...@gtri.gatech.edu>, "Travis Oliphant" <tra...@continuum.io>, Vikas Sindhwani/Watson/IBM@IBMUS, "Yves Ineichen1" <ZRL...@ch.ibm.com>, "<elemen...@googlegroups.com>" <elemen...@googlegroups.com>
Date:	03/29/2013 10:01 AM
Subject:	[elemental] Re: Python interface to Elemental
Sent by:	elemen...@googlegroups.com

---

) does it. It seems to be quite a comfortable arrangement to have a separate Python package.

--
You received this message because you are subscribed to the Google Groups "elemental-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elemental-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jack Poulson]

Just for the record, I have no qualms with there being several different Python interfaces in the official repo, I just think that it isn't the best use of everyone's time.

If anyone sends a patch with a reasonably useful implementation, I will happily accept it. All that I ask is that, if there is an added dependency (e.g., boost or Cython), that the patch also include the necessary CMake code to make sure the module is only built if the dependency is met.

Jack

[Michael C. Grant]

Anju, since you are offering, I think you should go ahead with a boost::python implementation. This resource may help you with CMake:

http://preney.ca/paul/archives/107

But of course, while SWIG offers an alternative path to Python, boost::python does not offer an alternative path to other languages. So moving forward with this will not, in my view, significantly change decision-making about how to support other languages.

On Mar 29, 2013, at 1:39 PM, Jack Poulson <jack.p...@gmail.com>

 wrote:

Just for the record, I have no qualms with there being several different Python interfaces in the official repo, I just think that it isn't the best use of everyone's time.

If anyone sends a patch with a reasonably useful implementation, I will happily accept it. All that I ask is that, if there is an added dependency (e.g., boost or Cython), that the patch also include the necessary CMake code to make sure the module is only built if the dependency is met.

Jack

On Fri, Mar 29, 2013 at 11:26 AM, ANJU KAMBADUR <pkam...@us.ibm.com> wrote:

Hey Michael ---

You bring out excellent points; I think the purpose of this discussion is mostly to give Jack additional information so that he can take a decision (hopefully soon) about what he wants to support. Once Jack fixes what the *official* support for python bindings is going to be, we can either decide to contribute to that effort or decide to maintain a separate set of bindings outside the trunk. Given most of our time constraints and that Elemental is itself in flux, it makes sense to contribute to the *official* effort, which is my personal opinion.

Regarding SWIG:
There are several internet forums dealing with the "SWIG vs boost::python", "Cython vs boost::python", etc debates. There is no hard and fast rule about which one approach to pick. I've already stated the reasons why I think boost::python might be a better choice. I am willing to contribute a small "first-cut" prototype of the boost::python-based bindings for Elemental in the next few days if people are interested in seeing how this might look like. A similar effort by the SWIG-lovers might help Jack (and us) make up our minds.

Regarding making approaches independent:
(Repeated) There is another parallel issue at stake here. Let's say Elemental decides to use SWIG (noooooo :)); this might force budding libraries --- like ours --- to move over to SWIG as well. Now say that there is a rogue C++ library --- Satan --- that refuses to switch to SWIG and insists on using boost::python (or something else for that matter). But Satan is a very powerful library and it would be nice to not antagonize it. There should theoretically be a way for Satan's python bindings to interact freely with Elemental's python and C++ bindings.

- Anju
--------------------------------------------
Prabhanjan Kambadur
Research Staff Member
Business Analytics and Mathematical Sciences
IBM TJ Watson Research Center
Room 30-229 A

<graycol.gif>"Michael C. Grant" ---03/29/2013 10:01:03 AM---I think there are a couple issues here that might be worth separating---or at least talking about se

<ecblank.gif> From:	<ecblank.gif> "Michael C. Grant" <m...@cvxr.com>
<ecblank.gif> To:	<ecblank.gif> ANJU KAMBADUR/Watson/IBM@IBMUS
<ecblank.gif> Cc:	<ecblank.gif>

Andy Ray Terrel <andy....@gmail.com>, Aron Ahmadia <aron.a...@gmail.com>, "barry...@gtri.gatech.edu" <barry...@gtri.gatech.edu>, Dag Sverre Seljebotn <d.s.se...@astro.uio.no>, Jack Poulson <jack.p...@gmail.com>, "richar...@gtri.gatech.edu Boyd" <richar...@gtri.gatech.edu>, "Travis Oliphant" <tra...@continuum.io>, Vikas Sindhwani/Watson/IBM@IBMUS, "Yves Ineichen1" <ZRL...@ch.ibm.com>, "<elemen...@googlegroups.com>" <elemen...@googlegroups.com>

<ecblank.gif> Date:	<ecblank.gif> 03/29/2013 10:01 AM
<ecblank.gif> Subject:	<ecblank.gif>

[elemental] Re: Python interface to Elemental

<ecblank.gif> Sent by:	<ecblank.gif> elemen...@googlegroups.com

---

I think there are a couple issues here that might be worth separating---or at least talking about separately.

POINT 1: What interfaces should *Elemental* support? Put another way, which interfaces should be contained within the main Elemental repository? At least initially, it seems to me that the right answer might be C++, C, and perhaps SWIG. This seems like the quickest and easiest-to-maintain way to provide multi-language support for Elemental. Given that Jack is likely to remain the primary developer for some time, I would say that limiting the scope of what he would be directly involved in is worthwhile.

I know that those of us who want a first-class Python interface to start with might not prefer SWIG. But C/C++/SWIG immediately covers not just Python but Fortran, Java, Scala, C#, R, Octave... (HPC in Tcl, anyone?) While I suspect that in many of these languages, there will be a benefit to having a higher-level layer that provides native language abstractions, at least it will *work*.

POINT 2: Building on Andy's last point. Does a decision like the above mean we can't build a boost::python or Cython interface to Elemental? Of course not! Before I began toying with Cython for Elemental, I looked at how petsc4py (http://code.google.com/p/petsc4py/

) does it. It seems to be quite a comfortable arrangement to have a separate Python package.

So if some of us want work on a Python interface outside of Elemental, great! If it ends up working very well, it could eventually be merged into the Elemental tree alongside SWIG, in glorious redundancy. But in all likelihood, it would simply continue to flourish as its own project.

I think accepting this "fork" frees everyone up to to be flexible. Those of us who are ready to dive into a higher-level Python interface can do so without feeling like we're stepping on the toes of Jack or anyone else with interests in other language interfaces.

--
You received this message because you are subscribed to the Google Groups "elemental-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elemental-de...@googlegroups.com.
For more options, visit

https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elemental-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elemental-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
You received this message because you are subscribed to a topic in the Google Groups "elemental-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elemental-dev/J7Q31ZRHf50/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to elemental-de...@googlegroups.com.

[Dag Sverre Seljebotn]

Hi,

(I'm answering Jack first and then moved Michael's reponse down bottom
to attempt to keep things together,)

> On Thursday, March 28, 2013 6:57:02 PM UTC-5, Jack Poulson wrote:
>
>> Dear all,
>>
>> A number of research groups have recently been asking about plans for
>> a Python interface to Elemental, and so I thought I would ask the many
>> Python experts on this list for their opinions on what the current and
>> future best approaches are for exposing templated C++ libraries in
>> Python.
>>
>> The history of previous and current efforts and participants is a bit
>> complicated, so here goes:
>>
>> - About a year ago, Dag Sverre Seljebotn [1] began working on such an
>> interface, elemental4py [2], using Cython [3], but he has (rightly so)
>> been busy finishing up his PhD and so the project stalled about 11
>> months ago.

I think what was important about that experiment hasn't been mentioned
yet.

elemental4py is a set of Jinja templates that generates a C shim around
Elemental -- doing away with compiling the template combinatorial
explosion and the C++ exception cathing once and for all there, in a
single code base for C, Fortran and Python. One could thus have a single
libelemental.so that compiles in all the templates once, rather than
getting a copy in each wrapper etc.

The Cython wrapper around that C shim is almost a side-effect -- all
the real stuff is in making sure C++ exceptions are captured etc. etc.

So note that even with a ctypes wrapper, you should benefit from 80% of
what's there in elemental4py if you want to make sure exceptions are
properly handled, have a single C API for Python and Fortran wrappers
and C code and so on.

I realize that some may think the code could enter a programming
obfuscation contest, since it is Jinja generating C++ -- it was
abandoned before I got to document it which probably doesn't help. But
I'm very happy to explain more about it if anybody think that is the way
forward. I think the state is that all the "though" issues has been
solved and the rest is just filling in the various operations, and the
templating make sure you don't have to repeat yourself too much in that
process.

>> Given the significant amount of interest in such an interface at the
>> moment, the largest obstacle right now is to make sure that everyone
>> is not working on their own separate implementations. It would thus be
>> helpful to start by collectively deciding on an approach. As of now,
>> at least one person has suggested each of the following:
>> - ctypes
>> - CFFI [15]
>> - Cython
>> - Numba
>>
>> Andy, Aron, and Dag, do you have any input on this?

1) General approach

Well, I'll be immodest and say that I believe the elemental4py approach
(though not necesarily the execution) is the right one:

a) get rid of C++ "ABI", exceptions and templates once and for all
using some kind of template metaprogramming and expose a clean C API
(i.e., that you would want to use from C), where the template arguments
are turned into run-time variables with run-time dispatches

(I believe for the usecases we are talking about when using a wrapper,
the overhead is utterly and completely negligible)

b) wrap this to get your Python interface

Now, whether the elemental4py code is too obfuscated to be suitable for
this, I cannot judge -- I like it myself but I'm obviously biased in
more than one way (I also used to be very active in Cython development).
But the idea could be carried out in other ways if you didn't like
elemental4py.

2) Which wrapper language to use

I don't have much experience with the C++ breed of wrappers (not even
the C++ support in Cython). I don't like that way of doing things --
because it means you need to cross the messy bridge to C++ again and
again from every language you want to have a wrapper for, better to do
that once. But I can't comment on Boost vs SWIG vs Cython here.

For wrapping a C shim, I think it doesn't really matter that much:

- Numba has an LLVM dependency, which seems overkill for such a simple
purpose as what we are doing here
- ctypes is loosing popularity and is slow
- CFFI may be good, and will allow Numba code to inline calls to it,
and allow PyPy to use it
- Cython is the de facto standard for these things as far as I can
see; to me Cython is the null hypothesis you'd need to invalidate by
pointing out a real argument in favour of the others

The advantage with Cython and Numba is that you can write non-trivial
code to make things more Pythonic, and do that on the compiled side of
the compiled-vs-interpreted divide, so that in some situations you can
make high-level functions that do a lot of lower-level calls and thus
get a lot lower overhead than with ctypes (and CFFI + CPython). I'm not
sure if that applies to the elemental API though.

The advantage with CFFI and ctypes is that PyPy users can use the
wrapper in a good way. However, NumPy doesn't exist for PyPy, so I
really don't think is an argument.

>> I had originally hoped that work on such an interface would have the
>> byproduct of a full-fledged C interface, which could then easily
>> result in full-fledged F90 and R interface, but it seems that many of
>> the Python wrappers would not provide this.

elemental4py does!

> As Jack indicated I've been working on a Cython-based interface. But
> in fact, today I investigated what it would take to do one in
> boost::python instead, and frankly it looks reasonably promising as
> well. I too have a keen interest in not duplicating other people's
> work, so if there is already a somewhat complete "direct" interface
> implemented in boost::python, I would be happy to suspend at least
> part of my work. Or if there is not yet an interface, but the
> expertise is there to get it done more quickly, I'm happy to jump over
> to boost::python. I do not want to be the sole and primary developer
> and maintainer of the Python interface in the long run, so I need to
> be flexible.
>

> However, one of the things that I am keen to see is a _Python-native_

> interface. We can certainly expose all of the objects and methods of
> the C++ interface to Python via Cython or boost::python, and one could
> learn to be quite productive with that. But Elemental's structure is
> very C++-like, which is to say it's not very Python-like. It
> implements its own Matrix<T> class; and while theoretically that can
> be ported directly to Python, it seems a shame to do so when there is
> a perfectly serviceable matrix class in NumPy.
>
> What about DistMatrix<T>, then? Well, what I would like to see a
> Python "distmatrix" class that wraps the Elemental DistMatrix class,
> but presents an interface that is as familiar to NumPy users as
> possible. I think this is entirely possible---and yet, it could be

> done _in addition _or _on top of _a Python interface. I can readily

> see a two-layered approach emerging: a low-level Python interface that
> mirrors the C++ API very closely; and an additional layer that
> re-creates DistMatrix in a Python-friendly way.
>
> One thing I would like to point out is that the folks at Continuum
> Analytics (http://continuum.io) have expressed a very strong interest
> in seeing Elemental work smoothly with Python, and eventually with
> their Blaze and Numba numerics efforts (http://blaze.pydata.org,
> http://numba.pydata.org/). Continuum was founded by some pretty heavy
> NumPy/SciPy contributors, including Travis Oliphant. They are based
> here in Austin, where I live, and they have encouraged me to visit
> when I want and consult with them when I need.

As far as Continuum goes I think it'd be good to know exactly why you'd
want to consult with them first. For wrapper technology, I wrote the
NumPy integration in Cython and I mentored Mark Florisson for two Google
Summer of Code programs on Cython before he went to Continuum to work on
Numba, and I'm happy to answer anything on the technical side for Cython
or CPython, or ask Mark about anything for Numba.

However you may want to talk to them about Blaze integration etc. etc.,
Andy would know more about that. My understanding is that what they are
doing don't necesarrily mesh that well with the "grid computing"/MPI way
of doing things, but that may be completely wrong.

I really think going this route should be done AFTER there's a Python
wrapper ready, and if Continuum has a keen interest in it themselves.
One don't want to get hung up into too much talk before getting to the
coding part.

>
> Blaze and Numba seem a bit too preliminary to provide us with any

> direct benefits for this effort. But the _expertise _that Continuum

> brings to the table could be valuable int he long run. And I think

> that if we do a Python interface _well_, it will influence the

> development of Blaze and Numba tremendously, which in turn would
> provide considerable performance boosts for numeric software built
> around NumPy and Elemental.
>
> As for the C interface: I think that is an admirable goal, and
> something that would still be worthwhile to pursue, but I do not think
> that Python in particular would benefit from it. I think that a direct
> connection to the C++ interface is the right approach, whether it be
> with Cython, boost::python, or Numba.

I already argued against this above.

But to be honest, I sort of feel like this is being made a lot larger
than what I think it is -- we're talking writing a wrapper, it should be
a relatively small and straightforward project code-wise, and I feel
we're about to spend almost as much time discussing it as it would take
to write it.

I think I spent 2 days on elemental4py so far, 2-3 days more should
easily finish it, but since the need for distributed dense algebra
disappeared in my PhD I didn't think much about it..

So really, I'd just say "go with the tool whoever is doing the job
prefers". And if anybody would like to pick up the elemental4py approach
I'm happy to Skype etc.

Dag Sverre

[Nichols A. Romero]

This is my experience FWIW
I wrote a Python-C interface to ScaLAPACK for use in GPAW, a real-space DFT code.

Before writing my own Python-C interface, I did shop around to see what was out there. I found that I could not use other peoples Python-C interface to ScaLAPACK for two reasons.
1. They depending on mpi4py or some other MPI binding and GPAW code had its own. It would have required much effort to get these different interfaces to interact properly.

2. The Python-C interface was missing uncommonly used function from BLACS.

Overall, I still spent more time try to figure out what DESCA is that writing the interface myself.

If I were to add Elemental to GPAW now, I would just ask Jack for a C interface as I am unfamiliar with interface with Python to C++.

---

--
You received this message because you are subscribed to the Google Groups "elemental-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elemental-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

--

Nichols A. Romero, Ph.D.
Argonne Leadership Computing Facility
Argonne National Laboratory
Building 240 Room 2-127
9700 South Cass Avenue
Argonne, IL 60490
(630) 252-3441

[ANJU KAMBADUR]

I'm thoroughly confused :).

(1) Jack, is there going to be an *official* set of python bindings for Elemental?  Would the current ctypes interface be the only one available in the near future? Who is going to maintain the python bindings?

(2) I am willing to write a prototype in boost::python only if the choice of boost::python is still open. Python user base is an important one and I defer to the opinion of Andy et al. about the unpopularity of Boost in the python community. We have zero interest in having our own version of Elemental's python bindings, so is there any point in spending time on a prototype?

Looks like most everyone has joined the elemental-dev mailing list. Should we remove the CC from the next mail onwards?

- Anju
--------------------------------------------
Prabhanjan Kambadur
Research Staff Member
Business Analytics and Mathematical Sciences
IBM TJ Watson Research Center
Room 30-229 A

Dag Sverre Seljebotn ---03/29/2013 12:38:51 PM---Hi, (I'm answering Jack first and then moved Michael's reponse down bottom

From:	Dag Sverre Seljebotn <d.s.se...@astro.uio.no>
To:	Michael Grant <m...@cvxr.com>
Cc:	<elemen...@googlegroups.com>, ANJU KAMBADUR/Watson/IBM@IBMUS, Vikas Sindhwani/Watson/IBM@IBMUS, Yves Ineichen1 <ZRL...@ch.ibm.com>, <richar...@gtri.gatech.edu>, Dag Sverre Seljebotn <d.s.se...@astro.uio.no>, <barry...@gtri.gatech.edu>, Andy Terrel <andy....@gmail.com>, Aron Ahmadia <aron.a...@gmail.com>
Date:	03/29/2013 12:38 PM
Subject:

Re: Python interface to Elemental

[Dag Sverre Seljebotn]

I forgot one important thing:

Using Cython, it is somewhat easy to dig into mpi4py and extract the
underlying C communicator from an mpi4py wrapped communicator. For
others wrapper languages one should at least figure out how to do this
step first. (mpi4py should probably get the feature of exposing the
communicator as a Python capsule, but not sure if that is in there now,
and you can't rely on users updating their mpi4py)

IMO, you absolutely do want to allow people using their own Python MPI
communicators.

Dag Sverre

[Jack Poulson]

On Fri, Mar 29, 2013 at 1:08 PM, ANJU KAMBADUR <pkam...@us.ibm.com> wrote:

I'm thoroughly confused :).

(1) Jack, is there going to be an *official* set of python bindings for Elemental?  Would the current ctypes interface be the only one available in the near future? Who is going to maintain the python bindings?

I should clarify that I would very much like to move towards a development model where significantly more people are actively contributing code to Elemental. Taking a somewhat hands off approach to the Python bindings seems like a nice stepping stone towards this goal.

I would prefer that things be kept simple and that there be a single official wrapper, but I do would prefer someone else to take the lead, and I am therefore trying to be as flexible as possible.

The moment someone sends me a functioning replacement I will incorporate it. If it appears to be more useful than my current ctypes implementation (which is very likely), it will be the "official" interface. Hopefully the people that write such an interface would be willing to maintain it, but I'm willing to shoulder the burden of manually propagating any changes made to the core library into the wrappers.

(2) I am willing to write a prototype in boost::python only if the choice of boost::python is still open. Python user base is an important one and I defer to the opinion of Andy et al. about the unpopularity of Boost in the python community. We have zero interest in having our own version of Elemental's python bindings, so is there any point in spending time on a prototype?

The option is certainly open, but there needs to be additional logic added into the CMake build system to ensure that it is only compiled if Boost support is detected.

 

Looks like most everyone has joined the elemental-dev mailing list. Should we remove the CC from the next mail onwards?

Sure!

Jack

[ANJU KAMBADUR]

Thanks for your answers! I guess this is decision time. Should we all briefly list our preferences and what we are willing contribute?

- Anju
--------------------------------------------
Prabhanjan Kambadur
Research Staff Member
Business Analytics and Mathematical Sciences
IBM TJ Watson Research Center
Room 30-229 A

Jack Poulson ---03/29/2013 01:20:48 PM---On Fri, Mar 29, 2013 at 1:08 PM, ANJU KAMBADUR <pkam...@us.ibm.com> wrote: > I'm thoroughly confuse

From:	Jack Poulson <jack.p...@gmail.com>
To:	"elemen...@googlegroups.com" <elemen...@googlegroups.com>
Cc:	Dag Sverre Seljebotn <d.s.se...@astro.uio.no>, Andy Terrel <andy....@gmail.com>, Aron Ahmadia <aron.a...@gmail.com>, "barry...@gtri.gatech.edu" <barry...@gtri.gatech.edu>, Michael Grant <m...@cvxr.com>, "richar...@gtri.gatech.edu" <richar...@gtri.gatech.edu>, Vikas Sindhwani/Watson/IBM@IBMUS, Yves Ineichen1 <ZRL...@ch.ibm.com>
Date:	03/29/2013 01:20 PM
Subject:	Re: [elemental] Re: Python interface to Elemental

Sent by:

elemen...@googlegroups.com

--

You received this message because you are subscribed to the Google Groups "elemental-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elemental-de...@googlegroups.com.
For more options, visit

https://groups.google.com/groups/opt_out.

[Michael C. Grant]

I've decided to pause my Cython effort and work on a SWIG port over the next few days.

I know this may seem like it's coming out of nowhere, but my thinking has changed as a result of this discussion. While I am keen to see the best possible Python port, I recognize that a long-term goal for Elemental should be accessibility on other numerical platforms. In fact, the notion of having a JVM-compatible version might be attractive to some of the Java-loving performers in XDATA :-) (There seem to be none here in this discussion, I admit.)

I've already done some preliminary work on SWIG and it is clear that I can get something running relatively quickly. In fact I've already ported some of the global functions over to Python.

Keep in mind that my first objective here is *not* to get the best Python implementation. If someone feels like Cython, boost::python, Jinja, etc. is the better way to go for Python, I encourage them to do it. Given that these other platforms will need an alternative solution *anyway*, I do not feel there is going to be significant duplicated effort.

In fact, even if SWIG turns out to be a great way to do a direct Python port, I remain convinced that the right approach is to go *above and beyond* such a port, and make something that is NumPy-aware and Python friendly.

[Vikas Sindhwani]

Hi Michael:

This sounds great. If you have an early version available, I would love to try it out.

best,
Vikas

[Michael C. Grant]

I will be happy to share as soon as I can!

--
You received this message because you are subscribed to a topic in the Google Groups "elemental-dev" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/elemental-dev/J7Q31ZRHf50/unsubscribe?hl=en-US.
To unsubscribe from this group and all its topics, send an email to elemental-de...@googlegroups.com.

[ANJU KAMBADUR]

Thanks Michael!

On my part, I will try to run toy experiments to see how such an arrangement can work when using SWIG. I don't know if it makes much sense for me to butt in this early, but if you want to --- you and I could figure out that SWIG looks promising w.r.t our intended use case, I would be happy to help or comment with this (private git branch on XDATA repository?). In particular, I can see if I adopted SWIG as well, we could interface with Elemental. To repeat --- the intended use case is the ability to communicate with Elemental's C++ and python bindings harmoniously. I am pasting the example I sent here for everyone's benefit:

namespace skylark {
  // Fully generic implementation
  template <typename MatrixType>
  struct foo_t {
    typedef MatrixType matrix_type;
    void bar(const matrix_type& A) { ... }
  };

  // Specialization for elem::DistMatrix<double, VR, STAR>
  template <>
  struct foo_t <elem::DistMatrix<double, VR, STAR> > {
    typedef elem::DistMatrix<double, VR, STAR> matrix_type;
    void bar(const matrix_type& A) { ... }
  };
}

Suppose skylark::foo_t<elem::DistMatrix<double, VR, STAR> were exposed in python, we should be able to instantiate a elem::DistMatrix<double, VR, STAR> python object (it might be called something else in python) and use it in the call to bar().

- Anju
--------------------------------------------
Prabhanjan Kambadur
Research Staff Member
Business Analytics and Mathematical Sciences
IBM TJ Watson Research Center
Room 30-229 A

"Michael C. Grant" ---03/29/2013 02:22:47 PM---I will be happy to share as soon as I can! On Mar 29, 2013, at 4:20 PM, "Vikas Sindhwani" <vikas.sin

From:

"Michael C. Grant" <m...@cvxr.com>

To:	"<elemen...@googlegroups.com>" <elemen...@googlegroups.com>

Date:

03/29/2013 02:22 PM

Subject:	Re: [elemental] Re: Python interface to Elemental
Sent by:	elemen...@googlegroups.com

--

You received this message because you are subscribed to the Google Groups "elemental-dev" group.

To unsubscribe from this group and stop receiving emails from it, send an email to elemental-de...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Travis Oliphant]

Hello all, 

I just found out about this thread and wanted to chime in.     There have been some excellent points made.    Generally, I very much favor a C-API to a library like Elemental.    This is the interface that will be easiest to wrap for any other language including Python.    The C++ ABI is too non-standard for the kind of dynamic library generation and interop that I like to enable in Python. 

I don't have much experience with using SWIG or Cython for wrapping C++ as my approach has traditionally been to hand-wrap C++ code to get exactly the interface I would like.   I have heard others who have used Cython and then reverted to hand-wrapping to get the performance they need.    Mark Wiebe has done this to expose his C++ library (DyND) to Python and will hopefully have time to comment on this thread in the coming days.   DyND is a C++-library for dynamic N-dimensional arrays (as opposed to static compile-time N-dimensional arrays) with nice bindings to Python and very useful inter-op with NumPy.   It is available here:   https://github.com/ContinuumIO/dynd with Python bindings here:  https://github.com/ContinuumIO/dynd-python It is part of the Blaze project but also usable separately.   We are using DyND as a local run-time library for Blaze. 

I know some people who like boost::python quite a bit, but I don't share their enthusiasm.    I find boost::python bindings hard to reason about when something goes wrong.   But, my experience is limited with Boost in general and Boost::Python in particular.  

At Continuum we are very interested in distributed dense-linear algebra (using MPI or otherwise) and are building Blaze precisely to bring a common distributed array concept to Python so that many libraries can share a common structure.   We will be creating bindings between Elemental and Blaze which  may borrow from any of the work done here but will likely also explore approaches that use direct compiler technology (LLVM and libclang).   This is experimental and should not dissuade anyone from working on the wrapping choice they prefer.

One of the challenges of a C++ library (especially one that makes heavy use of templates) is that you rarely create a machine-code library and then call it from another run-time (which relies on a stable and known ABI).   But, this is precisely the mode of operation that interactive environments like Python, R, Matlab, etc. prefer.     So, to "wrap" the C++ library you use the C++ compiler to instantiate the templates.   I think the best way to do this is for the library author to build the C-API (to expose how they think the library should be called from another run-time --- i.e. one that may not have access to a C++ compiler to re-build the templates).    

As a result, my opinion is that the C-API and ctypes wrapper that is currently present is sufficient and a SWIG wrapper or boost::Python wrapper might be  interesting but is not really needed.    But, I like to be proven wrong :-) 

Effort could go into improving the C-API perhaps to ensure that as many features as possible are exposed through the API. 

Best,

-Travis

On Thursday, March 28, 2013 6:57:02 PM UTC-5, Jack Poulson wrote:

[drak...@gmail.com]

Thought I'd chime in here for GA Tech:

So far I've been a lurker, but am very keen on this discussion. At GA Tech/GTRI we've chosen the Cython route to this point. Our stack is
a Python framework (Anaconda-based)/Cython/high level numerical algorithms (e.g., NMF clustering; topic analysis, etc.)/elemental/numerical libraries.

In making this decision, I trolled the 'nerd' sites and found that Cython came out on top. This is after
investigating boost-python and ctypes. Boost-python seemed daunting in its complexity.
Especially, since I've only sporadically used C/C++ for the past several years (Richard Boyd, my colleague,
is our expert in all things C/C++). On the otherhand, Cython appeared to be more 'Pythonic' although it may be somewhat
limited in terms of scalability to larger efforts and 'closeness' to the interesting C/C++ libraries; don't know this for sure. Aren't there always trade-offs?
I found the comments regarding hooks to the JVM interesting, which seems to me, would require a deeper dive than Cython would offer. Not sure
about this either at this point. But, hooks to the JVM would be an extremely important and interesting feature for our other XDATA friends and the java-based data
analytics community at large.

Thus, although we've put a stake in the ground with Cython at the moment for the first deliverable, we are completely open to moving to a more viable
solution going forward and the opinions/expertise expressed in this ongoing discussion will weigh heavily in our final decision.

We are interested in delivering the best solution that offers as seamless Python2C/C++/elemental integration as possible from the point of view of
numerical analysis/scientific computing/data analytics researchers who don't want to be distracted by the details of the C/C++ interface or underlying libraries.
Of course, they should be willing to switch from Matlab/Octave to Python ;-)

Thank you all for taking the time to post here and for Jack Poulson getting the conversation rolling.

Cheers!
Barry Drake
GA Tech/GTRI

[Vikas Sindhwani]

Given Travis and Andy's comments - which confirm some of my own impressions as a Python user -  I am wondering why not go with Elemental's current ctypes based interface and develop a C-layer for our C++ libraries.  This seems like the quickest and simplest way to get started. Any other cons to be aware of concerning ctypes?

Jack, if another candidate official version comes along, would you still intend to keep the C/ctypes interface intact and maintained?

ctypes is part of the Python language and seems easier to work with since one doesnt even have to leave the Python environment, once a C layer is written (which could be useful in itself).

best,
Vikas

[Robert van de Geijn]

Folks,

I will be giving a talk at ARITH 21 in about a week, titled "Managing the Dense Linear Algebra Software Stack"

Since ARITH is mostly focused on very low level stuff, I will likely spend most of the talk on our BLIS adventures.

Elemental is at the top of the stack. It would be good to give some evidence of its impact to date. Perhaps people on this group can help out with this.

Please send me a message with

- Institution
- Research group
- Application
- Target architectures
- Did your group use ScaLAPACK previously?
o if so, why did you switch?

(Feel free to point out other interesting details that I should have requested.)

Robert

[Jack Poulson]

On Sat, Mar 30, 2013 at 7:19 PM, Vikas Sindhwani <vikas.s...@gmail.com> wrote:

Jack, if another candidate official version comes along, would you still intend to keep the C/ctypes interface intact and maintained?

 

It depends on how serious the other candidate is and what the interests of users are. If there is a sufficient need for it to remain, then I would be happy to keep it around.

Jack

[Jack Poulson]

Hi Robert,

Would you prefer those sent privately to you or to the list? I assume that you are familiar with most of the applications directly related to my work, but I could come up with a list if you would like.

Jack

[Robert van de Geijn]

Privately would be fine.  I don't meant to clog up the forum.

If you, too,  could send me your list, that would be great.

[Mark Wiebe]

Hey all,

I've been reading through this thread, and have a few ideas to contribute. Prior to working in the NumPy codebase, all the wrapping I did was with boost::python, and my experience is that it`s easier to work with than all the other alternatives. The biggest disadvantages I experienced were how heavyweight the wrappers were, adding overhead, and the lack of control over exact memory layout and behavior. In NumPy, directly using the CPython API allows it to integrate much more seamlessly with the language. For DyND, the choice of Cython allows for a similar tight integration, but with the benefit that Cython generates much of the boilerplate required to define C extension modules and classes. I haven't used SWIG, so I don't know very much about how it works.

One important feature I think the bindings should support is the buffer protocol (http://docs.python.org/3.3/c-api/buffer.html, PEP 3118). This is a C API which allows the object to communicate the data type, striding, and memory pointers of array data, and allows libraries like NumPy, Cython, DyND, and others to work with the data directly as a matrix.

 - From ctypes, it almost looks like one could populate a Py_buffer struct, and call the PyMemoryView_FromBuffer function  to create an object which exposes this protocol, but the documentation warns that this is only for temporary memoryviews, and doesn't seem to have a way to properly support the protocol this way. One way to get this from a ctypes-wrapped library might be using a NumPy array subclass for the Matrix wrapper, but I think there will be tricky edge cases there too.

 - From cython, one can implement __getbuffer__ and __releasebuffer__ (http://docs.cython.org/src/userguide/special_methods.html#buffer-interface-pep-3118-no-python-equivalents-see-note-1), which is pretty straightforward.

 - I'm not sure how SWIG or boost::python code can expose the buffer interface. There are probably ways, but simple googling didn't find them.

Some of the work I've done in DyND might inform how other wrapped C++ code could access the wrapped elemental objects directly. I've written up an overview of how DyND uses Cython here:

https://github.com/ContinuumIO/dynd-python/blob/master/doc/source/dev.rst

In particular, here's how the C++ code is accessing the Cython-generated objects:

https://github.com/ContinuumIO/dynd-python/blob/master/doc/source/dev.rst#accessing-type-objects-from-c

For the ctypes case, it would be possible to write some code which uses the CPython API to retrieve the wrapped pointers to elemental objects, though with lower performance than in the other approaches.

Overall, my inclination would be to go with Cython, because it makes supporting the buffer protocol easy, and the extension classes it creates can be efficieintly accessed from other libraries without too much trouble. Correctly managing the ownership of elemental objects within the ctypes approach is harder as well. There's the __del__ method (http://docs.python.org/3/reference/datamodel.html#object.__del__), but the documentation warns about problems with destruction order when using it.

Thanks,
Mark

[Dag Sverre Seljebotn]

On 04/02/2013 10:38 PM, Mark Wiebe wrote:
> Hey all,
>
> I've been reading through this thread, and have a few ideas to
> contribute. Prior to working in the NumPy codebase, all the wrapping I
> did was with boost::python, and my experience is that it`s easier to
> work with than all the other alternatives. The biggest disadvantages I
> experienced were how heavyweight the wrappers were, adding overhead, and
> the lack of control over exact memory layout and behavior. In NumPy,
> directly using the CPython API allows it to integrate much more
> seamlessly with the language. For DyND, the choice of Cython allows for
> a similar tight integration, but with the benefit that Cython generates
> much of the boilerplate required to define C extension modules and
> classes. I haven't used SWIG, so I don't know very much about how it works.
>
> One important feature I think the bindings should support is the buffer
> protocol (http://docs.python.org/3.3/c-api/buffer.html, PEP 3118). This
> is a C API which allows the object to communicate the data type,
> striding, and memory pointers of array data, and allows libraries like
> NumPy, Cython, DyND, and others to work with the data directly as a matrix.

I think the natural way to do this (and as earlier mentioned by someone
else in this thread I think) is simply to expose the local buffer as a
NumPy array, i.e.,

>>> type(my_dist_array.local)
numpy.ndarray

I think the simplest and cleanest way to do this is simply to have every
node use NumPy for allocation and then pass the data pointer to
Elemental (in particular because you also want the option to have the
DistArray constructor take an existing local buffer to use as an argument).

And in that case, exposing a PEP 3118 buffer as the .local attribute is
automatic. I believe supporting PEP 3118 access directly on a DistArray
instance would be confusing since PEP 3118 is unable to give a view to
the global distributed array, which is what my_dist_array represents
(i.e., you may want my_dist_array.shape to give the global shape,
whereas my_dist_array.local.shape is delegated to NumPy).

Dag Sverre

> <http://eli.thegreenplace.net/2013/03/09/python-ffi-with-ctypes-and-cffi/>
>

[Dag Sverre Seljebotn]

I.e., this:

https://github.com/dagss/elemental4py/blob/master/elemental/elemental.pyx#L140

Building on this, you could see local_buf being an optional constructor
argument in case the local array was already assembled, and so on.

Dag Sverre

[Michael C. Grant]

Yes, I did suggest returning numpy.matrix objects for calls to Matrix::GetLocal(). I agree that it would not be appropriate to do the same for DistMatrix.

I see you're already thinking in terms of having a "Pythonic" interface to elemental---using attributes, for instance, for const C++ methods. I also agree that this is the right approach.

I am still working on a SWIG port, but short of a breakthrough I will probably be giving up in a couple of days. It is simply too cumbersome to have to deal with SWIG's idiosyncracies in order to get the behavior I want. Now, that may be due to my inexperience with SWIG; so be it. But it doesn't bode well for future, collaborative maintenance. Besides, it also looks like it is going to be horribly inefficient---the wrapper code is far larger and more complex than it should be. For this much overhead I'd want a fully "Python/NumPy"-friendly port! (And I'd like it to make my tea.)

I am now leaning towards the following three-prong strategy for a Python interface:
--- 1) A dynamically typed API, probably implemented in C++.
--- 2) A C wrapper
--- 3) A Python wrapper around the C API.

About #1. There seems to be at least a plurality among us who agree that a dynamically typed approach is better suited for Python implementation. In theory, this could be accomplished entirely at the Python level, but it would require duplicating information that could easily be exposed at very low cost by the C++ API. I've already suggested to Jack a way to refactor the current AbstractDistMatrix class a bit to provide for this. This would in no way compromise the performance of a standard, statically-typed C++ usage approach. For instance, dynamic type and distribution information could be provided via virtual function calls, not added data elements; and said virtual function overhead would not be incurred for stack allocated objects or pointers to fully instantiated types.

I think it would be good to do most of the heavy lifting for a dynamically typed interface to Elemental within C++ itself, and I am happy to contribute to that effort. But strictly speaking, it could be done without any changes to the current code. It would simply require the storage of type information in the Python wrapper objects.

About #2: The C API is going to be needed for other languages anyway, and as others have pointed out. Using it for the Python interface avoids the idiosyncracies we are likely to encounter with a more direct C++ wrapping. That said, if we *only* exposed the dynamically typed interface, then I think most of the wrapper difficulties go away, because they are generally caused by the use of templates, and the limitations of various automated approaches to handling those well.

Anyway, I shall now return to SWIG for a bit longer.

On Apr 2, 2013, at 4:13 PM, Dag Sverre Seljebotn <d.s.se...@astro.uio.no>
wrote:

[Andy Ray Terrel]

There is a reason I didn't even mention SWIG in my suggestions =P.  Getting the typemaps correct is a nightmare.  This was released today.... https://s3.amazonaws.com/xdress/index.html

-- Andy

[Dag Sverre Seljebotn]

On 04/02/2013 11:38 PM, Michael C. Grant wrote:
> Yes, I did suggest returning numpy.matrix objects for calls to Matrix::GetLocal(). I agree that it would not be appropriate to do the same for DistMatrix.
>
> I see you're already thinking in terms of having a "Pythonic" interface to elemental---using attributes, for instance, for const C++ methods. I also agree that this is the right approach.

I'm not "already thinking"; I implemented elemental4py almost exactly
one year ago with a ".local" attribute.

To be clear, cholesky(), gemm() and set_identity() (and access to
arr.local) works, there's even som unit-tests (against the Elemental
version released one year ago). It's just a matter of filling in blanks
to support more methods.

> I am still working on a SWIG port, but short of a breakthrough I will probably be giving up in a couple of days. It is simply too cumbersome to have to deal with SWIG's idiosyncracies in order to get the behavior I want. Now, that may be due to my inexperience with SWIG; so be it. But it doesn't bode well for future, collaborative maintenance. Besides, it also looks like it is going to be horribly inefficient---the wrapper code is far larger and more complex than it should be. For this much overhead I'd want a fully "Python/NumPy"-friendly port! (And I'd like it to make my tea.)
>
> I am now leaning towards the following three-prong strategy for a Python interface:
> --- 1) A dynamically typed API, probably implemented in C++.
> --- 2) A C wrapper
> --- 3) A Python wrapper around the C API.

To me, this edges its way exactly to the elemental4py approach.
elemental4py doesn't use Jinja and code-generation because it is fun --
it uses it to side-step the lack of 1).

In my opinion that's a fair first move that gets results quickly by
focusing on 2) and 3) above. THEN, when things work, you can implement
1), which gets rid of the Jinja templates and makes the code easier to
read, if you wish so.

As I said I'm more than happy to provide "support" on elemental4py if
people want to push that forward; I can spend the time myself to write
some docs on it and make sure it builds against latest version on
Elemental and so on (a CMake build system I can't promise, but people
are using CMake with Cython all the time so should be easy).

Don't worry, I'll now shut up about elemental4py unless encouraged (or I
magically find time to get working on it), lest I sound like a broken
record.

Dag Sverre