Google summer of code project

[Will Levine]

Hi all,
I've been accepted to the Google summer of code program to work on
SciRuby and nmatrix this summer. I'm a physics PhD student at Carnegie
Mellon University. My plan for the summer is to make nmatrix easier to
install and use by allowing it to be used with a greater variety of
external linear algebra libraries, or, if desired, without any
external libraries at all. I've posted my full proposal here:
https://github.com/wlevine/nmatrix/wiki/Summer-of-Code-2015-proposal

I'll be posting more to the list to get feedback on some of the some
of the ideas I'm proposing, so y'all will hear more from me, but for
now I just wanted to introduce myself.

I look forward to working with everyone!

Will

[Carlos Agarie]

Hey Will, congratulations! I'm really excited to have someone dedicated to improving NMatrix and its install process.

Cheers :)

-----

Carlos Agarie

+55 11 97320-3878 | @carlos_agarie

Will

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[John Woods]

Hey Will,

Could you please update the posted version of your proposal with the things we discussed last week?

Thanks for introducing yourself, and congrats. We're glad to have you involved. =)

John

[John Woods]

Hi all,

Just a reminder that Will is looking for some feedback on his project proposal. Now's a good time to raise any concerns you might have, since he hasn't started coding yet.

Sameer, I'm especially looking at you, since you wrote an excellent proposal for this same project. =) I know you have some thoughts.

John

On Mon, Apr 27, 2015 at 10:02 PM Will Levine <wle...@gmail.com> wrote:

[Pjotr Prins]

It is important to realise that GSoC is about coding and not about
deployment.

One thing would be interesting is to add CUDA support to nmatrix. That
comes also with dependency challenges and would be a very good to
have.

Pj.

On Thu, May 07, 2015 at 06:40:45PM +0000, John Woods wrote:
> Hi all,
> Just a reminder that Will is looking for some feedback on his project
> proposal. Now's a good time to raise any concerns you might have, since he
> hasn't started coding yet.
> Sameer, I'm especially looking at you, since you wrote an excellent
> proposal for this same project. =) I know you have some thoughts.
> John

> On Mon, Apr 27, 2015 at 10:02 PM Will Levine <[1]wle...@gmail.com> wrote:
>
> Hi all,
> I've been accepted to the Google summer of code program to work on
> SciRuby and nmatrix this summer. I'm a physics PhD student at Carnegie
> Mellon University. My plan for the summer is to make nmatrix easier to
> install and use by allowing it to be used with a greater variety of
> external linear algebra libraries, or, if desired, without any
> external libraries at all. I've posted my full proposal here:

> [2]https://github.com/wlevine/nmatrix/wiki/Summer-of-Code-2015-proposal

>
> I'll be posting more to the list to get feedback on some of the some
> of the ideas I'm proposing, so y'all will hear more from me, but for
> now I just wanted to introduce myself.
>
> I look forward to working with everyone!
>
> Will
>
> --
> You received this message because you are subscribed to the Google
> Groups "SciRuby Development" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [3]sciruby-dev...@googlegroups.com.
> For more options, visit [4]https://groups.google.com/d/optout.

>
> --
> You received this message because you are subscribed to the Google Groups
> "SciRuby Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [5]sciruby-dev...@googlegroups.com.
> For more options, visit [6]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:wle...@gmail.com
> 2. https://github.com/wlevine/nmatrix/wiki/Summer-of-Code-2015-proposal
> 3. mailto:sciruby-dev%2Bunsu...@googlegroups.com
> 4. https://groups.google.com/d/optout
> 5. mailto:sciruby-dev...@googlegroups.com
> 6. https://groups.google.com/d/optout

--

[Sameer Deshmukh]

Yes John I'll go through it and let you guys know.

By the way, could you please finish off (merge or close) all the pending Pull Requests on NMatrix before work starts? That would save a lot of hard work from going to waste. 

>    5. mailto:sciruby-dev+unsub...@googlegroups.com
>    6. https://groups.google.com/d/optout

--

[li...@blackwinter.de]

Hi Will,

I'm the maintainer of the rb-gsl gem [1] which currently depends on
NArray for some of its functionality. And while I'm not involved in
the SciRuby project, I'm watching your GSoC project with high hopes
that it might lead to merging rb-gsl and gsl-nmatrix into one single
project [2], replacing NArray with NMatrix. For this to be feasible,
I'm particularly interested in the "no further dependencies" variant
of NMatrix, just like NArray doesn't have any external dependencies.

As for your proposal, I think it looks good and well thought-out. I
wish you all the best for your GSoC!

[1] <https://rubygems.org/gems/rb-gsl>
[2] <https://github.com/SciRuby/rb-gsl/issues/14>

Cheers,
Jens

[John Woods]

I'm still struggling with not having a machine that can actually compile NMatrix at this time. I'm also dealing with some hard deadlines for work in the next few weeks. However, both Colin and Carlos now have merge permissions, so hopefully some of the work will get done before GSoC actually starts.

I'd like to also do a release before the end of the month.

John

>    5. mailto:sciruby-dev...@googlegroups.com
>    6. https://groups.google.com/d/optout

--

--

You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sameer Deshmukh]

Couple of suggestions:

1. Could you elaborate what measures you'll be taking to help future plug-in writers with writing more plug-ins for nmatrix? Will there be something in the code that will enforce a particular workflow, architecture and style for future plug-ins?

2. NMatrix has long term objectives of supporting multiple libraries that can do the same thing. For example, LAPACK or Accelerate for Linear Algebra. Say I want to leverage the benefits of both by choosing certain functions from each (assuming that there are nmatrix wrappers for both), how can I do that?

3. What interface will you be defining to let users easily load and unload plug-ins (or extensions, whatever you call them)?

4. Magdalen Berns, in previous year's GSOC has made a FFTW interface to nmatrix (https://github.com/thisMagpie/fftw). Currently it requires some effort to install. Could you further demonstrate your nmatrix extension system by incorporating Magdalen's FFTW wrapper as another extension for nmatrix? This would make nmatrix twice as usable and better equipped for the real world. Also this would be a great demo of your nmatrix extension system. I dont think this should be too difficult either because most of the work is already done.

5. You're saying you'll be using LAPACKE. But LAPACKE is quite different from the current combination of CLAPACK/ATLAS. When I submitted my proposal I was told that we should go ahead with CLAPACK/ATLAS first for backward compatibility. Could you make it clear if you're going to replace the current CLAPACK/ATLAS with LAPACKE, or are you going to create two separate extensions, one with LAPACKE and the other with CLAPACK/ATLAS?

6. Last but not the least, you might want to go through my proposal and see if you get any new ideas. Heres the link: http://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2015/v0dro/5673385510043648

If you have any queries please let me know.

Cheers!

[Sameer Deshmukh]

By the way I really liked your idea of keeping multiple gems in one repo without polluting the code.

Guess I'll take some lessons from that :)

On Tuesday, April 28, 2015 at 7:32:51 AM UTC+5:30, Will Levine wrote:

[Will Levine]

Hi Sameer,
Thanks for your comments. You've given me a lot to think about! Some
comments below.

> 1. Could you elaborate what measures you'll be taking to help future plug-in
> writers with writing more plug-ins for nmatrix? Will there be something in
> the code that will enforce a particular workflow, architecture and style for
> future plug-ins?

Certainly my work can serve as a template for other extensions. Though
one issue I just thought of is that my work will be done in the same
repository as the main nmatrix, but there will be additional obstacles
for an extension author who wants to work in their own repository:
they may need access to the nmatrix header files, but rubygems doesn't
provide for installing C header files. I am not sure of an elegant way
to solve this.

> 2. NMatrix has long term objectives of supporting multiple libraries that
> can do the same thing. For example, LAPACK or Accelerate for Linear Algebra.
> Say I want to leverage the benefits of both by choosing certain functions
> from each (assuming that there are nmatrix wrappers for both), how can I do
> that?

It seems like everybody in this discussion thinks that more backends
is better, but I'm not convinced of this. All other things being
equal, it's better to have more choices, but nmatrix is a small
project with limited resources, and for every option you add you pay a
price in terms of making the code more complex and difficult to
maintain.

That said, if we do end up having multiple linear algebra extensions,
I don't really see your scenario (choosing specific functions from
multiple backends) as a common use-case.

> 3. What interface will you be defining to let users easily load and unload
> plug-ins (or extensions, whatever you call them)?

Like above, I'd like to understand the use case for this before I
think about it much. For extensions that provide different
functionality, you can just load both of them like normal. For
extensions that provide similar, or conflicting, functions, just use
one of them. I'm not convinced we need to accommodate unloading.

> 4. Magdalen Berns, in previous year's GSOC has made a FFTW interface to
> nmatrix (https://github.com/thisMagpie/fftw). Currently it requires some
> effort to install. Could you further demonstrate your nmatrix extension
> system by incorporating Magdalen's FFTW wrapper as another extension for
> nmatrix? This would make nmatrix twice as usable and better equipped for the
> real world. Also this would be a great demo of your nmatrix extension
> system. I dont think this should be too difficult either because most of the
> work is already done.

I will take a look at this.

> 5. You're saying you'll be using LAPACKE. But LAPACKE is quite different
> from the current combination of CLAPACK/ATLAS. When I submitted my proposal
> I was told that we should go ahead with CLAPACK/ATLAS first for backward
> compatibility. Could you make it clear if you're going to replace the
> current CLAPACK/ATLAS with LAPACKE, or are you going to create two separate
> extensions, one with LAPACKE and the other with CLAPACK/ATLAS?

My hope is that the transition between ATLAS and LAPACKE will be
seamless and that no one will have to change either their code or
their build environment (I want to emphasize that my LAPACKE
proof-of-concept code still supports ATLAS, it's just that it doesn't
use any ATLAS-specific code). If this is the case then it would be my
preference to remove the current ATLAS code entirely, but I don't
think everyone else is convinced on this yet. I'm sure there will be
more discussion of this.

> 6. Last but not the least, you might want to go through my proposal and see
> if you get any new ideas. Heres the link:
> http://www.google-melange.com/gsoc/proposal/review/student/google/gsoc2015/v0dro/5673385510043648

your link doesn't work for me.

Will

>
> If you have any queries please let me know.
>
> Cheers!
>
> On Tuesday, April 28, 2015 at 7:32:51 AM UTC+5:30, Will Levine wrote:
>>
>> Hi all,
>> I've been accepted to the Google summer of code program to work on
>> SciRuby and nmatrix this summer. I'm a physics PhD student at Carnegie
>> Mellon University. My plan for the summer is to make nmatrix easier to
>> install and use by allowing it to be used with a greater variety of
>> external linear algebra libraries, or, if desired, without any
>> external libraries at all. I've posted my full proposal here:
>> https://github.com/wlevine/nmatrix/wiki/Summer-of-Code-2015-proposal
>>
>> I'll be posting more to the list to get feedback on some of the some
>> of the ideas I'm proposing, so y'all will hear more from me, but for
>> now I just wanted to introduce myself.
>>
>> I look forward to working with everyone!
>>
>> Will
>

[Sameer Deshmukh]

Heres the full proposal:

https://github.com/v0dro/nmatrix/wiki/GSOC-project-proposal

I think its VERY VERY important to make it possible for ANYONE to cook up  an nmatrix plug-in easily. Imagine nmatrix like an iPhone, with apps (plug-ins) that make the real magic happen. I think you should really look into creating a formal interface for that purpose and spend a good amount of time on it. This project wont be worth much if other people cant build on top of it. Maybe make it agnostic to having all extensions in the same repo. Or maybe incorporate only the important ones in the repo.

And about C header files: the C headers of a gem (nmatrix.h in this case) are installed automatically when you run 'gem install nmatrix'. In rvm the path is so: .rvm/gems/ruby-2.2.1/extensions/x86-linux/2.2.0/nmatrix-0.1.0 

On Tuesday, April 28, 2015 at 7:32:51 AM UTC+5:30, Will Levine wrote:

[Sameer Deshmukh]

Ok one more suggestion regarding keeping all the gems in the single repo.

You can actually specify the git repo from which you want to include the gem in your Gemfile. So maybe we can get away with decentralizing repos and keeping each extension in a separate repo.

See this: http://stackoverflow.com/questions/11427922/how-to-specify-path-for-dependent-gem-in-gemspec-file

On Tuesday, April 28, 2015 at 7:32:51 AM UTC+5:30, Will Levine wrote:

[John Woods]

A few comments below.

On Sun, May 10, 2015 at 2:59 PM Will Levine <wle...@gmail.com> wrote:

Hi Sameer,
Thanks for your comments. You've given me a lot to think about! Some
comments below.

> 1. Could you elaborate what measures you'll be taking to help future plug-in
> writers with writing more plug-ins for nmatrix? Will there be something in
> the code that will enforce a particular workflow, architecture and style for
> future plug-ins?

Certainly my work can serve as a template for other extensions. Though
one issue I just thought of is that my work will be done in the same
repository as the main nmatrix, but there will be additional obstacles
for an extension author who wants to work in their own repository:
they may need access to the nmatrix header files, but rubygems doesn't
provide for installing C header files. I am not sure of an elegant way
to solve this.

I agree that it needs to be in the main NMatrix repo.

I'm not sure I necessarily agree that there needs to be a "standard plugin architecture." I want it to be as simple as it possibly can be to include these types of extensions. That means:

require "nmatrix"

require "nmatrix-lapacke"

OR

require "nmatrix"

require "nmatrix/lapacke"

 

> 2. NMatrix has long term objectives of supporting multiple libraries that
> can do the same thing. For example, LAPACK or Accelerate for Linear Algebra.
> Say I want to leverage the benefits of both by choosing certain functions
> from each (assuming that there are nmatrix wrappers for both), how can I do
> that?

It seems like everybody in this discussion thinks that more backends
is better, but I'm not convinced of this. All other things being
equal, it's better to have more choices, but nmatrix is a small
project with limited resources, and for every option you add you pay a
price in terms of making the code more complex and difficult to
maintain.

That said, if we do end up having multiple linear algebra extensions,
I don't really see your scenario (choosing specific functions from
multiple backends) as a common use-case.

I will put in here that I think we need to support ATLAS (at least as much as we currently do) for backwards compatibility unless LAPACKE implements *every single* ATLAS functionality we have in there currently.

 

> 3. What interface will you be defining to let users easily load and unload
> plug-ins (or extensions, whatever you call them)?

Like above, I'd like to understand the use case for this before I
think about it much. For extensions that provide different
functionality, you can just load both of them like normal. For
extensions that provide similar, or conflicting, functions, just use
one of them. I'm not convinced we need to accommodate unloading.

> 4. Magdalen Berns, in previous year's GSOC has made a FFTW interface to
> nmatrix (https://github.com/thisMagpie/fftw). Currently it requires some
> effort to install. Could you further demonstrate your nmatrix extension
> system by incorporating Magdalen's FFTW wrapper as another extension for
> nmatrix? This would make nmatrix twice as usable and better equipped for the
> real world. Also this would be a great demo of your nmatrix extension
> system. I dont think this should be too difficult either because most of the
> work is already done.

I will take a look at this.

Does Magdalen's interface work? I was under the impression it was not complete.

 

> 5. You're saying you'll be using LAPACKE. But LAPACKE is quite different
> from the current combination of CLAPACK/ATLAS. When I submitted my proposal
> I was told that we should go ahead with CLAPACK/ATLAS first for backward
> compatibility. Could you make it clear if you're going to replace the
> current CLAPACK/ATLAS with LAPACKE, or are you going to create two separate
> extensions, one with LAPACKE and the other with CLAPACK/ATLAS?

My hope is that the transition between ATLAS and LAPACKE will be
seamless and that no one will have to change either their code or
their build environment (I want to emphasize that my LAPACKE
proof-of-concept code still supports ATLAS, it's just that it doesn't
use any ATLAS-specific code). If this is the case then it would be my
preference to remove the current ATLAS code entirely, but I don't
think everyone else is convinced on this yet. I'm sure there will be
more discussion of this.

Good. =)

[Will Levine]

Hi all,
I finally got around to starting a blog:
https://wlevine.github.io
John invited me to add some of my posts to the main SciRuby blog, so I
will do that, but some of my posts will be more log-like and more
appropriate for my personal blog, so I just wanted to share the link
here in case anyone was interested.

Will

[Yoong Kang Lim]

How's it going with this? It would be really great if it becomes straightforward to get NMatrix to use OpenBLAS instead of ATLAS.

[Will Levine]

OpenBLAS support is still part of the plan. I wouldn't expect it to be
done until August.