Thanks for core.matrix!

[Rich Hickey]

I finally got to dig into core.matrix this weekend and found it a real pleasure.

Thanks very much to Mike et al for all the hard work that went into it. I really appreciate it.

Rich

[Mike Anderson]

Thanks very much for the kind words Rich!

Of course this has only been feasible because of the brilliant design of Clojure protocols :-)

We genuinely have a great story for array programming in Clojure: to my knowledge core.matrix is the only example of a high-performance n-dimensional matrix library for any language that allows open extension to arbitrary backend implementations without changing the API. This has been extremely useful in my data science work.

I'm wondering if it might be a good time to officially move core.matrix into Clojure contrib? The API has be mostly stable for a couple of years: recent changes have mostly been new function additions and occasionally needing to resolve some corner cases (e.g. around the complexity of specifying exactly which type of parameters are allowed for certain API functions).

[Rich Hickey]

Yes, we should definitely start working on getting this into contrib (once Alex has recovered from Strange Loop :).

One API question:

I couldn't find mutating ops that took a target (vs in-place on one of the operands). E.g. this is common in DSP libs like:

https://developer.apple.com/reference/accelerate/1450344-vdsp_vmul

and

https://developer.apple.com/reference/accelerate/1450665-vdsp_vma

from:

https://developer.apple.com/reference/accelerate/1652565-vdsp

something like assign-product!, assign-product-and-sum!

Without it, my options were:

a) temporaries and assign!

b) zeroing out my register vectors with fill! prior to using add-product!

c) most efficiently, using refs and rotating around the semantics of my registers, combined with add-product! This introduced a good bit of complexity and might not be a good fit for the GPU (I'm very keen to try vectorz-opencl).

d) something I missed :)

I'm not sure how well this target-vector approach aligns with the underlying protocol, but if it's not a conflict it might be something to consider.

Thanks again.

[Mike Anderson]

On Monday, 19 September 2016 21:00:26 UTC+8, Rich Hickey wrote:

Yes, we should definitely start working on getting this into contrib (once Alex has recovered from Strange Loop :).

Cool. No particular hurry, but I think would be good to do sometime soon.

 

One API question:

I couldn't find mutating ops that took a target (vs in-place on one of the operands). E.g. this is common in DSP libs like:

https://developer.apple.com/reference/accelerate/1450344-vdsp_vmul

and

https://developer.apple.com/reference/accelerate/1450665-vdsp_vma

from:

https://developer.apple.com/reference/accelerate/1652565-vdsp

something like assign-product!, assign-product-and-sum!

There are a few operations like this, e.g. set-inner-product!, add-outer-product! or the various functions in clojure.core.matrix.blas which emulate the equivalent BLAS operations. No fundamental objection to these, though I've tended to include them only in cases where there is a concrete use case to avoid explosion of API functions and/or arity overrides. 

Theoretically any operation has three possible variants:

a) Creating a new array for the result (implies allocation overhead, but plays nicely with immutable arguments)

b) Updating one of the arguments in-place with the result (the common case for accumulators etc.)

c) Putting the result in a separate mutable destination array (arguably the most general operation, but requires an extra argument for everything)

The challenge is that c) is complicated by other factors - e.g. do you want to add to the destination array or overwrite whatever is there? This seems to be fairly use case dependent, e.g. for neural networks we do a lot of stuff with accumulators so adding to the destination is often the most natural operation. 

 

Without it, my options were:

a) temporaries and assign!

b) zeroing out my register vectors with fill! prior to using add-product!

c) most efficiently, using refs and rotating around the semantics of my registers, combined with add-product! This introduced a good bit of complexity and might not be a good fit for the GPU (I'm very keen to try vectorz-opencl).

d) something I missed :)

I find the following pattern to be quite common:

(assign! dest a)

(mul! dest b)

There is a bit of extra overhead of copying with the first assign, but this is essentially equivalent to (set-product! dest a b). It also nicely handles the cases where dest, a and b are all from different implementations, which I find is quite common in practice (e.g. you will often want to use something like vectorz-clj for mutable destination arrays, but immutable Clojure vectors or Java lists for input data).

 

I'm not sure how well this target-vector approach aligns with the underlying protocol, but if it's not a conflict it might be something to consider.

I don't think it is a fundamental conflict. We could add a whole bunch of operations like add-xxx! and set-xxx! / assign-xxx!. The trade-off is that it adds API complexity so I've avoided doing this so far for the majority of operations. Also each permutation seems like it would need a separate protocol in order for implementations to make these work efficiently (unless there is some implementation trick I have missed?).

As always, I would be most persuaded by concrete use cases. 

[Alex Miller]

On Monday, September 19, 2016 at 11:05:10 AM UTC-5, Mike Anderson wrote:

On Monday, 19 September 2016 21:00:26 UTC+8, Rich Hickey wrote:

Yes, we should definitely start working on getting this into contrib (once Alex has recovered from Strange Loop :).

Cool. No particular hurry, but I think would be good to do sometime soon.

Hey Mike, Rich pinged me about this and I'd be happy to put some work in on it.

Some things that need to be done:

- CA verification - me

- Offer of source on clojure-dev - contributors

- Moving the repo to org.clojure (several ways this could be done) - ?

- Integration into Clojure CI - me

- Create JIRA project - me

I can start working on some of these. Curious about your thoughts on the repo move - could be moved via just uploading a git repo, forked, actually transferred, etc. I haven't done one like this for contrib in the past.

Alex

[Mike Anderson]

On Wednesday, 21 September 2016 02:30:51 UTC+8, Alex Miller wrote:

On Monday, September 19, 2016 at 11:05:10 AM UTC-5, Mike Anderson wrote:

On Monday, 19 September 2016 21:00:26 UTC+8, Rich Hickey wrote:

Yes, we should definitely start working on getting this into contrib (once Alex has recovered from Strange Loop :).

Cool. No particular hurry, but I think would be good to do sometime soon.

Hey Mike, Rich pinged me about this and I'd be happy to put some work in on it.

Some things that need to be done:

- CA verification - me

- Offer of source on clojure-dev - contributors

This sounds fine, I have asked contributors to indicate their willingness to contribute the code to contrib in the past in anticipation of this requirement. All have seemed agreeable.

Just one question - if we can't contact the contributor for any reason, what is the necessary step? Remove the relevant code and rewrite?

 

- Moving the repo to org.clojure (several ways this could be done) - ?

- Integration into Clojure CI - me

- Create JIRA project - me 

I can start working on some of these. Curious about your thoughts on the repo move - could be moved via just uploading a git repo, forked, actually transferred, etc. I haven't done one like this for contrib in the past.

Not done one either, but I think transferring the repo itself to clojure.org is fine as long as a link is maintained at the current location?

This seems to be the case according to: https://help.github.com/articles/about-repository-transfers/

One point I would like to confirm: as maintainer would I still have the capability to make releases at any time? We are using core.matrix in production/client code so it is very important to me to be able to get releases out fast (especially bugfixes) without any unnecessary process delays.

 

[Alex Miller]

On Tuesday, September 20, 2016 at 11:19:18 PM UTC-5, Mike Anderson wrote:

On Wednesday, 21 September 2016 02:30:51 UTC+8, Alex Miller wrote:

On Monday, September 19, 2016 at 11:05:10 AM UTC-5, Mike Anderson wrote:

On Monday, 19 September 2016 21:00:26 UTC+8, Rich Hickey wrote:

Yes, we should definitely start working on getting this into contrib (once Alex has recovered from Strange Loop :).

Cool. No particular hurry, but I think would be good to do sometime soon.

Hey Mike, Rich pinged me about this and I'd be happy to put some work in on it.

Some things that need to be done:

- CA verification - me

- Offer of source on clojure-dev - contributors

This sounds fine, I have asked contributors to indicate their willingness to contribute the code to contrib in the past in anticipation of this requirement. All have seemed agreeable.

Just one question - if we can't contact the contributor for any reason, what is the necessary step? Remove the relevant code and rewrite?

I've reviewed the existing contributors list and while most have signed the CA, not all have. I have sent requests to a few people and have at least one that is not able to sign the CA and a couple that I have no contact info.

If we cannot get a signed CA then there contributions will need to be removed. I'll be in touch off-list about this if necessary.

 

- Moving the repo to org.clojure (several ways this could be done) - ?

- Integration into Clojure CI - me

- Create JIRA project - me 

I can start working on some of these. Curious about your thoughts on the repo move - could be moved via just uploading a git repo, forked, actually transferred, etc. I haven't done one like this for contrib in the past.

Not done one either, but I think transferring the repo itself to clojure.org is fine as long as a link is maintained at the current location? 

This seems to be the case according to: https://help.github.com/articles/about-repository-transfers/

Yep, that is my understanding.

 

One point I would like to confirm: as maintainer would I still have the capability to make releases at any time? We are using core.matrix in production/client code so it is very important to me to be able to get releases out fast (especially bugfixes) without any unnecessary process delays.

Yes, you'll have an account on the Clojure CI Hudson system and can initiate a release anytime.