Simultaneous update of arbitrary elements in core.matrix

[Mars0i]

I'd like to modify an arbitrary number of elements in a core.matrix matrix (in vectorz-clj) at one time.  I can go through and make the changes individually with 'mset', but changing, say, 30 elements in a 300x300 matrix using 'mset' seems like it's going to do a lot of unnecessary copying.  (Is that correct? mset copies the rest of the matrix?)   Or I can use 'mset!', which would be efficient, but I thought I would try to do it functionally, and then benchmark the functional version against a mutating version.

It looks like 'set-indices' does what I want:

-------------------------
clojure.core.matrix/set-indices
([a indices values])
  like select-indices but sets the elements at the specified indices to values.
   Leaves a unchanged and returns a modified array

clojure.core.matrix/select-indices
([a indices])
  returns a one-dimensional array of the elements which are at the specified
   indices. An index is a one-dimensional array which element-count matches the
   dimensionality of a. Examples:
   (select-indices [[1 2] [3 4]] [[0 0][1 1]]) ;=> [1 4]

This seems to work on small matrices with ndarray, but not vectorz-clj at the moment:

(set-indices (matrix :ndarray [[1 2][3 4]])  [[0 1][1 0]]  [10 20])
#<NDArray [[1 10] [20 4]]>

(set-indices (matrix :vectorz [[1 2][3 4]]) [[0 1][1 0]] [10 20])
RuntimeException Can't do 2D set on nil  clojure.core.matrix.impl.default/eval5591/fn--5594 (default.clj:204)

Although set-indices! works on vectorz-clj:

(set-indices! (matrix :vectorz [[1 2][3 4]]) [[0 1][1 0]] [10 20])
#<Matrix22 [[1.0,10.0],[20.0,4.0]]>

But my real question is about the syntax of set-indices and set-indices! for larger matrices.  Maybe there's a bug, but I suspect that I just don't understand.

(set-indices (matrix :ndarray [[1 2][3 4][5 6]])  [[0 1][1 0][2 1]]  [10 20 30])
RuntimeException Incompatible shapes, cannot broadcast [3] to [3 2]  clojure.core.matrix.impl.persistent-vector/eval4676/fn--4677 (persistent_vector.clj:156)

popco.core.popco=> (set-indices (matrix :ndarray [[1 2][3 4][5 6]])  [[0 1][1 0][2 1]]  [[10 20] [30 40] [50 60]])
#<NDArray [[1 10] [20 4] [5 30]]>


Is set-indices part of the recent selector functionality, and is in flux?

Thanks.

[Mars0i]

On Thursday, April 24, 2014 8:38:52 PM UTC-5, Mars0i wrote:

(Is that correct? mset copies the rest of the matrix?)

Sorry, should have this one investigated myself before asking.  For vectorz-clj, it appears: Yes. A brand new matrix is created, and is then mutated in one place.  Something similar usually happens in persistent-vector and ndarray.  (So I don't want to use mset to create a matrix modified in 150 places from a prior matrix.)
(I have been unsuccessful so far finding the answers to my other questions.)

[Mike Anderson]

Yeah mset isn't generally a good way to do large numbers of updates to a matrix. It's possible that matrcies implemented with persistent data structures could allow O(log N) updates, but otherwise it will be O(n).

For the moment, using mset! with mutable arrays is the recommended way to do lots of small / scattered updates.

The set / select indices functionality is still under development, we haven't quite finalised the API. Please consider it somewhat experimental at the moment. 

It's non-trivial to get this right for a number of reasons:

- It needs to be possible to implement efficiently

- It needs to be sane for arbitrary dimension arrays

- It needs to line up with other parts of the API, e.g. Maik's selector functionality and other index-handling functionality

[Maik Schünemann]

That seems like a bug in the set-indices functionality, I'll investigate it today. Thanks for reporting.

Otherwise your assumptions about how it should work are correct.

--
You received this message because you are subscribed to the Google Groups "Numerical Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to numerical-cloj...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Maik Schünemann]

Okay, I pushed  a PR to fix this issue

On Fri, Apr 25, 2014 at 3:38 AM, Mars0i <mars...@logical.net> wrote:

I'd like to modify an arbitrary number of elements in a core.matrix matrix (in vectorz-clj) at one time.  I can go through and make the changes individually with 'mset', but changing, say, 30 elements in a 300x300 matrix using 'mset' seems like it's going to do a lot of unnecessary copying.  (Is that correct? mset copies the rest of the matrix?)   Or I can use 'mset!', which would be efficient, but I thought I would try to do it functionally, and then benchmark the functional version against a mutating version.

It looks like 'set-indices' does what I want:

-------------------------
clojure.core.matrix/set-indices
([a indices values])
  like select-indices but sets the elements at the specified indices to values.
   Leaves a unchanged and returns a modified array

clojure.core.matrix/select-indices
([a indices])
  returns a one-dimensional array of the elements which are at the specified
   indices. An index is a one-dimensional array which element-count matches the
   dimensionality of a. Examples:
   (select-indices [[1 2] [3 4]] [[0 0][1 1]]) ;=> [1 4]

This seems to work on small matrices with ndarray, but not vectorz-clj at the moment:

(set-indices (matrix :ndarray [[1 2][3 4]])  [[0 1][1 0]]  [10 20])
#<NDArray [[1 10] [20 4]]>

(set-indices (matrix :vectorz [[1 2][3 4]]) [[0 1][1 0]] [10 20])
RuntimeException Can't do 2D set on nil  clojure.core.matrix.impl.default/eval5591/fn--5594 (default.clj:204)

This fails because vectorz returns nil on mp/set-nd and doesn't fall back to set-2d here.

What is the contract for the set-nd implementation if the underlying implementation does only support 1 and 2 dimensional matrices? 

Should if fall back to set-1d and set-2d where possible or should the default implementation make sure only to call mp/set-nd in the case for more than two dimensions? 

Although set-indices! works on vectorz-clj:

(set-indices! (matrix :vectorz [[1 2][3 4]]) [[0 1][1 0]] [10 20])
#<Matrix22 [[1.0,10.0],[20.0,4.0]]>

But my real question is about the syntax of set-indices and set-indices! for larger matrices.  Maybe there's a bug, but I suspect that I just don't understand.

(set-indices (matrix :ndarray [[1 2][3 4][5 6]])  [[0 1][1 0][2 1]]  [10 20 30])
RuntimeException Incompatible shapes, cannot broadcast [3] to [3 2]  clojure.core.matrix.impl.persistent-vector/eval4676/fn--4677 (persistent_vector.clj:156)

That works now. 

popco.core.popco=> (set-indices (matrix :ndarray [[1 2][3 4][5 6]])  [[0 1][1 0][2 1]]  [[10 20] [30 40] [50 60]])
#<NDArray [[1 10] [20 4] [5 30]]>


Is set-indices part of the recent selector functionality, and is in flux?

Thanks.

[Mike Anderson]

On Friday, 25 April 2014 09:29:24 UTC+1, Maik Schünemann wrote:

Okay, I pushed  a PR to fix this issue

On Fri, Apr 25, 2014 at 3:38 AM, Mars0i <mars...@logical.net> wrote:

I'd like to modify an arbitrary number of elements in a core.matrix matrix (in vectorz-clj) at one time.  I can go through and make the changes individually with 'mset', but changing, say, 30 elements in a 300x300 matrix using 'mset' seems like it's going to do a lot of unnecessary copying.  (Is that correct? mset copies the rest of the matrix?)   Or I can use 'mset!', which would be efficient, but I thought I would try to do it functionally, and then benchmark the functional version against a mutating version.

It looks like 'set-indices' does what I want:

-------------------------
clojure.core.matrix/set-indices
([a indices values])
  like select-indices but sets the elements at the specified indices to values.
   Leaves a unchanged and returns a modified array

clojure.core.matrix/select-indices
([a indices])
  returns a one-dimensional array of the elements which are at the specified
   indices. An index is a one-dimensional array which element-count matches the
   dimensionality of a. Examples:
   (select-indices [[1 2] [3 4]] [[0 0][1 1]]) ;=> [1 4]

This seems to work on small matrices with ndarray, but not vectorz-clj at the moment:

(set-indices (matrix :ndarray [[1 2][3 4]])  [[0 1][1 0]]  [10 20])
#<NDArray [[1 10] [20 4]]>

(set-indices (matrix :vectorz [[1 2][3 4]]) [[0 1][1 0]] [10 20])
RuntimeException Can't do 2D set on nil  clojure.core.matrix.impl.default/eval5591/fn--5594 (default.clj:204)

This fails because vectorz returns nil on mp/set-nd and doesn't fall back to set-2d here.

What is the contract for the set-nd implementation if the underlying implementation does only support 1 and 2 dimensional matrices? 

Should if fall back to set-1d and set-2d where possible or should the default implementation make sure only to call mp/set-nd in the case for more than two dimensions? 

set-nd should fall back to set-1d / set-2d where appropriate. You can thing of set-1d and set-2d as specialised versions of set-nd. So set-nd should work for any 1 and 2 dimensional arrays providing the implementation supports them.

set-1d and set-2d are are mainly useful for better performance - they should be much more efficient than wrapping the indices in a collection whenever you call the function). It would be pretty wasteful if "(mset some-vector i value)" needed to wrap "i" in a persistent vector, for example.....

 

Although set-indices! works on vectorz-clj:

(set-indices! (matrix :vectorz [[1 2][3 4]]) [[0 1][1 0]] [10 20])
#<Matrix22 [[1.0,10.0],[20.0,4.0]]>

But my real question is about the syntax of set-indices and set-indices! for larger matrices.  Maybe there's a bug, but I suspect that I just don't understand.

(set-indices (matrix :ndarray [[1 2][3 4][5 6]])  [[0 1][1 0][2 1]]  [10 20 30])
RuntimeException Incompatible shapes, cannot broadcast [3] to [3 2]  clojure.core.matrix.impl.persistent-vector/eval4676/fn--4677 (persistent_vector.clj:156)

That works now. 

popco.core.popco=> (set-indices (matrix :ndarray [[1 2][3 4][5 6]])  [[0 1][1 0][2 1]]  [[10 20] [30 40] [50 60]])
#<NDArray [[1 10] [20 4] [5 30]]>


Is set-indices part of the recent selector functionality, and is in flux?

Thanks.

--
You received this message because you are subscribed to the Google Groups "Numerical Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to numerical-clojure+unsubscribe@googlegroups.com.

[Mars0i]

Thanks Mike, Maik.

After thinking about the fact that many of the functional update functions in core.matrix are in effect mutating update functions applied to a clone of the original matrix, it became obvious that I can just clone and then call mset! as many times as I want, if I want a functional multi-element update.  Sometimes it takes time to see the obvious.  (And benchmarking the number of times I'd have to clone a matrix showed that the cost is small.)

[Maik Schünemann]

Hi, 

On Fri, Apr 25, 2014 at 4:41 PM, Mars0i <mars...@logical.net> wrote:

Thanks Mike, Maik.

After thinking about the fact that many of the functional update functions in core.matrix are in effect mutating update functions applied to a clone of the original matrix, it became obvious that I can just clone and then call mset! as many times as I want, if I want a functional multi-element update.  Sometimes it takes time to see the obvious.  (And benchmarking the number of times I'd have to clone a matrix showed that the cost is small.)

I think that we should review the default implementations and see what implementations uses loop or recur with functional updates functions instead of cloning and using the destructive operations. And rewrite them to the second approach.

I'll also update the default implementation of set-selection to use the second approach calling set-indices is the same as cloning and mset! ing for the default implementation and optimally faster for certain implementations.

--
You received this message because you are subscribed to the Google Groups "Numerical Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to numerical-cloj...@googlegroups.com.

[Mike Anderson]

On Friday, 25 April 2014 16:17:27 UTC+1, Maik Schünemann wrote:

Hi, 

On Fri, Apr 25, 2014 at 4:41 PM, Mars0i <mars...@logical.net> wrote:

Thanks Mike, Maik.

After thinking about the fact that many of the functional update functions in core.matrix are in effect mutating update functions applied to a clone of the original matrix, it became obvious that I can just clone and then call mset! as many times as I want, if I want a functional multi-element update.  Sometimes it takes time to see the obvious.  (And benchmarking the number of times I'd have to clone a matrix showed that the cost is small.)

I think that we should review the default implementations and see what implementations uses loop or recur with functional updates functions instead of cloning and using the destructive operations. And rewrite them to the second approach.

Improving default implementations is always welcome! There are probably quite a few different ways they can be improved with a bit of thought. The only thing to remember is that you can't assume very much.... they must work with anything that satisfies the minimal set of mandatory protocols.

Though I hope there are not many that are *quite* as naive as to do a lot of functional updates in a loop :-)

 

I'll also update the default implementation of set-selection to use the second approach calling set-indices is the same as cloning and mset! ing for the default implementation and optimally faster for certain implementations.

--
You received this message because you are subscribed to the Google Groups "Numerical Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to numerical-clojure+unsubscribe@googlegroups.com.