Redefining Breeze's basic vector operations to perform replication (like in R)

[Martin Studer]

Hi,

I'm trying to redefine Breeze's basic arithmetic operations to perform vector replication like in R. I.e. given two operands in an operation such as addition, subtraction, multiplication or division, the resulting vector will have the length of the longest operand - shorter operands will be replicated to match the length of the longer operand. The way I tried to do this is by defining the following implicit (initial naive implementation):

import breeze.math.PowImplicits._

  @expand

  @expand.valify

  implicit def v_v_Op[@expand.args(Int, Double) T,

  @expand.args(OpAdd, OpSub, OpDiv, OpSet, OpMod, OpPow) Op <: OpType]

  (implicit @expand.sequence[Op]({_ + _}, {_ - _}, {_ / _}, {(a, b) => b}, {_ % _}, {_ pow _})

  op: Op.Impl2[T, T, T]): Op.Impl2[Vector[T], Vector[T], Vector[T]] = {

    new Op.Impl2[Vector[T], Vector[T], Vector[T]] {

      def apply(a: Vector[T], b: Vector[T]): Vector[T] = {

        (a.length, b.length) match {

          case (0, _) =>

            a

          case (_, 0) =>

            b

          case (1, bl) =>

            val s = a(0)

            val result = Vector.zeros[T](bl)

            var i = 0

            while (i < bl) {

              result(i) = op(s, b(i))

              i += 1

            }

            result

          case (al, 1) =>

            val s = b(0)

            val result = Vector.zeros[T](al)

            var i = 0

            while (i < al) {

              result(i) = op(a(i), s)

              i += 1

            }

            result

          case (al, bl) if al == bl =>

            val result = a.copy

            var i = 0

            while (i < al) {

              result(i) = op(a(i), b(i))

              i += 1

            }

            result

          case (al, bl) if al < bl =>

            val result = Vector.zeros[T](bl)

            var i = 0

            while (i < bl) {

              result(i) = op(a(i % al), b(i))

              i += 1

            }

            result

          case (al, bl) =>

            val result = Vector.zeros[T](al)

            var i = 0

            while (i < al) {

              result(i) = op(a(i), b(i % bl))

              i += 1

            }

            result

        }

      }

      implicitly[BinaryRegistry[Vector[T], Vector[T], Op.type, Vector[T]]].register(this)

    }

  }

With the above implicit in scope, however, Breeze will still use its built-in implicit and a such result in a "java.lang.IllegalArgumentException: requirement failed: Vectors must be the same length!"

What puzzles me is that with a similar (more concrete) definition for OpAdd on Semirings I seem to indeed get the desired behavior:

implicit def opAdd[T : ClassTag : Semiring]: OpAdd.Impl2[Vector[T], Vector[T], Vector[T]] =

    new OpAdd.Impl2[Vector[T], Vector[T], Vector[T]] {

      val r = implicitly[Semiring[T]]

      def apply(a: Vector[T], b: Vector[T]): Vector[T] = { ... }

    }

However, I would ideally like to use the @expand macros to come up with one definition for a whole set of arithmetic operations.

It seems like I am missing something. Are there other implicits that will need to be "masked" with the macro version above? Is "masking" implicits even the suggested approach or is there a better alternative?

Thanks & Best regards,

Martin

[David Hall]

First, I consider R's behavior an abomination, but what you do in the privacy of your own repo is not my business. :-)

I guess my first question is whether or not you're using things that are statically DenseVectors, or statically Vectors, in the code using it? If statically DenseVectors, then it's going to prefer those operators to your new Vector operators.

If statically vectors, Breeze uses a multimethod BinaryRegistries so that (dv: Vector[Double]) + (dv: Vector[Double]) will use the faster Dense/Dense operator, even though statically they're Vectors. I don't think that should be happening here, but it's my best explanation.

If you can publish something that reproduces, I can take a look.

-- David

--
You received this message because you are subscribed to the Google Groups "Scala Breeze" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-breeze...@googlegroups.com.
To post to this group, send email to scala-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/scala-breeze/359a8a53-8fc5-4787-a93c-9dc598e61f06%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Martin Studer]

Hi David,

thanks for your feedback. I agree that R's behavior takes getting used to ;-) I'm trying to implement an R interpreter in Scala which is backed by Breeze for numerical things - so I want to mimic R's replication "abomination".

With respect to your question: Statically I have Vectors and I intend to use both DenseVectors and SparseVectors going forward (for now just DenseVectors for my tests).

I put together a small reproducible example (sbt script runner script; see attachment) which shows the essence of what I'm trying to do. Any insight you can provide on how I could get the 3rd case running is very much appreciated!

BTW: While playing around with the macros I also ran into some interesting AssertionErrors during compilation for which I started a thread at scala-internals: https://groups.google.com/forum/#!topic/scala-internals/gnvcWsmAPaY

Thanks & Best regards,

Martin

[David Hall]

cool, I'll take a look soon.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-breeze/4320b640-77fb-4e48-80d0-66a9cb013ad2%40googlegroups.com.

[David Hall]

Hrm, it works for me. Pasting your exact code into a suitably named object and importing it, I get:

scala> import breeze.linalg.Abomination._

import breeze.linalg.Abomination._

scala> val v = Vector(1.0, 2.0, 3.0)

v: scala.collection.immutable.Vector[Double] = Vector(1.0, 2.0, 3.0)

scala> import breeze.linalg._

import breeze.linalg._

scala> val v = Vector(1.0, 2.0, 3.0)

v: breeze.linalg.Vector[Double] = DenseVector(1.0, 2.0, 3.0)

scala> val v2 = Vector(4.0)

v2: breeze.linalg.Vector[Double] = DenseVector(4.0)

scala> v + v2

res0: breeze.linalg.Vector[Double] = DenseVector(5.0, 6.0, 7.0)

scala>

[Martin Studer]

Having both implicits in scope these worked for me as well. However, the following didn't:

def plus[T : ClassTag : Semiring](v1: Vector[T], v2: Vector[T]): Vector[T] = v1 + v2

plus(v, v2)

[David Hall]

I see, so you want to dynamic override the default implicits in all scopes, and not just place where you explicitly import them

--
You received this message because you are subscribed to the Google Groups "Scala Breeze" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-breeze...@googlegroups.com.
To post to this group, send email to scala-...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/scala-breeze/a4153831-c67b-4c4a-ba44-baa68209be1f%40googlegroups.com.

[David Hall]

actually, wait, are you asking for 

def plus[T : ClassTag : Semiring](v1: Vector[T], v2: Vector[T]): Vector[T] = v1 + v2

to pick up the implicits you defined for Vector[Int] etc?

Implicits don't work that way. You defined the method for any type T with a Semiring and a ClassTag, which means that it has to use the slow-path Semiring operators since, statically, the implicit resolution mechanisms doesn't know that T is an Int, or whatever.

If you want that, you instead need to explicitly ask for implicits you intend to use:

def plus[T](v1: Vector[T], v2: Vector[T])(implicit add: OpAdd.Impl2[Vector[T], Vector[T], Vector[T]]): Vector[T] = v1 + v2

Or you can make a VectorSpace (or equivalent) operator bundle, and import that into scope:

def plus[T](v1: Vector[T], v2: Vector[T])(implicit ops: Module[Vector[T], T]): Vector[T] = { import ops._; v1 + v2 }

[Martin Studer]

Oh, right! I expected it to pick up my implicits but it obviously won't due to the Semiring as you just explained. I think

def plus[T](v1: Vector[T], v2: Vector[T])(implicit add: OpAdd.Impl2[Vector[T], Vector[T], Vector[T]]): Vector[T] = v1 + v2

is exactly what I'm looking for.

Thanks for pointing that one out!