Breeze Survey Followup

227 views
Skip to first unread message

David Hall

unread,
Apr 25, 2013, 2:07:45 PM4/25/13
to scala-...@googlegroups.com
Hi everyone,

So, I got 14 responses to the user survey (Thanks!), 12 of whom use
Breeze, so lower than I expected. (I expected in the neighborhood of
30 to 40. Oh well.)

A few usage observations:

* Most everyone uses it for research and/or hobbies. A few people use
it for commercial development; one brave soul uses it in production.
* Almost everyone has used Scala for < 2 years. Breeze/Scalala use is
kind of scattered all over the 0 to 1 year range.
* Almost everyone uses the linear algebra component. Roughly a third
of people use each of optimization, distributions, and plotting.
* People often have to go back to other tools for completeness, which
is expected at this point.

Requests:

* Everyone wants more documentation, especially API documentation and
examples/tutorials.
* The most requested code-related feature is FFT stuff. Second is GPU
acceleration. Third is better sparse matrix support.

In response to the request for documentation, I've made an effort to
add more and do a better job communicating what documentation there
is.

The main wiki page (https://github.com/scalanlp/breeze/wiki) now links
to the API documentation (which is reasonably current now), and I
started a new User Guide
(https://github.com/scalanlp/breeze/wiki/UserGuide), which includes an
overview of the various tensor types, UFuncs, URFuncs, and two
mini-tutorials on writing generic functions (over Vector-type things)
and defining your own operators.

Several people offered to help, but I foolishly didn't ask for contact
info. Please step forward if you want to contribute. Pretty much
anything is appreciated! In particular, I am more than happy to help
get everyone comfortable with how pieces fit together. (For example,
if you want to make a new matrix type, or add a new capability, I
would be happy to walk you through adding all the various implicits or
whatever else.)

David Hall

unread,
Apr 30, 2013, 12:58:46 PM4/30/13
to scala-...@googlegroups.com
great thanks!

I don't know that there's anything special that needs to be done with
the ffts in terms of Breeze integration. These functions (or I guess
probably this API with maybe JTransform backends) would be fine?

Maybe these go in a new package object? breeze.signal? breeze.dsp?

Thoughts?

-- David

On Tue, Apr 30, 2013 at 8:34 AM, René Treffer <rtre...@gmail.com> wrote:
> Hi,
>
> I'm currently using 2 small hacked function for FFT/wavelet because I had no
> idea how to add them to breeze. I'll read the docs soon. jtransforms looks
> like a good candidate for fft.
>
> import breeze.linalg._
> import breeze.math._
> import edu.emory.mathcs.jtransforms.fft._
>
> /**
> * Fast fourier transform via jtransforms.
> */
> def fft(v : DenseVector[Double]) : DenseVector[Double] = {
> val fft_instance = new DoubleFFT_1D(v.size)
> val d = v.data.clone
> fft_instance.realForward(d)
> DenseVector[Double](d)
> }
>
>
> sbt: "com.github.rwl" % "jtransforms" % "2.4.0"
>
> The most simple haar transforms:
>
> private def padOrCopy(v : DenseVector[Double]) = {
> if ((v.length & -v.length) == v.length) {
> v.copy
> } else {
> val length = 1 << (32 - Integer.numberOfLeadingZeros(v.length))
> val r = new DenseVector(new Array[Double](length))
> r.slice(0, v.length) := v
> r
> }
> }
>
> /**
> * Normalized Fast Walsh–Hadamard transform.
> * @see
> https://en.wikipedia.org/wiki/Fast_Walsh%E2%80%93Hadamard_transform
> */
> def fwht(v : DenseVector[Double]) : DenseVector[Double] = {
> def _fwht(v : DenseVector[Double]) : DenseVector[Double] = {
> v.length match {
> case 0 => v
> case 1 => v
> case _ => {
> val high = v.slice(0, v.length / 2)
> val low = v.slice(v.length / 2, v.length)
> val h = _fwht(high + low)
> val l = _fwht(high - low)
> high := h
> low := l
> v
> }
> }
> }
>
> val r = _fwht(padOrCopy(v))
>
> val l = 31 - Integer.numberOfLeadingZeros(r.length)
> if (l % 2 == 0) {
> r :*= 1d / (1 << (l / 2))
> } else {
> r :*= 1d / (Math.sqrt(2) * (1 << (l / 2)))
> }
> }
>
> /**
> * Fast haar transform.
> * @see https://en.wikipedia.org/wiki/Haar_wavelet
> */
> def fht(v : DenseVector[Double]) : DenseVector[Double] = {
> def _fht(v : DenseVector[Double]) : DenseVector[Double] = {
> if (v.length > 1) {
> val p = v.toArray.grouped(2).toList
> v.slice(0, v.length / 2) := _fht(new DenseVector(p.map(e =>
> (e(0) + e(1)) / 2).toArray))
> v.slice(v.length / 2, v.length) := new DenseVector(p.map(e
> => (e(0) - e(1)) / 2).toArray)
> }
> v
> }
>
> _fht(padOrCopy(v))
> }
>
> I'm also not sure how numpy etc. pad the date (leading or trailing).
> --
> You received this message because you are subscribed to the Google Groups
> "Scala Breeze" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scala-breeze...@googlegroups.com.
> To post to this group, send email to scala-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msg/scala-breeze/-/fc-0VYlGAP8J.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

David Hall

unread,
May 5, 2013, 12:29:59 PM5/5/13
to scala-...@googlegroups.com
So, this looks basically good!

My main comment is that I think we can just put breeze.signal as a new
package in the math subproject, rather than its own new project.
JTransforms is a pretty small library (500KB), so it shouldn't matter
much in terms of dependency bloat. (Unless there are objections?)

The style/implementation is mostly fine. Don't use v.data directly on
input vectors, because they might have a non-zero offset or non-unit
stride. (i.e. they might be a slice of a matrix or something.) Vectors
you create you have control over and so you can access their data
directly.

Also, you can use slices to avoid this loop:

val tempArr = new Array[Double](v.length*2)
for (n <- 0 to v.length) {
tempArr(n*2) = v.data(n)
//tempIn(n*2 + 1) = 0 ...not necessary, arrays initialized to zero
}

val tempArr = DenseVector.zeros[Double](v.length * 2)
tempArr(0 until tempArr.length by 2) := v

// then do the fft on tempArr.data

As for going forward, I'd happily take this code as a PR that goes
into breeze-math. We can't accept code that is directly based on
implementation details of GNU Octave or other GPL/LGPL code, since
Breeze is Apache-licensed. We can depend on LGPL code (hence
JTransforms is ok). Copying APIs is apparently fine, given the Oracle
v. Google ruling (IANAL, and it could still be overturned), however.

Thanks!

-- David


On Sun, May 5, 2013 at 4:03 AM, ktakagaki <kentaroh...@gmail.com> wrote:
> Hi everyone,
>
> I'm a scientific end-user, and I'm migrating from Java to Scala for my
> backend. I especially need some equivalents to the Matlab DSP toolbox, like
> fft, ifft, filt, etc. ScalaLab looks promising, but just personally, it
> seems a bit confusing and perhaps too complex (just one matrix library is
> enough for me, and having 0-indexed and 1-indexed matrix classes in the same
> package is a bit scary). The small contributing community also worries me.
>
> I would really like to contribute to breeze, but I am very new to Scala (-3
> months, I am doing the Odersky Coursera course now). This weekend, as a
> Scala exercise, I rigged up a simple "breeze.signal":
> https://github.com/ktakagaki/breezeTemp.git
>
> Could somebody with experience be so kind as to take a peek, and see if the
> structure of what I put in place might be of any use? I am very new to
> Scala, and the overall project design (the handling of Types, for instance),
> goes a bit over my head at this point. If what I set up has potential for
> breeze, I would expand, document, unit test, etc., and my goal would be to
> slowly incorporate functionality equivalent to the Matlab DSP toolbox, using
> JTransform as well as translations of some scripts from the GNU Octave DSP
> toolbox. Alternatively, I would love to contribute to a better-structured
> branch set up by somebody who knows what they're doing. (I can also just
> patch the libraries ad hoc, directly into my own new Scala backend, but it
> would be super if my work were shareable.)
>
> Thanks to all for what seems like a great project!
> Kenta
> https://groups.google.com/d/msg/scala-breeze/-/mj4Vhu4IxS0J.

ktakagaki

unread,
Jun 13, 2013, 8:18:21 AM6/13/13
to scala-...@googlegroups.com
Hello,

I've incorporated a 1D DenseVector fft and ifft from the JTransforms package. It passes basic tests.

However, I've been struggling with type erasure. To be precise, JTransforms provides optimized algorithims for fft'ing real vectors (vs. complex vectors), so I've been playing around trying to distinguish DenseVector[Double] vs. DenseVector[Complex]... the best I could do was to use java.object.getClass on the DenseVector.data array.
  • There must be a better way to do this, can anybody help? 
  • Is there any other class in breeze where such reflection is done, where I could copy from?
Cheers,
Kenta

David Hall

unread,
Jun 13, 2013, 2:47:10 PM6/13/13
to scala-...@googlegroups.com
Great!

I think the way I would do it is to add an implicit

trait CanFFT[InputType, OutputType] {
   def apply(input: InputType):OutputType
}

object CanFFT {
  implicit def dvDoubleFFT:CanFFT[DenseVector[Double], DenseVector[Complex]] = ???
  //etc.
}

def fft[Input,Output](v: Input)(implicit canFFT: CanFFT[InputType, OutputType]):Output = canFFT(input)

-- David


ktakagaki

unread,
Jun 27, 2013, 8:25:25 AM6/27/13
to scala-...@googlegroups.com
Hello All,

I apologize for the delay, I've been swamped.
fft/ifft is now wrapped for 1D and 2D as David suggested with implicits, and is passing tests (i.e. agrees with MatLab output).


I have three elementary questions...

1. I'm new to contributing to these things, am I supposed to file a pull request? Should I keep wrapping all of the rest of the JTransforms library and only put in a request once it's done?

2. What about the documentation? For ScalaDoc, am I supposed to put end user-documentation in the main fft() function scaladoc, or should I document the implicit delegated functions? Or is the wiki the main documentation?

2. I haven't wrapped Float ffts, it should be straight forward now that I understand the implicit structure that David suggested. Is breeze committed to having Float equivalents for all of its functionality?

Cheers,
Kenta

T Schulte

unread,
Jun 27, 2013, 11:16:39 AM6/27/13
to scala-...@googlegroups.com
I would also like to note that I would like to use Breeze with the Spark Project--

(aside): It seems they have not moved to 2.10 yet, so there is an issue there-- it it possible to build the latest Breeze with 2.9?...

David Hall

unread,
Jun 27, 2013, 12:02:09 PM6/27/13
to scala-...@googlegroups.com
On Thu, Jun 27, 2013 at 8:16 AM, T Schulte <anthony...@gmail.com> wrote:
> I would also like to note that I would like to use Breeze with the Spark
> Project--
>
> (aside): It seems they have not moved to 2.10 yet, so there is an issue
> there-- it it possible to build the latest Breeze with 2.9?...

It would be good to have Breeze work with Spark, for sure. I imagine
they'll be upgrading soon, but... It is not possible to build Breeze
with 2.9. It should not be too hard. Most of the errors should be that
it can't find ClassTag, which you can replace with ClassManifest.
(This will require a massive find/replace.)

-- David
> --
> You received this message because you are subscribed to the Google Groups
> "Scala Breeze" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to scala-breeze...@googlegroups.com.
> To post to this group, send email to scala-...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/scala-breeze/f7bed342-a27f-4454-be30-50ff27cba9c0%40googlegroups.com.

David Hall

unread,
Jun 27, 2013, 12:09:14 PM6/27/13
to scala-...@googlegroups.com
Great!

On Thu, Jun 27, 2013 at 5:25 AM, ktakagaki <kentaroh...@gmail.com> wrote:
> Hello All,
>
> I apologize for the delay, I've been swamped.
> fft/ifft is now wrapped for 1D and 2D as David suggested with implicits, and
> is passing tests (i.e. agrees with MatLab output).
>
> It is in https://github.com/ktakagaki/breeze.git MASTER.
>
> I have three elementary questions...
>
> 1. I'm new to contributing to these things, am I supposed to file a pull
> request? Should I keep wrapping all of the rest of the JTransforms library
> and only put in a request once it's done?

PR is great. I think it's fine to issue a PR request against what you
have. Some basic docs would be nice, but that's it.

>
> 2. What about the documentation? For ScalaDoc, am I supposed to put end
> user-documentation in the main fft() function scaladoc, or should I document
> the implicit delegated functions? Or is the wiki the main documentation?

In principle, I try to (but certainly do not always manage to) have
some scaladoc for the
function and to add something to the wiki. I would document fft directly.

For the wiki, if you would just add a small entry to
https://github.com/scalanlp/breeze/wiki/Breeze-Linear-Algebra I would
appreciate it.

If you want to do something more, adding a page off of
https://github.com/scalanlp/breeze/wiki/UserGuide would be great.

>
> 2. I haven't wrapped Float ffts, it should be straight forward now that I
> understand the implicit structure that David suggested. Is breeze committed
> to having Float equivalents for all of its functionality?

Eventually, I'd like to make Float better supported in all of Breeze,
but don't worry about that for now.
> https://groups.google.com/d/msgid/scala-breeze/bc3ad485-4ac0-431c-b5c4-96e6d32dd26c%40googlegroups.com.

ktakagaki

unread,
Jun 28, 2013, 7:33:51 AM6/28/13
to scala-...@googlegroups.com
Thanks for the response. I hope to put in a pull request by the end of the weekend.

One more question regarding syntax convention choice...

fft( X ) in MatLab for a 2D matrix gives column-wise 1D fft, and fft2( X ) gives the 2D fft over both dimensions.
I personally feel that this MatLab syntax is a bit hackish, and I would prefer fft (DenseMatrix) in breeze to give a 2D fft. Column-wise 1D ffts could be done with slices, I feel that this might be more natural.

Any thoughts re this syntax vis-a-vis other similar functions in the breeze package overall?

Cheers,
Kenta

David Hall

unread,
Jun 30, 2013, 8:56:52 PM6/30/13
to scala-...@googlegroups.com

Sorry for slow response.

I don't have strong opinions here, but I agree with you to the extent I have one. For column wise... It probably makes sense to use the m(::, *) syntax when we get it.

MLnick

unread,
Jul 9, 2013, 5:36:23 AM7/9/13
to scala-...@googlegroups.com
I'm using Breeze and Spark currently, and am just using the old 0.2 Breeze (2.9.2) for now until Spark moves to 2.10. Hopefully they will move soon.

Jason Baldridge

unread,
Jul 9, 2013, 10:37:43 AM7/9/13
to scala-...@googlegroups.com
FWIW, there's a 2.10 branch for Spark: https://github.com/mesos/spark/tree/scala-2.10



--
You received this message because you are subscribed to the Google Groups "Scala Breeze" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scala-breeze...@googlegroups.com.
To post to this group, send email to scala-...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 



--
Jason Baldridge
Associate Professor, Dept. of Linguistics, UT Austin
Co-founder & Chief Scientist, People Pattern
http://www.jasonbaldridge.com
http://twitter.com/jasonbaldridge

Francisco José Valverde Albacete

unread,
Jul 9, 2013, 2:49:33 PM7/9/13
to scala-...@googlegroups.com
And we have had it installed, up and running!

Cheers,

F. Valverde

Reply all
Reply to author
Forward
0 new messages