GSoC Project [Port Nmatrix to JRuby] [Prasun Anand -> Charles Nutter/ Francesco Strozzi/ Pjotr Prins/ John Woods/ Rodrigo Botafogo]

[Prasun Anand]

Hi

I am Prasun Anand.  I am very grateful to Sciruby Community for selecting my proposal for the project 'Port Nmatrix to JRuby'. 

I am a fourth-year student at BITS Pilani KK Birla Goa Campus, Goa, India pursuing a dual degree in M.Sc. in Biological Sciences and B.E. in Chemical Engineering. I am inclined more towards Biological Sciences. My research interests lie in Graph Theory, Neural Signal Processing. I like FOSS and previously I have made a few contributions to Jquery foundation and BioJavascript( BioJS ). My first encounter with Ruby was when I had to work on jekyll. Ruby as a programming language never cease to amaze me (Since, I come from a Javascript background). This is a great opportunity for me to learn and contribute to FOSS.

About Project: NMatrix for MRI has become a fairly well-established project. However with JRuby+Truffle+Graal becoming very fast compared to MRI, a lot of Ruby developers are switching to JRuby. This project aims to port NMatrix to JRuby.

My application can be found here.

Looking forward to a great learning experience in summers.

Regards

Prasun Anand

[Prasun Anand]

Hi mentors

The coding period has started and I would like to start with achieving these milestones.

1. The java backend will be added to the existing repo. I will pushing my code on jruby_port branch of the NMatrix forked repo.

Setting up the environment for java backend:

• New java backend will be placed in ext/java directory.

• Add a rake file to compile .java files to .class files and zip them into a jar file.

• The external jar file Apache Commons Math  would in the ext/java/vendor  directory.

To begin with NMatrix creation I would like the mentors to suggest me whether I use generic classes for NMatrix to handle the data-types. Generic classes like ArrayList are extremely powerful and they ideal for building “tools”. Should I be using JNMatrix<DType> or should I work without generics.

2. After creating NMmatrix backend for NMatrix of an arbirtrary dimension and specified dtype I will Plug NMatrix frontend  to java backend and run a few test and see if it passes.

• In the frontend environment, detecting if jruby is used:

if defined?(RUBY_ENGINE) && RUBY_ENGINE == "jruby"

            require "jar files"

       end

• Modify Rakefile to add script for tests .

• test_helper.rb

if RUBY_PLATFORM  =~ /jruby

                  require "java"

           end

Regards

Prasun Anand

[Pjotr Prins]

Hi Prasun,

I think you should try with generics first to make use of modern JVM
libraries. See where you get stuck.

I guess our target is matrices of doubles really, so it may well be
you can do without when it makes sense.

Pj.

On Sun, May 22, 2016 at 10:54:54AM -0700, Prasun Anand wrote:
> Hi mentors
> The coding period has started and I would like to start with achieving
> these milestones.
>
> 1. The java backend will be added to the existing repo. I will pushing my

> code on jruby_port branch of the NMatrix forked [1]repo.

>
> Setting up the environment for java backend:
>

> * New java backend will be placed in ext/java directory.
>
> * Add a rake file to compile .java files to .class files and zip them
> into a jar file.
>
> * The external jar file Apache Commons Math  would in the

> ext/java/vendor  directory.
>
> To begin with NMatrix creation I would like the mentors to suggest me
> whether I use generic classes for NMatrix to handle the data-types.
> Generic classes like ArrayList are extremely powerful and they ideal for
> building “tools”. Should I be using JNMatrix<DType> or should I work
> without generics.
>
> 2. After creating NMmatrix backend for NMatrix of an arbirtrary dimension
> and specified dtype I will Plug NMatrix frontend  to java backend and run
> a few test and see if it passes.
>

> * In the frontend environment, detecting if jruby is used:

>
> if defined?(RUBY_ENGINE) && RUBY_ENGINE == "jruby"
>
>             require "jar files"
>
>        end
>

> * Modify Rakefile to add script for tests .
>
> * test_helper.rb

>
> if RUBY_PLATFORM  =~ /jruby
>
>                   require "java"
>
>            end
>
> Regards
>
> Prasun Anand
>
> On Sunday, April 24, 2016 at 9:44:35 PM UTC+5:30, Prasun Anand wrote:
>
> Hi
> I am Prasun Anand.  I am very grateful to Sciruby Community for
> selecting my proposal for the project 'Port Nmatrix to JRuby'. 
> I am a fourth-year student at BITS Pilani KK Birla Goa Campus, Goa,
> India pursuing a dual degree in M.Sc. in Biological Sciences and B.E. in
> Chemical Engineering. I am inclined more towards Biological Sciences. My
> research interests lie in Graph Theory, Neural Signal Processing. I like
> FOSS and previously I have made a few contributions to Jquery foundation
> and BioJavascript( BioJS ). My first encounter with Ruby was when I had
> to work on jekyll. Ruby as a programming language never cease to amaze
> me (Since, I come from a Javascript background). This is a great
> opportunity for me to learn and contribute to FOSS.
> About Project: NMatrix for MRI has become a fairly well-established
> project. However with JRuby+Truffle+Graal becoming very fast compared to
> MRI, a lot of Ruby developers are switching to JRuby. This project aims
> to port NMatrix to JRuby.

> My application can be found [2]here.

> Looking forward to a great learning experience in summers.
> Regards
> Prasun Anand
>

> --
> You received this message because you are subscribed to the Google Groups
> "SciRuby Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [3]sciruby-dev...@googlegroups.com.
> For more options, visit [4]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. http://github.com/prasunanand/nmatrix
> 2. https://docs.google.com/document/d/1EOvrH8AgYReVSmX8IsSYX8fl9CtJfHRDIgBsuMqeSIU/edit?usp=sharing
> 3. mailto:sciruby-dev...@googlegroups.com
> 4. https://groups.google.com/d/optout


--

[Prasun Anand]

Hi Pjotr

Thanks. I will start with generic classes.

I hope I can solve any issues with Type Erasure if they arise.

Regards

Prasun

>    3. mailto:sciruby-dev+unsub...@googlegroups.com
>    4. https://groups.google.com/d/optout


--

[Pjotr Prins]

On Sun, May 22, 2016 at 11:55:19AM -0700, Prasun Anand wrote:
> Thanks. I will start with generic classes.
> I hope I can solve any issues with Type Erasure if they arise.

Well, assuming they are doubles they will remain so ;). I think we can
track type at a higher level (i.e., the matrix); but maybe I am wrong.

Pj.

[Kenta Murata]

2016年5月23日(月) 2:54 Prasun Anand <prasunan...@gmail.com>:

• New java backend will be placed in ext/java directory.

"java" is very common name, so I think you should avoid to use this in top-level namespace of the load path.

I recommend you to change the directory to have nmatrix prefix like ext/nmatrix/java or ext/nmatrix_java.

Kenta Murata

[Prasun Anand]

Day 2 Report

I have pushed my code to the (repo https://github.com/prasunanand/nmatrix/tree/jruby_port).

I got double matrix working. Now I am implementing casts between the storage and data-types

by using 2 data-types int and double.

Also, I am trying to implement enumerators/iterators along with it; that will be useful in slicing.

Day 2 Report

I implemented the following features:

1. Added cast_interpret method

2. registered storage in a separate object

3. switching object type by detecting object_type at runtime

4. tests => I am using Junit for running tests. The tests are still not complete

My current code can be improved by using Java Metaprogramming(using Reflection). I am learning it in depth :) . I have 

used this in WrapperType.java. This will take care of argument types at runtime and it doesn't have 

type-erasure issues and take cares of Auto-boxing and Out-boxing.

Day3 Tasks

I would continue to work with iterators and slicing.

I would like to know how people use NMatrix for data types? I think double data-type is mostly used for computation.

Regards

Prasun

[Pjotr Prins]

Hi Prasun,

Better make your E-mails less 'noisy' if there is a question in there:

On Thu, May 26, 2016 at 03:06:47AM -0700, Prasun Anand wrote:
> I would like to know how people use NMatrix for data types? I think double
> data-type is mostly used for computation.

So, our question is: is anyone here really interesting in using
Nmatrix other than for doubles? At this stage it is easiest for us to
do a 'double' implementation first.

Who here is using Nmatrix for integer/long values only?

Pj.

[Prasun Anand]

Hi Pjotr

I will post my question again :) .

Regards

Prasun

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Prasun Anand]

Hi mentors

I would like to know how people use NMatrix for data types.

Is anyone here really interesting in using Nmatrix other than for doubles? 

At this stage it is easiest for us to do a 'double' implementation first.

Who here is using Nmatrix for integer/long values only?

Regards

Prasun

[John Woods]

I think probably complex128 gets used for computations a lot.

--

[Prasun Anand]

Hi John,

Thanks.

I will be working on Complex Matrices in the coming week. 

Regards,

Prasun

[Prasun Anand]

Hi mentors

I need help regarding slicing of multi-dimensional array .

I used xslice to get the slice coordinates and lengths. However, how do I get the slice or the element 

at the given position?

Code which I used from ruby_nmatrix.c

 def xslice(args)

    result = nil

    s = @elements

    if @dim < args.length

      raise Exception.new("wrong number of arguments (%d for %lu)", args, effective_dim(s))

    else

      result = Array.new()

      slice = get_slice(@dim, args, @shape);

      if slice[:single]

        if (@dtype == "RUBYOBJ") 

          # result = *reinterpret_cast<VALUE*>( ttable[NM_STYPE(self)](s, slice) );

        else                                

          result = slice

        end 

      else

        # NMATRIX* mat  = NM_ALLOC(NMATRIX);

        # mat->stype    = NM_STYPE(self);

        # mat->storage  = (STORAGE*)((*slice_func)( s, slice ));

        # nm_register_nmatrix(mat);

        # result        = Data_Wrap_Struct(CLASS_OF(self), nm_mark, delete_func, mat);

      end

    end

    return result

  end

  def get_slice(dim, args, shape_array)

    slice = {}

    slice[:coords]=[]

    slice[:lengths]=[]

    slice[:single] = true

    argc = args.length

    t = 0

    (0...dim).each do |r|

      v = t == argc ? nil : args[t]

      if(argc - t + r < dim && shape_array[r] ==1)

        slice[:coords][r]  = 0

        slice[:lengths][r] = 1

      elsif v.is_a?(Fixnum)

        v_ = v.to_i.to_int

        if (v_ < 0) # checking for negative indexes

          slice[:coords][r]  = shape_array[r]+v_

        else

          slice[:coords][r]  = v_

          slice[:lengths][r] = 1

        t+=1

        end

      elsif (v.is_a?(Symbol) && v.__id__ == "*")

        slice[:coords][r] = 0

        slice[:lengths][r] = shape_array[r]

        slice[:single] = false

        t+=1

      # elsif condition

        # not implemented currently

        # for range

        # if condition

          

        # elsif condition

          

        # slice[:single] = false

        # t++

        # end

      else

        raise Exception.new("expected Fixnum or Range for slice component instead of")

      end

      if (slice[:coords][r] > shape_array[r] || slice[:coords][r] + slice[:lengths][r] > shape_array[r])

        raise Exception.new("slice is larger than matrix in dimension %lu (slice component %lu)", r, t);

      end

    end

    return slice

  end

For example

m = NMatrix.new([2,2,2], [1,2,3,4,5,6,7,8], dtype: :int64)

puts m[1,1,1]  # prints {:coords=>[1, 1, 1], :lengths=>[1, 1, 1], :single=>true} as the slice object.

Regards

Prasun

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev+unsubscribe@googlegroups.com.

[John Woods]

I'm having a little trouble deciphering your question. Keep in mind I haven't looked at the slicing code in about two or three years, give me a little background, and then ask again?

John

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.

[Prasun Anand]

Hi John

I had to implement  def [ ](args)  function for a matrix. This function pass the args to xslice function. xslice function passes the args to get_slice function which returns a slice object slice={ coords => [ ], position =>[ ] , single : true/false } to xslice.

Now xslice uses this slice object to return a result that is an array or element.

Now, I don't know how to proceed from here. I don't understand the significance of "stride" .

 void* nm_dense_storage_ref(const STORAGE* storage, SLICE* slice) {

  DENSE_STORAGE* s = (DENSE_STORAGE*)storage;

  if (slice->single)

    return (char*)(s->elements) + nm_dense_storage_pos(s, slice->coords) * DTYPE_SIZES[s->dtype];

  else {

    nm_dense_storage_register(s);

    DENSE_STORAGE* ns = NM_ALLOC( DENSE_STORAGE );

    ns->dim        = s->dim;

    ns->dtype      = s->dtype;

    ns->offset     = NM_ALLOC_N(size_t, ns->dim);

    ns->shape      = NM_ALLOC_N(size_t, ns->dim);

    for (size_t i = 0; i < ns->dim; ++i) {

      ns->offset[i] = slice->coords[i] + s->offset[i];

      ns->shape[i]  = slice->lengths[i];

    }

    ns->stride     = s->stride;

    ns->elements   = s->elements;

    s->src->count++;

    ns->src = s->src;

    nm_dense_storage_unregister(s);

    return ns;

  }

}

I tried to find out myself how I can access elements of a multidimensional matrix but I couldn't find any helpful resource.

Regards

Prasun

[John Woods]

Ah! Stride, I believe, tells you how far to skip as you iterate through the rows of a slice. So, if my slice length is 2,4 and my matrix is 4x4, my stride tells me how many elements I need to move forward in order to get from the last column of row i to the first column of row i+1.

Or so I remember. Make sense?

[Prasun Anand]

Thanks :)
I think it makes sense now. I will retry implementing [ ] method.

Can you point me to some resource like a book or an article which
deals with this algorithm?
It will be very helpful.

Regards,
Prasun

[John Woods]

There's really no such resource. It just seemed like the most optimal way to implement slicing with as little overhead as possible.

[Prasun Anand]

Ok. :)

[Pjotr Prins]

On Thu, Jun 02, 2016 at 09:21:39PM +0000, John Woods wrote:
> There's really no such resource. It just seemed like the most optimal way
> to implement slicing with as little overhead as possible.

Scipy also uses stride. It makes sense to go through data like that,
though you will get cache misses on the CPU. For some algorithms it
can make sense to transform/reduce the matrix first for that reason.

But that is a case by case thing.

Pj.

[Prasun Anand]

End of Week 2 Report

1. Nmatrix_Initializations

 https://github.com/prasunanand/nmatrix/commit/08f424ce675ab79da5449013e1ead121cbd499e8

 ==> jruby: modified jnmatrix.java to store shape as an array and
worked on nmatrix class methods
 shape, dim , slice ,[] , []= , effective_dimension, etc.

Accessors [ ] => Currently, I am successful able to get the element at a given coordinate. I am working

on how to get the elements when a range is supplied as an argument.

2.  Tests for java backend ; I am using junit to test the java backend for nmatrix. Currently I have not

my own tests and plan to do it in the next week.

The junit tests will be run as a rake task.

3.element_wise operations. I implemented other element-wise operators(unary and binary)

Binary => All the binary operators have been implemented.
Unary => Unary operators except for error functions(erf, erfc) and round, gamma are not supported for now.
I need to implement a univariate function using erf provided by special package
org.apache.commons.math3.special.Erf

While running tests for math functions, I wasn't able to clear the tests. I tried to
find out the reason by manual testing. The lhs and rhs are equal, still they fail.
Need to look again.

Commit => https://github.com/prasunanand/nmatrix/commit/2e691ff3fc4557fb730bf83a1ea15011ab7f3495

Tests

00_shortcuts.rb => 12 Tests are succeeding currently.

I am trying to get more tests succeed and benchamark the nmatrix functionalities and hope to share it

with the community as soon as possible.

In this week, I will be working on 2D Matrices.

[Prasun Anand]

Hi

I have published my blog :).

It summarises my progress on the project.

Link => http://www.prasunanand.com/ruby/gem/2016/06/16/gsoc16-sciruby-nmatrix-jruby.html

Regards

Prasun

[John Woods]

Couple of comments:

* Why are you passing in "FLOAT32" and "DENSE_STORE" as strings instead of symbols? It'll be faster with symbols, since strings require N comparisons but symbols only require 1.

* Looking at your definition of +, and wondering: why aren't you raising the exact same exceptions as MRI NMatrix here? The exceptions need to match.

* lapacke has no effect on the addition function. Lapacke does not provide an addition functionality. Same goes for subtraction. Multiplication, yes.

* CamelCase and camelCase variable names are a no-no in our style guide, unless necessary for proper code functioning. Is this a JRuby requirement? Please use snake_case instead.

* Looks like you fixed a few of the other things I was going to point out between the beginning and end of the post, so good job with the progress. =)

* Does JRuby matrix addition/subtraction handle upcasting?

* Can you implement elementwise operations (addition, subtraction, etc.) in a way that is simple and doesn't require you to repeat a lot of code, but nonetheless runs quickly?

Good blog entry.

John

--

[Prasun Anand]

Hi John,

I will correct my mistakes and will work on solving the issues which
you pointed.
In the coming weeks I intend to get all the tests succeed :) .

Thanks and Regards,
Prasun

[Prasun Anand]

Hi mentors,

NMatrix for double data type is almost complete. To check how nmatrix integrates with other sciruby gems,

I have started porting mixed-models gem to jruby.

I have hit a blocker which I think will be resolved in a few days.

For the blog I started running the examples that Alexej discussed in his blog[0].

The first example for analysis of 'blog data'[1] doesn't work for ruby as well as jruby. 

I tried every possible way I could. The example gets stuck at line[2]. I used daru-0.1.0

as daru-0.1.1 is not compatible with 'mixed-models'[3].

Next I started running example "LMM.rb"[4]. I ran the example using
both ruby and jruby and compared output at every stage. Here I found
two issues.

1. When we take dot product of two matrices C = A.dot B
   If Aij and Bij are both smaller than |1| we get Cij = 0
   I solved this by
        C = ((A * 5).dot B)/5

2. When LMM does optimisation line [5] it calls NelderMead.minimize
   that uses deviation and autoboxing leads to 0 as element output.
   Therefore a diagonal matrix gets reduced to a singular matrix  and 

   Cholesky solve throws "singular matrix" error [6].

Solution : I will have a look at jruby tests to figure how it handles dtypes.

Regards,

Prasun

[0]. http://sciruby.com/blog/2015/08/19/gsoc-2015-mixed-models/
[1]. https://github.com/agisga/mixed_models/blob/master/examples/blog_data.rb

[2]. https://github.com/agisga/mixed_models/blob/master/examples/blog_data.rb#L31
[3]. https://github.com/agisga/mixed_models/blob/master/examples/LMM.rb
[4]. https://github.com/agisga/mixed_models/issues/4
[5]. https://github.com/agisga/mixed_models/blob/master/lib/mixed_models/LMM.rb#L95
[6]. https://github.com/agisga/mixed_models/blob/master/lib/mixed_models/Deviance.rb#L43

[Prasun Anand]

Hi Alexej

I am unable to figure out the issue with example 'blog_data' [1] ?
Can you suggest why it gets stuck on Ruby MRI?

Regards,
Prasun


[1]. https://github.com/agisga/mixed_models/blob/master/examples/blog_data.rb

[Alexej Gossmann]

Hi Prasun,

If I'm not mistaken, I ran that particular example with what was the development version of daru at that time (rather than the daru-0.1.0 stable release).

Let me try out a few things, and I'll get back to you later today.

Alexej

[Prasun Anand]

Thanks Alexej :)

[Alexej Gossmann]

Prasun,

Try to use mixed_models from this branch: https://github.com/agisga/mixed_models/tree/daru-0.1.1-compatibility

In that branch I have adjusted all examples, tests, and notebooks to run with the newest version of daru. Among others, the blog data example works fine on my machine with MRI ruby 2.3.1p112.

Best,

Alexej

[Prasun Anand]

Hi Alexej,

When running mixed_models test using jruby-9.0.5.0, I get a warning message

"Process.fork is not supported by this Ruby"

Also parallel gem is not supported by jruby. I am not able to identify why is parallel gem used?

Is it critical for LMM examples because I don't get the correct output when LMM examples are run using rspec?

In example "blog_data.rb", the heap size exceeds 14gb at line 71[0]. Before this step 11gb of RAM is already

utilized. Can you suggest how I can utilize memory efficiently?

[0]https://github.com/agisga/mixed_models/blob/master/examples/blog_data.rb#L71

Regards,

Prasun

[Alexej Gossmann]

Hi Prasun,

> When running mixed_models test using jruby-9.0.5.0, I get a warning message "Process.fork is not supported by this Ruby"

I never had this problem. Does it happen on MRI ruby too for you, or only on jruby?

> Also parallel gem is not supported by jruby. I am not able to identify why is parallel gem used?

Parallel gem is used to speed up computation of bootstrap confidence intervals and bootstrap hypotheses tests (none of that is done in the blog_data.rb example by the way).

> In example "blog_data.rb", the heap size exceeds 14gb at line 71[0]. Before this step 11gb of RAM is already utilized. Can you suggest how I can utilize memory efficiently?

Not sure about that. If you just want to test things, maybe you should look at an example that uses a smaller dataset (the blog_data.rb example has the purpose to show of the mixed_models gem on a real dataset of respectable size). By the way, do you have the memory efficiency problem with jruby only, or also with MRI? I just ran the blog_data.rb example on my work machine (ruby 2.2.1p85 on Ubuntu 14.04), and it used a maximum of about 4gb of RAM.

Best,

Alexej

[Pjotr Prins]

On Wed, Jul 06, 2016 at 11:16:06AM -0500, 'Alexej Gossmann' via SciRuby Development wrote:
> Hi Prasun,
>
> > When running mixed_models test using jruby-9.0.5.0, I get a warning
> message "Process.fork is not supported by this Ruby"
>
> I never had this problem. Does it happen on MRI ruby too for you, or only
> on jruby?

parallel gem does not work on JRuby I think. Can we disable it?

Pj.

[Alexej Gossmann]

> parallel gem does not work on JRuby I think. Can we disable it?

Yes, it can be disabled in gemspec. Only consequence is that a few tests will fail, and the confidence intervals and hypotheses test methods requiring bootstrap will not work.

Alexej

[Prasun Anand]

Hi Alexej,

> I never had this problem. Does it happen on MRI ruby too for you, or
> only on jruby?
>

It doesn't happen on Ruby MRI, only for JRuby.

> ...

> Not sure about that. If you just want to test things, maybe you should
> look at an example that uses a smaller dataset (the blog_data.rb example
> has the purpose to show of the mixed_models gem on a real dataset of
> respectable size). By the way, do you have the memory efficiency problem
> with jruby only, or also with MRI? I just ran the blog_data.rb example
> on my work machine (ruby 2.2.1p85 on Ubuntu 14.04), and it used a
> maximum of about 4gb of RAM.

Yeah, for MRI it uses less memory(around 4GB).

I am close to finishing 'Mixed-Models' port :) .

Thanks and Regards,
Prasun

[Prasun Anand]

Hi mentors,

I have ported mixed_models to JRuby and published a blog.

Link : http://www.prasunanand.com/jruby/ruby/gem/2016/07/09/gsoc16-sciruby-mixedmodels-jruby.html

Regards,
Prasun

[Alexej Gossmann]

Nice! I'm glad that the port was successful!

Alexej

[Prasun Anand]

Hi,

I have published my blog and submitted the final evaluation.

Blog=> http://www.prasunanand.com/jruby/2016/08/22/gsoc16-project-report-port-nmatrix-to-jruby.html

Regards,

Prasun