Ruby netcdf wrapper

[Edmund H]

Hi all,

I maintain a package called coderunner for the running and analysis of large simulations. It is currently built on stack of narray/rb-gsl/ruby-netcdf for data handling (projects from the GFD Dennou Ruby club). I'd like to migrate to SciRuby as it seems more active.

However, the sticking point is the lack of a ruby-netcdf interface that is compatible with NMatrix. First, does anyone know of one? Have I missed one?

I'm happy to try and address this, but there are a variety of strategies and I'd like any thoughts people have on what would be best.

(1) I could use the narray-nmatrix gem which in theory has an nmatrix compatible version of narray. I could then make a new ruby-netcdf which used this narray-nmatrix gem, and read the data as NArrays, which could then be converted to NMatrices as required (is there a function which converts between the two without a memory copy?) I have tried this, but when I change ruby-netcdf to depend on the narray-nmatrix gem instead of the narray gem, I always get the mysterious message Could not find a valid gem 'narray-nmatrix' (>= 0) in any repository even though the narray-nmatrix gem is actually installed.

(2) I could convert the existing ruby-netcdf package (including the C extension) to use NMatrix directly, instead of NAarray. Is this easy? Could someone point me to a page where this has been done/explained?

(3) I could fork this project: https://github.com/grim7reaper/netcdf, which has started from scratch using NMatrix and FFI, but this would be a lot of work as the project hasn't got far.

Any thoughts very welcome!

Edmund

[John Woods]

I don't think it should be particularly difficult to adapt anything to use NMatrix. It's probably going to involve fewer copies than using narray-nmatrix. But the layout for the dense storage type is pretty similar to NArray, so it's really just about switching the API you use.

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Edmund H]

Hi John

Thanks for the reply. I’m figuring that it’s probably best to do option (2), so what I really need to do is to convert the C functions in the ruby-netcdf gem to use the NMatrix C API. Based on your reply, it seems that I won’t have to muck about too much with the actual storage, just the wrapping around it (a C array of floats is a fairly simple thing ;-) ).

Has anyone migrated any C extensions from NArray to NMatrix before? I’d welcome any insights before I get stuck in.

Edmund

You received this message because you are subscribed to a topic in the Google Groups "SciRuby Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sciruby-dev/A1K6LI1xvMw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sciruby-dev...@googlegroups.com.

[Rodrigo Botafogo]

Hi Edmund,

MDArray is based on NetCDF-Java and can read and write NetCDF-3 files.  I don´t know if it would be easy to migrate your application to JRuby and MDArray, but this could be an alternate solution.  If I´m not mistaken MDArray also reads NetCDF-4 files, but cannot write then.  To write NetCDF-4 files requires installation of a C library which I haven´t done.

MDArray can interface with Parallel Colt (with MDMatrix) without copying making it easy to add basic statistical analysis to the data. MDArray can also interface with SciCom (R interpreter for the JVM) without data copying, allowing powerful data analytics to the data.

Good luck with your project,

Rodrigo

[Edmund H]

Hi Rodrigo,

Thanks very much for the suggestion. My whole framework has a lot of C extensions and I need it to run on a variety of supercomputers all with varying system environments. I am also looking to meet it generally compatible with SciRuby. Given this, I’m not sure migrating to MDArray and JRuby meets my needs at the current time, but it’s good to know that there are other options out there.

Edmund

[Pjotr Prins]

(1) should be solvable but it requires writing conversions and lots of
in-RAM copying, I agree with John.

For (2) I just looked at the ruby-netcdf code and it is pretty small.
I think you can port that to nmatrix. Be sure to add more tests ;)

Note that we also have a JRuby Nmatrix port in the works this Google
summer of code.

Pj.

> (3) I could fork this project: [2]https://github.com/grim7reaper/netcdf,

> which has started from scratch using NMatrix and FFI, but this would be
> a lot of work as the project hasn't got far.
>
> Any thoughts very welcome!
>
> Edmund
>
> --
> You received this message because you are subscribed to the Google
> Groups "SciRuby Development" group.
> To unsubscribe from this group and stop receiving emails from it, send

> an email to [3]sciruby-dev...@googlegroups.com.
> For more options, visit [4]https://groups.google.com/d/optout.

>
> --
> You received this message because you are subscribed to the Google Groups
> "SciRuby Development" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to [5]sciruby-dev...@googlegroups.com.
> For more options, visit [6]https://groups.google.com/d/optout.
>
> References
>
> Visible links
> 1. mailto:sciru...@googlegroups.com
> 2. https://github.com/grim7reaper/netcdf
> 3. mailto:sciruby-dev...@googlegroups.com
> 4. https://groups.google.com/d/optout
> 5. mailto:sciruby-dev...@googlegroups.com
> 6. https://groups.google.com/d/optout

--

[Edmund H]

Hi everyone,

I have now adapted the ruby-netcdf gem to work with nmatrix, and have created a new gem netcdf-nmatrix (https://rubygems.org/gems/netcdf-nmatrix).

Since the ability to read/write netcdf files is pretty useful for scientific computing applications, I'd be happy for my git repository to be transferred to the SciRuby organisation if ppl are interested.

https://github.com/edmundhighcock/netcdf-nmatrix

I don't know what the licensing strategy is for SciRuby, but I though it was safest to stick to the previous license for the gem.

[John Woods]

Hey Edmund,

This is great! Well done.

Here's our checklist for becoming a SciRuby project:

https://github.com/SciRuby/sciruby/wiki/Gem-and-Subproject-Checklist

John

--
You received this message because you are subscribed to the Google Groups "SciRuby Development" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sciruby-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Edmund Highcock]

Hi John,

Thanks for the link. ruby-netcdf was an old gem and it'll take me a while to get it up to scratch.

 

However, I'm wondering if the license can be converted without getting agreement from all the original authors. It's currently licensed under the "GFD Dennou Club" custom open source license.

https://github.com/edmundhighcock/netcdf-nmatrix/blob/master/LICENSE.txt

To my inexpert eye, it looks as though it would be fine to redistribute the code under a different open source license, but I'd welcome any thoughts from people who have more expertise. 

Edmund

Sent from my iPhone

You received this message because you are subscribed to a topic in the Google Groups "SciRuby Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sciruby-dev/A1K6LI1xvMw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sciruby-dev...@googlegroups.com.

[Youhei SASAKI]

Hi,

I'm a member of GFD-Dennou Club and a maintainer of ruby-netcdf.

The license of ruby-netcdf is not custom
license, it's "Ruby's" license. So you can create derivative as GPL-2 or "Ruby's".

Best wishes,
Youhei

[Sameer Deshmukh]

Hello,

Have you seen the rb-gsl gem? It works with both narray and nmatrix (though not at the same time) and there's lots of code there that does the same job but with narray or nmatrix.

Here's an example:

https://github.com/SciRuby/rb-gsl/blob/master/ext/gsl_native/linalg.c#L77

https://github.com/SciRuby/rb-gsl/blob/master/ext/gsl_native/linalg.c#L39

[Sameer Deshmukh]

Hello,

Have you seen the rb-gsl gem? It works with both narray and nmatrix (though not at the same time) and there's lots of code there that does the same job but with narray or nmatrix.

Here's an example:

https://github.com/SciRuby/rb-gsl/blob/master/ext/gsl_native/linalg.c#L77

https://github.com/SciRuby/rb-gsl/blob/master/ext/gsl_native/linalg.c#L39

[Youhei SASAKI]

Hi Sameer, Edmund,

On Sun, 15 May 2016 02:47:40 +0900,
Sameer Deshmukh <sameer.d...@gmail.com> wrote:
>
> Have you seen the rb-gsl gem? It works with both narray and nmatrix (though
> not at the same time) and there's lots of code there that does the same job
> but with narray or nmatrix.

Yes (of course) I know.

Currently, we have a plan about next release.

1. License.

From https://github.com/SciRuby/sciruby/wiki/Gem-and-Subproject-Checklist,
"GPL is not permissible."

The ruby-netcdf were released under GPL-2 or Ruby's License because
of compatibility for MRI. It's the same license of old MRI <= 1.9.2,
*not custom open source license* !.

We have plan to change license same as MRI >= 1.9.3 (BSD or Ruby's).

2. NMatrix support

Not yet Implement. Because of the speed of nmatrix, the nmatrix
support is postponed any other (real) work, unfortunatly.

Edmund, if you have to plan to update add nmatrix support into
ruby-netcdf, I'm very happy. Please let me know your work.

> > Hi John,
> >
> > Thanks for the link. ruby-netcdf was an old gem and it'll take me a while
> > to get it up to scratch.
> >
> > However, I'm wondering if the license can be converted without getting
> > agreement from all the original authors. It's currently licensed under the
> > "GFD Dennou Club" custom open source license.
> >
> > https://github.com/edmundhighcock/netcdf-nmatrix/blob/master/LICENSE.txt
> >
> > To my inexpert eye, it looks as though it would be fine to redistribute
> > the code under a different open source license, but I'd welcome any
> > thoughts from people who have more expertise.
> >
> > Edmund
> >
> >
> >
> > Sent from my iPhone
> >

> > On 13 May 2016, at 19:39, John Woods <john.o...@gmail.com <javascript:>>

> > wrote:
> >
> > Hey Edmund,
> >
> > This is great! Well done.
> >
> > Here's our checklist for becoming a SciRuby project:
> >
> > https://github.com/SciRuby/sciruby/wiki/Gem-and-Subproject-Checklist
> >
> > John
> >
> > On Fri, May 13, 2016 at 12:16 PM 'Edmund H' via SciRuby Development <
> > sciru...@googlegroups.com <javascript:>> wrote:
> >
> >> Hi everyone,
> >>
> >> I have now adapted the ruby-netcdf gem to work with nmatrix, and have
> >> created a new gem netcdf-nmatrix (
> >> https://rubygems.org/gems/netcdf-nmatrix).
> >>
> >> Since the ability to read/write netcdf files is pretty useful for
> >> scientific computing applications, I'd be happy for my git repository to be
> >> transferred to the SciRuby organisation if ppl are interested.
> >>
> >> https://github.com/edmundhighcock/netcdf-nmatrix
> >>
> >> I don't know what the licensing strategy is for SciRuby, but I though it
> >> was safest to stick to the previous license for the gem.

---
Youhei SASAKI, Ph.D.
Department of Mathematics, Kyoto University
Kitashirakawa Oiwake-chou, Sakyou-ku, Kyoto, 606-8502 JAPAN
E-mail: <uwa...@gfd-dennou.org>
<uwa...@math.kyoto-u.ac.jp>
TEL:+81-(0)75-753-3731(direct) FAX:+81-(0)75-753-3711
GPG fingerprint:
4096/RSA: 66A4 EA70 4FE2 4055 8D6A C2E6 9394 F354 891D 7E07

[Sameer Deshmukh]

I think we can release a new gem and release it under the BSD 2 clause license (same as nmatrix). I don't think getting permission from the authors should be an issue.

About the speed of nmatrix, where do you think it is slower than narray? This is a very (_very_) important issue for us so if you could explicitly point out speed bottlenecks it would go a long way in improving nmatrix.

Regards,

Sameer Deshmukh

[Edmund Highcock]

Hi Sameer and Youhei,

Thanks for your feedback. Just to reiterate what we were discussing earlier in the thread, I have already released a new gem

netcdf-nmatrix which is compatible with nmatrix (adapted from the latest ruby-netcdf gem). The source is on my GitHub page. The changes I had to make were not too extensive so it should be possible to combine my version with the official ruby-netcdf repo and make a version that can support both NArray and nmatrix going forward.

There is a separate question of whether to target making the project officially associated with SciRuby and whether the project could be moved to a SciRuby compatible license.

Edmund

Sent from my iPhone