Clojure with Tensorflow, Torch etc (call for participation, brainstorming etc)

[kovas boguta]

Anyone seriously working on deep learning with Clojure?

I'm working with Torch at the day job, and have done work integrating Tensorflow into Clojure, so I'm fairly familiar with the challenges of what needs to be done. A bit too much to bite off on my own in my spare time. 

So is anyone out there familiar enough with these tools to have a sensible conversation of what could be done in Clojure?

The main question on my mind is: what level of abstraction would be useful?

All the existing tools have several layers of abstraction. In Tensorflow, at the bottom theres the DAG of operations, and above that a high-level library of python constructs to build the DAG (and now of course libraries going higher still). In Torch, its more complicated: there's the excellent tensor library at the bottom; the NN modules that are widely used; and various non-orthogonal libraries and modules stack on top of those. 

One could try to integrate at the bottom layer, and then re-invent the layers above that in Clojure. Or one could try to integrate at the higher layers, which is more complicated, but gives more leverage from the existing ecosystem. 

Any thoughts?

[Mikera]

I've been working with a number of collaborators on a deep learning library for Clojure. 

Some key features:

- An abstract API for key machine learning functionality

- Ability to declare graphs / stacks of operations (somewhat analogous to tensorflow)

- Support for multiple underlying implementations (ClojureScript, JVM, CPU, GPU)

- Integration with core.matrix for N-dimensional data processing

We intend to release as open source. We haven't released yet because we want to get the API right first but it is looking very promising.

[Christian Weilbach]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 31.05.2016 07:17, Mikera wrote:
> I've been working with a number of collaborators on a deep
> learning library for Clojure.
>
> Some key features: - An abstract API for key machine learning
> functionality - Ability to declare graphs / stacks of operations
> (somewhat analogous to tensorflow) - Support for multiple
> underlying implementations (ClojureScript, JVM, CPU, GPU) -
> Integration with core.matrix for N-dimensional data processing
>
> We intend to release as open source. We haven't released yet
> because we want to get the API right first but it is looking very
> promising.

Almost all of the development in deep learning is done in Python, so
having to reproduce this work on a different runtime (and language)
seems non-Clojure-like for me (compared to being hosted on the JVM and
leveraging this ecosystem). deeplearning4j is already attempting it
from the Java side with the argument of distributed scaling, but
again, there is a ton of work done for the Python toolkits including
massive scaling (e.g. with tensorflow) and most research is there, so
my question would be what the actual goals for a Clojure stack would be?

Machine learning systems usually can be implemented very well as a
"microservice" because they have no state (assumed the model is
trained and constant) and just need to return an output given a sample
(act like a pure function). This has worked out fine for me in Clojure
so far.
There are many other important machine learning concepts and papers
beyond deep learning, which can be used that way. The network IO can
be problematic in some cases ofc., e.g. small models+low-latency
requirement, but I think the leverage is much higher and will increase
unless the ML community switches away from Python to the JVM (highly
unlikely atm.). I personally will rather focus on ML papers and
improving the algorithm on paper and in Python than reimplementing a
ton of work in Clojure just to get on parity in my favourite language.

Another point is performance. For many areas it doesn't matter too
much whether you are 5 or 10% faster as long as you can scale. In
machine learning it is often critical though as these 5-10% are
already hours and scaling can be hard model-wise. I have tried to get
performance on par with Theano out of Clatrix (both on CPU) a year ago
and Theano was 2-4x times faster on my machine. Even if a Clojure
stack would address this, there is a lot of work necessary to
constantly optimize the stack for different GPU architectures and
compute graphs. Theano has atm. 40 contributors or so, not to speak of
TensorFlow being backed by Google & DeepMind now.

I think a much better approach would be to bring Python native
bindings to the JVM. There is http://www.jyni.org/ and they seem to be
getting closer to numpy support, but are only two guys atm. (I think).
While this is a more low-level effort and has little to do with
Clojure, it would allow to use all Python libraries natively from
Clojure (together with core.matrix backends for numpy, etc.). Jython
is considered as slow as CPython, so one could expect to have
Python-like performance with a small wrapper library, because most of
the performance critical code is already in the native libraries. From
there emancipating with Clojure through a core.matrix based-stack
would be a non-uphill battle similar to the development of the Clojure
ecosystem.

What points would speak against an approach like this?


Christian



-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)

iQIcBAEBAgAGBQJXTXq+AAoJEICbLivqiPOFR1sQAKZYWzGz3mEFVQuItOUgFz8p
/zRh3oj2jLYOT5rHxEehZkZfQEjuRVMn6NW5nPR8c6mEzUc2FRNUTJHbDAgqaWSp
LxIOy5qfqzuA1J1x/hlsn1JRGMrvjZv+NvW2PpG8WSgZYwblIxdzzcRCYiRQ4+tQ
IhGDg1CKc2awGOJHLuJmzTuHtI+fIhvhDxRBEjvlfTdIInKugS5K0rwyiXr50jcx
zoO5jnhibcZB9LxskmW0J/8kH/hT2RD8mwjeI5oQYZuHZ/LvZX0+U9ocihyxoL2B
YFd2TwDc7ebgx71gsAnSTPcrIOfIwItprP4ka2gWtmXGJR3PZxfm5JlBir/gJIYf
Aa3uqR19qOFCgxwUCqFCWTVgojVFcF4F+VU7dXtfrQE5hkmBycSwbXiDh4CC11jV
Lhlff+yjv4OL2IYrPMBbVVU/KeWH+o4ETR0GaePRfGuBOEc04048F4Xz84NBZ6ke
lJhGL63JpUKqBJAPjZlU57VMoNMIczHdlMGF1oRhqkWzo0gD4ygX5C8g90xXGXLq
NjF/GiFEUWR1xzPvqLTNIX2kTveW46ZBDTiYCYCD8j+8yxGw04ow5wEoflbhz3Gd
PdWk7wb9bXlSHm6+b0Ax8CGQqMeDbb/RXqHneTDCQBjDW4olmzWfpRokUGj+K0ne
/1dgMs5AjIB+QHMW+PHZ
=3hFS
-----END PGP SIGNATURE-----

[Bobby Bobble]

- Ability to declare graphs / stacks of operations (somewhat analogous to tensorflow)

I'd be interested to know more as I've been working with factor graphs in Clojure with core.matrix, and it sounds related -- have you done anything like message-passing on graphs ?

[atucker]

Given that the TensorFlow website invites people to build interfaces from other languages using SWIG, I guess they feel that access to the C++ component is the major thing.  So while I agree with Christian about reinventing the wheel, it may be that to interface at that level would involve reinventing only a relatively few spokes.

  I did an MSc in neural networks some years ago (before it was fashionable) and could do with brushing up on the latest.  I might be able to find some time for a straightforward task such as translating TensorFlow's Python into Clojure.

  A

[kovas boguta]

On Tue, May 31, 2016 at 1:17 AM, Mikera <mike.r.an...@gmail.com> wrote:

I've been working with a number of collaborators on a deep learning library for Clojure. 

Some key features:

- An abstract API for key machine learning functionality

- Ability to declare graphs / stacks of operations (somewhat analogous to tensorflow)

- Support for multiple underlying implementations (ClojureScript, JVM, CPU, GPU)

- Integration with core.matrix for N-dimensional data processing

We intend to release as open source. We haven't released yet because we want to get the API right first but it is looking very promising.

Looking forward to seeing this. 

 

[kovas boguta]

On Tue, May 31, 2016 at 7:51 AM, Christian Weilbach <white...@polyc0l0r.net> wrote:

Almost all of the development in deep learning is done in Python, so
having to reproduce this work on a different runtime (and language)
seems non-Clojure-like for me (compared to being hosted on the JVM and
leveraging this ecosystem). deeplearning4j is already attempting it
from the Java side with the argument of distributed scaling, but
again, there is a ton of work done for the Python toolkits including
massive scaling (e.g. with tensorflow) and most research is there, so
my question would be what the actual goals for a Clojure stack would be?

Good question. Basic answer is that we are in the early stages of discovering the right abstractions for these systems; Clojure provides a strong platform for exploring the basic issues. The biggies for me are:

1. Model composition. Most of these systems have some flavor of compilation, often with a heavy dose of mutability in the construction of the graph. I'd argue its easier and simpler to explore the design space starting from immutable data representations. Given the speed of evolution in model architectures, this seems pretty important. 

2. Introspection & interactivity. With Clojurescript (and even just Clojure) we have excellent options for creating interactive UIs on top of the models. Yes, theres IPython, but thats not a really equivalent to the power provided by Clojurescipt. 

Now, I'm all in favor of not reinventing the wheel, thats why I'm wondering what the best available foundation would be. JyNI is quite interesting and I hope it takes off. But just having a wrapper on a high-level Python lib probably isn't worth the hassle, unless you get some fundamentally new leverage. 

Poking around I actually discovered something closer to what I want: https://github.com/dmlc/mxnet

Its pretty much the only DL platform intended to be consumed as a library, from any language. And there are already sensible JVM bindings to it. 

[kovas boguta]

On Tue, May 31, 2016 at 9:36 AM, atucker <agjf....@gmail.com> wrote:

Given that the TensorFlow website invites people to build interfaces from other languages using SWIG, I guess they feel that access to the C++ component is the major thing.  So while I agree with Christian about reinventing the wheel, it may be that to interface at that level would involve reinventing only a relatively few spokes.

I looked into this and unfortunately Tensorflow is not as language-agnostic as it might seem. 

If you just want predictions from a trained model, you can connect with SWIG.  

However, for training, it is required to use Python because Python installs a bunch of backprop definitions into the graph. It should be possible to pull this off (and then call into it from Clojure). 

The difficulty of doing so is hard for me to assess, so I'm hoping someone can chime in on this. 

[Dragan Djuric]

Hey Mike,

A friend asked me if I know of any good (usable) deep learning libraries for Clojure. I remembered you had some earlier neural networks library that was at least OK for experimenting, but seems abandoned for your current work in a similar domain. A bit of digging lead me to this post.

I understand that this library may not be completely ready yet, but I wandered wheter now you were able to give a better estimation of where it stands in comparison with other DL offerings, like what deeplearning4j guys are doing, or even with the established non-Java libraries such as Theano, Torch, Caffe, and TensorFlow. What is the chance of you releasing it even if it is not 100% ready? 

I get the reluctance to commit to a certain API, but I don't think everyone will rush to commit their code to the API you release anyway, and the open development will certainly help both the (potential) users and your team (by returning free testing & feedback).

[kovas boguta]

+1 to Dragan's inquiry. 

FWIW, was reviewing the state of affairs the other day:

- MXNet currently has the best JVM interop story, among DL frameworks that have competitive perf. - DL4J has improved a lot recently but still looks like it has a ways to go in terms of perf. 

Right now I'm more interesting in word2vec type things, which don't require a deep net, so I might give Neanderthal a shot. 

By the way, I'd love to see matrix/tensor benchmarks of Neanderthal and Vectorz vs ND4J, MXNet's NDArray, and BidMat..  :)

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Dragan Djuric]

Hi Kovas,

By the way, I'd love to see matrix/tensor benchmarks of Neanderthal and Vectorz vs ND4J, MXNet's NDArray, and BidMat..  :)

I don't have exact numbers, but will try to give you a few pointers to help you if you decide to investigate this further:

0. Neanderthal's scope is matrices and linear algebra. NNs and other stuff is something that could be built on top of it (assuming that the features needed are implemented, which may or may not be true yet), but certainly not in Neanderthal.

1. Neanderthal is a 100% Clojure solution. One of the main goals, other than the speed, of course, is that it is simple and straightforward, with no overhead. That means that you always know what backend you are using, and you get exactly the speed of that backend. If it works, you are sure it works at the full speed, with no slow fallback. Theoretically, of course, there is always some overhead of FFI, but in Neanderthal it is so miniscule that you can ignore it for all uses that come to my mind. So, basically, Neanderthal is as fast as ATLAS on CPU and CLBlast on GPU (both offer state-of-the-art speed) or any (not yet existing) pure java engine that I might plug in in the future if necessary.

2. All those other libraries, besides not targeting Clojure at all except for general "you can call Java from Clojure", are trying to be everything for everyone. That has its strengths, because you are, generally, able to accommodate more use cases. On the other hand, it complicates things too much, and can lead to overblown beasts. For example, it might seem good to support MKL, and, ATLAS, and OpenBLAS, and netlib BLAS, and some imaginary fallback solution, like ND4J does (or tries to do), but what's the point of it when today they have more or less the same performance (MKL being a bit faster - in percentages only - but requires $$), and supporting all that stuff makes code AND the installation much, much, more compelex. BLAS is so mature that I think it is better to choose one solution and offer it out of the box. Technically, neanderthal can support all other native blas libraries too, but I intentionally restricted that option because I think fiddling with it does more harm than good. I prefer to give users one Ford model T, than let them choose between 20 different horse carriages. And, if they can even choose the color, provided that their choice is black :)

3. ND4J is, in my opinion, a typical overblown solution. deeplearning4j guys got the investment dollars, and have to rush to the market with business-friendly solution, which usually favors having a checklist of features regardless of whether those features make sense for the little guy. I hope they succeed in the business sense, but the code I'm seeing from them does not seem promising to me regarding Java getting a great DL/NN library.

4. NDArray is actually, as the name suggests, a nd array library, and not a matrix library. Why is this important?  Vectors and matrices are something that has been very well researched through decades. The ins and outs of algorithm/architecture fit are known and implemented in BLAS libraries, so you are sure that you get the full performance. N-dimensional vectors (sometime referred as tensors, although that name is not accurate IMO) not so much. So, it is easy that an operation that looks convenient does not lead to a good performance. I do not say it is bad, because if you need that operation, it is better to have something than nothing, but for now I decided to not support 3+ dimensions. This is something that might belong to Neanderthal or on top of it. A long term goal, to be sure. Another aspect of that story is knowledge: most books that I read from the fields of ML/AI give all formulas as vectors of matrices. Basically, matrices are at least 95% (if not more) of potential users need or even understand!

5. BidiMat seems to have much larger scope. For example, at their benchmark page, I see benchmarks for machine learning algorithms, but for nothing matrix-y.

The speed comparison; it boils down to this: both Neanderthal and those libraries use (or can be linked to use) the same native BLAS libraries. I took great care to make sure Neanderthal does not incur any copying or calling overhead. From what I saw glancing at the code of other libraries, they didn't. They might support that if you set up everything well among lots of options, or they don't if you do not know how to ensure this. So I doubt any of those could be noticeably faster, and they can be much slower if you slip somewhere.

I would also love to see straightforward numbers, but I was unable to find anything like that for those libraries. BidiMat, for example, gives benchmarks of K-Means on MNIST dataset - I do not know how this can be used to discern how fast is it with matrices, other that it is, generally, fast at K-Means.

[Dragan Djuric]

Just a small addition: I looked at BidiMat's code and even at the JNI/C level they are doing some critical things that work on small scale but byte unexpectedly when JVM needs to rearrange memory and also may trigger copying.

[kovas boguta]

On Thu, Oct 6, 2016 at 4:46 PM, Dragan Djuric <drag...@gmail.com> wrote:

 

s more harm than good. I prefer to give users one Ford model T, than let them choose between 20 different horse carriages. And, if they can even choose the color, provided that their choice is black :)

Thanks the for the comments, which I largely agree with. 

I understand your design philosophy and agree its a useful point in the space to have. 

One question:

Is it possible to feed Neanderthal's matrix representation (the underlying bytes) into one of these other libraries, to obtain computations Neanderthal doesn't support? 

My situation: Neanderthal covers some of the algorithms I'd like to implement. It looks easier to get started with and understand than the alternatives. But down the line I'll likely want to do convolution, sigmoid, the typical Neural Network layers. Note, I don't care about 'tensors' exactly; the bookkeeping to simulate a tensor can be automated, but the fundamental operations cannot. So there is a question of whether to just absorb the learning curve and shortcomings of these more general libraries rather than to risk switching horses midstream. 

I imagine I'm not alone in this.. if there was a straightforward story for how to interop Neanderthal when necessary with some more general library that would be powerful. Unfortunately I'm not sufficiently expert to evaluate which of the choices would be most pragmatic and how to pull it off. 

[Mikera]

Hi Dragan,

We have things working quite well (including stuff like cuDNN integration for convolution networks on the GPU). We also have all of the standard stuff (many different layer types, dropout, noise function, regularisation etc.). However I think it still needs a bunch of work before we stabilise on the core API.

Opening the source is not entirely my decision, this is a collaboration with the Thinktopic folks (Jeff Rose et al.). I'm personally in favour of being pretty open about this stuff but I do think that it would be a mistake if people build too much stuff on top of the wrong API which is likely to happen if we release prematurely.

Things I'm paricularly keen to have nailed down in particular before we go public:

1. A tensorflow-like DAG model that allows arbitrary operations to be composed compose into (possibly nested) graphs

2. Enough abstraction that different backends work (ClojureScript, Pure-JVM, Native, GPU etc.). core.matrix provides most of this, but there are still some deep-learning specific operations that need tuning and can be quite backend-specific, e.g. cuDNN has some specific ways of dealing with mini-batches which we need to get right. I'd love to try Neanderthal with this too if we can get the core.matrix integration working.

3. Decent, stable APIs for the algorithms that you typically want to run for mini-batch training, cross-validation etc.

4. Pluggable gradient optimisation methods (we currently have stuff like ADADELTA, SDG, ADAM etc. but would like to make sure this is sufficiently general to support any optimisation method)

I'll have a talk with the Thinktopic folks and see if we can come up with a timeline for a open source release. In the meantime, if anyone is *really* interested then we may be able to arrange collaboration on a private basis.

[Mikera]

I'm hoping to work with Dragan to get core.matrix integration working with Neanderthal, now that Windows support is finally arriving. This would get you a few big advantages:

1. Neanderthal could be used as a drop-in replacement for other core.matrix implementations if it fits your use case

2. You would get extra core.matrix features (additional operations, higher dimensional array operations, type conversion, broadcasting etc. essentially for free)

3. We minimise the risk of fragmentation in the Clojure numerical ecosystem :-)

4. We could use Neanderthal as a optional backend for libraries like Incanter or our new NN libraries without any major re-engineering

I'm certainly willing to contribute a fully compliant core.matrix implementation for Neanderthal (will take me a couple of days as long as the Windows build works smoothly). This might not optimise all of the possible operations, but should get high performance in most of the common use cases.

Hope you will all support this effort!

 

 

[kovas boguta]

On Thu, Oct 6, 2016 at 9:20 PM, Mikera <mike.r.an...@gmail.com> wrote:

Hi Dragan,

We have things working quite well (including stuff like cuDNN integration for convolution networks on the GPU). We also have all of the standard stuff (many different layer types, dropout, noise function, regularisation etc.). However I think it still needs a bunch of work before we stabilise on the core API.

 

Things I'm paricularly keen to have nailed down in particular before we go public:

FWIW it sounds like you've achieved a huge amount already. 

There are many people in the Clojure community who can come up with a DAG abstraction. There are very, very few who have the skill and time to assess and integrate the various native libs necessary to achieve the fundamental operations in a clojure-friendly way. 

If people build on the 'wrong' api, thats a good problem to have. The field is so in flux anyway. The problem can also be mitigated through minimalism in what is released in the beginning. 

In any case, looking forward to hopefully seeing this stuff one day. 

 

[kovas boguta]

On Thu, Oct 6, 2016 at 9:26 PM, Mikera <mike.r.an...@gmail.com> wrote:

I'm hoping to work with Dragan to get core.matrix integration working with Neanderthal, now that Windows support is finally arriving. This would get you a few big advantages:

Yes, I can see how my problem relates to the core.matrix vision. The only missing piece is the actual operations I want (nn layers) :)

[Sunil S Nandihalli]

I think deeplearning4J is a contender for deeplearning in clojure. I have not used it .. but I repeatedly see the sponsored link on clojure-reddit. Since nobody mentioned it .. I thought of mentioning it

[Joachim De Beule]

For those interested: here's some work on using deeplearning4j in clojure (no abstractions added, simply a port of the nd4j and deeplearning4j API's to 'clojuresque' functions and multimethods.)

https://github.com/engagor/dl4clj

If anybody wants to contribute they're welcome of course!

Op vrijdag 7 oktober 2016 06:22:48 UTC+2 schreef Sunil Nandihalli:

I think deeplearning4J is a contender for deeplearning in clojure. I have not used it .. but I repeatedly see the sponsored link on clojure-reddit. Since nobody mentioned it .. I thought of mentioning it

On Fri, Oct 7, 2016 at 7:40 AM, kovas boguta <kovas....@gmail.com> wrote:

On Thu, Oct 6, 2016 at 9:26 PM, Mikera <mike.r.an...@gmail.com> wrote:

I'm hoping to work with Dragan to get core.matrix integration working with Neanderthal, now that Windows support is finally arriving. This would get you a few big advantages:

Yes, I can see how my problem relates to the core.matrix vision. The only missing piece is the actual operations I want (nn layers) :)

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Dragan Djuric]

Hi Kovas,

One question:

Is it possible to feed Neanderthal's matrix representation (the underlying bytes) into one of these other libraries, to obtain computations Neanderthal doesn't support? 

There are two parts to that question, I think: 1) How can you make Neanderthal work with any other library that you need to interoperate with? 2) Does Neanderthal have the operation that you need, and what to do if it doesn't?

I'll start with 2:

2) Currently, Neanderthal supports all BLAS 1, 2 and 3 operations for vectors and dense matrices on CPU & GPU. The ultimate goal is to support all other standard matrix formats (TR, sparse, etc.) AND LAPACK, which has extensive support for linear algebra. The good news is that what is there works really, really well, because I concentrated my efforts on solving the hardest problem first (and succeeded!). Now it is a matter of putting the grunt work to repeat what's done for the existing things to cover more of CBLAS and LAPACK API, and even to do the integration with CUDA in a similar way I did the OpenCL integration. I could have even done it by now, but I preferred to work on other things, one of those being a bayesian data analysis library Bayadera, that puts what Neanderthal offers to great use :) I have also seen that the Yieldbot people forked Neanderthal and implemented some part of LAPACK, but did not release anything nor issued a PR. So, if the methods you need fall into the scope of matrices and linear algebra (BLAS + LAPACK), there is a good chance it will be supported, either by you or some other user providing it, or bugging me often enough that I realize it is urgent that I add it :)

1) There are at least two parts to the interoperation story - API for operations (like mm vs mmult or whatever) and that is the very easy part. The hard part is a multitude of matrix formats and those formats' representations in memory. This is what makes or breaks your performance, and not by a few percents but by a few orders of magnitude. The sad part is that almost all focus is always on the easy part, completely ignoring the hard part or just thinking that it will magically solve itself. So, suppose that you have data laid out in memory in the format A. That format may or may not be suitable for operation X, and if it is not, it is often a bad idea to shoehorn it in for convenience, instead of thinking harder about data flow and transition data to format B to be used appropriately. That means that even inside the same library, you often need to do the data transformations to best suit what you want to do with the data. Long story short, you need to do data transformations anyway, so having Neanderthal and ND4J support core.matrix mmult operation won't help you a bit here. You'll have to transform data from the one to the other. If you are lucky, they use the same underlying format, so the transformation is easy or even automatic, or can be, but the point is that someone needs to create explicit transformations to ensure the optimal way instead on relying on generic interoperability layer (at least for now).

 

My situation: Neanderthal covers some of the algorithms I'd like to implement. It looks easier to get started with and understand than the alternatives. But down the line I'll likely want to do convolution, sigmoid, the typical Neural Network layers. Note, I don't care about 'tensors' exactly; the bookkeeping to simulate a tensor can be automated, but the fundamental operations cannot. So there is a question of whether to just absorb the learning curve and shortcomings of these more general libraries rather than to risk switching horses midstream. 

I think it is important to note that the operations you mentioned are not in the scope of a matrix library, but in the scope of a neural networks library. Neanderthal is simply not a library that should have those operations, nor it can have all operations for all ML techniques (that are countlesss :)

 

On the other hand, what I created Neanderthal for, is exactly as a building block for such libraries. The focus is on: if you need to build a NN library, Neanderthal should (ideally) give you a standard matrix methods for computations and data shuffling, Clojure should enable you to create a great interface layer, and (if needed) ClojureCL should help you write custom optimized low-level algorithms for GPU and CPU. 

I imagine I'm not alone in this.. if there was a straightforward story for how to interop Neanderthal when necessary with some more general library that would be powerful. Unfortunately I'm not sufficiently expert to evaluate which of the choices would be most pragmatic and how to pull it off. 

Today's state (in Clojure, Java, and probably elsewhere) is, IMO: if you need a ready-made solution for NNs/DL, you have to pick one library that has the stuff that you need, and go with what they recommend completely. For example, let's say that you need to use deeplearning4J. It requires ND4J matrices. Let's also say that someone creates a decent core.matrix wrapper for ND4J, and you use it. Now, 95% of your code will be tied to DL4J, and you won't be able to just swap ND4J and use Vectorz instead, unless you create ND4J<->Vectorz transformation to help you on the API side. Even then, ND4J does not care about Vectorz, so it will require you to give it ND4J data. It may or may not be made to work, but it will not magically work well.

On the other hand, if you need to implement your own algorithms and help building a Clojure-based high-performance computing ecosystem, Neanderthal is exactly what I'd have in mind. 

[Dragan Djuric]

Hi Mike,

Thanks for the update.

Opening the source is not entirely my decision, this is a collaboration with the Thinktopic folks (Jeff Rose et al.). I'm personally in favour of being pretty open about this stuff but I do think that it would be a mistake if people build too much stuff on top of the wrong API which is likely to happen if we release prematurely.

I believe that number of users that would try this is small anyway, and they will use exclusively for evaluation and hobby at first, so, as Kovas said, having people build too much on it is something we can only hope for! :) I hope the Thinktopic folks would support opening it...

 

Things I'm paricularly keen to have nailed down in particular before we go public:

1. A tensorflow-like DAG model that allows arbitrary operations to be composed compose into (possibly nested) graphs

2. Enough abstraction that different backends work (ClojureScript, Pure-JVM, Native, GPU etc.). core.matrix provides most of this, but there are still some deep-learning specific operations that need tuning and can be quite backend-specific, e.g. cuDNN has some specific ways of dealing with mini-batches which we need to get right. I'd love to try Neanderthal with this too if we can get the core.matrix integration working.

3. Decent, stable APIs for the algorithms that you typically want to run for mini-batch training, cross-validation etc.

4. Pluggable gradient optimisation methods (we currently have stuff like ADADELTA, SDG, ADAM etc. but would like to make sure this is sufficiently general to support any optimisation method)

I'll have a talk with the Thinktopic folks and see if we can come up with a timeline for a open source release. In the meantime, if anyone is *really* interested then we may be able to arrange collaboration on a private basis.

It's not urgent, so any rough estimate would be great! 

[Dragan Djuric]

If people build on the 'wrong' api, thats a good problem to have. The field is so in flux anyway. The problem can also be mitigated through minimalism in what is released in the beginning. 

This. 

[je...@thinktopic.com]

Hey,
  I'm glad this came up.  We were initially thinking we'd wait until the API and design had stabilized before releasing Cortex, but there is enough of value now that I think Kovas is right.  We should release it.

Cortex has been a collaboration between ThinkTopic and Mike Anderson, and it's design is somewhat similar to Torch.  (Neural network layers that implement protocols for forward computation and backward propagation of gradients along with a set of optimizers.)  Although we are already using Cortex models in production, it's definitely still a library in flux.  We had arrived at a pretty good set of base abstractions in pure Clojure using core.matrix, but then when we decided to add gpu support we realized we had to refactor things a bit.  The CuDNN and CuBLAS libraries from Nvidia provide a lot of useful functionality, but they also come with their own abstractions for matrix computation, neural network layers, optimizers, etc.  To take advantage of the GPU effectively we also need to be able to sequence and interleave tasks using streams, which requires a variety of design changes unless we want to maintain separate implementations of almost everything for both the CPU and the GPU.  You can build and run networks now, but we are still exploring how this should all come together.

So, as long as you keep in mind that things will change, it would be great to have you all join the conversation and help experiment with abstractions and apis. 

Give us a couple days to do some house keeping, and then we'll open source it.

-Jeff

[je...@thinktopic.com]

Hi,
  We've made cortex public:

                 https://github.com/thinktopic/cortex

Fork away, and we hope that this contributes to a growing ML community in Clojure.  Thoughts, ideas, feedback are welcome!

Cheers,
Jeff

[kovas boguta]

This is amazing!! Thanks so much for releasing it. Very excited to dig in. 

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+unsubscribe@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.

[Boris V. Schmid]

Small addition to this post:

There is a tiny library (toy project) of ddosic, who build a neural network with neanderthal. It might be interesting as a benchmark of what speed neanderthal can reach (although it might not currently be a good reflection of neanderthal), versus larger packages with more overhead.

https://github.com/ddosic/affe

[Dragan Djuric]

Please note that this is a really a TOY nn project, actually a direct translation of gigasquid's nn hello-world. It is ridiculous to compare it with a library that delegates nn work to cuDNN. And it is a really old version of affe, that ddosic has improved in the meantime, but have not yet released.

What could be fair, for example, is to compare affe with another toy library built with core.matrix, or to compare a cuDNN bindings built with help from neanderthal's CUDA engine (not yet available) with Cortex, or to compare Cortex with something that delegates to one of experimental OpenCL-based "real" nn libraries...