RNN for cortex

[jw]

trying to implement a simplified version of

https://github.com/fchollet/keras/blob/master/examples/addition_rnn.py

for cortex. 

[jw]

One existing implementation:

https://github.com/fchollet/keras/blob/master/keras/layers/recurrent.py

a few questions though:

1) I also looked at cudnn documentation and after I subscribed, I could load the shared objects/archives (.so/.a) files. 

However I did not find the source code? Is it available somewhere (would be interested in their implem. of rnn, which you find int the doc.)?

2) I am using vim for clojure. Is it realistic for dev/debugging/visualisation with cortex? If not, any recommendation?

Regards, 

J

[Chris Nuernberger]

Hey JW:

I think you will have to create a layer with a bunch of parameters in a similar vein as the cudnn API.  

Basically it should work like a linear layer or an number of other layers just with some complex details.

An interesting blog post to think about:

https://devblogs.nvidia.com/parallelforall/optimizing-recurrent-neural-networks-cudnn-5/

[jw]

Thank you!

I suggest that we start with "simple RNN" rather than a LSTM (see diagram in http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/).

For this and with your lib., one would need to generate/coordinate:

1) directed graphs with one "stream" flowing transversally within a layer (for forward and backward propag.)

2) nodes with one incoming stream and two going out

That kind of structure is needed (see http://d3kbpzbmcynnmx.cloudfront.net/wp-content/uploads/2015/10/rnn-bptt-with-gradients.png)

With these basic operations, it should be straightforward to do RNNs.

I have looked at concat, join, split layers, but how to use them for building the above (do you have examples)?

Kind Regards!