concatenating layers (say, linear + input)

[boris....@gmail.com]

I am looking through the code of cortex.nn.layers and cortex.graph, but it is not clear to me how I would go about concatenating the input layer to a network at some deeper layer. Looks possible though.

I was thinking of something like

(network/linear-network [(layers/input 8 1 1 :id :data)
                                       (layers/linear 40 :l2-regularization 0.05) (layers/prelu)
                                       (layers/linear 40 :l2-regularization 0.05) (layers/prelu)
                                       (layers/concat [(layers/linear 20 :l2-regularization 0.05) (layers/prelu)] [(layers/input 8 1 1 :id :data)])

                                       (layers/linear 40 :l2-regularization 0.05) (layers/prelu) 

                                       (layers/linear 2)
                                       (layers/softmax :id :labels)])

 where somewhere halfway the network, I feed the network the input layer again.

I can imagine you are thinking/working on a syntax that is more universal than just adding the input layer again, so maybe a layers/store and layers/retrieve command, together with a layers/concat command.

(network/linear-network [(layers/input 8 1 1 :id :data)
                                       (layers/linear 40 :l2-regularization 0.05) (layers/prelu)

                                       (layers/store :output_of_above_layer)

                                       (layers/linear 40 :l2-regularization 0.05) (layers/prelu)
                                       (layers/concat [(layers/linear 20 :l2-regularization 0.05) (layers/prelu)] [(layers/retrieve :output_of_above_layer)])

                                       (layers/linear 40 :l2-regularization 0.05) (layers/prelu) 

                                       (layers/linear 2)
                                       (layers/softmax :id :labels)])

[Harold]

Hi Boris,

That's a neat idea, thanks for sharing it. I don't know off-hand if that's possible and the syntax isn't quite right, or if doing that would require changes in cortex.

I created this issue:

https://github.com/thinktopic/cortex/issues/168

Chris or Charles there may know. Regardless, I'd like to see more documentation about the layer graph operations moving forward.

Take care,

-Harold

[Chris Nuernberger]

Hey Harold, Boris:

I replied on the issue.  Agreed it would be good to have a network that takes several distinct input streams and also an example of a multiple-target network.