How to implement this type of shared weight neural network in pylearn2?

[gammapoint]

Hi, I'm just getting started with pylearn2, and I followed the tutorials with no difficulty. Now I'm looking to construct a neural network of a particular topology, and I'm curious if what is needed is already implemented or if I would need to implement something myself. 

I'd like to have a network of the following form, where my input vector is broken up into chunks and each one passes through a neural net which shares the weights and biases along each path. At the end of the paths, the resultant values are simply summed to make what is the known target value for my training.  

Does this sort of functionality exist at present in pylearn2? Specifically, is it possible to share weights among different parts of the network, construct branching type topologies like the above, and then sum them back together? If not, any hints as to how I might begin going about implementing this would be very helpful!

Thanks,

GP

[Ian Goodfellow]

What is different between the two paths? If nothing is different, just run one of the networks and multiply its output by 2.

--
You received this message because you are subscribed to the Google Groups "pylearn-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylearn-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Sent from Gmail Mobile

[Brad Malone]

Hi Ian, thanks for the response.

What is different between the two paths? If nothing is different, just run one of the networks and multiply its output by 2.

The top 4 inputs are not the same as the bottom 4 inputs, so E_1 and E_2 are not the same. But my data set doesn't include E_1 or E_2, but only their sum. 

 

In a typical run, we don't have 2 such branches (as shown in the figure for simplicity), but many of them, all which sum to one value at the end that we know. For no two branches are E_i and E_j related by some simple integer. 

[Brad Malone]

To say it differently, my data set is composed of points consisting of an 8-dimensional input vector (green dots) and a one-dimensional output vector/scalar (red dot). 

[Ian Goodfellow]

I'm not sure what happens when you feed two copies of the same inner Layer to a CompositeLayer. It might just work, or it might tell you you're not allowed to do that. If the latter occurs, you could make a ReplicatedLayer subclass of CompositeLayer. You can use the CompositeLayer's routing feature to send one half of the input to each subnetwork.

If you want each subnetwork to output a sigmoid then sum their outputs, you can do the fprop using the Sum layer from the adversarial nets repo.

Training such a network will require a new cost method though, since your output is now a mixture of sigmoidal Bernoullis rather than a single sigmoidal Bernoulli. 

--
You received this message because you are subscribed to the Google Groups "pylearn-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylearn-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brad Malone]

Great, thanks for all the information Ian! I haven't dug enough into the code internals yet to fully appreciate your answer, but this will be very useful to get me going (at the very least I wanted to make sure all this functionality wasn't already included but under a name I wouldn't have grep'ed for). 

If you want each subnetwork to output a sigmoid then sum their outputs, you can do the fprop using the Sum layer from the adversarial nets repo.

By the adversarial nets repo you mean this (https://github.com/goodfeli/adversarial/search?p=1&q=Sum&ref=cmdform)? Because my simple search isn't showing me an obvious Sum layer. 

Thanks.

[Ian Goodfellow]

https://github.com/goodfeli/adversarial/blob/master/__init__.py

class Sum

Il venerdì 20 giugno 2014, Brad Malone <brad....@gmail.com> ha scritto:

--

You received this message because you are subscribed to the Google Groups "pylearn-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pylearn-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Brad Malone]

Ahh, thanks Ian.

You received this message because you are subscribed to a topic in the Google Groups "pylearn-users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/pylearn-users/JEVTpa5ZvDk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to pylearn-user...@googlegroups.com.