combining or concatenating from two separate networks?

[Chen-Ping Yu]

Hi all,

I've been trying to combine 2 networks into one with concat but I kept running into problems and can't seem to get it implemented...the network structure is attached, it is a very simple network, basically I have 2 separate networks, each has their own conv layers, then after reshaping/viewing their last conv layer, I'd like to concatenate their two 1d vectors into a single 1d vector, then that vector would continues to have the subsequent fully connected layers. anyone has a good way of implementing that? Thanks!

                           

[Vislab]

I think this is what you are looking for: https://github.com/torch/nn/blob/master/doc/table.md#jointable

You can check out this package for easier network management: https://github.com/torch/nngraph

[Chen-Ping Yu]

oooh jointable seems to be doing the trick!

yea I did find nngraph but I'm having a hard time connecting the example illustration and the example code, how did the node numbers get numbered in the example illustrations? 

[Vislab]

I've never used nngraph so I can't give you any assistance regarding that package. Hopefully someone else may know more about it.

[Koustav Mullick]

Hi,

The following code achieves what you want, but not using nngraph.

I am going to make some assumptions over here,

1) input1 = input2 = torch.Tensor(3, 16, 16)  -- RGB images of dimension 16X16.

2) Let's call the two parallel networks net1 and net2 respectively (as given by conv1a and conv 1b in your example).

3) Structure of net1 is convolution (3->16, kernel 3 X 3, stride 1 X 1), activaltion (RELU) followed by Max Pooling (2 X 2).

4) Structure of net2 is convolution (3->16, kernel 5 X 5, stride 1 X 1), activaltion (RELU) followed by Max Pooling (2 X 2).

5) The outputs froms net1 and net2 are of dimension (16 X 7 X 7) and (16 X 6 X 6) respectively, after applying nn.View() in each.

6) So your final concatenated output is of dimension (16 X 7 X 7) + (16 X 6 X 6) = 1360 (2a and 2b in your given example. You can insert Fully Connected layers over here if you want in each of net1 and net2. The output dimensions will also change accordingly then).

7) Finally an FC layer mapping 1360 to 100 (say). So your final output is of dimension 100.

The code to get the above described network is as follows :

net1 = nn.Sequential()
net1:add(nn.SpatialConvolution(3, 16, 3, 3))
net1:add(nn.ReLU())
net1:add(nn.SpatialMaxPooling(2, 2))
net1:add(nn.View(16*7*7))  --  flattening out


net2 = nn.Sequential()
net2:add(nn.SpatialConvolution(3, 16, 5, 5))
net2:add(nn.ReLU())
net2:add(nn.SpatialMaxPooling(2, 2))
net2:add(nn.View(16*6*6))  -- flattening out


parallel_model = nn.Parallel(1, 1)  -- model that concatenates net1 and net2
parallel_model:add(net1)
parallel_model:add(net2)


--  Putting everything together
model = nn.Sequential()

model:add(parallel_model)
model:add(nn.Linear(1360, 100))  -- the final FC layer



input = torch.Tensor(2, 3, 16, 16)
out = model:forward(input)



out:size()


100
[torch.LongStorage of size 1]

Note: For ease of understanding, in this I have considered both the inputs to net1 and net2 of same dimension (3 X 16 X 16), as can be seen from input = torch.Tensor(2, 3, 16, 16). If you want to have different sized inputs, have a look at nn.ParallelTable(). You van use that instead of nn.Parallel().

[Chen-Ping Yu]

HI Koustav,

Thank you for your example! I just wanted to make sure a few things:

I'm always kind of confused on how to use nn.parallel, and the other related containers; by looking at the way you're using it, can I assume that when you create a nn.parallel and add 2 networks to it, it will always concatenate the ending layer of the 2 networks together (concatenate the ending layer of the added networks)? Also, My 2 inputs are the same size and dimension yes, but different input images, but i guess I can just put them as the 1st and the 2nd 3x16x16 within the 2x3x16x16.

However, do you know how I can use mini-batch with your type of implementation? I get the following error if I try a mini-batch size of 128:

input = torch.Tensor(128,2,3,16,16)

out = model:forward(input)

/data/torch/install/share/lua/5.1/nn/PArallel.lua:18: attempt to index a nil value

Thanks!!

[Chen-Ping Yu]

I think I got it:

...

parallel_model = nn.Parallel(2, 2) -- split the input's 2nd dimension and pass each of them through the parallel modules, then concatenate the result's 2nd dimension together

... 

input = torch.Tensor(128, 2, 3, 16, 16)

out = model:forward(input)


out:size()

128
100
[torch.LongStorage of size 2]

[Koustav Mullick]

Hi,

Have a look at the doc for nn.Parallel() over here. You need to specify the input-dimension and output-dimension as parameters to it.

Input-dimension is the dimension along which you want to divide your input before sending in to your parallel modules. So since in my example I had no batch, a simple (2, no_channels, height, width), hence my input-dimension in nn.parallel() was 1. 

As you have figured out correctly, in case of batch, instead of 1, you want it to split along the second dimension of (batch_size, 2, no_channels, height, width) .

Also, yes, nn.Parallel() will always concatenate the ending layer of the networks you put in the module. You can specify along which dimension you want it to concatenate. In my example since I had flattened out the outputs using nn.View(), just concatenating along dimension 1 of output served the purpose. In case of batch similarly you need to concatenate along the second dimension. Also, instead of nn.View(), if my last layer is something else like a max pooling or convolution layer, then also I can specify the dimension along which I want to concatenate.

[Koustav Mullick]

Everything I said, you already seem to have figured out, from your last example. :)

[Chen-Ping Yu]

right, but hey still greatly appreciated your help Koustav!

[Chen-Ping Yu]

So something weird is happening..

if I convert the network into cudnn, like:

cudnn.convert(model,cudnn)

then I get the following error:

cudnn.convert(model, cudnn);

model = model:cuda();

t = torch.rand(10,2,3,16,16):cuda();

out = model:forward(t)

...torch/install/share/lua/5.1/cudnn/SpatialConvolution.lua:101: assertion failed!
stack traceback:
    [C]: in function 'assert'
    ...torch/install/share/lua/5.1/cudnn/SpatialConvolution.lua:101: in function 'createIODescriptors'
    ...torch/install/share/lua/5.1/cudnn/SpatialConvolution.lua:349: in function 'updateOutput'
    /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput'
    /data/torch/install/share/lua/5.1/nn/Parallel.lua:18: in function 'updateOutput'
    /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
    [string "out = mlp:forward(t:cuda());"]:1: in main chunk
    [C]: in function 'xpcall'
    /data/torch/install/share/lua/5.1/trepl/init.lua:648: in function 'repl'
    /data/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:185: in main chunk
    [C]: at 0x00406670    

The assertion is asserting that the input must have dim() == 4, and isContiguous( ), does it mean that nn.Parallel splits the input tensor into slices, and the slice is (10,1,3,16,16) become going into the conv layer, and since that is 5 dimensional it fails the assertion? If so, does it mean that I cannot use nn.Parallel with cudnn? that's going to be a very very slow training........

[soumith]

let me push a fix for this.

--
You received this message because you are subscribed to the Google Groups "torch7" group.
To unsubscribe from this group and stop receiving emails from it, send an email to torch7+un...@googlegroups.com.
To post to this group, send email to tor...@googlegroups.com.
Visit this group at https://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

[Chen-Ping Yu]

great thanks!

I guess I will be waiting for the fix and then update cudnn after it is done?

On Thursday, February 25, 2016 at 11:11:50 PM UTC-5, smth chntla wrote:

let me push a fix for this.

[soumith]

Ok I pushed a fix.

Now do: luarocks install cudnn

After that no error.

Fixed via: https://github.com/soumith/cudnn.torch/pull/125

On Thu, Feb 25, 2016 at 11:17 PM, Chen-Ping Yu via torch7 <torch7+APn2wQfWL8x7t9n3lwCNlISDs...@googlegroups.com> wrote:

great thanks!

I guess I will be waiting for the fix and then update cudnn after it is done?

On Thursday, February 25, 2016 at 11:11:50 PM UTC-5, smth chntla wrote:

let me push a fix for this.

[Chen-Ping Yu]

great than solved the cudnn/SpatialConvlution problem! Thanks!

By the way I'm also getting this strange issue, if I explicitly code my network as cudnn.SpatialconVolution...cudnn.SpatialBatchNormalization...etc, it works fine just now after your fix.

However, if my network is coded using nn.xxx, nn.xxx, then convert them using cudnn.convert( ), I get the following error:

...h/install/share/lua/5.1/nn/SpatialBatchNormalization.lua:84: attempt to index field 'THNN' (a nil value)
stack traceback:
        ...h/install/share/lua/5.1/nn/SpatialBatchNormalization.lua:84: in function 'updateOutput'

        /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput'
        /data/torch/install/share/lua/5.1/nn/Parallel.lua:18: in function 'updateOutput'
        /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'

        [string "out = mlp:forward(t);"]:1: in main chunk
        [C]: in function 'xpcall'

        /data/torch/install/share/lua/5.1/trepl/init.lua:648: in function 'repl'
        /data/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:185: in main chunk
        [C]: at 0x00406670

Any idea what is causing that? I did update cutorch, cunn, nn, cudnn to the latest version...

I also noticed that cudnn.convert did not convert nn.SpatialBatchNormalization into cudnn.SpatialBatchNormalization, is that something that might need to be updated as well for cudnn? Thanks.

On Thursday, February 25, 2016 at 11:28:25 PM UTC-5, smth chntla wrote:

Ok I pushed a fix.

Now do: luarocks install cudnn

After that no error.

Fixed via: https://github.com/soumith/cudnn.torch/pull/125

On Thu, Feb 25, 2016 at 11:17 PM, Chen-Ping Yu via torch7 <torch7+APn2wQfWL8x7t9n3lwCNlISDsxLM5IPD0mId5P4fyPO7rdnPMXmJRmYNf@googlegroups.com> wrote:

great thanks!

I guess I will be waiting for the fix and then update cudnn after it is done?

On Thursday, February 25, 2016 at 11:11:50 PM UTC-5, smth chntla wrote:

let me push a fix for this.

[soumith]

yes, reinstall nn and cunn, that is definitely the reason for the error.

luarocks install nn

luarocks install cunn

On Thu, Feb 25, 2016 at 11:51 PM, Chen-Ping Yu via torch7 <torch7+APn2wQfWL8x7t9n3lwCNlISDs...@googlegroups.com> wrote:

great than solved the cudnn/SpatialConvlution problem! Thanks!

By the way I'm also getting this strange issue, if I explicitly code my network as cudnn.SpatialconVolution...cudnn.SpatialBatchNormalization...etc, it works fine just now after your fix.

However, if my network is coded using nn.xxx, nn.xxx, then convert them using cudnn.convert( ), I get the following error:

...h/install/share/lua/5.1/nn/SpatialBatchNormalization.lua:84: attempt to index field 'THNN' (a nil value)
stack traceback:
        ...h/install/share/lua/5.1/nn/SpatialBatchNormalization.lua:84: in function 'updateOutput'
        /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput'
        /data/torch/install/share/lua/5.1/nn/Parallel.lua:18: in function 'updateOutput'
        /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
        [string "out = mlp:forward(t);"]:1: in main chunk
        [C]: in function 'xpcall'
        /data/torch/install/share/lua/5.1/trepl/init.lua:648: in function 'repl'
        /data/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:185: in main chunk
        [C]: at 0x00406670

Any idea what is causing that? I did update cutorch, cunn, nn, cudnn to the latest version...

I also noticed that cudnn.convert did not convert nn.SpatialBatchNormalization into cudnn.SpatialBatchNormalization, is that something that might need to be updated as well for cudnn? Thanks.

On Thursday, February 25, 2016 at 11:28:25 PM UTC-5, smth chntla wrote:

Ok I pushed a fix.

Now do: luarocks install cudnn

After that no error.

Fixed via: https://github.com/soumith/cudnn.torch/pull/125

[Chen-Ping Yu]

hmm, I followed your suggestion and the same error is still coming up,

in fact it was working before (sometimes it fails) but somehow after I did your two lines of updates, now i'm getting a bunch of /nn/THNN.lua errors, do you have any other suggestions..?

On Friday, February 26, 2016 at 12:08:25 AM UTC-5, smth chntla wrote:

yes, reinstall nn and cunn, that is definitely the reason for the error.

luarocks install nn

luarocks install cunn

On Thu, Feb 25, 2016 at 11:51 PM, Chen-Ping Yu via torch7 <torch7+APn2wQfWL8x7t9n3lwCNlISDsxLM5IPD0mId5P4fyPO7rdnPMXmJRmYNf@googlegroups.com> wrote:

great than solved the cudnn/SpatialConvlution problem! Thanks!

By the way I'm also getting this strange issue, if I explicitly code my network as cudnn.SpatialconVolution...cudnn.SpatialBatchNormalization...etc, it works fine just now after your fix.

However, if my network is coded using nn.xxx, nn.xxx, then convert them using cudnn.convert( ), I get the following error:

...h/install/share/lua/5.1/nn/SpatialBatchNormalization.lua:84: attempt to index field 'THNN' (a nil value)
stack traceback:
        ...h/install/share/lua/5.1/nn/SpatialBatchNormalization.lua:84: in function 'updateOutput'
        /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'updateOutput'
        /data/torch/install/share/lua/5.1/nn/Parallel.lua:18: in function 'updateOutput'
        /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
        [string "out = mlp:forward(t);"]:1: in main chunk
        [C]: in function 'xpcall'
        /data/torch/install/share/lua/5.1/trepl/init.lua:648: in function 'repl'
        /data/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:185: in main chunk
        [C]: at 0x00406670

Any idea what is causing that? I did update cutorch, cunn, nn, cudnn to the latest version...

I also noticed that cudnn.convert did not convert nn.SpatialBatchNormalization into cudnn.SpatialBatchNormalization, is that something that might need to be updated as well for cudnn? Thanks.

On Thursday, February 25, 2016 at 11:28:25 PM UTC-5, smth chntla wrote:

Ok I pushed a fix.

Now do: luarocks install cudnn

After that no error.

Fixed via: https://github.com/soumith/cudnn.torch/pull/125

[soumith]

Do you have multiple and old torch installs on your machine?
I suggest running: https://github.com/torch/ezinstall/blob/master/clean-old.sh

On Fri, Feb 26, 2016 at 12:27 AM, Chen-Ping Yu via torch7 <torch7+APn2wQfWL8x7t9n3lwCNlISDs...@googlegroups.com> wrote:

hmm, I followed your suggestion and the same error is still coming up,

in fact it was working before (sometimes it fails) but somehow after I did your two lines of updates, now i'm getting a bunch of /nn/THNN.lua errors, do you have any other suggestions..?

On Friday, February 26, 2016 at 12:08:25 AM UTC-5, smth chntla wrote:

yes, reinstall nn and cunn, that is definitely the reason for the error.

luarocks install nn

luarocks install cunn

[Chen-Ping Yu]

oooooh I found the problem !!!

I thought I only had to require 'nn' and require 'cudnn', but I didn't know I also had to require 'cunn'......requiring cunn solved this problem!! thanks!!

ok one last issue with the nn.Parallel framework, now everything trains and forward very smoothly with mini-batches of, say (128,2,3,16,16), assuming we're still using the example code that Koustav posted earlier. But If I want to test the network with a mini-batch size of 1: (1,2,3,16,16), I get the following error:

input = torch.rand(1,2,3,16,16):cuda();

out = model:forward(input)

/data/torch/install/share/lua/5.1/nn/Parallel.lua:20: bad argument #1 to 'size' (out of range)
stack traceback:
    [C]: in function 'size'
    /data/torch/install/share/lua/5.1/nn/Parallel.lua:20: in function 'updateOutput'

    /data/torch/install/share/lua/5.1/nn/Sequential.lua:44: in function 'forward'
    [string "out = mlp:forward(t);"]:1: in main chunk
    [C]: in function 'xpcall'
    /data/torch/install/share/lua/5.1/trepl/init.lua:648: in function 'repl'
    /data/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:185: in main chunk
    [C]: at 0x00406670    

 

so the 1st dimension of my input must be greater than 1, but cannot be 1, how should I correctly pass in a single input for a forward pass with the nn.Parallel container?

On Friday, February 26, 2016 at 12:44:32 AM UTC-5, smth chntla wrote:

Do you have multiple and old torch installs on your machine?
I suggest running: https://github.com/torch/ezinstall/blob/master/clean-old.sh

On Fri, Feb 26, 2016 at 12:27 AM, Chen-Ping Yu via torch7 <torch7+APn2wQfWL8x7t9n3lwCNlISDsxLM5IPD0mId5P4fyPO7rdnPMXmJRmYNf@googlegroups.com> wrote:

hmm, I followed your suggestion and the same error is still coming up,

in fact it was working before (sometimes it fails) but somehow after I did your two lines of updates, now i'm getting a bunch of /nn/THNN.lua errors, do you have any other suggestions..?

On Friday, February 26, 2016 at 12:08:25 AM UTC-5, smth chntla wrote:

yes, reinstall nn and cunn, that is definitely the reason for the error.

luarocks install nn

luarocks install cunn

[Koustav Mullick]

Hi,

Try using nn.View(-1, whatever_dimension).

-1 suggests the mini-batch dimension, otherwise nn.View() squeezes the output.

That is, an input of 1 X some_dimension is becoming just some_dimension. That is not the case is it's N X some_dimension, where N > 1. 

From the doc, "This makes it possible to use minibatch inputs when using a size -1 for one of the dimensions".

Thus in net1 it becomes 

net1:add(nn.View(-1, 16*7*7))

[Koustav Mullick]

If you find this approach kind of messy, you can use nn.Reshape() instead of nn.View().

[Francisco Vitor Suzano Massa]

I think it's easier to use :setNumInputDims(nbOfDims) in nn.View(). Something like

nn.View(-1):setNumInputDims(3) -- if the input is supposed to be spatial, it has 3 dimensions

View will then properly handle batched/non-batched inputs, because it knows that the data should have 3 dimensions originally, and if it's > 3 than it's batch mode.

[Chen-Ping Yu]

Hi guys,

I've tried both of your suggestions, and both worked like a charm for batch or non-batch forward pass, thanks!!

However, when I tried for a simulated training step to obtain the error gradient, during the model:backward part the nn.View complains that the input isn't contiguous, and this same error happens with both of your suggested use of nn.View:

net1 = nn.Sequential()

net1:add(nn.SpatialConvolution(3, 16, 3, 3))

net1:add(nn.SpatialBatchNormalization(16,1e-3))

net1:add(nn.ReLU())

net1:add(nn.SpatialMaxPooling(2, 2))

--net1:add(nn.View(-1):setNumInputDims(3))  --works well for forward pass

net1:add(nn.View(-1,16*7*7))                --also works well for forward pass

net2 = nn.Sequential()

net2:add(nn.SpatialConvolution(3, 16, 3, 3))

net2:add(nn.SpatialBatchNormalization(16,1e-3))

net2:add(nn.ReLU())

net2:add(nn.SpatialMaxPooling(2, 2))

--net2:add(nn.View(-1):setNumInputDims(3))

net2:add(nn.View(-1,16*7*7))

parallel_model = nn.Parallel(2, 2)  

parallel_model:add(net1)

parallel_model:add(net2)

model = nn.Sequential()

model:add(parallel_model)

model:add(nn.Linear(16*7*7*2, 100))

model:add(nn.LogSoftMax())

model = model:cuda()

cudnn.convert(model, cudnn)

inputs = torch.rand(10,2,3,16,16):cuda();

outputs = model:forward(inputs)

-- getting the error gradients

criterion = nn.ClassNLLCriterion():cuda()

parameters, gradParameters = model:getParameters()

gradParameters:zero();

targets = torch.ones(10):cuda();

f = criterion:forward(outputs, targets) 

df_do = criterion:backward(outputs, targets)

model:backward(inputs, df_do)   

/data/torch/install/share/lua/5.1/torch/Tensor.lua:457: expecting a contiguous tensor

stack traceback:

        [C]: in function 'assert'

        /data/torch/install/share/lua/5.1/torch/Tensor.lua:457: in function 'view'

        /data/torch/install/share/lua/5.1/nn/View.lua:90: in function 'updateGradInput'

        /data/torch/install/share/lua/5.1/nn/Sequential.lua:55: in function 'updateGradInput'

        /data/torch/install/share/lua/5.1/nn/Parallel.lua:53: in function 'updateGradInput'

        /data/torch/install/share/lua/5.1/nn/Module.lua:30: in function 'backward'

        /data/torch/install/share/lua/5.1/nn/Sequential.lua:88: in function 'backward'

        [string "_RESULT={model:backward(inputs, df_do)    }"]:1: in main chunk

        [C]: in function 'xpcall'

        /data/torch/install/share/lua/5.1/trepl/init.lua:630: in function 'repl'

        /data/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:185: in main chunk

        [C]: at 0x00406670

                                                         

is it because of the -1 part of nn.View that is making it non-contiguous? Thanks for all the help by the way, almost got this all the way...!

[Koustav Mullick]

Hi,

The error is not because of the -1 part. Try removing it and using the batch mode as you were doing before, even then it will throw the error.

Reason is, nn.View() expects contiguous gradOutputs, which you are losing while nn.Parallel() splits the gradOutputs it is receiving, among it's parallel modules (unless it is the first dimension. 2nd in your case).

I generally use nn.Reshape() in these situatons.

If Francisco or someone else can come up with a better solution, it will be helpful for me as well. :)

 

[Chen-Ping Yu]

Hi Koustav,

As you've suggested, I tried switching nn.View(-1,16*7*7) or nn.View(-1):setNumInputDims(3) with nn.Reshape(16*7*7), and both batch and non-batch mode worked, as well as the backward, nice!

I can use that for now, but I'm actually always confused of the difference between using nn.View and nn.Reshape, is nn.View supposed to be faster than Reshape? or, any other advantage of using nn.View instead of nn.Reshape in the case of flattening the outputs of conv layers into a 1d vector? Thanks.

[Koustav Mullick]

Here's the answer to your exact question.

https://groups.google.com/forum/#!topic/torch7/ocqTqLJiKOM

[Chen-Ping Yu]

oh ok, so I guess it is memory issue as nn.Reshape creates a new copy. thanks!

so hopefully someone (@Francisco ?) may have a good approach with nn.View that'll work with nn.Parallel's backward. 

[Francisco Vitor Suzano Massa]

Hi Chen-Pin,

You can add a nn.Contiguous() module just after each nn.View. It will only allocate new memory for the non-contiguous elements, so it won't be more memory consuming than using nn.Reshape

As a side note, in the past we considered making nn.View work with non-contiguous tensors https://github.com/torch/nn/pull/284 , but as torch.view doesn't allow non-contiguous tensors, we decided to drop it for consistency.

[Chen-Ping Yu]

Hi Francisco,

Your suggestion worked perfectly, how logical that nn.View( ) expects contiguous input and if we don't, make it contiguous by adding a nn.Contiguous( ) lol !!!

thanks!!

[Hang Gao]

On Sunday, February 21, 2016 at 8:42:27 PM UTC-5, Chen-Ping Yu wrote:

Hi all,

I've been trying to combine 2 networks into one with concat but I kept running into problems and can't seem to get it implemented...the network structure is attached, it is a very simple network, basically I have 2 separate networks, each has their own conv layers, then after reshaping/viewing their last conv layer, I'd like to concatenate their two 1d vectors into a single 1d vector, then that vector would continues to have the subsequent fully connected layers. anyone has a good way of implementing that? Thanks!