Split output layer

[Derk Mus]

Suppose I have an output layer with a number of outputs, 21 to be more specific.

Now one of the outputs represent a probability and has to be mapped to the [0,1] interval by a sigmoid function.

Another 20 outputs represents weights and have to sum to 1 (a softmax function has to be applied here).

How to implement this in Torch?

model = nn.Sequential()

....

... hidden layers etc

model:add(nn.Linear(hiddenSize, 21)) -- 21 outputs

-- what here??

nn.Sigmoid()

nn.SoftMax() 

I looked to SplitTable. However, I do not see how to apply this.

[Florient Chouteau]

Hi,

You could simply manually do the split outside the network if you think it's simpler.

Otherwise, just do:

local c1 = nn.Sequential():add(nn.SelectTable(1)):add(nn.Sigmoid()) -- This will select the first entry and apply Sigmoid
local c2 = nn.Sequential():add(nn.NarrowTablre(2,20)):add(nn.SoftMax()) -- This will narrow the table from index 2 to index 21 and apply softmax

local concat = nn.ConcatTable():add(c1):add(c2) This will apply the 21-output to both C1 and C2
model:add(concat)

So now when you do model:forward(input) you will have as output a table { SigmoidOutput [torch.Tensor(1)], softmaxOutput (torch.Tensor(20)}

Regards,

[Shuaib Ahmed S]

You can try this

hiddenSize = 3

data = torch.rand(5,3)

model = nn.Sequential()

model:add(nn.Linear(hiddenSize, 21)) -- 21 outputs

model:add(nn.SplitTable(2))

l=nn.ParallelTable()

l:add(nn.Sigmoid())

for i=1,20 do l:add(nn.SoftMax()) end

op=model:forward(data)

--
You received this message because you are subscribed to the Google Groups "torch7" group.
To unsubscribe from this group and stop receiving emails from it, send an email to torch7+unsubscribe@googlegroups.com.
To post to this group, send email to tor...@googlegroups.com.
Visit this group at https://groups.google.com/group/torch7.
For more options, visit https://groups.google.com/d/optout.

[Derk Mus]

Okay thanks, and what if the output tensor if of size 10x300x21 (batchSize x sequence length x outputs) in the case of an RNN?

Op maandag 19 september 2016 13:56:05 UTC+2 schreef Florient Chouteau:

[Florient Chouteau]

Okay I was wrong in my previous answer but this should work

require 'nn'

local model = nn.Sequential()

model:add(nn.Identity()) -- Your model that outputs 10,300,21

local c1 = nn.Sequential():add(nn.Narrow(3,1,1)):add(nn.Sigmoid())
local c2 = nn.Sequential():add(nn.Narrow(3,2,20)):add(nn.SoftMax())

local concat = nn.ConcatTable():add(c1):add(c2) --This will apply the 21-output to both C1 and C2
model:add(concat)

print(model)

local t = torch.randn(10,300,21)

local out = model:forward(t)

print(out)

print(out[1])
print(out[2])

[Derk]

This works well, except the softmax is not applied to the right numbers. It seems like it is applied to the first dimension instead of the 20 numbers in the third dimension. Do you know how to fix this? Thanks :)

Op maandag 19 september 2016 15:29:37 UTC+2 schreef fchouteau: