Multiple Activation Functions in a Hidden Layer

[m.a.n...@gmail.com]

Hi, I was wondering if it is possible to have different activation functions in a hidden layer in Keras? For example, in a hidden layer with two units, how can I have Sigmoid activation for one unit and RELU for the other one? Many thanks

[Matias Valdenegro]

Sure, the easiest way is to use the functional API, split this into two layers with different activations, and then merge the tensors to generate a single output. Something like:

 

x = some tensor

rx = Dense(1, activation = "relu")(x)

sx = Dense(1 activation = "sigmoid")(x)

out = merge([rx, sx], mode = "concat")

[m.a.n...@gmail.com]

Thanks a lot for your help. I applied what you suggested, but still it doesn't work! I keep receiving this error: IndexError: list index out of range.

Here is how I build my NN. The error happens in the last line of the code:

model0 = Sequential()
model0.add(Dense(1, input_dim=d, activation = L0))
model1 = Sequential()
model1.add(Dense(1, input_dim=d, activation = L1))
model2 = Sequential()
model2.add(Dense(1, input_dim=d, activation = L2))
model3 = Sequential()
model3.add(Dense(1, input_dim=d, activation = L3))
model4 = Sequential()
model4.add(Dense(1, input_dim=d, activation = L4))
model5 = Sequential()
model5.add(Dense(1, input_dim=d, activation = L5))
merged_model = Sequential()
merged_model.add(Merge([model0, model1, model2, model3, model4, model5], mode='concat'))
merged_model.add(Dense(output_dim=1))

merged_model.compile(loss='mean_squared_error', optimizer='SGD')
merged_model.fit(x_train, y_train, nb_epoch=200, batch_size=64)

L0 through L5 are activation functions I have introduced. My input dimension is 4, and the output of the NN is a scalar. Basically, I am trying to build a one-hidden-layer NN in which there are 5 units in the hidden later, 4 units in the input layer, and one unit in the output layer.

I would really appreciate any help you could give me! Many thanks!

[Daπid]

Don't use sequential, use the functional API. Look at the example for the inception module, it is in essence similar to what you want to do. 

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/273728a6-8b07-4391-a148-40d9438784a1%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.