keras - NaN at output (network with two Dense layers)

[gaus...@gmail.com]

About task : I have class distances as input and want to get class confidences (number between 0.0 and 1.0). So I have something like :

[
  [
    0.0,
    0.0,
    0.0,
    6.371921190238224,
    0.0,
    3.3287083713830516,
    7.085957828217146,
    7.747408965761948,
    5.498717498872398,
    5.498717498872398,
    5.498717498872398,
    5.498717498872398,
    8.529725281060978
  ],
  [
    6.396501448825533,
    0.0,
    0.0,
    5.217483270813266,
    0.0,
    5.319046151560534,
    5.823161030197735,
    3.8991256371824976,
    6.269856323952211,
    5.517874167220461,
    6.396501448825533,
    5.328678274963717,
    3.8991256371824976
  ],
]

And as result

[
  [1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
  [0.0, 1.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0],
  ...
]

I have about 200 examples. My network building code is next :

def train(self, distances, classes):
    """
    Train network
    :param distances: array of distances to classes
    :type distances: list[list[float]]
    :param classes: array of class indicators
    :type classes: list[list[float]]
    """
    example_count, class_count = self._dimensions(distances, classes)
    self.model = Sequential()
    self.model.add(Dense(128, input_dim=class_count))
    self.model.add(Dense(class_count))
    self.model.compile(optimizer=SGD(), loss='mse')
    self.model.fit(array(distances), array(classes))

But during training I get next output :

Epoch 1/10
425/425 [==============================] - 0s - loss: nan     
Epoch 2/10
425/425 [==============================] - 0s - loss: nan     
Epoch 3/10
425/425 [==============================] - 0s - loss: nan     
Epoch 4/10
425/425 [==============================] - 0s - loss: nan     
Epoch 5/10
425/425 [==============================] - 0s - loss: nan     
Epoch 6/10
425/425 [==============================] - 0s - loss: nan     
Epoch 7/10
425/425 [==============================] - 0s - loss: nan     
Epoch 8/10
425/425 [==============================] - 0s - loss: nan     
Epoch 9/10
425/425 [==============================] - 0s - loss: nan     
Epoch 10/10
425/425 [==============================] - 0s - loss: nan    

And when I trying to use model.predict(numpy.array([[ 0.0, 0.0, 0.0, 6.371921190238224, 0.0, 3.3287083713830516, 7.085957828217146, 7.747408965761948, 5.498717498872398, 5.498717498872398, 5.498717498872398, 5.498717498872398, 8.529725281060978]])) (example from train set) - I getting [[ nan nan nan nan nan nan nan nan nan nan nan nan nan]]

What can be wrong in data or building code?

[gaus...@gmail.com]

Seems like I had wrong fit parameters (learning rate and other). Now I have next code (yes, I added neurons to hidden layer and increased train epochscount during testing):

    example_count, class_count = self._dimensions(distances, classes)
    self.model = Sequential()
    self.model.add(Dense(1024, input_dim=class_count))
    self.model.add(Dense(class_count))
    self.model.compile(optimizer=SGD(lr=0.002, momentum=0.0, decay=0.0, nesterov=True), loss='mse', metrics=['accuracy'])
    self.model.fit(array(distances), array(classes), nb_epoch=80)

And it gives

...
Epoch 79/80
425/425 [==============================] - 0s - loss: 0.0381 - acc: 0.6729     
Epoch 80/80
425/425 [==============================] - 0s - loss: 0.0382 - acc: 0.6871     
[[ 0.19048974  0.1585739   0.28798762 -0.23555818  0.4293299   0.10981751
-0.08614585 -0.06363138  0.05927059  0.07283521 -0.07852616 -0.02396417
-0.28515971]]

Not a good accuracity, but topic problem solved. Also, are there any way  to "normalize" output? (as I written - I need numbers in [0.0:1.0]) ?

[Klemen Grm]

As it is, your neural network is completely linear. You might consider different activation functions (eg: tanh, sigmoid, linear) for your hidden and output layers. This both lets you constrain the output range, and will probably improve the learning properties of your network.

[Daπid]

On 13 July 2016 at 13:17, Klemen Grm <kleme...@gmail.com> wrote:
> As it is, your neural network is completely linear. You might consider
> different activation functions (eg: tanh, sigmoid, linear) for your hidden
> and output layers. This both lets you constrain the output range, and will
> probably improve the learning properties of your network.

In addition to what Klemen says, for the last one you want a softmax,
that normalises the outputs into probabilities.

[Anh Vũ Mai Nguyễn]

I used tensorflow 2.7.0 to build model:

```

initializer = tf.keras.initializers.Zeros()

inputs = Input(shape=data.shape)

x = Flatten()(inputs)

x = Dense(data.shape[0],activation='relu',kernel_initializer='random_normal')(x)

x = Dense(data.classnum, kernel_initializer=initializer)(x)

x = tf.keras.layers.Softmax()(x)

self.model = tf.keras.Model(inputs=inputs, outputs=x)

```
Before fitting, I used model.predict with my train set, It was ensure that train set was free of nan values. Next, I printed output of each layer:

```

for idx in range(len(model.layers)):

# print(model.layers[idx].name,model.layers[idx].get_weights())

intermediate_layer_model = keras.Model(inputs=model.input,

outputs=model.layers[idx].output)

intermediate_output = intermediate_layer_model.predict(data.train_dataset)

print(idx,intermediate_output)

```

 The output is:

```

0 [[ 1.000000e-03 -1.310000e-01 -9.504200e-02 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 [ 0.000000e+00 -1.000000e-02  1.556190e+00 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 [ 1.000000e+00  5.763000e+00  3.027034e+00 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 ...
 [ 8.930000e-01 -5.760000e-01  4.266911e+00 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 [ 1.000000e+00  5.978000e+00  4.650479e+00 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 [ 8.500000e-01  8.910000e-01  7.704330e-01 ...  0.000000e+00
   0.000000e+00  1.000000e+00]]
1 [[ 1.000000e-03 -1.310000e-01 -9.504200e-02 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 [ 0.000000e+00 -1.000000e-02  1.556190e+00 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 [ 1.000000e+00  5.763000e+00  3.027034e+00 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 ...
 [ 8.930000e-01 -5.760000e-01  4.266911e+00 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 [ 1.000000e+00  5.978000e+00  4.650479e+00 ...  0.000000e+00
   0.000000e+00  1.000000e+00]
 [ 8.500000e-01  8.910000e-01  7.704330e-01 ...  0.000000e+00
   0.000000e+00  1.000000e+00]]
2 [[nan nan nan nan]
 [nan nan nan nan]
 [nan nan nan nan]
 ...
 [nan nan nan nan]
 [nan nan nan nan]
 [nan nan nan nan]]

```

Please help me to solve that. Thank you.

Vào lúc 18:44:47 UTC+7 ngày Thứ Tư, 13 tháng 7, 2016, Daπid đã viết: