Loss always Nan

[Oliver Lippert]

I know there are several Topics around "nan" loss, but I think I tried everything.

I do dive into NN at the Moment and I think keras is a good start Point.

While getting into it, I have an single Topic, but Analyse more and more Inputs to get better results.

I startet with 8 and got quiet interesting results so I improved and labeled more date.

Now since the last data increasement (32 Inputs), I get "Nan" as loss all the time (from first Iteration). 

I tried a lot of Things (switching optimiziers and loss function for example).

Here's an data sample

0.015625,0.0087890625,0.8125,0.1875,0,0.047058823529412,0,1,1,0,0.4,1,0,0,0,0,0,0,0,0.01,0.02,0.01,0,0,0,0,0,0,0.0185546875,0.0185546875,0.0185546875,0
0.060546875,0.0234375,0.87096774193548,0.12903225806452,0,0.098039215686275,0,1,1,0,0,0,0,1,0,0.5,0,0,0,0.01,0.08,0.07,0,0,0,0,0,0,0.0205078125,0.046875,0.02734375,0
0.115234375,0.046875,0.83050847457627,0.14406779661017,0.025423728813559,0.10588235294118,0,1,1,0,0,0,0,0,0,0,0,0,0,0.01,0,0,0,0,0,0,0,0,0.021484375,0.017578125,0.017578125,1
0.0458984375,0.0234375,0.82978723404255,0.17021276595745,0,0.082352941176471,0,1,1,0,0,1,0,1,0,1,0,0,0,0.01,0,0,0,0,0.01,0.03,0.02,0,0.01953125,0.08203125,0.0244140625,0
0.0439453125,0.017578125,0.66666666666667,0.33333333333333,0,0.082352941176471,0,1,0.4,0,0,1,1,0,0,0,0,0,0,0.01,0,0,0,0,0,0,0,0,0.01953125,0.0224609375,0.0224609375,1
0.05078125,0.0205078125,0.67307692307692,0.32692307692308,0,0.090196078431373,0,0.5,0.66,0,0,1,1,0,0,0,0,0,0,0.01,0,0,0,0,0,0,0,0,0.01953125,0.021484375,0.021484375,1

 Last index is the "result". The data field itself is mostly statistical Information generated by one of my Systems, it is already normalized (between 0 and 1).

Here's how I do build up the keras model:

def build_model(self):
    model = tf.keras.Sequential([
        Dense(100, input_shape=(self.input_size,), activation=tf.keras.activations.relu),
        Dense(30, activation=tf.keras.activations.relu),
        Dense(10, activation=tf.keras.activations.relu),
        Dense(1, activation=tf.keras.activations.relu)
    ])

    loss = tf.keras.losses.MeanSquaredError()
    optimizer = tf.keras.optimizers.SGD(lr=0.0001)

    model.compile(optimizer=optimizer, loss=loss, metrics=[tf.keras.metrics.Accuracy()])
    return
 
 
def load_data(self, data_raw):
    x = []
    y = []
    for line in data_raw.split("\n"):
        line_data = self.convert_line(line)
        if len(line_data) != 32:
            print(len(line_data))
            print(line_data)

        x.append(line_data[:32])
        y.append(line_data[32:])

    training_input = numpy.array(x)
    training_labels = numpy.array(y)
    return
 
 
def training(self):
    print(numpy.any(numpy.isnan(training_input)))
    print(numpy.any(numpy.isnan(training_labels)))
    print(numpy.any(numpy.isinf(training_input)))
    print(numpy.any(numpy.isinf(training_labels)))

    model.fit(training_input, training_labels, epochs=10,
                   batch_size=1, verbose=1)

    # evaluate the model
    scores = model.evaluate(training_input, training_labels)
    print("\n%s: %.2f%%" % (model.metrics_names[1], scores[1] * 100))

    model.summary()
    tf.contrib.saved_model.save_keras_model(model, model_path)
    return

 

The "print" in Training() are just for Debugging purposes to make sure no "nan" or "infinity" values are in the dataset, all four return "false").

This is the Output of the first epoch

Epoch 1/1000

 100/1869 [>.............................] - ETA: 3s - loss: nan - accuracy: 0.0000e+00
1869/1869 [==============================] - 0s 108us/sample - loss: nan - accuracy: 0.0000e+00

I searched around all day, mostly the comments where like changing loss function or Optimizer. I tested all this but Nothing changed.

I think my data is not that complex, does anybody has an Idea (or an link / tutorial) what I need to change here?

I appreciate any reply.

[Oliver Lippert]

Small hint: I also gave a small Addition to the 0 values, so instead of 0 I used 0.000000001 because 0 could trigger infinite for some optimizers, I read.

I forgot to mention it. It did not change anything.

[Lance Norskog]

Small questions:

1) I don't really understand the Sequential tool. Is the model more complex than it needs to be? Does it have layers that you can take out for while debugging?

2) One numerical technique is to clamp all input & output values to the range [0.01 -> 0.99]. It avoids divide-by-zero cases.

3) What happens with other activation functions? 

Cheers,

Lance Norskog

On Mon, Apr 22, 2019 at 8:56 AM Oliver Lippert <lippe...@gmail.com> wrote:

Small hint: I also gave a small Addition to the 0 values, so instead of 0 I used 0.000000001 because 0 could trigger infinite for some optimizers, I read.

I forgot to mention it. It did not change anything.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/77dbccb0-7405-45fe-8f14-02dba0118b08%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Lance Norskog
lance....@gmail.com
Redwood City, CA

[Oliver Lippert]

Thanks for your Reply.

1) I testet a variable set of layers. 1, 2, 3, 5, 10. Having max 10, 100 or 1000 Neurons -> no change

2) I resized -> no change

3) I tested softmax, selu, tanh, sigmoid, linear -> no change

[Lance Norskog]

The last value is the label to predict? I think there's an off-by-one here.

        x.append(line_data[:32])

        y.append(line_data[32:])

A quick python2 session:

>>> a = [0, 1, 2]

>>> a[2:]

[2]

>>> a[:2]

[0, 1]

This should be 

      x.append(line_data[:31])     

      y.append(line_data[31:])

Data should be an array of [[0.01,...0.302],...]

The labels should be an array of [[0],[1],[0],[0]...]

We don't want to feed the label as an input, the point is to guess the label from the other inputs.

Lance

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/601f3b36-edbc-4a17-bb6e-46be43d57d42%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Oytun Turk]

If this is a binary classification problem, you should be using sigmoid activation in final dense layer and binary cross-entropy loss.

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/CAP0ykh9J2BsTamf%2BRud%3DOqNY4myDAzyF4yQ3Fdaw47Vs6x53Rg%40mail.gmail.com.

[Oliver Lippert]

Wow that was the dumbest possible mistake I could make. Shame on me.

Thanks for Opening my eyes :)

Now the loss is as usual.

To unsubscribe from this group and stop receiving emails from it, send an email to keras...@googlegroups.com.

To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/601f3b36-edbc-4a17-bb6e-46be43d57d42%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Lance Norskog]

off-by-one errors are I think the most common programming error.

Also, what "Oytun Turk" said is worth looking into.

AutoKeras is a project to do automated parameter search. It is also use-case-based, so if you search its different use cases you can often find the right base design for a project. It's a "Design Pattern" archive.

Lance Norskog

To unsubscribe from this group and stop receiving emails from it, send an email to keras-users...@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/62312b97-bfca-4ec3-a813-fca951d87b42%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.