Stuck while writing custom loss

[kushag...@iiitd.ac.in]

Hi.

I am using writing a custom loss for an LSTM, logically defined as abs(y_true - y_pred) * weight_dict[round(y_pred)]

Here, y can take real values between [-3, 3], weight_dict has keys (-3, -2, -1, 0, 1, 2, 3) and is precomputed.

I wrote my function in the following manner

def custom_loss(y_true, y_pred):
   
    yp = K.eval(y_pred)
    yt = K.eval(y_true)
   
    error = 0.
   
    for i in range(yp.shape[0]):
        yp[i] = (np.abs(yp[i] - yt[i]) * class_weights[round(yp[i])])

    K.set_value(y_pred, yp)
    return K.mean(y_pred)

I tested this function outside net, and it was working fine.

K.eval(my_loss(K.variable(np.array([[0], [1], [2], [3]])), K.variable(np.array([[0], [3], [2], [1]]))))

However, when I try to compile my model,

model.compile(loss = my_loss, optimizer = "RMSprop", metrics = ['mae', 'mse'])

It throws the following error. Can someone help me with this?

InvalidArgumentError: You must feed a value for placeholder tensor 'keras_learning_phase' with dtype bool
	 [[Node: keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
	 [[Node: add_343/_409 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_669_add_343", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

Caused by op u'keras_learning_phase', defined at:
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/__main__.py", line 3, in <module>
    app.launch_new_instance()
  File "/usr/local/lib/python2.7/dist-packages/traitlets/config/application.py", line 589, in launch_instance
    app.start()
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelapp.py", line 405, in start
    ioloop.IOLoop.instance().start()
  File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/ioloop.py", line 162, in start
    super(ZMQIOLoop, self).start()
  File "/usr/local/lib/python2.7/dist-packages/tornado/ioloop.py", line 883, in start
    handler_func(fd_obj, events)
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 440, in _handle_events
    self._handle_recv()
  File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 472, in _handle_recv
    self._run_callback(callback, msg)
  File "/usr/local/lib/python2.7/dist-packages/zmq/eventloop/zmqstream.py", line 414, in _run_callback
    callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/tornado/stack_context.py", line 275, in null_wrapper
    return fn(*args, **kwargs)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 260, in dispatcher
    return self.dispatch_shell(stream, msg)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 212, in dispatch_shell
    handler(stream, idents, msg)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/kernelbase.py", line 370, in execute_request
    user_expressions, allow_stdin)
  File "/usr/local/lib/python2.7/dist-packages/ipykernel/ipkernel.py", line 175, in do_execute
    shell.run_cell(code, store_history=store_history, silent=silent)
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2723, in run_cell
    interactivity=interactivity, compiler=compiler, result=result)
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2825, in run_ast_nodes
    if self.run_code(code, result):
  File "/usr/local/lib/python2.7/dist-packages/IPython/core/interactiveshell.py", line 2885, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-54-a236ccf6cc3b>", line 1, in <module>
    model = getModel()
  File "<ipython-input-53-b4296039aff1>", line 6, in getModel
    model.add(Dropout(0.3))
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 327, in add
    output_tensor = layer(self.outputs[0])
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 569, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 632, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/topology.py", line 164, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/usr/local/lib/python2.7/dist-packages/keras/layers/core.py", line 90, in call
    x = K.in_train_phase(K.dropout(x, self.p, noise_shape), x)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 1877, in in_train_phase
    if learning_phase() is 1:
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/tensorflow_backend.py", line 73, in learning_phase
    name='keras_learning_phase')
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 1587, in placeholder
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 2043, in _placeholder
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 759, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2240, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1128, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): You must feed a value for placeholder tensor 'keras_learning_phase' with dtype bool
	 [[Node: keras_learning_phase = Placeholder[dtype=DT_BOOL, shape=[], _device="/job:localhost/replica:0/task:0/gpu:0"]()]]
	 [[Node: add_343/_409 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/cpu:0", send_device="/job:localhost/replica:0/task:0/gpu:0", send_device_incarnation=1, tensor_name="edge_669_add_343", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"]()]]

[stela...@gmail.com]

Hi,

i've got a similar issue with my custom_loss function,

did you manage to find a solution to the issue?

Many Thanks.

Kind Regars.

[stela...@gmail.com]

Hi,

the best comment i got on this was one from this thread: https://github.com/keras-team/keras/issues/4075

"Keep in mind that the python function you write (custom_loss) is called to generate and compile a C function. The compiled function is what is called during training. When your python custom_loss function is called, the arguments are tensor objects that don't have data attached to them. The K.eval call will fail, as will the K.shape call. The only thing you can really know about your arguments is the number of dimensions. You must write your function in such a fashion that it deals with things in a more symbolic fashion. Look at the source for the different loss functions provided by Keras for inspiration."

Which actually lead me to use a tf.cond function and feeding tf.bool to pick a defined function to do some tensorflow functions that otherwise were throwing a None error in Keras.

Hope this is useful for someone.

Regards,

[Sergii Myskov]

Your custom loss function will be called just once, returning a part of computational graph, returning a tensor, which will be fed to optimizer.

I cannot explain the behaviour of Keras on this error, but it looks like indirect problem.

Try something like this:

def custom_loss(y_true, y_pred):
    # Here I expect your class_weights is just a plain list or numpy array.
    weights = [class_weights[i] for i in range(-3, 4)]
    weights = K.constant(weights)

    # I expect y_pred shape is (batch_size, 1), need to be rounded and
    # casted to integral type.
    pred_indices = K.cast(K.round(y_pred), dtype='int32')

    # Shift indices by 3 on lookup, as we converted weights from -3-based to 0-based.
    lookup = K.gather(weights, pred_indices + 3)

    return K.mean(K.abs(y_true - y_pred) * lookup)

I did not tested with model, but should work.