add tensorflow scalar summary to keras program ?

[boole...@gmail.com]

Hi,

I have a Keras program with a tensorflow backend.  Tensorboard shows

the graph and the losses.

I tried to add a scalar_summary to plot another variable of interest.

To do this, I just did

  import tensorflow as tf

  #.... later

  tf.scalar_summary('myvar', var_of_interest)

The program ran without errors, but the new variable was not visible in tensorboard.

Probably something more needs to be done?  Can anyone give a very minimal example

of adding a scalar_summary in Keras?

[Daπid]

You can do this adding your variable as a custom metric.

--
You received this message because you are subscribed to the Google Groups "Keras-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to keras-users+unsubscribe@googlegroups.com.
To view this discussion on the web, visit https://groups.google.com/d/msgid/keras-users/87f080e3-b59f-4391-8c55-1b5d79986f70%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[boole...@gmail.com]

The reply isn't quite enough to help me yet (I'm a newbie).

In the documentation example, the custom metric is a fucntion of y_true and y_pred.

In my case I want to visualize something that is not a function of either of these - specifically,

one of the terms in the loss.

Perhaps this could be done with a global variable, i.e.,

1. while assembling the loss, do a sess.run() to get the value of the perticular term of interest

2. assign it to a global variable

3. reference this global variable inside the custom metric callbacl.

Is this the right way to do this?

[max....@gmail.com]

Hi!

I've faced the same problem and it took me an evening to figure out how to implement it properly. 

So, I've decided to reply to your question, which may be not relevant, but the answer still be preserved 

in google's search result. Maybe, it will help somebody else :)

So, to add custom metric to your keras model you need the following:

1. add your tensors to summary collection

2. create a summary writer

3. inject aggregated summary into session.run() calls in your optimisation process

4. add summary result to writer

First steps is simple, just add something like this to your model:

x_entropy_t = K.sum(p_t * K.log(K.epsilon() + p_t), axis=-1, keepdims=True)
full_policy_loss_t = -res_t + X_ENTROPY_BETA * x_entropy_t
tf.summary.scalar("loss_entropy", K.sum(x_entropy_t))
tf.summary.scalar("loss_policy", K.sum(-res_t))
tf.summary.scalar("loss_full", K.sum(full_policy_loss_t))

Second is also straightforward. Just create writer:

summary_writer = tf.summary.FileWriter("logs/" + args.name)

Third step is a bit magical. What you need to do is to pass a custom metric function to model's compile() call 

which returns aggregated summary tensor. It can be done like this:

def summary(y_true, y_pred):
    return tf.summary.merge_all()

value_policy_model.compile(optimizer=Adagrad(), loss=loss_dict, metrics=[summary])

The idea is to make keras pass your aggregated summary operation to every session.run call and return it's result as metric. 

Second part is essential, as we need to add result to summary writer to make our values visible in tensorboard.

On fourth step you need to extract summary protobuf value from your model's fit() or train_on_batch() result. In my code I do this like this:

l = value_policy_model.train_on_batch(x_batch, y_batch)
l_dict = dict(zip(value_policy_model.metrics_names, l))

summary_writer.add_summary(l_dict['value_summary'], global_step=iter_idx)
summary_writer.flush()

Hope this will help.

[sebastien...@gmail.com]

Hi max,

First, thank you for your code, this is exactly what I need

Yet, what is iter_idx ?

Thank you,

Sebastien

[Max Lapan]

Hi!

Just an iteration index, integer you should pass to tf summary writer.

--

wbr, Max Lapan

[sebastien...@gmail.com]

Shouldn't a new summary be created after each epoch rather than after each batch ?

[sebastien...@gmail.com]

I get an error when passing summary as a metric because its output type is non numeric :/

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-14-fac1a5253ff5> in <module>()
      1 vae = Model(x, x_decoded_mean)
----> 2 vae.compile(optimizer='rmsprop', loss=vae_loss, metrics=[summary])

/home/david/machineLearning/DeepLearningProteins/venv/lib/python2.7/site-packages/keras/engine/training.pyc in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, **kwargs)
    954                     metric_fn = metrics_module.get(metric)
    955                     masked_metric_fn = _masked_objective(metric_fn)
--> 956                     metric_result = masked_metric_fn(y_true, y_pred, mask=masks[i])
    957                     metric_result = {
    958                         metric_fn.__name__: metric_result

/home/david/machineLearning/DeepLearningProteins/venv/lib/python2.7/site-packages/keras/engine/training.pyc in masked(y_true, y_pred, mask)
    487             score_array /= K.mean(mask)
    488 
--> 489         return K.mean(score_array)
    490     return masked
    491 

/home/david/machineLearning/DeepLearningProteins/venv/lib/python2.7/site-packages/keras/backend/tensorflow_backend.pyc in mean(x, axis, keepdims)
   1121     if x.dtype.base_dtype == tf.bool:
   1122         x = tf.cast(x, floatx())
-> 1123     return tf.reduce_mean(x, reduction_indices=axis, keep_dims=keepdims)
   1124 
   1125 

/home/david/machineLearning/DeepLearningProteins/venv/lib/python2.7/site-packages/tensorflow/python/ops/math_ops.pyc in reduce_mean(input_tensor, axis, keep_dims, name, reduction_indices)
   1308       _ReductionDims(input_tensor, axis, reduction_indices),
   1309       keep_dims,
-> 1310       name=name)
   1311 
   1312 

/home/david/machineLearning/DeepLearningProteins/venv/lib/python2.7/site-packages/tensorflow/python/ops/gen_math_ops.pyc in _mean(input, reduction_indices, keep_dims, name)
   1531   result = _op_def_lib.apply_op("Mean", input=input,
   1532                                 reduction_indices=reduction_indices,
-> 1533                                 keep_dims=keep_dims, name=name)
   1534   return result
   1535 

/home/david/machineLearning/DeepLearningProteins/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.pyc in apply_op(self, op_type_name, name, **keywords)
    583               _SatisfiesTypeConstraint(base_type,
    584                                        _Attr(op_def, input_arg.type_attr),
--> 585                                        param_name=input_name)
    586             attrs[input_arg.type_attr] = attr_value
    587             inferred_from[input_arg.type_attr] = input_name

/home/david/machineLearning/DeepLearningProteins/venv/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.pyc in _SatisfiesTypeConstraint(dtype, attr_def, param_name)
     59           "allowed values: %s" %
     60           (param_name, dtypes.as_dtype(dtype).name,
---> 61            ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
     62 
     63 

TypeError: Value passed to parameter 'input' has DataType string not in list of allowed values: float32, float64, int64, int32, uint8, uint16, int16, int8, complex64, complex128, qint8, quint8, qint32, float16

[Max Lapan]

From performance perspective, it should. But in keras you have little control over session.run() arguments. 

Of course, you can always add summary result to writer only once in an epoch.

[Max Lapan]

Could you show the code?

[sebastien...@gmail.com]

Sorry for the late answer and thank you for yours

Actually I don't think the code would help you, it's just the compiling that does not work, because of the output type of the summary metrics

[za...@okcupid.com]

Were you ever able to solve this? I'm running into this issue...

[mj...@google.com]

I had the same problem and solved it by extending the TensorBoard callback. The callback normally evaluates all summaries, including user-defined ones, only if histogram_freq is greater than zero. With my extension, the user-defined summaries are evaluated independently of histogram_freq.

class CustomTensorBoard(tf.keras.callbacks.TensorBoard):
  """Extends the TensorBoard callback to allow adding custom summaries.


  Arguments:
      user_defined_freq: frequency (in epochs) at which to compute summaries
          defined by the user by calling tf.summary in the model code. If set to
          0, user-defined summaries won't be computed. Validation data must be
          specified for summary visualization.
      kwargs: Passed to tf.keras.callbacks.TensorBoard.
  """


  def __init__(self, user_defined_freq=0, **kwargs):
    self.user_defined_freq = user_defined_freq
    super(CustomTensorBoard, self).__init__(**kwargs)


  def on_epoch_begin(self, epoch, logs=None):
    """Add user-def. op to Model eval_function callbacks, reset batch count."""


    # check if histogram summary should be run for this epoch
    if self.user_defined_freq and epoch % self.user_defined_freq == 0:
      self._epoch = epoch
      # pylint: disable=protected-access
      # add the user-defined summary ops if it should run this epoch
      self.model._make_eval_function()
      if self.merged not in self.model._eval_function.fetches:
        self.model._eval_function.fetches.append(self.merged)
        self.model._eval_function.fetch_callbacks[
            self.merged] = self._fetch_callback
      # pylint: enable=protected-access


    super(CustomTensorBoard, self).on_epoch_begin(epoch, logs=None)


  def on_epoch_end(self, epoch, logs=None):
    """Checks if summary ops should run next epoch, logs scalar summaries."""


    # pop the user-defined summary op after each epoch
    if self.user_defined_freq:
      # pylint: disable=protected-access
      if self.merged in self.model._eval_function.fetches:
        self.model._eval_function.fetches.remove(self.merged)
      if self.merged in self.model._eval_function.fetch_callbacks:
        self.model._eval_function.fetch_callbacks.pop(self.merged)
      # pylint: enable=protected-access


    super(CustomTensorBoard, self).on_epoch_end(epoch, logs=logs)

[ivanov...@gmail.com]

Thanks, but still get the same error 

TypeError: Value passed to parameter 'input' has DataType string not in list of allowed values: float32, float64, int32, uint8, int16, int8, complex64, int64, qint8, quint8, qint32, bfloat16, uint16, complex128, float16, uint32, uint64

How did you apply this callback? Do you specify tf.summary.scalar and metrics=[summary]?

[mj...@google.com]

What line throws this error?

In your model code, simply add summaries like this (based on the Keras examle):

from keras.layers import Input, Dense
from keras.models import Model

# This returns a tensor
inputs = Input(shape=(784,))

# a layer instance is callable on a tensor, and returns a tensor
x = Dense(64, activation='relu')(inputs)
x = Dense(64, activation='relu')(x)

tf.summary.scalar(tf.reduce_mean(x))

predictions = Dense(10, activation='softmax')(x)

# This creates a model that includes
# the Input layer and three Dense layers
model = Model(inputs=inputs, outputs=predictions)

Then, use the callback like any other Keras callback. Add it to the `callbacks` argument when you fit the model:

model.fit(x=x_data, y=y_data, callbacks=[CustomTensorboard(...)])

You don't have to do anything else. The summary will automatically added to the default graph and will show up in TensorBoard.