Autograph error when iterating over tensor

[George Sterpu]

Hi,

I am seeing this error in TensorFlow 2.1 nightly (15 Jan) when trying a new model:

  .../scr.py:108 _train_step  *

        pred_output, _ = self.model(data, training=True)

    /home/_/miniconda3/envs/tfn/lib/python3.7/site-packages/tensorflow_core/python/keras/engine/base_layer.py:823 __call__  **

        outputs = self.call(cast_inputs, *args, **kwargs)

    .../scr2.py:196 call

        for t1 in tf.range(ts):

    /home/_/miniconda3/envs/tfn/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:561 __iter__

        self._disallow_iteration()

    /home/_/miniconda3/envs/tfn/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:554 _disallow_iteration

        self._disallow_when_autograph_enabled("iterating over `tf.Tensor`")

    /home/_/miniconda3/envs/tfn/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:532 _disallow_when_autograph_enabled

        " decorating it directly with @tf.function.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed: AutoGraph did not convert this function. Try decorating it directly with @tf.function.

The _train_step function is decorated with @tf.function (otherwise the code would run just fine in Eager mode).

If I additionally decorate the call method of my tf.keras.Model subclass, as instructed in the error log, the error goes away.

My other models which instantiate layers subclassed from tf.keras.layers.Layer and make use of the same for loop construct (for t in tf.range(ts) ) don't trigger this Autograph error, and no call method needs to be decorated except the main train_step function.

Without proving a code to reproduce this issue, do you have any insight that might help me understand where the error comes from ?

Thanks,

George

[Dan Moldovan]

+Tom O'Malley +Francois Chollet 

Judging from the call stack, it seems that Keras is attempting to run the call method using eager execution (https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/engine/base_layer.py#L823), but _train_step seems to run in graph mode. Out of curiosity, if you add `print(type(data))` before calling self.model, what does it print?

Dan

--
You received this message because you are subscribed to the Google Groups "TensorFlow Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to developers+...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/developers/da812e8a-979c-4d5b-91c5-a8116c302479%40tensorflow.org.

[George Sterpu]

Thanks for the response, Dan.

I just got home and reproduced the error on my gpu-less laptop as well with tf-nightly-cpu.

print(type(data))
<class 'avsr.io_utils.BatchedData'>

where

BatchedData = collections.namedtuple(
    "BatchedData",
    (
        "inputs",
        "inputs_length",
        "inputs_filenames",
        "labels",
        "labels_length",
        "labels_filenames",
        "payload",
    ))

The elements are a mix of float32, string, and int32 Tensors
Full info here:
https://pastebin.com/Un3UDcAv

Thanks,

George

[George Sterpu]

An important difference between my two scenarios is that the error-free model extracts the data from the BatchData instance and feeds it to several layers (which have for loops over a Tensor) as a tuple of Tensors. Whereas the faulty model has the for loop right inside its call method which takes as input a class instance, not a tuple.

[Dan Moldovan]

So namedtuple subclasses should be supported and treated like regular tuples, but looking at the point where the decision to run things eagerly is made: https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/keras/engine/base_layer.py#L657, it seems that you want to avoid non-Tensor values. In our case the payload field seems to be None - setting it to some dummy tensor value should help (as well as double-checking that all other fields contain a Tensor of some kind).

[George Sterpu]

You are right, Dan.

Setting payload=tf.zeros(0) successfully bypassed the issue.

Thanks for your help and for identifying the problem so quickly.

George