Been working on this problem for a fortnight, really need help

213 views
Skip to first unread message

zjk

unread,
Apr 23, 2022, 1:49:41 PM4/23/22
to Protocol Buffers
Hi, I am not a direct user of protobuf, I was training my model with tensorflow using Python, while attempting to save my checkpoint model, this error related to protobuf occurred:

[libprotobuf ERROR google/protobuf/wire_format_lite.cc:577] String field 'tensorflow.TensorShapeProto.Dim.name' contains invalid UTF-8 data when parsing a protocol buffer. Use the 'bytes' type if you intend to send raw bytes.
Traceback (most recent call last):
  File "train.py", line 77, in <module>
    train(model)
  File "train.py", line 24, in train
    model.optimize()
  File "/user-data/HyperBox-main/script/model/box_model.py", line 329, in optimize
    self.save_model(itr)
  File "/user-data/HyperBox-main/script/model/box_model.py", line 140, in save_model
    self.saver.save(self.sess, filename)
  File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1208, in save
    self.export_meta_graph(
  File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/training/saver.py", line 1254, in export_meta_graph
    graph_def=ops.get_default_graph().as_graph_def(add_shapes=True),
  File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3345, in as_graph_def
    result, _ = self._as_graph_def(from_version, add_shapes)
  File "/opt/conda/lib/python3.8/site-packages/tensorflow/python/framework/ops.py", line 3262, in _as_graph_def
    graph.ParseFromString(compat.as_bytes(data))
google.protobuf.message.DecodeError: Error parsing message

from the information above, we can see that:
1)  some so-called 'invalid UTF-8 data' was sent to protobuf;
2)  Use the 'bytes' type if you intend to send raw bytes.

the function at the second to last line compat.as_bytes(data) is defined as:
 def as_bytes(bytes_or_text, encoding='utf-8'):
        if isinstance(bytes_or_text, bytearray):
            return bytes(bytes_or_text)
         elif isinstance(bytes_or_text, _six.text_type):      ##_six.text_type = unicode
            return bytes_or_text.encode(encoding)
         elif isinstance(bytes_or_text, bytes):
            return bytes_or_text
         else:
            raise TypeError('Expected binary or unicode string, got %r' %
(bytes_or_text,))

we can see that this function makes sure that what will be passed into graph.ParseFromString(which later passed to protobuf) is definitely utf-8 data or bytes, however, the protobuf keeps claiming the data passed to it is not valid utf-8 data. Taking one step back, even if the data was not valid utf-8, it is definitely bytes, which it should take in normally.

And I printed some information before the execution of graph.ParseFromString(compat.as_bytes(data)):

pprint(type(data))
graph.ParseFromString(compat.as_bytes(data))

and here's the result:
<class 'bytes'>
the data has already been bytes even before it is passed into compat.as_bytes!

I am sorry for sounding emotional, it's just that I've been stuck with this problem for too long and it really gets me frustrated.

Deanna Garcia

unread,
May 2, 2022, 1:42:19 PM5/2/22
to Protocol Buffers
I think you should file this bug with tensorflow since you are only using protobufs through them.
Reply all
Reply to author
Forward
Message has been deleted
Message has been deleted
0 new messages