Parsing .prototxt file in python

[Aleš Žurek]

Hi,

I have some problem with reading .solvertxt file in python.
I want to get paths from imagenet_solver.prototxt and imagenet_train_val.prototxt (both from caffe/examples/imagenet folder).
I found this protobuf python tutorial: https://developers.google.com/protocol-buffers/docs/pythontutorial

And I have code like this:

def _readProtoSolverFile(self, filepath):
    solver_config = caffe.proto.caffe_pb2.SolverParameter()
    #TODO how to read proto file?
    return self._readProtoFile(filepath, solver_config)
#enddef

def _readProtoFile(self, filepath, parser_object):
    file = open(filepath, "rb")
    if not file:
        raise self.ProcessException("ERROR (" + filepath + ")!")
    file_content = file.read()
    parser_object.ParseFromString(file_content)
    file.close()
    return parser_object
#enddef

But when I run this code I get this error:
Exception occured: Tag had invalid wire type.
Script Traceback: EXC: <class 'google.protobuf.message.DecodeError'> Tag had invalid wire type.: Traceback (most recent call last):
  File "/usr/lib/pymodules/python2.7/szn_utils/daemon.py", line 432, in _childIteration
    self._processIteration()
  File "/www/picturedetector/daemon/bin/daemon.py", line 283, in _processIteration
    self.__startLearningProcess(queue_info)
  File "/www/picturedetector/daemon/bin/daemon.py", line 124, in __startLearningProcess
    pid = self._startCaffeLearning(queue_info['neural_network_id'], queue_info['picture_set_id'], queue_info['start_iteration'])
  File "/www/picturedetector/daemon/bin/daemon.py", line 251, in _startCaffeLearning
    solver_config = self._readProtoSolverFile(network['solver_config_path'])
  File "/www/picturedetector/daemon/bin/daemon.py", line 500, in _readProtoSolverFile
    return self._readProtoFile(filepath, solver_config)
  File "/www/picturedetector/daemon/bin/daemon.py", line 508, in _readProtoFile
    parser_object.ParseFromString(file.read())
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/message.py", line 182, in ParseFromString
    self.MergeFromString(serialized)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/python_message.py", line 795, in MergeFromString
    if self._InternalParse(serialized, 0, length) != length:
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/python_message.py", line 819, in InternalParse
    new_pos = local_SkipField(buffer, new_pos, end, tag_bytes)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/decoder.py", line 716, in SkipField
    return WIRETYPE_TO_SKIPPER[wire_type](buffer, pos, end)
  File "/usr/local/lib/python2.7/dist-packages/google/protobuf/internal/decoder.py", line 685, in _RaiseInvalidWireType
    raise _DecodeError('Tag had invalid wire type.')
DecodeError: Tag had invalid wire type.

File is loaded corectly, when I print file_content variable I will get content of imagenet_solver.prototxt.
It is a problem that ParseFromString() method read only serialized binary files and not these text files?
Is there any other way how to load paths from these files?
Yes I know that I could parse those files with regex. In case of imagenet_solver.prototxt it is easy, but imagenet_train_val.prototxt file is quite complicated.

Thanks in advance.

Regards,
Ales

[Mohamed Omran]

This should work:

prototxt_solver = "solver.prototxt"
solver = caffe_pb2.SolverParameter()
f = open(prototxt_solver, 'r')
Merge(str(f.read()), solver)

[Yangqing Jia]

You are seeing this error because protobuffer deals with text format and binary format differently. If you are loading a binary format protobuffer (like the pretrained model), use ParseFromString(). Otherwise, use the one provided by google.protobuf.text_format.Merge, see the docs here:

https://developers.google.com/protocol-buffers/docs/reference/python/google.protobuf.text_format-module

Yangqing

--
You received this message because you are subscribed to the Google Groups "Caffe Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to caffe-users...@googlegroups.com.
To post to this group, send email to caffe...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/caffe-users/1d870810-7e63-4610-9c7f-f156c239d11c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Aleš Žurek]

Thank you. It looks like it is working now.
I suppose that you talk about the same Merge function.

So my code now looks like this.

import caffe
from caffe.proto import caffe_pb2
from google.protobuf import text_format

def _readProtoSolverFile(self, filepath):
    solver_config = caffe.proto.caffe_pb2.SolverParameter()
    #TODO how to read proto file?
    return self._readProtoFile(filepath, solver_config)
#enddef

def _readProtoFile(self, filepath, parser_object):

    file = open(filepath, "r")

    if not file:
        raise self.ProcessException("ERROR (" + filepath + ")!")

    text_format.Merge(str(file.read()), parser_object)
    file.close()
    return parser_object
#enddef

Thanks for quick answer :)

[Junhao Wen]

Sorry to reopen this issue, I have the same error even if I read the .proto file( not prototxt file) and here is the error:

raise _DecodeError('Tag had invalid wire type.')

google.protobuf.message.DecodeError: Tag had invalid wire type.

here is my code:

try:
  f = open(sys.argv[1], "rb")
  # f = open('/Users/junhao.wen/Hao/tutorial/Protocol_buffers_tutorial/tutorial/Protocol_buffers_basic_python/address_book/addressbook.proto', 'rb')
  address_book.ParseFromString(f.read())

Can you help me??? I really get stucked here for a long time...

Thanks in advance

[Shiv Kumar]

Did you find a solution back, I am having the same problem?