error when using nested and packed in python

[Dan]

Hello protobuf gurus-
I'm just getting started with the python version of protobuf, using
v2.10. I want to store some lists of floats in a database's blob
field. When using a nested Message structure with the "packed=true"
option, I'm getting errors that look like this:

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "build/bdist.linux-i686/egg/google/protobuf/message.py", line
160, in ParseFromString
File "build/bdist.linux-i686/egg/google/protobuf/reflection.py",
line 1215, in MergeFromString
File "build/bdist.linux-i686/egg/google/protobuf/reflection.py",
line 1075, in _DeserializeOneEntity
File "build/bdist.linux-i686/egg/google/protobuf/reflection.py",
line 899, in _RecursivelyMerge
File "build/bdist.linux-i686/egg/google/protobuf/internal/
decoder.py", line 181, in ReadMessageInto
google.protobuf.message.DecodeError: Submessage told to deserialize
from 14-byte encoding, but used only 12 bytes

If I use a flat (not nested) message structure, I do not get these
errors. If I use a nested, packed=false approach, I do not get these
errors. The error happens only when both nested and packed.

Here is some code to reproduce the problem. Note that the
deserialization seems to succeed in putting the correct values into
the list (at least in this test), in spite of the error message.

my.proto file contains:

message TestNest {
optional ListData list_data = 1;
message ListData {
repeated float float_packed = 1 [packed=true];
repeated float float_notpacked = 2 [packed=false];
repeated int32 int_packed = 3 [packed=true];
repeated int32 int_notpacked = 4 [packed=false];
}
}

and then I run this set of commands for each of the fields
[float_packed, float_notpacked, int_notpacked, int_packed] and see
that the error happens only for the two fields that use packed=true.

import my_pb2
a = my_pb2.TestNest()
a.list_data.float_notpacked.extend([1,2,3])
a_string = a.SerializeToString()
a_deserialized = my_pb2.TestNest()
a_deserialized.ParseFromString(a_string)
str(a_deserialized)

I can think of three options to proceed, 2 of which I'm able to do
myself:
1) don't use the packed=true option
2) catch the errors and ignore them
3) if this is actually a bug, ask you gurus to fix the protobuf code

What do you recommend?
Dan

[Kenton Varda]

Yeah, I think it's broken.

In reflection.py there is a line:

  return decoder.Position() - content_start

I think (untested) that it should actually be:

  return decoder.Position() - initial_position

I have no idea how this got past testing and I have asked the people responsible to fix it.  Surprisingly you are the first to report this, so maybe there's a mitigating factor that makes it unusual for the bug to harm anything.  Sorry for the trouble.