C++ SerializeToArray

[Ryan]

I have been working with a Group Communication System and Protocol
Buffers.

I have an issue where the C++ SerializeToArray call on one of my
messages is occasionally appending Bytes {1,0,0,0,0,0,0,0,0} to the
end of the returned character array?

Any ideas on what might be causing this? I can Marshall/Unmarshall
fine using the java api but the C++ call above has the odd quirk
mentioned.

The Java parsingFrom fails on the C++ generated messages that have the
above bytes appended.

Any suggestions much appreciated.

[Caleb]

Ryan wrote:

> I have an issue where the C++ SerializeToArray call on one of my
> messages is occasionally appending Bytes {1,0,0,0,0,0,0,0,0} to the
> end of the returned character array?

Are you sure you have sized your array correctly (e.g. maybe thats
just garbage at the end of an overly-long buffer)? If not, can you
come up with a test-case that reproduces this behavior? I'm certainly
not seeing anything like that in my use of the library in C++.

[Kenton Varda]

Hi Ryan,

What does your code look like that calls SerializeToArray()?  It should be something like:

  int size = message.ByteSize();

  char* array = new char[size];

  message.SerializeToArray(array, size);

[Ryan]

char mess[MAX_MESSLEN] ;
.
//Load some properties for the group com call
.
.
bplMessage.SerializeToArray(mess,bplMessage.ByteSize());

//what I am using to detect trailing bytes
for(int i =0; i< bplMessage.ByteSize();i++) {
std::cout << (int) mess[i] << std::endl;

}



On Dec 17, 8:32 am, Kenton Varda <ken...@google.com> wrote:
> Hi Ryan,
> What does your code look like that calls SerializeToArray()?  It should be
> something like:
>
>   int size = message.ByteSize();
>   char* array = new char[size];
>   message.SerializeToArray(array, size);
>

[Kenton Varda]

If you write a program demonstrating the problem and send it to me, I can debug it.

[Ryan]

I will try. A good debugging step for myself if not anything else.
This system is of course distributed so it may be difficult to isolate
it down into something I can send you easily (also lots of
dependencies). I have been trying to identify the messages that are
causing this but it appears to be happening independent of the message
contents.

In a particular sequence of messages the same messages always get the
erroneous bytes appended. However, if I take one of those messages and
send it independently - works fine.

Weird...

Thanks for the dialog.

On Dec 17, 9:36 am, Kenton Varda <ken...@google.com> wrote:
> If you write a program demonstrating the problem and send it to me, I can
> debug it.
>

[Ryan]

Ok - found it. The problem was not on the serialization but on the
message creation/parsing. The buffer the C++ code was parsing the
protocol buffer message from was not properly initialized causing the
ByteSize() function to return an incorrect value based on the junk
padded past the typical sentinel.

I did a call to memset prior and all is good.

Two additional thoughts:

1) I am surprised the C++ parsing succeeded?

2) Why was the serialization reproducing the bad input? I would think
the process of Marshalling/Unmarshalling would of cleaned it up.

[Kenton Varda]

When you call ParseFromArray(), you must pass the exact size of the message being parsed.  You cannot simply give it the size of the buffer if there are extra bytes after the end of the message.

What happened here is that the parser, after parsing the bytes of your actual message, continued to interpret the following bytes as more data.  The next byte was a 1, which the parser thought was indicating the beginning of a 64-bit fixed-width field with field number zero.  Since your message does not declare a field number zero (you actually aren't allowed to), it treated this as an unknown field and stored the value in the UnknownFieldSet.  Later, when you serialized the message, the value in the UnknownFieldSet was written out again.

The moral of the story is that you must transmit the size of your message along with the data, so that you can pass it to ParseFromArray() on the receiving end.

[Ryan]

Ok, that was my real mistake, i.e. passing the buffer size vs. the
message size to ParseFromArray.

This also saves me initializing the memory because you know all your
bytes will be set.

Funny, ParseFromArray gets the right message length when passed a
buffer 0 padded at the end.

Thanks

On Dec 17, 12:45 pm, Kenton Varda <ken...@google.com> wrote:
> When you call ParseFromArray(), you must pass the exact size of the message
> being parsed.  You cannot simply give it the size of the buffer if there are
> extra bytes after the end of the message.
> What happened here is that the parser, after parsing the bytes of your
> actual message, continued to interpret the following bytes as more data.
>  The next byte was a 1, which the parser thought was indicating the
> beginning of a 64-bit fixed-width field with field number zero.  Since your
> message does not declare a field number zero (you actually aren't allowed
> to), it treated this as an unknown field and stored the value in the
> UnknownFieldSet.  Later, when you serialized the message, the value in the
> UnknownFieldSet was written out again.
>
> The moral of the story is that you must transmit the size of your message
> along with the data, so that you can pass it to ParseFromArray() on the
> receiving end.
>

[Kenton Varda]

On Wed, Dec 17, 2008 at 4:03 PM, Ryan <ryanr...@gmail.com> wrote:

Funny, ParseFromArray gets the right message length when passed a
buffer 0 padded at the end.

Odd, it should fail (return false) in this case.