Message missing required fields exception when parsing a message that has required fields defaulted

[locky]

I have a question regarding required string properties and default
values.

Consider the following message definition.

message StringTestProto {
required stringProperty = 1;
}

I have a C++/Java setup where C++ app is producing a message and a
Java app is consuming it.

If the stringProperty value is set to an empty string I get the
following exception when trying to parse the
message once I receive it in the the Java app.

com.google.protobuf.InvalidProtocolBufferException: Message missing
required fields:

The message is being read as follows:

StringTestProto proto = StringTestProto.parseFrom(data);

From what I can see the issue is as follows:
As the stringProperty value is the same as the default (an empty
string) it is not sent over the wire. When the generated Java code
tries to build the proto from a byte[ ] (in the buildFrom method), a
check is done to see if the proto is initialised, this check sees if
the required fields have been set. As the stringProperty was never
sent, the proto does not think it has been set (as it never was) and
therefore the exception is thrown.

So, how are you supposed to decode a byte[] that contains required
properties, that may have default values?

[Paul]

I've gotten this message for reasons other than the field not being
set. make sure that you have the number of bytes on the delimiters
set correctly on the C++ side. also, maybe stringProperty should be
string Property (two words).

[locky]

Hi Paul.

Apologies for the message definition. It was a typo and should have
read like so...

message StringTestProto {
required string property = 1;
}

The C++ side is setting things correctly. My understanding is that
default values are not sent over the wire. When building a received
message from a byte[ ] a check is done to see if required fields have
been set. Any required field that was not sent due to having a
default value on the other side is not marked as being set and the
exception gets thrown.

Any ideas or further input appreciated.

[Evan Jones]

On Oct 26, 2010, at 4:13 , locky wrote:
> The C++ side is setting things correctly. My understanding is that
> default values are not sent over the wire. When building a received
> message from a byte[ ] a check is done to see if required fields have
> been set. Any required field that was not sent due to having a
> default value on the other side is not marked as being set and the
> exception gets thrown.

This is exactly correct. You should do two things:

1. Set this field on the sending side, but you mentioned that you are
already doing this.

2. Verify that the bytes you are reading in on one side match the
bytes being sent. I usually get this error when there is some sort of
message handling error. For example, if you pass protobuf an empty
array, you'll get this error message. You should write out the bytes
that you are writing, and the bytes that you are reading and verify
that they match. Also verify that the size you are passing in matches.

There is a difference between an unset field with a default value of
"" and a set field with a value of "". The .hasProperty() method will
return true for the set field, and false for the unset field. Thus,
these messages are serialized differently.

Hope this helps,

Evan

--
Evan Jones
http://evanjones.ca/

[Kenton Varda]

On Fri, Oct 22, 2010 at 9:12 AM, locky <true...@gmail.com> wrote:

From what I can see the issue is as follows:
As the stringProperty value is the same as the default (an empty
string) it is not sent over the wire.

No, this is not correct.  Whether or not a value is sent on the wire depends only on whether or not you explicitly called the setter.  It doesn't matter if you set it to its default value.  If you called set_property(), the field will be sent, even if you set it to the empty string.  If you did not call set_property(), the field will not be sent.  You can call has_property() to determine whether or not it was set.

I strongly suspect that your code contains some other bug, and the bytes being received in Java do not actually match the bytes sent from C++.  The most common problem we see is that people try to store the bytes in a C-style NUL-terminated string, like:

  std::string str = message.SerializeAsString();

  char* cstr = str.c_str();   // DOES NOT WORK

  send(cstr);  // DOES NOT WORK

The problem is that encoded protobufs contain arbitrary bytes, some of which may be zero.  A C-style "char*" string is considered to end at the first zero byte, but the encoded protobuf may extend beyond that.  You need to use code like this instead:

  std::string str = message.SerializeAsString();

  send(str.data(), str.size());

[locky]

The producers of the message are actually C# and not C++ as I first
mentioned (I only deal with the Java implementation). The scenario
however, is the same.

@Kenton, I thought default values were never sent over the wire.
According to the 'Updating A Message Type' section of the language
guide ..."Changing a default value is generally OK, as long as you
remember that default values are never sent over the wire...."

Are you saying this is not the case?


On Oct 27, 6:13 am, Kenton Varda <ken...@google.com> wrote:

[Henner Zeller]

On Wed, Oct 27, 2010 at 08:55, locky <true...@gmail.com> wrote:
> The producers of the message are actually C# and not C++ as I first
> mentioned (I only deal with the Java implementation).  The scenario
> however, is the same.
>
> @Kenton, I thought default values were never sent over the wire.
> According to the 'Updating A Message Type' section of the language
> guide ..."Changing a default value is generally OK, as long as you
> remember that default values are never sent over the wire...."

If you _set_ the value, not matter its content (yes, even if it
happens to be the default value), it is sent over the wire. Always.
How would you otherwise be able to distinguish between a not-set value
and the default value ?
The default values and its behaviors only apply to fields that are not set.

Almost certainly, you don't receive the data you think you sent. The
way to debug your problem is:
- actually make sure, that the value is set on the sending side
(check if has_property() returns true)
- serialize the value and make sure that on the receiving side you
actually get the same bytes back (print it, print its length, create a
hash from it and print that - whatever you can do to debug that). In
C/C++ it is a usual mistake to treat the binary data as \0 delimited
string, hence Kenton's suspicion.

(Note, that the Google implementation provides C++, Java and Python
but does not come with a C# version, so you're probably using one of
the 3rd party implementations.)

-h

> --
> You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
> To post to this group, send email to prot...@googlegroups.com.
> To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.
>
>

[Kenton Varda]

On Wed, Oct 27, 2010 at 8:55 AM, locky <true...@gmail.com> wrote:

The producers of the message are actually C# and not C++ as I first
mentioned (I only deal with the Java implementation).  The scenario
however, is the same.

It is possible that the C# implementation (which is not written by Google) accidentally interpreted default values differently.  You should ask the author.

 

@Kenton, I thought default values were never sent over the wire.
According to the 'Updating A Message Type' section of the language
guide ..."Changing a default value is generally OK, as long as you
remember that default values are never sent over the wire...."

Are you saying this is not the case?

What the documentation means is that the value you write in the .proto file is never used directly when serializing a message.  However, if you explicitly set a field to a value that happens to be equal to the default value, it *is* sent.  In this case we don't think of the value as being the default value.

Seems like the wording of that text needs to be improved to avoid ambiguity.

[AlexK]

On Oct 27, 7:55 am, locky <true.n...@gmail.com> wrote:
> The producers of the message are actually C# and not C++ as I first
> mentioned (I only deal with the Java implementation).  The scenario
> however, is the same.
>

If you are using protobuf-net on the sending side, then that is
possibly due to bug http://code.google.com/p/protobuf-net/issues/detail?id=136