optional vs required

[Colin Deasy]

Hey,

When reading https://developers.google.com/protocol-buffers/docs/proto#simple I see a stark warning indicating that "Required is Forever" advocating the use of optional with additional application level validation routines. This is because if at some point a required field is no longer written, the readers will break.

However IMO, there are common cases where 'required' is a good thing - given that it's enforced only during encoding/decoding.
For example there may be some field that is 'required' (right now) to both the reader and writer. Even if that changes at some point in the future to become optional, the reader would likely have to be updated regardless of the protocol decoding routine as it may make assumptions (reasonable considering it was required in the first place) on the presence of the field (e.g. the field being a key to a certain bit of data). In this case the approach would be to update the .proto of all readers to make that field optional, followed by updating all writers to remove the field.

Given this, I feel that the current language of the linked document gives the impression that the 'required' attribute is a Bad Thing and should be avoided. I hope you can clarify if I'm missing some crucial bit of information regarding it's usage.

Thanks
Colin

[Feng Xiao]

On Mon, Jun 8, 2015 at 7:43 AM, Colin Deasy <cde...@demonware.net> wrote:

Hey,

When reading https://developers.google.com/protocol-buffers/docs/proto#simple I see a stark warning indicating that "Required is Forever" advocating the use of optional with additional application level validation routines. This is because if at some point a required field is no longer written, the readers will break.

However IMO, there are common cases where 'required' is a good thing - given that it's enforced only during encoding/decoding.
For example there may be some field that is 'required' (right now) to both the reader and writer. Even if that changes at some point in the future to become optional, the reader would likely have to be updated regardless of the protocol decoding routine as it may make assumptions (reasonable considering it was required in the first place) on the presence of the field (e.g. the field being a key to a certain bit of data). In this case the approach would be to update the .proto of all readers to make that field optional, followed by updating all writers to remove the field.

In simpler scenarios, yes, it's possible to migrate a required field to optional even though it's an incompatible change, but in a more complicated system, where you have many different binaries using the same proto file running on thousands of machines, it's hard to tell whether all readers of a proto has been updated or not. You have to be very careful with such changes, and if you miss one, bad things can happen.

 

Given this, I feel that the current language of the linked document gives the impression that the 'required' attribute is a Bad Thing and should be avoided. I hope you can clarify if I'm missing some crucial bit of information regarding it's usage.

Thanks
Colin

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Colin Deasy]

On Tuesday, 9 June 2015 01:27:24 UTC+1, Feng Xiao wrote:

On Mon, Jun 8, 2015 at 7:43 AM, Colin Deasy <cde...@demonware.net> wrote:

Hey,

When reading https://developers.google.com/protocol-buffers/docs/proto#simple I see a stark warning indicating that "Required is Forever" advocating the use of optional with additional application level validation routines. This is because if at some point a required field is no longer written, the readers will break.

However IMO, there are common cases where 'required' is a good thing - given that it's enforced only during encoding/decoding.
For example there may be some field that is 'required' (right now) to both the reader and writer. Even if that changes at some point in the future to become optional, the reader would likely have to be updated regardless of the protocol decoding routine as it may make assumptions (reasonable considering it was required in the first place) on the presence of the field (e.g. the field being a key to a certain bit of data). In this case the approach would be to update the .proto of all readers to make that field optional, followed by updating all writers to remove the field.

In simpler scenarios, yes, it's possible to migrate a required field to optional even though it's an incompatible change, but in a more complicated system, where you have many different binaries using the same proto file running on thousands of machines, it's hard to tell whether all readers of a proto has been updated or not. You have to be very careful with such changes, and if you miss one, bad things can happen.

My point is that regardless of the size of the cluster, you will need to update every reader - it doesn't matter whether the 'required' constraint is within the protobuf deserialization logic or within the application logic itself.

Similar to missing an update to a certain application instance's proto file, you could miss an update the to application's binary itself. This is precisely why I don't understand the logic behind deprecating the 'required' constraint.

 

[Feng Xiao]

On Tue, Jun 9, 2015 at 2:29 AM, Colin Deasy <cde...@demonware.net> wrote:

On Tuesday, 9 June 2015 01:27:24 UTC+1, Feng Xiao wrote:

On Mon, Jun 8, 2015 at 7:43 AM, Colin Deasy <cde...@demonware.net> wrote:

Hey,

When reading https://developers.google.com/protocol-buffers/docs/proto#simple I see a stark warning indicating that "Required is Forever" advocating the use of optional with additional application level validation routines. This is because if at some point a required field is no longer written, the readers will break.

However IMO, there are common cases where 'required' is a good thing - given that it's enforced only during encoding/decoding.
For example there may be some field that is 'required' (right now) to both the reader and writer. Even if that changes at some point in the future to become optional, the reader would likely have to be updated regardless of the protocol decoding routine as it may make assumptions (reasonable considering it was required in the first place) on the presence of the field (e.g. the field being a key to a certain bit of data). In this case the approach would be to update the .proto of all readers to make that field optional, followed by updating all writers to remove the field.

In simpler scenarios, yes, it's possible to migrate a required field to optional even though it's an incompatible change, but in a more complicated system, where you have many different binaries using the same proto file running on thousands of machines, it's hard to tell whether all readers of a proto has been updated or not. You have to be very careful with such changes, and if you miss one, bad things can happen.

My point is that regardless of the size of the cluster, you will need to update every reader - it doesn't matter whether the 'required' constraint is within the protobuf deserialization logic or within the application logic itself.

Similar to missing an update to a certain application instance's proto file, you could miss an update the to application's binary itself. This is precisely why I don't understand the logic behind deprecating the 'required' constraint.

If you update your proto files in a backward compatible way, you do not need to update every reader when you change your protos. I think this is one of the main reasons why protobuf is created in the first place. You are assuming we have to update all readers any way for every proto change, but in some cases that's not even possible. And whether the "required" constraint is with in the protobuf deserialization logic or in the application logic has a crucial difference: if it's in protobuf deserialization logic, it's enforced for all applications no matter whether the application cares about the content of this proto. For example, a log service may receive log messages from clients. It parses the message but only inspect a few fields of it to determine how and where the message will be stored persistently. Clients can define their own log entries and add these entries to the log message. The log service may very likely have a different release schedule from its clients. If a client decides to remove a required field, the log service will start to fail on parsing such messages even though it does not access these required fields by itself. Basically besides the direct producer and consumer of a proto, there may be some other binaries that need to parse/serialize this proto. Changing a proto in an incompatible way is risking breaking these binaries because it's hard to figure out what binaries are parsing your protos in a complex environment.