[capnproto] Marking deleted fields (was: Inline structs and lists)

[Alek Storm]

On Tue, Apr 30, 2013 at 9:39 PM, Keir Mierle <mie...@gmail.com> wrote:

On Tue, Apr 30, 2013 at 9:19 PM, Kenton Varda <temp...@gmail.com> wrote:

On Tue, Apr 30, 2013 at 8:51 PM, Mark S. Miller <eri...@google.com> wrote:

The "same direction" constraint is surprising. Is this because you don't allow holes in the numbering? If you did, and you never reused an old number for something else, would the "same direction" constraint go away? Are you disallowing this only to avoid a minor representation cost?

So, JSON, Protobufs, and everything else all have the same directionality constraint, but only in that you can't reuse the same field identifier (name in JSON, number in protobufs) for a differently-typed field.  I assume you only find the constraint surprising in the case where reuse isn't happening, right?  Otherwise, if compatibility were fully transitive and commutative, then every struct would be compatible with every other struct, since every struct is clearly compatible with the empty struct.

I think what you're missing about Cap'n Proto is that the field numbers do not directly translate to positions.  Different fields have different widths, and pointers are actually segregated from flat data.  Therefore, the position of any particular field necessarily depends on what fields were declared "before" it.  The purpose of the field numbers in Cap'n Proto is only to provide a stable ordering -- one which developers are unlikely to "mess up" by adding new fields somewhere other than at the end of the struct.

The restriction that field numbers cannot have "holes" follows from this.  Removing an earlier field would cause all later fields to shift forward, so this cannot be allowed.

In practice, no one ever removes fields from protobufs, because doing so leads to the risk that the field number will be accidentally reused.  Instead, they rename the field to something like "OBSOLETE".  So, Cap'n Proto is merely enforcing what was already best practice.

While I did see the "OBSOLETE_foo" technique at Google, this was not the only way. Both my group and the protos from other groups I used often followed a pattern like this:

message AnOldAndCruftyMessage {

  optional int32 a_field = 3; 

  // ...many more fields

  // Deleted fields: 10, 17, 22, 3.

}

The advantage is that the deleted fields don't clutter the file. The disadvantage is that it is possible to accidentally reuse a field. This approach won't work for capnproto because without the sizing information, the compiler can't generate backwards compatible code. But perhaps we could have a more compact "remove" annotation instead of keeping the entire field around. Currently, to delete a field you need to do something like:

struct Person {

  name @0 :Text;

  birthdate @3 :Date;

  email @1 :Text;

  phones @2 :List(PhoneNumber);

  DELETED_crufty_extras_here @10 :DELETED_CruftyStruct;

}

struct DELETED_CruftyStruct {

  // This used to have lots of crufty fields, but is deleted.

}

What about instead something like:

struct Person {

  name @0 :Text;

  birthdate @3 :Date;

  email @1 :Text;

  phones @2 :List(PhoneNumber);

  Deleted(10 :Struct, 5 :Int32);

}

which makes it possible to remove the dependent struct definitions as well. Furthermore, if we assume that deleted fields are always 8-sized (so structs) then potentially even the type of the deleted field could be omitted unless it is a different size (e.g. bool). I don't care especially about the particular syntax for indicating deletes, but I do want a clean way of removing old fields.

While I'm in favor of a $deprecated("use foo instead") annotation, I'm concerned that letting users break backwards compatibility at will by "deleting" fields (in favor of replacement ones?) seems at odds with the design philosophy of Cap'n Proto. The fact that fields marked as deleted will be ignored, rather than cause a parsing error, actually makes things worse, since older clients will continue to set the deleted fields with the expectation that those fields will be honored by any future receiver, when in reality, they are being silently misinterpreted.

The contract I feel we should encourage is that if you add a field to a struct, then it must be interpreted the same way by all future receivers of that struct, forever. Breaking backwards compatibility is a fact of life, but it should result in the user creating a new struct (sans the unwanted fields), with the ordinal numbering reset to 0. This will force clients to handle incompatible structs separately, rather than handling one large struct representing the sum of all its past versions.

What do others think?

Alek

On Tue, Apr 30, 2013 at 8:46 PM, Kenton Varda <temp...@gmail.com> wrote:

Compatibility is commutative, but is only transitive if all changes are in the same "direction" -- upgrade or downgrade.  If you perform a series of upgrades to a type, the result is compatible with the original type.  If you apply a series of downgrades, they are also compatible.  But if you apply a mixture of upgrades and downgrades, the result is not compatible.

For example, if you start with struct A, remove its last field (a downgrade) to produce B, and then add a different field (an upgrade) to produce C, A is compatible with B and B is compatible with C but A is not compatible with C.

Our tool would want to enforce that any changes are strictly in the upgrade direction.

On Tue, Apr 30, 2013 at 6:51 PM, Mark Miller <eri...@gmail.com> wrote:

Is compatibility commutative? Assuming not, is it a partial order? A lattice? Assuming yes, now that you also have interfaces, new services can invoke old services and vice versa. So interface compatibility may need to account for co-variance and contra-variance of argument and return compatibility. 

(There was an old Sun Labs tech report from the Spring project that tried to map compatibility relationships onto subtyping. It was unsound but it may be relevant. Unfortunately, I don't know what to search for to find it. A bit of searching just now was fruitless.)

On Tue, Apr 30, 2013 at 6:28 PM, Kenton Varda <temp...@gmail.com> wrote:

On Tue, Apr 30, 2013 at 2:15 PM, Andrew Lutomirski <an...@luto.us> wrote:

This is, I think, just complicated enough to inspire a feature

request: can we have something like capnpc
--are-definitions-compatible?

Yes!  This is on my list, although it's relatively low priority compared to actually implementing the system.  :)  (Maybe someone would like to contribute?)

Ideally you'd be able to set things up so that git (or whatever you use) will automatically check for compatibility whenever you commit changes to a .capnp file.

--

You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto?hl=en-US.
 
 

--
Text by me above is hereby placed in the public domain

  Cheers,
  --MarkM

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto?hl=en-US.
 
 

--

    Cheers,
    --MarkM

--
You received this message because you are subscribed to the Google Groups "Cap'n Proto" group.
To unsubscribe from this group and stop receiving emails from it, send an email to capnproto+...@googlegroups.com.
Visit this group at http://groups.google.com/group/capnproto?hl=en-US.
 
 

[Kenton Varda]

On Thu, May 2, 2013 at 4:42 PM, Alek Storm <alek....@gmail.com> wrote:

While I'm in favor of a $deprecated("use foo instead") annotation, I'm concerned that letting users break backwards compatibility at will by "deleting" fields (in favor of replacement ones?) seems at odds with the design philosophy of Cap'n Proto. The fact that fields marked as deleted will be ignored, rather than cause a parsing error, actually makes things worse, since older clients will continue to set the deleted fields with the expectation that those fields will be honored by any future receiver, when in reality, they are being silently misinterpreted.

The contract I feel we should encourage is that if you add a field to a struct, then it must be interpreted the same way by all future receivers of that struct, forever. Breaking backwards compatibility is a fact of life, but it should result in the user creating a new struct (sans the unwanted fields), with the ordinal numbering reset to 0. This will force clients to handle incompatible structs separately, rather than handling one large struct representing the sum of all its past versions.

What do others think?

I think that's unrealistic.  In the real world, it is very common to want to remove features that have high maintenance cost and few users.  You contact the existing users, get them to stop using the feature, and then you stop honoring the field.  This works fine in practice.  But if every time you did this, you had to replace the whole protocol with a new one and get every client to update...  well, that would suck.