Inheriting protobuf messages

[Saurabh Kumar]

Hi,

This question is regarding inheritance in C++ protobuf library. I will explain by giving a concrete example.

 message MSG1

 {

   required string account = 0;

   required string symbol = 1;

 }

 

 message MSG2

 {

   required string account = 0;

   required string symbol = 1;

 }

[Saurabh Kumar]

Hi,

This question is regarding inheritance in protobuf C++ library. I will explain what I am looking for with a concrete example.

I have this message definition:

 message MSG1

 {

   required string account = 0;

   required string symbol = 1;

 }

Now I want to extend this message and define a new message like this:

 message MSG2

 {

   required string account = 0;

   required string symbol = 1;

   required int32  id          = 2;

 }

You will notice that first two fields of MSG2 are exactly same as MSG1 (they are intended to be like that). But here I had to copy paste the common fields again.

Can I do something like this?

 message MSG2 extends MSG1

 {

   required int32  id          = 2;

 }

I have already thought about using it like:

 message MSG2

 {

   required MSG1 msg1 = 0;

   required int32  id          = 2;

 }

But this is not really what I want.

What's the best way to achieve this?

Thanks,

Saurabh

[Feng Xiao]

Protobuf doesn't support inheritance. Having a common header and using composition is the best solution.

 

Thanks,

Saurabh

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Saurabh Kumar]

Thanks for the reply. What exactly do you mean by common header?

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

[Feng Xiao]

I meant something like:

message Header {

  string account = 1;

  string symbol = 1;

}

message Msg1 {

  Header header = 1;

  ...

}

message Msg2 {

  Header header = 1;

  ...

}

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.

[Saurabh Kumar]

Understood but this is not what I wanted in the first place. 

Does someone has any idea about what makes it difficult to implement this? Also, is there a clever way to have the same behaviour?

Basically, here I want to avoid copy pasting same fields over and over again (makes code less maintainable).

Any ideas are welcome.

I meant something like:

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

[Marcelo Cantos]

It probably wouldn’t be difficult to implement, but it’s not, afaik, a design goal for protocol buffers because it is almost never (if ever) necessary.

There are two composition approaches available, depending on what your needs are. Contain the common stuff:

message Common {
  required string account = 1;
  required string symbol = 2;
}

message MSG1 {
  required common = 1
}

message MSG2 {
  required common = 1

  required int32 id = 2;
}

Or contain the variable stuff:

message MSG {
  required string account = 1;
  required string symbol = 2;
  optional Extra1 extra1 = 3;
  optional Extra2 extra2 = 4;

  message Extra1 {
    required int32 id = 1;
  }

  message Extra2 {
    required string foo = 1;
  }
}

with proto3, you can do slightly better:

message MSG {
  string account = 1;
  string symbol = 2;
  oneof extra {
    Extra1 extra1 = 3;
    Extra2 extra2 = 4;
  }

  message Extra1 {
    int32 id = 1;
  }

  message Extra2 {
    string foo = 1;
  }
}

If composition is not what you want, then why not? What real-world problem do you have that cannot be effectively solved with one of the above strategies?

[Saurabh Kumar]

Hi,

Thanks for the suggestions below. I agree with you that these approaches address most  of the real life scenarios. What I am really looking for is more of a syntactic sugar. In my case, I want to avoid any kind of nesting as a personal preference. I just want to avoid copy pasting same members at multiple places and have the overhead of keeping them updated everywhere.

I don't even mind if we had a certain kind of preprocessor to achieve this i.e.

#define common_fields required int32 a; \
                                         required int32 b; \

message msg1 {

common_fields;

required int32 c;

}

message msg2 {

common_fields;

required int32 d;

}

Does it make sense?

[Marcelo Cantos]

I know exactly what you're trying to achieve, but you haven't presented a strong case for why you want this. Both the approaches I've suggested avoid the copy pasting you're worried about, and personal preference isn't an argument.

[Saurabh Kumar]

Sorry about that. I had written 'personal preference' because I thought it is hard to convey why I have to do it.

I think simplest way to convey would be, there is one proto definition say commonmsg.proto. I do not own this and can not make any changes to this.

Now I want to define many messages (that I own) and have some additional optional/required parameters and common ones from commonmsg.proto. So the suggestion about variable stuff is not applicable anymore:

message MSG { required string account = 1; required string symbol = 2; optional Extra1 extra1 = 3; optional Extra2 extra2 = 4; message Extra1 { required int32 id = 1; } message Extra2 { required string foo = 1; } }

Nesting (containing) is not what I want, because the nature of fields is such that it does not make much sense. All of them belong to same level.

I guess it might still not make sense to you, but  you know what I am looking for and it could be a hack :)

[Marcelo Cantos]

Why do they "belong" on the same level? Does containment prevent you from building the system you want (and how?), or are we'll still really talking about personal preference?

[Dov Wasserman]

Hello, Saurabh. I understand what you're trying to do, and that it seems to make perfect sense. Every instance of a Msg2 also has all the fields of a Msg1, so why can't it be a special subtype with just some extra data? Ploymorphism is meant primarily for types where the difference among parents and subtypes is behavior, but not for differering data fields. See here for a more detailed explanation why. 

[Pankaj Garg]

I think it can result in in-efficiencies. One of the core feature of porto-buffers is to pack things efficiently. The lower value of integer assigned to a variable ensures optimised packing and secondly the integers have to be unique but not necessarily continuous.

With multiple level of inheritance, it will be difficult to keep track of the integers assigned to properties in the base structures.

[SangemV]

I see two problems with this approach. It is neither cut & paste nor personal preference issue as I see. The real issues I see are:

1) The base message (MSG in the example)  and the extended message (MSG2 in the example ) can belong to different package owned by different group/org. The package defining base message (MSG) (call it Pkg1) does not event know the existing of the package defining extended message (MSG2) (call it Pkg2). How can the base message foresee all the extended message from it? Even if Pkg1 and Pkg2 are owned by the same group/org, making Pkg1 aware of Pkg2 is not a good idea as it can potentially create cyclic dependencies.

2) Polymorphic Lists: I have a list of MSG types which can potentially have both MSG and MSG2 types. How is this modeled in proto3 using the proposed solution?

These are real issues which I am facing right now. Any suggestions to handle the above problems would be of great help. 

On Friday, October 21, 2016 at 1:42:36 AM UTC+5:30, Marcelo Cantos wrote:

[Marc Gravell]

I'm going to share some thoughts here simply for discussion purposes - I don't expect them to be directly applicable.

FWIW, protobuf-net has spoofed inheritance for many many years. I'm able to do this because protobuf-net only needs to target .NET, which has good inheritance support.

I don't really expect the main "protobuf" project will add inheritance in any timescale, because it might need to target platforms that don't have inheritance support.

It would also require ".proto" language considerations.

That said... if anyone at Google does ever want to reignite the discussion around inheritance, I'm all ears :)

---

The way it protobuf-net does this can be *essentially* represented by "oneof" - so going back to the original example from (wow, a long time ago):

message MSG1

 {

   required string account = 1;

   required string symbol = 2;

 }

message MSG2 extends MSG1

 {

   required int32  id          = 2;

 }

the way protobuf-net does this is essentially:

 message MSG1

 {

   required string account = 1;

   required string symbol = 2;

   oneof _subtype { // keep in mind that this is actually "zero-or-one-of"

      MSG2 msg2 = 3; // this field-number needs to be unique in MSG1

    }

 }

 message MSG2

 {

   required int32  id          = 2; // no restrictions on this field-number

 }

but protobuf-net does some tricks so that when deserializing a MSG1, it might end up *actually* creating an instance of MSG2 instead (where MSG2 : MSG1).

For LSP purposes, everything is *always* serialized from the outside in, so if you had a list/array of MSG2, they would still write the MSG1 fields first - essentially

it would be like a list/array of MSG1, but where everything *happens to be* an MSG2. You're right to say that the base type needs to know about the derived types,

or at least have lib support to help it there, with some kind of deferred registration.

But: I wonder whether the same approach - just without the actual inheritance in the generated types - might still be useful to you.

Concretely, from a code-first angle: https://gist.github.com/mgravell/a33755800e823ee77ba01183be7084df

Or maybe it is completely unhelpful and you're worse off for reading this... who knows!

Marc

On Mon, 3 Sep 2018 at 11:34, SangemV <venkann...@gmail.com> wrote:

I see two problems with this approach. It is neither cut & paste nor personal preference issue as I see. The real issues I see are:

1) The base message (MSG in the example)  and the extended message (MSG2 in the example ) can belong to different package owned by different group/org. The package defining base message (MSG) (call it Pkg1) does not event know the existing of the package defining extended message (MSG2) (call it Pkg2). How can the base message foresee all the extended message from it? Even if Pkg1 and Pkg2 are owned by the same group/org, making Pkg1 aware of Pkg2 is not a good idea as it can potentially create cyclic dependencies.

2) Polymorphic Lists: I have a list of MSG types which can potentially have both MSG and MSG2 types. How is this modeled in proto3 using the proposed solution?

These are real issues which I am facing right now. Any suggestions to handle the above problems would be of great help. 

​

--

You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Marc

[John Lonergan]

The flexibility and expressiveness you are after can be found in Avro.
Having been involved in a pair of big APIs recently, one using proto and the other Avro, the groups found it significant easier and more sustainable when using avro.
A key advantage of avro was abstractions.

[Venkanna Sangem]

John,

  Could you please throw some light on how the problem is solved in Avro?

Thanks,

--
You received this message because you are subscribed to a topic in the Google Groups "Protocol Buffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/protobuf/ojpYHqx2l04/unsubscribe.
To unsubscribe from this group and all its topics, send an email to protobuf+u...@googlegroups.com.