Generating an (C#) Interface and extendable data model from .proto, namespacing and inheritence

[KarrokDC]

Hi all,

I was wondering if any thought has gone into generating an interface to go along with the generated data classes from a proto.

And if it had been considered were there any reasons to not do it?

Quick sketch of my current situation:

I have a project that has a complex and messy internal data structure (assume for now its too expensive to refactor).

On top of this I created an interface that already bears semblance to a .proto model we use. And an implementation of this interface which has a reference to an internal object and 'transmutes' the data from the internal model to the external format. So aside from the reference to the internal object, no data is held by the interface implementation. It basically acts as an adapter.

The implemented getters/setters allow the transmutation of internal data structures to 'external' structures

I am now however looking into simply generating this interface and part of the implementation to allow this model to be exported easily, while still having a clean API, which would (in time) also allow for code written against the generated API (also C++/python) to be portable.

For this I've been hacking away a bit at the C# proto generation code in an effort to generate something that would fit into the design below:

What I've done so far

• Added a mirror of csharp_reflection_class as csharp_reflection_interface (ReflectionInterfaceGenerator)  to generate what I would like my interface to look like
  
• Added interface-ended generation options to protoc cli ie. --csharp_opt=generate_interface=true
  
• Changed/added namespaces to use in generation (because in my case, the interface is in a different namespace than the implementation)
• Added a way to extract (ie not generate) property get;set; (similar to how https://silentorbit.com/protobuf/ generates from proto), the developer will need to implement these themselves, but having the interface enforces it.

Now there are a number of open questions/issues that I thought proto devs/experienced C# developers might be able to weigh in.

1. In various areas, the package name defined in the .proto is used as a way to create the namespace, types etc which aren't even overwritten by the --csharp_opt=base_namespace=... option. and it (seems) requires a C# specific option in the .proto (option csharp_namespace =...) Is this something I should definitely not overwrite/force, or can I freely change namespaces of generated files without problems in later serialization/deserialization?
2. Proto doesn't actively support inheritance causing each C# object to hold derived object instances. This is currently all generated out, but for any OO language it doesn't make any sense to have this in a design. But these are still generated as-is. I've considered custom proto markup extensions to mark these types as being derived to ignore them in generation of the interface (not implementation), but I'm unsure whether there are better ways to go about this.
3. The implementation explicitly generates private fields for each property and some code (Equals / GetHash etc) use these fields directly. For my purpose however I don't need/want these fields, instead the implemented accessors should be used. Would this cause any problems with the generated code (after moving all field usage to accessor usage)?

Are there any alternative solutions that I've missed, would any of this work be in any way/shape form be useful for the Proto C# generation mainline?

Thoughts, ideas, suggestions?

 

[Marc Gravell]

I can't speak for the design choices - but *as I understand it*, the key point of protobuf is to enable things to work well cross-platform. That means that at the DTO level, things need to be *possible* to implement in a wide range of languages, and the reality is that not all frameworks would allow what you describe. So building them into the protocol would limit the target scope, or force several to be second-class citizens.

In the case of C#, we are blessed by things like "partial classes" and "extension methods" (and hopefully "extension everything" in a future language version); that combined with encapsulation means you already have a huge range of options. But - something I've said time and time again, and coming from someone who *deliberately set out* to create a protobuf implementation that works well against your pre-existing POCOs:

Serializing your domain model is fine, as long as it works perfectly. As soon as it gets *even remotely* complicated - or you start having to fight either your POCO model or the serializer, then the best thing to do is to very deliberately and neatly disconnect the two worlds. Have one object model that is your domain model - the POCO world. And have a *completely separate* model for serialization - the DTO world. How you map between them is an uninteresting implementation detail, IMO. There are tools that help, or static conversion operators, or whatever else you like.

That said: if altering the code generator works for you: great! do that!

As to your questions:

1: if I understand you correctly, then that won't impact the serialization or reflection/descriptor APIs at all

2: as it happens, protobuf-net already models inheritance via optional members, and I will be migrating that to "oneof" in proto3; as part of this, it is fully my plan to implement a custom markers (via the extension API) that work in my own codegen, to help it recognize when to emit C# inheritance from a .proto; so ... you wouldn't be alone in that idea (note: my implementation is not the official Google one)

3:  what generator are you using, specifically (including version number)? Is it possible to give a minimal example of a .proto and the C# that is offending you, and what you want it to be instead? It would be much easier to discuss with a concrete example.

Marc

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Marc

[KarrokDC]

Hey Marc,

Thanks for the feedback.

I'm currently mangling the "official" proto C# generator that was recently ported from https://github.com/jskeet/protobuf-csharp-port by Jon Skeet and brought into the google proto mainline @ https://github.com/google/protobuf/tree/master/csharp

As for an example:

example.proto:

syntax = "proto3";

package pb;

message Color {

  int32 R = 1;

  int32 G = 2;

  int32 B = 3;

}

message Dog {

  int32 dogProperty = 1;

}

message Cat {

  int32 catProperty = 1;

}

message Animal {

  // trivial data

  bool canSwim          = 1;

  

  // complex data (composition)

  Color skinColor       = 2;

  

  enum AnimalType {

    animalTypeDog = 0;

    animalTypeCat = 1;

  }

  

  AnimalType animalType = 3;

  

  // complex data (inheritance)

  Dog dog               = 4;

  Cat cat               = 5; 

}

This generates an implementation (or in my case also my half-working interface generation) like this:

In IExample.g.cs

public partial interface IAnimal

  {

    /// <summary>

    /// trivial data

    /// </summary>

    bool CanSwim { get; set; }

    /// <summary>

    /// complex data (composition)

    /// </summary>

    IColor SkinColor { get; set; }

    Animal.Types.AnimalType AnimalType { get; set; }

    /// <summary>

    /// complex data (inheritance)

    /// </summary>

    IDog Dog { get; set; }

    ICat Cat { get; set; }

  }

  public partial interface IDog

  {

    int DogProperty { get; set; }

  }

etc

In Example.g.cs

internal partial class Animal : pb::IMessage<Animal>, IAnimal 

{

   ..stuff

}

Now obviously the generation of the interface is far from done, but as far as the generator is concerned, IDog and IColor are both composites because there is no way to distinguish between the two.

I will have to find a good way to specify this difference so the parser can catch it.

This would then (maybe) also allow for the generator to create:

  public partial interface IDog : IAnimal

  {

    int DogProperty { get; set; }

  }

This would require 2 properties. One above a message to say whether it derives off other message ( to add f/e let IDog inherit from IAnimal).

And another to mark a message member (f/e Dog dog = 4;) as being derived from.. as all messages are handled individually.

A secondary problem comes from the boiler plate code including the above in writing and reading mechanics. 

In my case, I create external objects with an internal object reference as constructor parameter, ie I would have in Animal.cs (manual implementation stuff)

internal partial class Animal

{

  internal Animal(InternalObjectType internalObject)

  {

    // do stuff with internalObject

  }

  

  bool CanSwim

  {

    get { return internalObject.TerribleInternalFuncionToGetSwimInfo(); }

    set { internalObject.SomeArcaneWizardryToSetSwimming(value); }

  }

  

  IColor SkinColor

  {

    get { var color = new Color {R = internalObject.Redish, B = ... etc}; return color; }

    set { internalObject.Redish = value.R; ...etc

  }

} 

Now the generation as-is does provide a partial method OnConstruction, which I could implement and use to do the usual work post-construction, but it makes nested properties a PITA :)

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--

Regards,

Marc