Protobuf : How to generate the descriptor file dynamically from .proto

[jigmesh...@gmail.com]

Hi,

I wanted to generate FileDescriptor dynamically from .proto without referencing, as in the given example we have reference Student.Interop.DTO so that we can get the StudentDTO.Descriptor and EmployeeDTO.Descriptor, so i wanted to avoid this reference.

Example :

namespace Student.WebApi

{    

    using System.Web.Http;

    using Framework.WebApi.Formatters;

    using Google.Protobuf.Reflection;

    using Repository;  

    using Student.Interop.DTO;  

    public class WebApiApplication : System.Web.HttpApplication

    {        

        protected void Application_Start()

        {

            GlobalConfiguration.Configure(WebApiConfig.Register);

            var json = GlobalConfiguration.Configuration.Formatters.JsonFormatter;

            json.SerializerSettings.Converters.Add(new AnyJsonConverter(TypeRegistry.FromMessages(StudentDTO.Descriptor, EmployeeDTO.Descriptor)));

            

        }      

    }

}

So to achieve this following few steps i tried :

Step 1 : Sample Proto i.e Student.proto which consist of message type i.e StudentDTO and EmployeeDTO

syntax = "proto3";
package AnalysisConsoleApplication;
import "google/protobuf/any.proto";
option csharp_namespace = "AnalysisConsoleApplication";

message StudentDTO
{ google.protobuf.Any Any = 1;
string USN = 2;
string Name = 3;
string University = 4;
}
message EmployeeDTO
{
string Name = 1;
string Gender = 2;
}

Step 2 : I generated Student.pb by using below command i.e 

.\protoc.exe --proto_path=D:\Proto Proto\Student.proto --include_imports --descriptor_set_out=D:\AnalysisConsoleApplication\Student.pb

Step 3 : If i reference Google.Protobuf.3.0.0-beta2 in my SampleConsoleApplication and try to use FileDescriptor class then it will give error as recently in updated version FileDescriptor  is of Internal sealed class. so i clone the Google.Protobuf repository which was of Google.Protobuf.3.0.0-beta2 release time, at that time FileDescriptor class was public sealed class. Now my SampleConsoleApplication  solution will have another project which is of Google.Protobuf.

Note: SampleConsoleApplication zip file is been attached for detail understanding.

This code is from Actual generated C# file(Student.cs) from the given proto, but now next step we will create this below descriptorData dynamically from Student.pb

byte[] descriptorData = global::System.Convert.FromBase64String(

          string.Concat(

            "Cg1TdHVkZW50LnByb3RvEhpBbmFseXNpc0NvbnNvbGVBcHBsaWNhdGlvbhoZ",

            "Z29vZ2xlL3Byb3RvYnVmL2FueS5wcm90byJeCgpTdHVkZW50RFRPEiEKA0Fu",

            "eRgBIAEoCzIULmdvb2dsZS5wcm90b2J1Zi5BbnkSCwoDVVNOGAIgASgJEgwK",

            "BE5hbWUYAyABKAkSEgoKVW5pdmVyc2l0eRgEIAEoCSIrCgtFbXBsb3llZURU",

            "TxIMCgROYW1lGAEgASgJEg4KBkdlbmRlchgCIAEoCUIdqgIaQW5hbHlzaXND",

            "b25zb2xlQXBwbGljYXRpb25iBnByb3RvMw=="));

      descriptor = pbr::FileDescriptor.FromGeneratedCode(descriptorData,

          new pbr::FileDescriptor[] { global::Google.Protobuf.WellKnownTypes.AnyReflection.Descriptor, },

          new pbr::GeneratedCodeInfo(null, new pbr::GeneratedCodeInfo[] {

            new pbr::GeneratedCodeInfo(typeof(global::AnalysisConsoleApplication.StudentDTO), global::AnalysisConsoleApplication.StudentDTO.Parser, new[]{ "Any", "USN", "Name", "University" }, null, null, null),

            new pbr::GeneratedCodeInfo(typeof(global::AnalysisConsoleApplication.EmployeeDTO), global::AnalysisConsoleApplication.EmployeeDTO.Parser, new[]{ "Name", "Gender" }, null, null, null)

          }));

Step 4: Now to generate descriptorData  dynamically from Student.pb, i have written the following code which gives the descriptorData in byte[]

//get the path of Student.pb

var path = string.Format("{0}/Student.pb", Path.GetDirectoryName(new Uri(Assembly.GetExecutingAssembly().CodeBase).LocalPath)); 

//create messageParser

MessageParser<FileDescriptorSet> messageParser = new MessageParser<FileDescriptorSet>(() => new FileDescriptorSet());

// open and read it

CodedInputStream stream = new CodedInputStream(new FileStream(path, FileMode.Open));

//Parses a message from the given coded input stream.

var data = messageParser .ParseFrom(stream);

//Covert to ByteString

var descriptorData  = data.File[0].ToByteString();   

Step 5: Last step to generate filedescriptor by calling FileDescriptor.FromGeneratedCode() which accepts 3 parameters

Parameter 1:  byte[] descriptorData -> this parameter we are able to generate dynamically as done in step 4 i.e   descriptorData.ToByteArray()       

Parameter 2:  FileDescriptor[] dependencies-> this takes all the dependencies.

Parameter 3: GeneratedCodeInfo generatedCodeInfo-> This has GeneratedCodeInfo(Type clrType, MessageParser parser, string[] propertyNames, string[] oneofNames, Type[] nestedEnums, GeneratedCodeInfo[] nestedTypes) out of this parameters i am not able to get clrtype dynamically.

Here is the brief example of what i am expecting:

typeof(global::SampleConsoleApplication.StudentDTO): this gives the type i.e clrtype of StudentDTO which is of public sealed class, which will be generated in  Student.cs by executing the Student.proto. But I am trying to get the clrtype without generating Student.cs, so because of which typeof(global::SampleConsoleApplication.StudentDTO) can't be used here.

[osu...@gmail.com]

This works in Java, perhaps you can adapt it to C# (?)
https://github.com/os72/protobuf-dynamic

Is there DynamicMessage in the C# API?

[Benjamin Krämer]

I had a similar discussion some months ago. What you want is to generate the TypeRegistry at Runtime. I did this by using the following approach:

https://groups.google.com/d/msg/protobuf/fnhsS9tVhyc/9pkapOl-AQAJ

Therefore you don't have to gereate the file descriptors but just use the ones you already have at runtime. This worked, but needed to also generate partial classes for the Reflection classes which implement just the IReflection interface (just containing the Descriptor property). This would work as general solution, but right now I'm not using it since I found it better to just have a method in my code to return the FileDescriptors they are needing to have registered. Since I know all modules I don't have to rely on reflection.

[jigmesh...@gmail.com]

Hi,

Thank you So much for the reply.

Actually i have already implemented this approach but in this we need to put the dll but i wanted to avoid this step also.  We want to create the filedescriptor from the .pb file.

[jigmesh...@gmail.com]

Hi,

Thank you so much for the reply.

No i cant find it.