Dynamic/run-time decoding

[Jan Kyjovský]

Hi,

I have very specific problem. I have data and proto file available and my application should take both and based on external configuration determine how to interpret data (many different types/messages in proto). Yet that can be determine only during run. My question is if there is any support for that, I mean that I will be able to parse proto and decode data based on content of interpret intermediate structures.

I have been trying to analyze this possibility directly from codes but not with much success. I would be glad for any guidance.

[Feng Xiao]

Protobuf supports creating message types dynamically at runtime and use them for parsing/serialization/etc.

First you need to build up a DescriptorPool that contains all types that you may want to use. There are two approaches to construct this pool. One is to call DescriptorPool::BuildFile() directly with parsed proto files. For example:

  // Convert .proto files into parsed FileDescriptorProto

  bool ParseProtoFile(string filename, FileDescriptorProto* result) {

    FileInputStream stream(filename);

    google::protobuf::io::Tokenizer tokenizer(&stream);

    google::protobuf::compiler::Parser parser;

    return parser.Parse(&tokenizer, result);

  }

  // Build the descriptor pool

  DescriptorPool pool;

  for (string filename : proto_files) {

    FileDescriptorProto proto;

    ParseProtoFile(filename, &proto);

    pool.BuildFile(proto);

  }

After you have the pool, you can query for a type by its name. For example, DescriptorPool::FindMessageTypeByName().

Then to actually parse/serialize/use message types in the pool, you need to construct message objects around them. DynamicMessage is used for that:

  // Suppose you want to parse a message type with a specific type name.

  Descriptor* descriptor = pool.FindMessageTypeByName(message_type_to_parse);

  DynamicMessageFactory factory;

  unique_ptr<Message> message = factory.GetPrototype(descriptor)->New();

  // Use the message object for parsing/etc.

  message->ParseFromString(input_data);

  // Access a specific field in the message

  FieldDescriptor* field = descriptor->FindFieldByName(field_to_read);

  switch (field->type()) {

    case TYPE_INT32: message->GetReflection()->GetInt32(*message, field); break;

    ...

  }

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Jan Kyjovský]

Hi,

sorry for delay. I've been experimenting with that eve since I could return to this issue. Mostly about composition of project since whenever I try separate your sources from projects and use them individually I§ve been getting tons of errors (linking one included). In the end I get along with static library linked in my project. But putting that aside.

I haven't get as far as to use all your example. I am stuck and fact that when asking pool for specific message I am getting empty default descriptor as result. I am attaching my code so far could you please advise where I am making mistake?

Its just short program where I was trying to figure out how to use it.

#include <io.h>

#include <fcntl.h>

#include <stdio.h>

#include <string>

#include <vector>

#include "descriptor.h"

#include "descriptor.pb.h"

#include "zero_copy_stream_impl.h"

#include "tokenizer.h"

#include "parser.h"

#include "substitute.h"

#include "dynamic_message.h"

using namespace google::protobuf;

class ParserErrorCollector : public io::ErrorCollector

{

public:

ParserErrorCollector() {}

~ParserErrorCollector() {}

std::string text_;

// implements ErrorCollector -------------------------------------

void AddError(int line, int column, const std::string& message)

{

strings::SubstituteAndAppend(&text_, "$0:$1: $2\n", line + 1, column + 1, message);

}

void AddWarning(int line, int column, const std::string& message)

{

AddError(line, column, "WARNING:" + message);

}

};

bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result);

void main()

{

std::string strFile = "C:\\Users\\kyjovjan\\Desktop\\ProtoExperiment\\addressbook.proto";

// Build the descriptor pool

DescriptorPool pool;

FileDescriptorProto proto;

ParseProtoFile(strFile, &proto);

pool.BuildFile(proto);

// Suppose you want to parse a message type with a specific type name.

const Descriptor* descriptor = pool.FindMessageTypeByName("AddressBook");

DynamicMessageFactory factory;

Message *message = factory.GetPrototype(descriptor)->New();

}

// Convert .proto files into parsed FileDescriptorProto

bool ParseProtoFile(std::string strFileName, FileDescriptorProto* result)

{

  int file_descriptor = open(strFileName.c_str(), O_RDONLY);

ParserErrorCollector errCollector;

io::FileInputStream stream(file_descriptor);

stream.SetCloseOnDelete(true);

io::Tokenizer tokenizer(&stream, &errCollector);

compiler::Parser parser;

return parser.Parse(&tokenizer, result);

return true;

[Feng Xiao]

On Mon, Sep 8, 2014 at 10:32 PM, Jan Kyjovský <jan.ky...@tieto.com> wrote:

Hi,

pool.BuildFile(proto);

Have you checked the return value of these two method calls?

 

// Suppose you want to parse a message type with a specific type name.

const Descriptor* descriptor = pool.FindMessageTypeByName("AddressBook");

Does this return NULL or non-NULL? If it's the later, it means you have successfully constructed the descriptor for AddressBook. 

[Jan Kyjovský]

Variable proto contains some information about structure although proto.name is blank (don't know why).

BuildFile returns NULL as well as FindMessageTypeByName. Whey I went deeper in debugging I noticed that BuildFile is trying to look up for that file I provide in its tables but these are empty as well.

[Feng Xiao]

On Tue, Sep 9, 2014 at 11:03 PM, Jan Kyjovský <jan.ky...@tieto.com> wrote:

Variable proto contains some information about structure although proto.name is blank (don't know why).

You need to set the proto.name yourself:

https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.cc#L147

I guess this is the reason why it doesn't work. Sorry that I missed this in my code example.

[Jan Kyjovský]

Hi,

Thank you that was one thing I tried but was not successful up until now. Also one more issue was that i had to ask for type by full name. It took me some time to figure it our.

Now I got different sort of questions. Since I can now progress further I am finding new "problems". As you suggested i will be using ParseFromString method. Although I am not entirely sure what this string is. Is it represented by hexdump or is it some other format?

I am also wondering about imports. If one proto file imports other do I have to take that consideration and manually handle that file as well or is it done automatically while processing file I have provided? In my implementation all proto files are in one folder with possible imports between each other.

Thank you in advance for your response.

 

[Feng Xiao]

On Wed, Sep 10, 2014 at 10:23 PM, Jan Kyjovský <jan.ky...@tieto.com> wrote:

Hi,

Thank you that was one thing I tried but was not successful up until now. Also one more issue was that i had to ask for type by full name. It took me some time to figure it our.

Now I got different sort of questions. Since I can now progress further I am finding new "problems". As you suggested i will be using ParseFromString method. Although I am not entirely sure what this string is. Is it represented by hexdump or is it some other format?

ParseFromString() accepts protobuf wire-format data (which is the output of protobuf serialization routines).

 

I am also wondering about imports. If one proto file imports other do I have to take that consideration and manually handle that file as well or is it done automatically while processing file I have provided? In my implementation all proto files are in one folder with possible imports between each other.

You can use the DescriptorDatabase implementation we use for protocol compiler:

https://github.com/google/protobuf/blob/master/src/google/protobuf/compiler/importer.h#L76

Basically rather than adding individual files to the pool, you provide a DescriptorDatabase to the pool and the pool will use this DescriptorDatabase to get the FileDescriptorProtos it needs. One culprit of this approach is that, you need to call DescriptorPool::FindFileByName() on the .proto files before you can use DescriptorPool::FindMessageTypeByName() to search for types.

 

Thank you in advance for your response.

 

--

[Jan Kyjovský]

Hi,

due to organization changes and budget cuts this task is put on hold. Thank you for all your guidance up until now. If any further help is needed regarding this issue in the future me or someone else will continue with this thread.

[Jan Kyjovský]

Hi,

so we are somehow back in game. And now I am getting familiar what I have done more then half a year ago.

So for now I got some basic structure of code that will later be as a starting point for real implementation. I am trying to do some simple experiments to see how it works. Though I got problem that even if I get descriptor I am not getting any data. I have used example from encoding specification since I needed something really simple.

...
    const Descriptor* descriptor = pool.FindMessageTypeByName("tutorial.Test1");
    if (descriptor == NULL)
    {
        return;

    }
    DynamicMessageFactory factory;
    Message *message = factory.GetPrototype(descriptor)->New();

    // Use the message object for parsing/etc.

    std::string input_data = "089601";

    message->ParseFromString(input_data);
    // Access a specific field in the message

    for (int i = 0; i < descriptor->field_count(); i++)
    {
        const FieldDescriptor* field = descriptor->field(i);
        switch (field->type())
        {
            case FieldDescriptor::TYPE_INT32:
                {
                    int nVal = message->GetReflection()->GetInt32(*message, field);
                    printf("%d\n", nVal);

                    break;
                }
...

But from reflection I am getting that value is 0. So either I have wrongly set input data considering that I misunderstand format or there is something else. Do you have any idea what may be cause?

Thank you in advance for your reply.

[Feng Xiao]

On Mon, May 4, 2015 at 11:36 PM, Jan Kyjovský <jan.ky...@tieto.com> wrote:

Hi,

so we are somehow back in game. And now I am getting familiar what I have done more then half a year ago.

So for now I got some basic structure of code that will later be as a starting point for real implementation. I am trying to do some simple experiments to see how it works. Though I got problem that even if I get descriptor I am not getting any data. I have used example from encoding specification since I needed something really simple.

...
    const Descriptor* descriptor = pool.FindMessageTypeByName("tutorial.Test1");
    if (descriptor == NULL)
    {
        return;
    }
    DynamicMessageFactory factory;
    Message *message = factory.GetPrototype(descriptor)->New();
    // Use the message object for parsing/etc.
    std::string input_data = "089601";

ParseFromString() expects protobuf binary format data (stored in a C++ string), which I believe 089601 is not.

 

    message->ParseFromString(input_data);

You could check the return value of ParseFromString() to see whether the parsing is a success.

 

    // Access a specific field in the message

    for (int i = 0; i < descriptor->field_count(); i++)
    {
        const FieldDescriptor* field = descriptor->field(i);
        switch (field->type())
        {
            case FieldDescriptor::TYPE_INT32:
                {
                    int nVal = message->GetReflection()->GetInt32(*message, field);
                    printf("%d\n", nVal);

                    break;
                }
...

But from reflection I am getting that value is 0. So either I have wrongly set input data considering that I misunderstand format or there is something else. Do you have any idea what may be cause?

Thank you in advance for your reply.

--

[Jan Kyjovský]

Hi,

This is most likely last post. After converting these example data to binary and making sure that that parsing was successful (return true), I am finally getting decoded data. From this point on I can finish what was requested.

Once again thank you for all your advices and patience.

[Jan Kyjovský]

Hi,

sorry for asking for advice so soon again.

I have progressed a bit futher and I am now trying to display decoded data. For that purpose I have prepared some data and tried to decode them but I have encountered problem with repeated structures and with nested messages. I dont know how to get count of repeats in case of repeated fields so its problematic to address them. In case of nested messages I have experiencing difficulties how to get message.

Here is function I have made so far. Note that I am suspecting that it will probably need some more tweaking so it can be called recursively (work still in progress). For now I am satisfied just by displaying data on first level.

void ExpandSubMessage(Message *message, Descriptor *descriptor)
{

    for (int i = 0; i < descriptor->field_count(); i++)
    {
        const FieldDescriptor* field = descriptor->field(i);
        switch (field->type())
        {
            case FieldDescriptor::TYPE_INT32:
            {
                int nVal = message->GetReflection()->GetInt32(*message, field);

                printf("%s = %d\n", field->name().c_str(), nVal);

                break;
            }
            case FieldDescriptor::TYPE_STRING:
            {
                std::string strVal = message->GetReflection()->GetString(*message, field);
                printf("%s = %s\n", field->name().c_str(), strVal.c_str());

                break;
            }
//            case FieldDescriptor::TYPE_ENUM: message->GetReflection()->GetEnum(*message, field); break;
            case FieldDescriptor::TYPE_MESSAGE:
            {
                Message *messVal;
                if (field->is_repeated())
                {
                    message->GetReflection()->GetRepeatedMessage(*message, field, 1);
                }
                else
                {
                    message->GetReflection()->GetMessage(*messVal, field);
                }
                printf("%s = %s\n", field->name().c_str(), /*messVal.DebugString().c_str()*/"Test");

                break;
            }
        }
    }
}

I have tried to look for details in specification on your pages but didn't find anything that would answer my question.

May I once again ask for your assistance?

[Feng Xiao]

What you are trying to do looks similar to what the TextFormat class does. You can refer to that class as an example to learn how to use protobuf reflection to traverse a message tree:

https://github.com/google/protobuf/blob/master/src/google/protobuf/text_format.cc#L1449

[Jan Kyjovský]

Hi,

yes thank you for that hint. I was able to get enough references to go through decoded data. I am more or less efficiently able to get values (in case of ENUM also original value not only symbolic meaning) but now I would require some auxiliary information which I am not sure are contained in these objects. I will need to get access to current byte offset (from data start) and size of element. Although size is something I can probably somehow derive. Is there any possibility to that?

[Feng Xiao]

On Thu, May 28, 2015 at 12:02 AM, Jan Kyjovský <jan.ky...@tieto.com> wrote:

Hi,

yes thank you for that hint. I was able to get enough references to go through decoded data. I am more or less efficiently able to get values (in case of ENUM also original value not only symbolic meaning) but now I would require some auxiliary information which I am not sure are contained in these objects. I will need to get access to current byte offset (from data start) and size of element.

No, it's not possible to get the byte offset with protobuf API. If you need that, you probably have to write your own parsing method to parse the data.

[Jan Kyjovský]

Never mind then. We can provide results in two formats. Offsets and sizes are requirement of one of these and after giving it some thought I get the feeling that it would be better to use d second format (formatted text), also it would require much less work.

Thank you very much four your help. This time it is really end of the line. I am able to decode data in run-time and that's what was desired from the start.

[Jan Kyjovský]

Hi again,

As I have stated before I am done with decoding, but now I am solving different type of problem. As I have mentioned before imports may prove problematic to our implementation.

Let me describe use-case how it will be used. There will be configuration stating for which data (message number, specific field) will be applied which proto file and structure (data type). Therefore there is no knowledge about other proto files up until now. In run time data are fetched and shipped to decoder. In current implementation is specified proto file loaded and specific type used for decoding. That much is clear and works. But if some parts of structures are imported. It will fail. That much is clear that using just one file descriptor will be not enough as you have mentioned earlier.

That leads to your proposition (posted earlier) to use DescriptorDatabase. I have looked on implementation of classes derived from DscriptorDatabase. I am more or less able to provide directory for SourceTree, but that doesn't answer question about import. I mean since I am aware of only one proto file. Is import done automatically or other files has to be also included manually into database?

Another question I have is bout order of operations. You mentioned that I have to first call FindFileByName() before callingFindMessageTypeByName(). That may be problem since I am not aware in which proto file may be located.

Also I have noticed in code note that proto files with types used in other proto files have to be "loaded" first. So is there any way how to ensure right order of loading?

I hope my question are not too strange but I my knowledge about google protocol buffers are a bit limited.

[Feng Xiao]

Most of the work is done by DescriptorPool. Basically when you call FindFileByName() on a .proto file that is not yet loaded into the pool, DescriptorPool will call FindFileByName() on its underlying DescriptorDatabase to get the FileDescriptorProto for file. Then DescriptorPool will check the imports of this FileDescriptorPool. If it finds unsatisfied imports (i.e., files imported but not yet in the pool), it will issue FindFileByName() to the DescriptorDatabase again for these files.

So basically DescriptorDatabase only needs to know how to load a single .proto file, while DescriptorPool will take care of the imports and ordering.

Depending on what DescriptorDatabase you are using, you may or may not need to call FindFileByName() first. If the DescriptorDatabase has implement FindFileContainingSymbol, then calling DescriptorPool::FindMessageTypeByName() directly will be able to find the file using this method. If not (the case with SourceTreeDescriptorDatabase), FindFileByName() must be called first. As DescriptorPool will be able to find imports by itself, you don't need to call FindFileByName() for every file, but just the file that you actually need to use.

 

I hope my question are not too strange but I my knowledge about google protocol buffers are a bit limited.

--

[Jan Kyjovský]

HI,

I somehow understand what you mean but still I lack in experience with it. I tried to switch to descriptor database but still I got some problems.

C_ProtoDecoder::C_ProtoDecoder(std::string strFile)
{
    m_strProtoFile = strFile;

    // Build the descriptor pool

    m_bConstructed = ParseProtoFile(&m_ProtoFileDescr);

    if (m_bConstructed)
    {
        m_ProtoDescrDatabase.Add(m_ProtoFileDescr);
        m_Pool = new DescriptorPool(&m_ProtoDescrDatabase);
    }
}

C_ProtoDecoder::~C_ProtoDecoder()
{
    delete m_Pool;
}

bool C_ProtoDecoder::DecodeDataAsTypeFromWireFormat(std::string strWireFormat, std::string strType)
{
    Descriptor *descriptor = (Descriptor *)m_Pool->FindMessageTypeByName(strType);
    Message *message = m_MessageFactory.GetPrototype(descriptor)->New();

    if (descriptor == NULL)
    {
        char szError[256];

        sprintf(szError, "Unknown data type %s!", strType.c_str());
        m_InternalPrinter.AddErrorMessage(szError);

        return false;
    }

    bool bFlag = message->ParseFromString(strWireFormat);

    if (!bFlag)
    {
        m_InternalPrinter.AddErrorMessage("Encoding error!");
    }
    else
    {
        m_Transformator.MorphToStructs(*message);
    }

    return true;
}

bool C_ProtoDecoder::ParseProtoFile(FileDescriptorProto *result)
{
  int file_descriptor = open(m_strProtoFile.c_str(), O_RDONLY);

    if (file_descriptor == -1)
    {
        char szError[256];

        sprintf(szError, "Invalid proto file \"%s\"!", m_strProtoFile.c_str());
        m_InternalPrinter.AddErrorMessage(szError);

        return false;
    }

    C_ParserErrorCollector errCollector;

    io::FileInputStream stream(file_descriptor);
    stream.SetCloseOnDelete(true);
    io::Tokenizer tokenizer(&stream, &errCollector);

    result->set_name(m_strProtoFile);

    compiler::Parser parser;

    return parser.Parse(&tokenizer, result);
}

I have used SimpleDescriptorDatabase since it looked like it has already implemented and its covering what I will be requiring from it (or at least I think it does)

Everything is fine until I try to get descriptor by FindMessageTypeByName from pool. It returns null. Am I committing some steps?

[Feng Xiao]

If you use SimpleDescriptorDatabase, you need to add all .proto files into the database explicitly. For example, if you have 2 .proto files foo.proto and bar.proto, where bar.proto imports foo.proto, you need to add both foo.proto and bar.proto to SimpleDescriptorDatabase. If you already have a list of all proto files, using SimpleDescriptorDatabase should be fine. 

 

Everything is fine until I try to get descriptor by FindMessageTypeByName from pool. It returns null. Am I committing some steps?

I don't see anything obviously wrong in your code.

[Jan Kyjovský]

I have remade code so I now add into descriptor database all protofiles specified by directory. There is but little problem with error handling. From nature how I am handling errors it seems that message is possible to decode but there is one error in error collector which is bothering me:

1:1: ERROR: Multiple package definitions.

My protos are defined as follows:
addressbook_fragmented.proto:
import "phone.proto";

package tutorial;

option java_package = "com.example.tutorial";
option java_outer_classname = "AddressBookProtos";

message Person {
  required string name = 1;
  required int32 id = 2;        // Unique ID number for this person.
  optional string email = 3;

//  enum PhoneType {
//    MOBILE = 0;
//    HOME = 1;
//    WORK = 2;
//  }

  message PhoneNumber {
    required string number = 1;
    optional PhoneType type = 2 [default = HOME];
  }

  repeated PhoneNumber phone = 4;
  optional int32 age = 5;
}

// Our address book file is just one of these.
message AddressBook {
  repeated Person person = 1;
}

phone.proto:
package tutorial;

enum PhoneType {
  MOBILE = 0;
  HOME = 1;
  WORK = 2;
}

Even though there is error in error collector I can ignore it and decode message successfully. Yet if I try to remove this package info from phone.proto itis not detecting any error but FindMessageTypeByName fails to find data type. I am suspecting since not all references for person data type I am asking for are resolved it fails.

So I am confused how exactly should be these package keywords be used?

[Feng Xiao]

I don't see a problem in your use of "package". You could try to run protoc on these proto files and see if it reports any error. I suspect it's not the .proto file's problem.

[Jan Kyjovský]

I did as you told me. And tried to compile my proto files with protoc. Well it eat them up without any problem so I guess that clear that there is no problem with them but in that case why is it causing such error to me.

As I gave some hint before problem is happening when I am trying to do following with phone.proto which is being added as second into descriptor database

    compiler::Parser parser;

    parser.RecordErrorsTo(&errCollector);
    bool bV = parser.Parse(&tokenizer, result);
    std::string strErr = errCollector.GetErrorAndWarnings();

    return bV;

Parse returns false and error collector holds following error:

1:1: ERROR: Multiple package definitions.

Do you have nay idea what that could meant?

[肖锋]

On Wednesday, June 10, 2015 at 9:49:03 PM UTC-7, Jan Kyjovský wrote:

I did as you told me. And tried to compile my proto files with protoc. Well it eat them up without any problem so I guess that clear that there is no problem with them but in that case why is it causing such error to me.

As I gave some hint before problem is happening when I am trying to do following with phone.proto which is being added as second into descriptor database

    compiler::Parser parser;

    parser.RecordErrorsTo(&errCollector);
    bool bV = parser.Parse(&tokenizer, result);
    std::string strErr = errCollector.GetErrorAndWarnings();

    return bV;

Parse returns false and error collector holds following error:
1:1: ERROR: Multiple package definitions.

Do you have nay idea what that could meant?

Could you post the whole content of your code? It would be easier for me to figure out what's wrong if I can run the code myself.

[Jan Kyjovský]

It has grown a bit since I started so I will attach zipped whole project.

[Jan Kyjovský]

Hello,

have you any luck finding where there may be problem?

[Mike Trienis]

Hello,

I would very much like to know if this is possible with the Python API?

• https://developers.google.com/protocol-buffers/docs/reference/python/ 
  

In other words, if I have a .proto schema file, do I have to use protoc to compile a python module in order to decode the protobuf message?

Any help would be appreciated, Mike. 

[Mike Trienis]

Hi,

Do you know if it's possible do dynamic decoding at run-time using the Python API? 

• https://developers.google.com/protocol-buffers/docs/reference/python/
  

Indeed there is a descriptor.FileDescriptor, however I'm not sure what the other classes correspond to?

Thanks, Mike. 

On Tuesday, August 12, 2014 at 10:36:13 AM UTC-7, Feng Xiao wrote:

Protobuf supports creating message types dynamically at runtime and use them for parsing/serialization/etc.

First you need to build up a DescriptorPool that contains all types that you may want to use. There are two approaches to construct this pool. One is to call DescriptorPool::BuildFile() directly with parsed proto files. For example:

  // Convert .proto files into parsed FileDescriptorProto

  bool ParseProtoFile(string filename, FileDescriptorProto* result) {

    FileInputStream stream(filename);

    google::protobuf::io::Tokenizer tokenizer(&stream);

    google::protobuf::compiler::Parser parser;

    return parser.Parse(&tokenizer, result);

  }

  // Build the descriptor pool

  DescriptorPool pool;

  for (string filename : proto_files) {

    FileDescriptorProto proto;

    ParseProtoFile(filename, &proto);

    pool.BuildFile(proto);

  }

After you have the pool, you can query for a type by its name. For example, DescriptorPool::FindMessageTypeByName().

Then to actually parse/serialize/use message types in the pool, you need to construct message objects around them. DynamicMessage is used for that:

  // Suppose you want to parse a message type with a specific type name.

  Descriptor* descriptor = pool.FindMessageTypeByName(message_type_to_parse);

  DynamicMessageFactory factory;

  unique_ptr<Message> message = factory.GetPrototype(descriptor)->New();

  // Use the message object for parsing/etc.

  message->ParseFromString(input_data);

  // Access a specific field in the message

  FieldDescriptor* field = descriptor->FindFieldByName(field_to_read);

  switch (field->type()) {

    case TYPE_INT32: message->GetReflection()->GetInt32(*message, field); break;

    ...

  }

On Mon, Aug 11, 2014 at 9:31 PM, Jan Kyjovský <jan.ky...@tieto.com> wrote:

Hi,

I have very specific problem. I have data and proto file available and my application should take both and based on external configuration determine how to interpret data (many different types/messages in proto). Yet that can be determine only during run. My question is if there is any support for that, I mean that I will be able to parse proto and decode data based on content of interpret intermediate structures.

I have been trying to analyze this possibility directly from codes but not with much success. I would be glad for any guidance.

--

[Josh Haberman]

Hi Mike,

Python indeed has descriptor classes that you can use to build proto definitions at runtime. However it does not have a parser for .proto files built in. So if you want to load .proto files at runtime, you'll need to use protoc to build a descriptor (for example, you could shell out to protoc).

Josh

[osu...@gmail.com]

At least in Java, you can do everything fully dynamically at runtime. The native API can get quite confusing (between all the protos/descriptors/files...). I distilled following wrapper to make it easier:
https://github.com/os72/protobuf-dynamic