Dynamic protocol buffer generation in Java

[Mick Killianey]

I'd like to use dynamically generate/use protocol buffer objects in a
running Java application. This would be running in a production
environment (jre, not jdk) where protoc is not available.

Has anyone written a utility in Java to parse a protocol buffer
definition and dynamically build Java classes at runtime using
something like asm?

Mick

[Adam Vartanian]

No need for asm or anything like that, the protobuffer libraries
already contain enough to do what you want to do.

To create a new message definition, you can use DescriptorProto to
create the message, put it in a FileDescriptorProto, and use
FileDescriptor.buildFrom() to build it. Now you can get the
descriptors out with FileDescriptor.getMessageTypes().

Once you have a message's Descriptor, you can parse messages using
DynamicMessage.parseFrom(), and you can create new messages using
DynamicMessage.Builder.

- Adam

[Robert]

Maybe this post helps you:

https://groups.google.com/d/msg/protobuf/2yxvLpKjaHU/hCdbnC3mk-EJ

 

Kind regards,

Robert

[Mick Killianey]

Hmm.  I can't "use DescriptorProto to create the message" -- I have no idea what kind of message I'll need to create until runtime.

In my application, one or more plaintext .proto definition files are loaded at runtime, and I need to generate messages based on the types in those files.  (If it helps, think of the .proto definition file as the PB equivalent of a WSDL file.  At runtime, I'll fetch the .proto definition from a server and give the user a UI that allows them to interact with whatever services are defined there.)

Is there a way to extract the types from .proto definition files at runtime (without invoking protoc)?  I assume that I'd need some kind of .proto file lexical parser, but I don't see one in the Java source.

Mick

[Benjamin Wright]

Unfortunately there is no java equivalent to protoc - it's a long standing feature request.  I can't blame them though - it's a lot more work to keep two compilers in sync with each other.  For now the best you can do is:

1) pre-compile the FileDescriptorSet representation of a proto file and use those to initialize
2) invoke protoc at run time from java (not as hard as it sounds)...

here's some snippets glued together...  protoc, gen_src, and protoFile are all File objects.  And notably, you can install proco in the PATH, you can provide the path to it to the app, or you can even bundle it inside a jar.

            final String command = protoc.getPath() + " -o" + gen_src.getName()
                    + " " + protoFile.getName();

            final Process process;
            try {
                process = Runtime.getRuntime().exec(command, null, workingDir);
            } catch (IOException e) {
            }

                    int result;
                    try {
                        result = process.waitFor();
                    } catch (InterruptedException e) {
                        continue;
                    }
                    if (result == 0) {
                        // Success!

                final FileInputStream input = new FileInputStream(gen_src);
                fds = FileDescriptorSet.parseFrom(input);
                input.close();


                    }

[Dmitriy Ryaboy]

Can't help you with the parsing, but as far as building the object dynamically given a set of fields and their types -- I recently wrote a converter for taking Thrift objects and turning them into dynamic protocol buffer messages. If you can parse a .proto file at runtime to determine what fields need to be created in what order, you should be able to do something very similar (even somewhat simpler as you won't have to fight Thrift's API). 

The various builders involved get a bit tricky. See if the code's helpful:

https://github.com/kevinweil/elephant-bird/blob/master/src/java/com/twitter/elephantbird/util/ThriftToDynamicProto.java

It currently only works with flat messages, but it'd be pretty straightforward to add support for nesting I think.

D

                    }

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.

To view this discussion on the web visit https://groups.google.com/d/msg/protobuf/-/piKV9a9TVPMJ.

To post to this group, send email to prot...@googlegroups.com.
To unsubscribe from this group, send email to protobuf+u...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/protobuf?hl=en.