Using protobuf-java to write .proto files

[ittai zeidman]

Hi,
I have proto files which contain valid proto messages and services and my own DSL which I want to transpile into proto messages and services.
My idea is to invoke protoc and have it process "my proto" with a protoc plugin that will emmit "standard proto" (which I'll later feed into protoc again).
The reason I'm going through the intermediate proto files is because we need them to integrate with other systems.

I'm trying to write this plugin using protobuf-java (jvm developer) but the proto file that is emitted is incorrect.

I think (and hope) I'm doing something stupid and maybe someone here can point me to the right direction.

Thanks in advance...

Some of my code (to generate a hardcoded proto file with one message and one field):

    DescriptorProtos.DescriptorProto protoMessage = DescriptorProtos.DescriptorProto.newBuilder()
      .setName("messageGreeting")
      .addField(DescriptorProtos.FieldDescriptorProto.newBuilder()
        .setName("greeting")
        .setType(FieldDescriptorProto.Type.TYPE_STRING)
        .setNumber(0)
        .setDefaultValue("hi")
        .build()
      )
      .build();
    DescriptorProtos.FileDescriptorProto proto = DescriptorProtos.FileDescriptorProto.newBuilder()
      .addMessageType(
        protoMessage
      )
      .build();
    ByteArrayOutputStream byteArrayOutputStream = new ByteArrayOutputStream();

    proto.writeTo(byteArrayOutputStream); // should be lazy?

    CodeGeneratorResponse.File file = CodeGeneratorResponse.File.newBuilder()
      .setName("yo.proto")
      .setContentBytes(ByteString.copyFrom(byteArrayOutputStream.toByteArray))

//      I also tried:

//      .mergeFrom(protoMessage)
//      .setContent(byteArrayOutputStream.toString) // should be lazy?
//      .mergeFrom(byteArrayOutputStream.toByteArray)
      .build();
    file.toByteArray

Unfortunately what I get is:

cat yo.proto:
"%
messageGreeting                                                                                                                                                                                                                                       greeting(        :hi%                   

and in a text editor:
2225 0a0f 6d65 7373 6167 6547 7265 6574
696e 6712 120a 0867 7265 6574 696e 6718
0028 093a 0268 69

[Nadav Samet]

Hi Ittai,

It sounds like you are expecting your plugin to emit proto files in text format. Your code is assigning binary data into the file's content, and what protoc does is just writing it to the files it creates. There's nothing that would automatically detect that you are passing a DescriptorProto and would transform that into text representation. If you want the output to be in text format you need to manually create a string with the file content you want.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/03f1f05d-a9ef-4490-bf6a-469f16212f9f%40googlegroups.com.

--

-Nadav

[Alex Van Boxel]

You can use this class from the metastore. A lot of work has pored into this class. The plan is to actually donate this to the proto-java implementation if it has been battle tested.

https://github.com/anemos-io/metastore/blob/master/putils/src/main/java/io/anemos/metastore/putils/ProtoLanguageFileWriter.java

It currently only support proto3.

 _/
_/ Alex Van Boxel

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CANZcNEryh5FBtZQ%3D4_EZuy5N21JeMxJ6NTjhR2T68DM5s9OWtw%40mail.gmail.com.

[ittai zeidman]

Nice! Thanks :)

I’ll try it out and report back...

[Alex Van Boxel]

Actually your question triggered something I wanted todo for a long time. You could join the discussion on this thread:

https://groups.google.com/forum/#!topic/protobuf/NrhCXiXIfxk

 _/
_/ Alex Van Boxel

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CAOfK4wVpcbhpsN4eDmfprt4w-cB251mhXcrWp5hid%2B%3DXqtbFDw%40mail.gmail.com.

[ittai zeidman]

Good news!

I was able to get this out:

syntax = "proto3";
message messageGreeting {
        string greeting = 1;
}
Thanks!
I'll report more after I'll have more time to test out the corner cases and richness.

If I find issues would you rather we continue here, github issue, somewhere else?
Also I was unable to find this artifact on maven central or jcenter. I copied the files for the meantime but I'd rather add a dependency and not fork the code.

[Alex Van Boxel]

Congratulations, cool to see other people using it ;-)

Indeed, I rather have it part protobuf-java before haven maven-artifact. You can fill issues on the metastore repo, it's not abandoned. It's activity developed (I'll copy issues over when a move is ever node.

 _/
_/ Alex Van Boxel

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CAOfK4wUveFVMubfbRTq3cDgaswG5qeS5mvsPHoXGRPeYSxe%2BrQ%40mail.gmail.com.

[Derek Perez]

I wrote a library for doing this: https://github.com/perezd/protopoet

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CALCMntvLHG6ArsMRfe0fGj3t08qwRtTRe-qCZ31yPxGQj6qRNw%40mail.gmail.com.

[ittai zeidman]

Awesome!

This thread is full of gems after the few days of me banging my head :)

How complete would you say protopoet is? Have you used it to generate proto files anywhere?

[Derek Perez]

I think it's 100% complete for proto3, checkout the javadoc and tests. Doesn't support proto2 syntax at all However. 

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/CAOfK4wVJWcbnCCRzQMfmy9FSKeJY-V77KV2XFhuyaUMWhpVvTg%40mail.gmail.com.

[ittai zeidman]

That’s ok. I don’t need proto2.

It seems however metastore/Putils fits me better because I’m transpiling and not just generating.

My input is a proto file and my output is a proto file.

Because metastore/Putils uses protobuf-Java’s model I can have protobuf-java and protoc do the heavy lifting of generating my input AST, I’ll manipulate it in memory and metastore/Putils will serialize it out.

Your library looks valuable however for someone generating proto “out of thin air” or from other non proto models. I think it might be useful for other teams internally and I’ll let them know.

+1 on bazel btw :)