Dynamically load .proto / .pb.go files

[Gary Lucas]

We're using protocol buffers primarily as the schemas in a schema registry.

For most of our components including a given version of schema as a library import makes  sense.  

However for a handful of our components (specifically our data lake loading components) it would be advantageous to load assets dynamically.  I'm reasonably sure that there is a way to do so, but the methods are eluding me in the go library.  In terms of loading data, we don't really care about the schema of the data other than validating that it conforms to the given schema.

Can someone elaborate if it's possible to dynamically load a proto / pb.go file rather than importing the generated assets? 

Thanks!

[Adam Cozzette]

I'm not very familiar at all with the Go protobuf implementation but in other languages we usually call this feature DynamicMessage. This lets you operate on a message type whose descriptor you access at runtime, without having to know about it at build time. I think this is probably the Go equivalent: https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/7ebc8ddc-7b16-4255-9a19-4e789575785bn%40googlegroups.com.

[Gary Lucas]

That makes sense and that's where I looked first.  However I didn't find the faculty to load a proto / pb.go file in that library.

As far as I can tell the intention of dynamicpb is to be able to create dynamic messages, I don't see the faculty for loading a fileDescriptor from a proto / pb.go file in that library.   

> This lets you operate on a message type whose descriptor you access at runtime.

Where in the API (for any language) is the call to load a descriptor dynamically?  

I'm looking here right now: https://pkg.go.dev/google.golang.org/protobuf/reflect/protoregistry#Files.RegisterFile and wondering if this is the method I'd be looking for.

Thank you for taking the time to reply!

Gary

[Adam Cozzette]

On Thu, Oct 1, 2020 at 3:48 PM Gary Lucas <gary...@invisionapp.com> wrote:

That makes sense and that's where I looked first.  However I didn't find the faculty to load a proto / pb.go file in that library.

It assumes you already have the descriptor you need (see below for how to get that).

 

As far as I can tell the intention of dynamicpb is to be able to create dynamic messages, I don't see the faculty for loading a fileDescriptor from a proto / pb.go file in that library.   

> This lets you operate on a message type whose descriptor you access at runtime.

Where in the API (for any language) is the call to load a descriptor dynamically? 

It's kind of up to you where you want to store the serialized descriptor. In principle it can come from anywhere, e.g. read from a file or passed over the network. The descriptors are themselves protos, so you can just parse them like any proto. In your case you probably want to call protoc with --descriptor_set_out to generate a serialized FileDescriptorProto for your .proto file. After you parse the FileDescriptorProto, you can call protodesc.NewFile to turn that into a FileDescriptor. I think from there you can find the message you want to work with inside the FileDescriptor and pass it to dynamicpb.NewMessage and go from there.

 

I'm looking here right now: https://pkg.go.dev/google.golang.org/protobuf/reflect/protoregistry#Files.RegisterFile and wondering if this is the method I'd be looking for.

I know little about the Go implementation so I could be wrong, but I think you can use dynamicpb without having to register the FileDescriptor first. That makes it sound like it will add the descriptor to a global registry, which may or may not be what you want.

 

Thank you for taking the time to reply!

Gary

On Thursday, October 1, 2020 at 10:13:25 AM UTC-7 acoz...@google.com wrote:

I'm not very familiar at all with the Go protobuf implementation but in other languages we usually call this feature DynamicMessage. This lets you operate on a message type whose descriptor you access at runtime, without having to know about it at build time. I think this is probably the Go equivalent: https://pkg.go.dev/google.golang.org/protobuf/types/dynamicpb

On Wed, Sep 30, 2020 at 3:52 PM Gary Lucas <gary...@invisionapp.com> wrote:

We're using protocol buffers primarily as the schemas in a schema registry.

For most of our components including a given version of schema as a library import makes  sense.  

However for a handful of our components (specifically our data lake loading components) it would be advantageous to load assets dynamically.  I'm reasonably sure that there is a way to do so, but the methods are eluding me in the go library.  In terms of loading data, we don't really care about the schema of the data other than validating that it conforms to the given schema.

Can someone elaborate if it's possible to dynamically load a proto / pb.go file rather than importing the generated assets? 

Thanks!

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/7ebc8ddc-7b16-4255-9a19-4e789575785bn%40googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/81b42e9a-d684-4ab7-8abb-9a2be4ba675cn%40googlegroups.com.