Protobuf and Schema Registry

[Ryan Morton]

m conducting an investigation into the use of Protobuf for my organization and have a need to perform schema discovery. I've seen a number of articles related to Confluent Schema Registry with Avro. What are folks using for Protobuf?

[Adam Cozzette]

Protocol buffer schemas are represented with what are called descriptors, and descriptors themselves can be serialized as protocol buffers so that you can save them on disk or send them over the network or whatever you like. If you want to have a sort of schema registry for protocol buffers then I would look into the DescriptorPool and DescriptorDatabase and related classes. These don't provide a full-blown schema server, but they provide all the core functionality that you would need to create one fairly easily. Those classes I linked to are in C++, but we have similar functionality for Java (I'm not sure offhand about other languages).

I don't think schema discovery is actually that useful for protocol buffers most of the time, though. I'm not very familiar with Avro but from reading about it today it appears that when you parse a message you need to have more-or-less the exact schema that was used to serialize it. In that case I can see why you might want to have a versioned schema registry. But with Protobuf it's generally not a problem at all if your schema is older or newer than that of the peer you're communicating with or the data you're reading from disk. The worst that can happen is that you come across some new fields that your schema doesn't know about, and when that happens they are just treated as "unknown fields" and are invisible to you unless you want to go out of your way to inspect them. With protobuf you almost always just use the code generated by the protocol compiler (protoc), and there's generally no need to even think about descriptors unless you're doing something unusual. (The main example would be needing to manipulate message types that are unknown at compile time.)

On Fri, Nov 18, 2016 at 7:59 AM, Ryan Morton <rmo...@zerofox.com> wrote:

m conducting an investigation into the use of Protobuf for my organization and have a need to perform schema discovery. I've seen a number of articles related to Confluent Schema Registry with Avro. What are folks using for Protobuf?

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Michal Borowiecki]

Hi Adam,

I think the use cases are a bit broader. If I'm not mistaken Avro's approach of having access to the original schema that was used to serialize the message allows you more flexibility in schema evolution, doing things that would break compatibility in protobuf. e.g. changing a field's number in protobuf is not a compatible change. But having access to both the sender's schema and the receiver's schema allows you to re-map them symbolically, using the name. I think there are more things like that you can do. But then again I don't have hands on avro experience, only protobuf, so treat it with a grain of salt.

Cheers,

Michał

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.