using a different serialization format (than protobufs)

[ziffusion]

I have a question about how gRPC would work with a serializing format other than protobufs.

Where would I find information on this? Examples maybe.

When one uses a different serialization, do you still have to use the IDL?

Help will be much appreciated!

[Carl Mastrangelo]

What language are you using?  In Java, you can define your own MethodDescriptor.Marshaller to do the conversion.

[Sanjay Bhandari]

Ah OK. We are intending to use msgpack.

Would you know, if we used an alternative serialization format, would we still need to use the IDL (protobuf IDL)?

Sanjay

--
You received this message because you are subscribed to a topic in the Google Groups "grpc.io" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/grpc-io/DPgipyU0g5Y/unsubscribe.
To unsubscribe from this group and all its topics, send an email to grpc-io+unsubscribe@googlegroups.com.
To post to this group, send email to grp...@googlegroups.com.
Visit this group at https://groups.google.com/group/grpc-io.
To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/d800d6ce-3968-4953-9b74-1d072445cd03%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Carl Mastrangelo]

No, gRPC is encoder agnostic, but we make your life a lot easier if you do use it.  By default reflection will be proto only.  This isn't to say that you couldn't implement it yourself though.

[Sanjay Bhandari]

Do you mind if I press you for some details? We are new to gRPC and protobufs.

1. Are you saying the support for reflection is for protobufs only? On a tangent, what is reflection used for there? I mean, if the protobuf IDL generates code for the involved types, then in what context is reflection necessary?
   
   
2. Do you mind (if you know this) expanding on what we would need to do to use, say, msgpack, with gRPC? I mean, broadly, at a high level - enough for me to dive into the right code.
   
   
3. Is the use of the protobuf IDL optional? If we need to plugin a different serializer, do we need to provide a means to generate the type marshaling code for all involved type?
   

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAAcqB%2BtNs9NqvZky9oxzoU%2BGc%3D9mXE2n4yvG8fSB1GgEmAP7RA%40mail.gmail.com.

[Carl Mastrangelo]

gRPC is designed as two parts:  a core library, and a code generator.  The core library is what does the actual network part of the RPC, taking care to do the interesting things like deadlines, cancellation, load balancing, name resolution, flow control, etc.  This is too specific for most users, so there needs to be a simpler API on top.  The code generator part takes in a Protobuf file and generates a simplified API for client and server applications to use.  It is on top of the core, and not strictly necessary to have.

The generated code makes things for you like strongly typed Stubs, automatic flow control, and of interest to you: Marshallers for the data.  Since the core library is encoder agnostic, a marshaller implementation is necessary to turn raw bytes into message objects.  Additionally, the generated code for protobuf is self describing, so if a client wishes to find out what services a server exports, the server can look at this reflection info to tell the client.  

We recognize that not everyone wants to use Protobuf, so we separate the two parts.  However, since Google is a Protobuf shop, and pretty much everything inside uses Protobuf, we provide the code generators to make it easy.  If you use Proto, you get the second part of gRPC for free.  If not, you can still implemented said features, but it will be up to you to write and maintain the code.

I am more familiar with Java, so I can speak to that.  MethodDescriptor is a class that defines how a method is named, what type it is, and how to marshal and unmarshal data from the wire.  You can look in the grpc-java github repo to find examples (in the tests and generated code) on how to use it.  I don't know about msgpack, so you would need to figure out how to adapt it to the MethodDescriptor.Marshaller interface.

[Sanjay Bhandari]

Thank you. Appreciate the help.

To view this discussion on the web visit https://groups.google.com/d/msgid/grpc-io/CAAcqB%2BsvZCgrj-oA2O1OpeCD68VJdV4Tb4zL5GxBR1Y5yPbQPw%40mail.gmail.com.