[Protobuf] Some questions

269 views
Skip to first unread message

clem.d...@gmail.com

unread,
May 2, 2016, 7:32:22 PM5/2/16
to golang-nuts
Hello Gophers,

I have bunch of questions trying to use Protobuf for the first time with Go and I wish you guys can help me with most of it. 

I. Type conversion :
  1. How would you convert a [][]string to protobuf ? I've read on a StackOverflow thread to use a repeated string_slice, the later being a message composed of a repeated string. The problem with that solution is that the conversion when generated to Go files isn't quite the same. 
  2. How would you convert a map[string]interface{} to protobuf ? The best solution I've come up with is using Any but I'm wondering if actually using a Struct isn't better. I tried to do a map<string, Any> but this doesn't quite translate neither. 
  3. Same question actually goes for []map[string]interface{} ? Since you can't do repeated map<string, Any>.
  4. Finally, how would you serialize some interface{} in an Any type since you have to provide a []byte. ==> json.Marshal ? That wouldn't make much sense to use that serializer to use Protobuf after.

II. Protobuf usage :
  1. When I use Any, I get this import in my Go generated file : import google_protobuf "google/protobuf/any" which obviously is not found when compiling the Go. Any way to prevent that at protoc time ?
  2. Do you recommend using only your Protobuf structure in all messages ? For example, if I had a previous structure used in my code everywhere, should I rather convert it to use the generated Go structure or try to build a converter between old and new structure ?
Any help on any of the question would be greatly appreciated.

Thank you in advance.
Clem.

clem.d...@gmail.com

unread,
May 12, 2016, 1:42:30 PM5/12/16
to golang-nuts
No one can actually answer any of the above questions ? :-(

Marcus Hines

unread,
May 13, 2016, 12:29:14 AM5/13/16
to golang-nuts, clem.d...@gmail.com

So let me start with here are just some rough analogies of what you are asking but I really wouldn't recommend using proto for these types of operations.
It really isn't built for unstructured data.  That is what JSON is for. Or in the case of direction serialization of go structs/data types - you might want to look at gob.


On Monday, May 2, 2016 at 4:32:22 PM UTC-7, clem.d...@gmail.com wrote:
Hello Gophers,

I have bunch of questions trying to use Protobuf for the first time with Go and I wish you guys can help me with most of it. 

I. Type conversion :
  1. How would you convert a [][]string to protobuf ? I've read on a StackOverflow thread to use a repeated string_slice, the later being a message composed of a repeated string. The problem with that solution is that the conversion when generated to Go files isn't quite the same. 

message RepeatedString {
  repeated string value = 1;
}

message Foo {
  repeated RepeatedString value = 1;
} 

  1. How would you convert a map[string]interface{} to protobuf ? The best solution I've come up with is using Any but I'm wondering if actually using a Struct isn't better. I tried to do a map<string, Any> but this doesn't quite translate neither.
 map<string, proto.Any>
 
would be as close as you can but your interface{} would have have a marshaler to create a concrete proto message.

  1.  
  2. Same question actually goes for []map[string]interface{} ? Since you can't do repeated map<string, Any>.
message StringAnyMap {
  map<string, proto.Any> value = 1;
}

message Foo {
  repeated StringAnyMap value = 1; 
}
  1. Finally, how would you serialize some interface{} in an Any type since you have to provide a []byte. ==> json.Marshal ? That wouldn't make much sense to use that serializer to use Protobuf after.

 
so you could do some really terrible things with reflection and a raw proto descriptor. However, I just wouldn't recommend it would be very error prone and fragile.

II. Protobuf usage :
  1. When I use Any, I get this import in my Go generated file : import google_protobuf "google/protobuf/any" which obviously is not found when compiling the Go. Any way to prevent that at protoc time ?
find where protoc's libraries are installed for me it is this:
cd $GOPATH/src
DIR=/usr/local/include;protoc -I $DIR $DIR/google/protobuf/any.proto $DIR/google/protobuf/source_context.proto $DIR/google/protobuf/type.proto

that should put the generated code in the right place
  1. Do you recommend using only your Protobuf structure in all messages ? For example, if I had a previous structure used in my code everywhere, should I rather convert it to use the generated Go structure or try to build a converter between old and new structure ?
I don't claim to use proto's right but I normally use proto for gRPC calls and internally i just use native data structures with Marshal() calls to turn them into the proper messages to be either serialized to disk or to be sent to gRPC calls.
Sometimes I have a type which wraps a proto message and attaches methods to the wrapped type

type Foo {
  pb pb.Foo
}

func (f *Foo) DoNormalStuff()

func (f *Foo) Proto() pb.Foo {
  return f.pb 
} 

type NativeFoo {
  v int64
  s string
}

func (nf *NativeFoo)Proto() pb.Foo {
  return pb.Foo{
    value: nf.v,
    string: nf.s,
  }
}

Using proto3 syntax - proto2 you would need the proto.Int64() proto.String() wrappers

But again working with interface{} and proto are just very opposite ends of the spectrum
Reply all
Reply to author
Forward
0 new messages