go-protobuf example with self-describing message ?

[Sebastien Binet]

hi there,

any chance somebody has a readily available example in go (of course!)
for the self-describing message technique evocated over there:

https://developers.google.com/protocol-buffers/docs/techniques#self-description

I'd like to add it to my little example:

https://github.com/sbinet/go-pb-example/blob/master/go-pb-example/main.go

(and eventually, if there is enough interest, push it to the official
go-protobuf tutorial...)

-s

[Kyle Lemons]

For what would you need such functionality?  I definitely don't see it as being something that's common enough to be in a tutorial.  One of the advantages of protocol buffers is that both sides know the structure of the message...  In any case, you would not be able to build a Go struct at run-time that matched the described protobuf, though you could theoretically use map[string]interface{} in a similar way to what JSON does.

[Sebastien Binet]

Kyle,

On Tue, Sep 11, 2012 at 8:41 PM, Kyle Lemons <kev...@google.com> wrote:
> For what would you need such functionality?

the plan is to at least be able to generate such self-describing
messages from Go and then consume them in C++ or python.

> I definitely don't see it as
> being something that's common enough to be in a tutorial. One of the
> advantages of protocol buffers is that both sides know the structure of the
> message... In any case, you would not be able to build a Go struct at
> run-time that matched the described protobuf, though you could theoretically
> use map[string]interface{} in a similar way to what JSON does.

there is also the possibility to have a 2-tier process, with 2
processes, one generating the code of the other one :)

my use case is converting data written in a high energy physics -based
file format to/from protobuf files and compare the figure of merit of
these 2 file formats (r/w speed, size, convenience, ...)

-s

[Kyle Lemons]

If you use the self describing proto to which you linked, you can get the FileDescriptorProto from "protoc -oFILE" and then decoding that file.  You can see the source for protoc-gen-go if you want to see how the code is generated or re-wrap it.

[Sebastien Binet]

Kyle,

> If you use the self describing proto to which you linked, you can get the
> FileDescriptorProto from "protoc -oFILE" and then decoding that file. You
> can see the source for

> protoc-gen-go<https://code.google.com/p/goprotobuf/source/browse/#hg/protoc-gen-go>if

> you want to see how the code is generated or re-wrap it.

thanks, that helped me bootstrap my scaffolding.

I noticed that once one enables the filedescriptor generation, one gets
this dubious go-import in the xyz.pb.go file:

import google_protobuf "google/protobuf/descriptor.pb"

which should read:
import google_protobuf "code.google.com/p/goprotobuf/protoc-gen-go/descriptor"


couldn't quickly find where it came from...
did I forget to do something ?

-s

--
#########################################
# Dr. Sebastien Binet
# Laboratoire de l'Accelerateur Lineaire
# Universite Paris-Sud XI
# Batiment 200
# 91898 Orsay
#########################################

[Kyle Lemons]

On Wed, Sep 12, 2012 at 7:35 AM, Sebastien Binet <seb....@gmail.com> wrote:

Kyle,

> If you use the self describing proto to which you linked, you can get the
> FileDescriptorProto from "protoc -oFILE" and then decoding that file.  You
> can see the source for

> protoc-gen-go<https://code.google.com/p/goprotobuf/source/browse/#hg/protoc-gen-go>if

> you want to see how the code is generated or re-wrap it.

thanks, that helped me bootstrap my scaffolding.

I noticed that once one enables the filedescriptor generation, one gets
this dubious go-import in the xyz.pb.go file:

import google_protobuf "google/protobuf/descriptor.pb"

That, I believe, is based on the import in your self-describing proto file.

which should read:
import google_protobuf "code.google.com/p/goprotobuf/protoc-gen-go/descriptor"

Actually, that file is not the canonical one; this is: http://code.google.com/p/protobuf/source/browse/trunk/src/google/protobuf/descriptor.proto

You'll probably need to edit the import before you check it in, because the protoc tool probably can't be taught to do it right.

[Sebastien Binet]

Kyle Lemons <kev...@google.com> writes:

> On Wed, Sep 12, 2012 at 7:35 AM, Sebastien Binet <seb....@gmail.com>wrote:
>
>> Kyle,
>>
>> > If you use the self describing proto to which you linked, you can get the
>> > FileDescriptorProto from "protoc -oFILE" and then decoding that file.
>> You
>> > can see the source for
>> > protoc-gen-go<
>> https://code.google.com/p/goprotobuf/source/browse/#hg/protoc-gen-go>if
>> > you want to see how the code is generated or re-wrap it.
>>
>> thanks, that helped me bootstrap my scaffolding.
>>
>> I noticed that once one enables the filedescriptor generation, one gets
>> this dubious go-import in the xyz.pb.go file:
>>
>> import google_protobuf "google/protobuf/descriptor.pb"
>>
>
> That, I believe, is based on the import in your self-describing proto
> file.

I have this little template:

``` go
const pb_pkg_templ = `package {{.Package}};

import "google/protobuf/descriptor.proto";

message {{.Message}} {
extensions 50000 to max;
{{with .Fields}}
{{range .}} {{.Modifier}} {{.Type}} {{.Name}} = {{.Id}}{{.Attr}};
{{end}}
{{end}}
}

extend google.protobuf.FieldOptions {
optional string root_branch = 50002;
}

message DataHeader {
// Set of .proto files which define the type.
required google.protobuf.FileDescriptorSet proto_files = 1;

// number of entries in the payload message
required int64 NbrEntries = 2;
}
`
```

which looks like what you pointed at...

-s

[Sharath Kumar R]

Did you added this tutorial Sebstian?