Re: [protobuf] protoc - how to generate code for proto and its imports

[Feng Xiao]

On Fri, Apr 12, 2013 at 9:14 AM, Leonid G <leonid...@gmail.com> wrote:

Hi,

I have 2 proto files: imported.proto and importing.proto.

imported.proto:
  package first;
  message first_msg { optional bool fld = 1; }

importing.proto:
  package second;
  import "test/imported.proto" ;
  message second_msg { optional first.first_msg field1 = 1; }

I have place both files into "test" directory and when invoking protoc like this:
  protoc --cpp_out=test -I ./test  -I . test/importing.proto test/imported.proto

You should invoke protoc using the following command:

protoc --cpp_out=. test/importing.proto test/imported.proto

Then compile the generated code like this:

g++ -I. -c test/importing.pb.cc test/imported.pb.cc

 

I getting errors:
  imported.proto:3:30: "first_msg.fld" is already defined in file "test/imported.proto".
  imported.proto:1:9: "first_msg" is already defined in file "test/imported.proto".

Which indicates to me that when protoc processes test/importing.proto it actually resolves and loads imported.proto. Is that correct?

You passed in both "-I ./test" and "-I .". That confused protoc. For every proto file protoc will use a relative path to identify it and that's also the path used in "import" statements. For the protos passed in command line, it will determine its path name by matching and stripping the prefix found in "-I" arguments. So the unique name for "test/importing.proto" will be "importing.proto" and "test/imported.proto" will be "imported.proto". Then when compiling "importing.proto", protoc try to find the proto file in the "import test/imported.proto" statement. "test/test/imported.proto" will be tried first and then it finds "test/imported.proto". In the eyes of protoc, there are 3 proto files: importing.proto, improted.proto, test/imported.proto. The first two are in the command line and the third is found through import statement. It then leads to the error message you see.

You can use the command line I suggested above, or you can change the "import test/imported.proto" to "import imported.proto". Both approaches can fix the problem. Their difference lies in the unique names of the proto files. In the former approach, "test/importing.proto" and "test/imported.proto" are their names and generated files will retain the name prefix (i.e., a directory named "test" will be created in the output path and importing.pb.cc|h imported.pb.cc|h will be put into that directory). If using the other approach, generated files will be put directly in the path you specified.

 

If that's the case, then theoretically it should not be necessary to provide both files on command line in order to generate all required code, as protoc has all required info already.

My question - is it possible to tell protoc that it should generate files for all "resolved" files by specifying in command line only top level one ("importing.proto") ?

No.

 

Strictly speaking, in case of C++ it is not actually needed to have individual .cc/.h files for every .proto, as everything could be placed into single .h/.cc. I understand that this might not be possible for other languages, for example java, which has to respect "java_outer_classname" and "java_package". But for language like C++ I do not see why it is important to keep separate files.

I am not saying that when c++ code is generated it should always place code generated from all included protos into single file, but merely that such option might be convenient in some cases.

Reason I am mentioning possibility of embedding all code into single .h/.cc is because currently, assuming that there are two separate invocations of protoc to generate imported.pb.* and importing.pb.*,  there is a sort of dependency - if I generate code like this:
protoc --cpp_out=test -I ./test  test/imported.proto
protoc --cpp_out=test -I -I .      test/importing.proto

then protoc in both case will succeed, but result would not compile because of protobuf_AssignDesc_XXXproto names would not match.
In other words, following has to match:
 1. what flags (-I) I use when compiling test/imported.proto
 2. and how test/imported.proto is imported in other files.

Situation is a bit worse in case of java, as for above example, java generated code would actually compile, but would give a java.lang.ExceptionInInitializerError at run time.

Thanks

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at http://groups.google.com/group/protobuf?hl=en.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

[Leonid G]

Hi Feng Xiao,

Thanks for prompt reply

You should invoke protoc using the following command:

protoc --cpp_out=. test/importing.proto test/imported.proto

 

I am actually working on a build system and in my case all file names are an absolute path, not relative. With absolute path method suggested above (without -I) fails with
$ google/protobuf/2.3.0/gcc_4.1.2/64/bin/protoc --cpp_out=/mypath/test /mypath/test/importing.proto /mypath/test/imported.proto
/mypath/test/importing.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file.  Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

Looks like in general case I do need to set -I to some value. The only remaining thing - understand what the correct value for -I.
 

You passed in both "-I ./test" and "-I .". That confused protoc. For every proto file protoc will use a relative path to identify it and that's also the path used in "import" statements. For the protos passed in command line, it will determine its path name by matching and stripping the prefix found in "-I" arguments. So the unique name for "test/importing.proto" will be "importing.proto" and "test/imported.proto" will be "imported.proto". Then when compiling "importing.proto", protoc try to find the proto file in the "import test/imported.proto" statement. "test/test/imported.proto" will be tried first and then it finds "test/imported.proto". In the eyes of protoc, there are 3 proto files: importing.proto, improted.proto, test/imported.proto. The first two are in the command line and the third is found through import statement. It then leads to the error message you see.

You can use the command line I suggested above, or you can change the "import test/imported.proto" to "import imported.proto". Both approaches can fix the problem. Their difference lies in the unique names of the proto files. In the former approach, "test/importing.proto" and "test/imported.proto" are their names and generated files will retain the name prefix (i.e., a directory named "test" will be created in the output path and importing.pb.cc|h imported.pb.cc|h will be put into that directory). If using the other approach, generated files will be put directly in the path you specified.

Thanks for detailed explanation. I have suspected something along those lines after brief glance thru protoc source code.
 

 

If that's the case, then theoretically it should not be necessary to provide both files on command line in order to generate all required code, as protoc has all required info already.

My question - is it possible to tell protoc that it should generate files for all "resolved" files by specifying in command line only top level one ("importing.proto") ?

No.

Do you know if such approach was ever considered? If it was considered and rejected - I'd appreciate if you can point me place where I can get more info on subject.

BTW, I could not locate any documentation that describes protoc command line arguments, default plugin's arguments and perhaps guidelines on usage.
Would appreciate if you can help me locate such docs.

Leonid

[Oliver Jowett]

On Fri, Apr 12, 2013 at 7:59 PM, Leonid G <leonid...@gmail.com> wrote:

I am actually working on a build system and in my case all file names are an absolute path, not relative. With absolute path method suggested above (without -I) fails with

$ google/protobuf/2.3.0/gcc_4.1.2/64/bin/protoc --cpp_out=/mypath/test /mypath/test/importing.proto /mypath/test/imported.proto
/mypath/test/importing.proto: File does not reside within any path specified using --proto_path (or -I).  You must specify a --proto_path which encompasses this file.  Note that the proto_path must be an exact prefix of the .proto file names -- protoc is too dumb to figure out when two paths (e.g. absolute and relative) are equivalent (it's harder than you think).

Looks like in general case I do need to set -I to some value. The only remaining thing - understand what the correct value for -I.

In your example, it would be -I /mypath

Oliver

[Leonid G]

Thanks, Oliver.

problem is that build system is such that those protos are necessarily under common root.
I guess I'll figure it out.

Still would appreciate any pointers on where I can find protoc documentation.
I suspect that what Feng Xiao explained in post above already covered in that documentation, but I cannot find it.

[Oliver Jowett]

On Fri, Apr 12, 2013 at 8:43 PM, Leonid G <leonid...@gmail.com> wrote:

Thanks, Oliver.

problem is that build system is such that those protos are necessarily under common root.

Not quite sure what you mean.

 

I guess I'll figure it out.

Well, you can perhaps think of it this way: you have one or more non-overlapping filesystem trees of protobuf files; you must specify the roots of those trees via -I arguments; imported files are searched for relative to the root of each tree; all command-line input files must lie within one of the trees.

If you already have suitable subdirectory layout(s) containing your protobuf files, it's probably just a case of telling your build system where the roots of those trees are (which will turn into -I arguments), then consistently using absolute paths everywhere on the command line.

(Personally I'd avoid placing your output anywhere near your input while you're at it ..)

Oliver

[soorya a]

Hi,

   I have a similar query, If im using a thirdparty proto file for server implementation, how can I generate grpc.pb.cc/pb.cc required for all related protos as well.Like envoy frontproxy control plane grpc protos, I want to implement only Endpoint Discovery Service. Doing proto generation for all available proto files in control_plane directory is the only way to go ahead?Please guide.

Thanks and regards,

A Sooryaa

On Saturday, April 13, 2013 at 12:03:23 AM UTC+5:30, Feng Xiao wrote:

To unsubscribe from this group and stop receiving emails from it, send an email to prot...@googlegroups.com.

[Derek Perez]

Check flags for --descriptor_set_out and --include_imports

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.

Visit this group at https://groups.google.com/group/protobuf.
To view this discussion on the web visit https://groups.google.com/d/msgid/protobuf/df5cf4f2-cce4-4900-b73c-1e8f672f08f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.