Best practices for proto file organization in large projects

[Justin Muncaster]

Hello,

I've been using protocol buffers for a while, and I love the library, however I find that when using them in large projects I generally have to fight with the compiler to get protoc to play nicely with my build system. My projects are organized as follows:

common/foo/foo.h

common/foo/foo.cpp

common/foo/foo.proto

...

common/bar/bar.h

common/bar/bar.cpp

common/bar/bar.proto

where bar.proto contains

  import "common/foo/foo.proto"

... and elsewhere...

app1/baz.proto

...

app1/fud.proto 

and baz.proto and fud.proto contain

  "import common/bar/bar.proto"

 

I'm currently changing our build system to be cmake based and I'm again finding myself fighting with the build system to get the .proto to be automatically generated in a way where they build correctly. This leads me to believe that I am doing something wrong or at least not organizing files in a way that is expected. In light of that,

How do you organize your proto files when you have many in common libraries? Do all .proto files live in one folder? Should one avoid "import a/b/c/d/f.proto"? Do you have any recommendations for how one ought one setup the cmake build system to work with proto files that are organized as they are above? Any general recommendations?

Thanks,

Justin 

[Evan Jones]

On Jun 18, 2012, at 22:49 , Justin Muncaster wrote:
> I'm currently changing our build system to be cmake based and I'm again finding myself fighting with the build system to get the .proto to be automatically generated in a way where they build correctly.

What specific problems are you having? Errors in clean builds? Errors when modifying a .proto and rebuilding?

> How do you organize your proto files when you have many in common libraries? Do all .proto files live in one folder? Should one avoid "import a/b/c/d/f.proto"? Do you have any recommendations for how one ought one setup the cmake build system to work with proto files that are organized as they are above? Any general recommendations?

What I've done on my last project was to put all the .proto source code in their own "proto" directory. But this was a cross-language project, so I was accessing them from both C++ and Java, so that seemed to make the most sense to me. I configured the build to generate all C++ files into build/*, and the java files into build/java, then I included/compiled them from there.

The Chrome browser organizes its .proto files in a very different way:

http://src.chromium.org/viewvc/chrome/trunk/src/chrome/common/metrics/proto/

Hope this helps,

Evan

--
http://evanjones.ca/

[xiongjun liang]

we are using protobuf in a very large system,
and we simply put all proto files in one directory like "./public/rpc"
and it works very well!

On Jun 19, 10:49 am, Justin Muncaster <justin.muncas...@gmail.com>
wrote:

[Justin Muncaster]

Thanks for your reply. Attached is a source tree I think should "just work". I get the error:

1>Running C++ protocol buffer compiler on common/bar/bar.proto

1>common/foo/foo.proto: File not found.

1>bar.proto: Import "common/foo/foo.proto" was not found or had errors.

I can fix the error by hacking FindProtobuf.cmake and passing in additional include directories, but I run into problems down the line, which leads me to think there must be a better way. Every example I see has all proto files in one folder and does not have cross-library protobuf message dependencies.

[Evan Jones]

On Jun 19, 2012, at 13:53 , Justin Muncaster wrote:
> 1>Running C++ protocol buffer compiler on common/bar/bar.proto
> 1>common/foo/foo.proto: File not found.
> 1>bar.proto: Import "common/foo/foo.proto" was not found or had errors.
>
> I can fix the error by hacking FindProtobuf.cmake and passing in additional include directories, but I run into problems down the line, which leads me to think there must be a better way. Every example I see has all proto files in one folder and does not have cross-library protobuf message dependencies.

This should work, and with the project you attached it does work (well, once I fixed a bad field number):

Yamnuska:project ej$ protoc --cpp_out=build common/bar/bar.proto
Yamnuska:project ej$


I don't know how CMake or this PROTOBUF_GENERATE_CPP rule works, but maybe you need to pass the appropriate --proto_path argument so it looks for the included .proto in the right place?

Good luck,

Evan

--
http://evanjones.ca/

[Justin Muncaster]

Okay, so I got this build tree working. I had to:

(1) Modify FindProtobuf.cmake (the build rule) so that I can specify the proto_path

(2) Modify each .proto import to specify the file to import using the full path from the root, even if the files are parallel to the .proto source.

(3) Add CMAKE_SOURCE_DIR (the source tree root) to the proto_path

(4) Add CMAKE_BINARY_DIR (the build root) to the gcc_path.

(5) Do a clean build. I had old generated .pb.h files in the source tree, and they were being included rather then the newly generated files that lives in the build-tree.

Thanks for your help.

Justin

[Andrew Hundt]

Do you possibly have any code you could make available for this?