Question about Go protobufs and import_prefix

[Zellyn]

Hi folks,

Apologies in advance for the complex email. It takes a bit of explaining to set up what we're having trouble with.

I would like to lay out how we generate protos in Go (using our unfortunately forked version of the protoc plugin), and ask for suggestions as to how to accomplish the same thing with the unforked version. In the process, I wish to ask questions about the post-vendor-experiment utility of the current behavior of import_prefix, hopefully generating more discussion than I managed to here.

Background:

• We have protos spread around all over the place: in our Go monorepo, in our Java monorepo, and in miscellaneous Ruby repos.
• We collect them all into a single directory before generating Go code. Call it $COLLECT_DIR
• We have to use go_package to reorganize things that are fine in Java/Ruby, but would be circular package imports in Go.
• For historical reasons, our Go repo is checked out to $GOPATH/src/square/up. I know it's a little weird, but it's logically equivalent to any other location. Pretend "square/up" is "square3" if you like… :-)
• Our monorepo's vendor folder lives at $GOPATH/src/square/up/vendor/...
• We want to generate protos into $GOPATH/src/square/up/protos to keep them all in one place.
• Most of our protos are in package "squareup.*", but we have some third-party protos that have their own packages. Or none (eg. nanopb).
• Right now we also have copies of WKT protos in $COLLECT_DIR, but I'd love to just import them from their vendor folder locations.

What we do now: full go_package including "protos" folder in protobufs

Add full package and path go_package to every protobuf:

eg. For a file with package "squareup.testing", we set option go_package = "square/up/squareup/protos/testing".

(We haven't actually added these declarations everywhere yet: this is what our forked plugin does: if go_package doesn't specify package path, we use square/up/protos + slash-separated-package. But it's about the same thing. Our fork was created long before go_package gained the ability to actually specify output package path)

Cons:

• We generate into the output directory of $GOPATH/src, which is terrible: we could accidentally generate stuff into any arbitrary go package in our GOPATH. :-(
  
• We're writing an implementation detail (where we currently store protos) into all the proto files

What I would like to do:

Generate into an output directory of $GOPATH/src/square/up/protos, and use the import_prefix option to ensure that things get imported from the right place.

Why that doesn't work:

• import_prefix is added to *every* import, including grpc packages, the context package, the proto runtime package, etc. This made sense when the prefix was intended to be something like "square/up/Godep/_workspace/" (eg. vendoring by import-path-rewriting), but doesn't make sense anymore.
  
• import_prefix is added to imports for ALL protos, including things like well-known-type protos that should just live under $GOPATH/src/square/up/vendor/github.com/golang/protobuf/proto

What I'd like to get out of this:

Either:

• a reworking of the way import_prefix works
• a new parameter with slightly different semantics: adds an import prefix only for protos, and only for protos in a particular list of folders:
• eg. proto_import_prefix_by_dir=$COLLECT_DIR:square/up/protos
• an explanation of how I'm Doing it Wrong - I'm happy to change things. While there's a lot of existing code, Go protos have caused me so much pain, I'm willing to invest pretty significant effort.

Why I'd like to defork

Our fork was created before go_package could rearrange output packages (commit). With the recent increase in grpc-related volatility of the protobuf package, it's increasingly painful to maintain our fork. And since grpc updates are often tied to proto updates, I can't simply avoid updating it. Plus all the usual reasons maintaining a fork is terrible :-)

Comments, suggestions, and help welcome.

Zellyn

cc dsymonds, matloob at Google, Josh at Square.

[Jie Luo]

+Ross who is working on Go protobuf.

Ross, do you have any comments or suggestions?

--
You received this message because you are subscribed to the Google Groups "Protocol Buffers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.
To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

[Zellyn Hunter]

Oh wow. Thanks for thinking about it so carefully!

I should mention: before you could actually specify full package paths to the Go proto plugin, earlier versions of our wrapper (https://github.com/square/goprotowrap) just forcibly manhandled things into the shape we wanted by explicitly specifying -M parameter import rewrites for every single import, and renaming the output files into place.

So it's definitely possible to shape things as you want. However, I would prefer to find a solution that works with the official plugin, because I don't believe our setup is remarkable or unusual: as GRPC sees an increase in use as a cross-language RPC mechanism, everyone else is going to struggle with the same things.

Zellyn

On Fri, Jul 1, 2016 at 7:32 PM Ross Light <li...@google.com> wrote:

Sorry for the delayed response!  I've written about 3-4 draft replies over this week that I've discarded because they don't meet your requirements.  I'll think about it more over the long weekend and get back to you.

Cheers,

-Ross

[Josh Haberman]

I don't know the background of the Go import system or go_package option. However this statement concerns me a little:

> We have to use go_package to reorganize things that are fine in Java/Ruby, but would be circular package imports in Go.

Is this implying that certain .proto files would generate invalid Go code until you manually insert some go_package statements? How tricky is it to do this manual untangling?

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+unsubscribe@googlegroups.com.

[Zellyn Hunter]

Usually not too tricky. The problem is that protos and most other programming languages prohibit circular imports at the file level, and Go does it at the package level. The fact that protoc and the other languages are okay with a topology ensures there is a valid Go repackaging that will work.

Zellyn

To unsubscribe from this group and stop receiving emails from it, send an email to protobuf+u...@googlegroups.com.

To post to this group, send email to prot...@googlegroups.com.
Visit this group at https://groups.google.com/group/protobuf.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Protocol Buffers" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/protobuf/W2zN-xKsdgk/unsubscribe.
To unsubscribe from this group and all its topics, send an email to protobuf+u...@googlegroups.com.

[Zellyn Hunter]

Yeah I can do anything with a wrapper. But to the extent that our concerns and structure are normal, it would be a shame to need a wrapper.

On Tue, Jul 12, 2016, 8:54 AM Ross Light <li...@google.com> wrote:

Zellyn and I talked a little bit at Gophercon about this. I think some kind of wrapper is in order for his use case: it may actually be a filter between protoc and protoc-gen-go.

On Tue, Jul 12, 2016, 8:49 AM Damien Neil <dn...@google.com> wrote:

If I'm following this correctly, your core problem is that protoc doesn't understand vendored paths as used by the go tool. For example:

- You have a proto file located in src/square/up/protos/square.proto.

- square.proto imports "github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".

- timestamp.proto is actually located in "src/square/up/vendor/github.com/.../timestamp.proto".

You can't run "protoc --go_out=. square/up/protos/square.proto", because protoc won't be able to locate the vendored copy of timestamp.proto. If it did, everything would work correctly without setting an import_prefix.

It would be nice for Go users if protoc did understand vendored paths. Failing that, however, I think you can get what you want with a very small wrapper around protoc. The wrapper can copy a proto and all its dependencies into a temp directory (pulling dependencies from vendored paths as necessary), run protoc there, and copy the result back. e.g., in the case of the above example:

$ mkdir -p /tmp/x/square/up/protos

$ cp src/square/up/protos/square.proto /tmp/x/square/up/protos

$ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp

$ cp square/up/vendor/github.com/golang/protobuf/ptypes/timestamp/timestamp.proto github.com/golang/protobuf/ptypes/timestamp

$ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto

$ cp /tmp/x/square/up/protos/square.pb.go square/up/protos

The generated .go files will reference non-vendored paths, which the Go compiler will resolve to the correct vendored directory.

Does that seem like it would work for you?

[Damien Neil]

If I'm following this correctly, your core problem is that protoc doesn't understand vendored paths as used by the go tool. For example:

- You have a proto file located in src/square/up/protos/square.proto.

- square.proto imports "github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".

- timestamp.proto is actually located in "src/square/up/vendor/github.com/.../timestamp.proto".

You can't run "protoc --go_out=. square/up/protos/square.proto", because protoc won't be able to locate the vendored copy of timestamp.proto. If it did, everything would work correctly without setting an import_prefix.

It would be nice for Go users if protoc did understand vendored paths. Failing that, however, I think you can get what you want with a very small wrapper around protoc. The wrapper can copy a proto and all its dependencies into a temp directory (pulling dependencies from vendored paths as necessary), run protoc there, and copy the result back. e.g., in the case of the above example:

$ mkdir -p /tmp/x/square/up/protos

$ cp src/square/up/protos/square.proto /tmp/x/square/up/protos

$ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp

$ cp square/up/vendor/github.com/golang/protobuf/ptypes/timestamp/timestamp.proto github.com/golang/protobuf/ptypes/timestamp

$ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto

$ cp /tmp/x/square/up/protos/square.pb.go square/up/protos

The generated .go files will reference non-vendored paths, which the Go compiler will resolve to the correct vendored directory.

Does that seem like it would work for you?

On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter <zel...@gmail.com> wrote:

[Damien Neil]

(Apologies if you get two copies of this mail; resending to the list because my first try bounced.)

If I'm following this correctly, your core problem is that protoc doesn't understand vendored paths as used by the go tool. For example:

- You have a proto file located in src/square/up/protos/square.proto.

- square.proto imports "github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".

- timestamp.proto is actually located in "src/square/up/vendor/github.com/.../timestamp.proto".

You can't run "protoc --go_out=. square/up/protos/square.proto", because protoc won't be able to locate the vendored copy of timestamp.proto. If it did, everything would work correctly without setting an import_prefix.

It would be nice for Go users if protoc did understand vendored paths. Failing that, however, I think you can get what you want with a very small wrapper around protoc. The wrapper can copy a proto and all its dependencies into a temp directory (pulling dependencies from vendored paths as necessary), run protoc there, and copy the result back. e.g., in the case of the above example:

$ mkdir -p /tmp/x/square/up/protos

$ cp src/square/up/protos/square.proto /tmp/x/square/up/protos

$ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp

$ cp square/up/vendor/github.com/golang/protobuf/ptypes/timestamp/timestamp.proto github.com/golang/protobuf/ptypes/timestamp

$ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto

$ cp /tmp/x/square/up/protos/square.pb.go square/up/protos

The generated .go files will reference non-vendored paths, which the Go compiler will resolve to the correct vendored directory.

Does that seem like it would work for you?

[Ross Light]

Zellyn and I talked a little bit at Gophercon about this. I think some kind of wrapper is in order for his use case: it may actually be a filter between protoc and protoc-gen-go.

On Tue, Jul 12, 2016, 8:49 AM Damien Neil <dn...@google.com> wrote:

If I'm following this correctly, your core problem is that protoc doesn't understand vendored paths as used by the go tool. For example:

- You have a proto file located in src/square/up/protos/square.proto.

- square.proto imports "github.com/golang/protobuf/ptypes/timestamp/timestamp.proto".

- timestamp.proto is actually located in "src/square/up/vendor/github.com/.../timestamp.proto".

You can't run "protoc --go_out=. square/up/protos/square.proto", because protoc won't be able to locate the vendored copy of timestamp.proto. If it did, everything would work correctly without setting an import_prefix.

It would be nice for Go users if protoc did understand vendored paths. Failing that, however, I think you can get what you want with a very small wrapper around protoc. The wrapper can copy a proto and all its dependencies into a temp directory (pulling dependencies from vendored paths as necessary), run protoc there, and copy the result back. e.g., in the case of the above example:

$ mkdir -p /tmp/x/square/up/protos

$ cp src/square/up/protos/square.proto /tmp/x/square/up/protos

$ mkdir -p /tmp/x/github.com/golang/protobuf/ptypes/timestamp

$ cp square/up/vendor/github.com/golang/protobuf/ptypes/timestamp/timestamp.proto github.com/golang/protobuf/ptypes/timestamp

$ cd /tmp/x && protoc -go_out=. square/up/protos/square.proto

$ cp /tmp/x/square/up/protos/square.pb.go square/up/protos

The generated .go files will reference non-vendored paths, which the Go compiler will resolve to the correct vendored directory.

Does that seem like it would work for you?

On Sun, Jul 10, 2016 at 12:32 PM, Zellyn Hunter <zel...@gmail.com> wrote:

[Damien Neil]

Simpler than maintaining a fork, though.

The more convenient alternative would be for protoc to understand Go vendor directories, but that would require putting some Go-specific logic in protoc itself.

[Zellyn]

Note that I'm not actually generating protobufs into a vendor folder; what I described has the same problems with or without vendoring.