Problem getting bazel go rules to build projects in external repositories

[David Santiago]

I've been trying to get a project working with go rules, and also using an external repository that needs to build go code. I extracted a project that shows this issue here: https://github.com/davidsantiago/bazel_go_rule_problem

Because the project itself has a go_prefix set, the go protobuf project can't build the protoc-gen-go binary. That binary imports packages under "github.com/golang/protobuf", which then can't be found when you try to compile it, because it considers the code to be under the parent project's go_prefix:

$ bazel build @go-protobuf//:protoc-gen-go --verbose_failures

INFO: Found 1 target...

INFO: From GoCompile external/go-protobuf/protoc-gen-go.a:

github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/main.go:49: package main; expected documentation

github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/main.go:55: can't find import: "github.com/golang/protobuf/proto"

ERROR: /private/var/tmp/_bazel_David/965f3b0a69b74add319e09e7417880fb/external/go-protobuf/BUILD:25:1: error executing shell command: 'rm -rf bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/...' failed: bash failed: error executing command

  (cd /private/var/tmp/_bazel_David/965f3b0a69b74add319e09e7417880fb/bazel_go_rule_problem && \

  exec env - \

    GOARCH=amd64 \

    GOOS=darwin \

  /bin/bash -c 'rm -rf bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf && ln -s ../../../../../../../../../../../bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/proto.a bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/proto.a && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go && ln -s ../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/doc.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/doc.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/generator && ln -s ../../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/generator/generator.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/generator/generator.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/internal/grpc && ln -s ../../../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go && ln -s ../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/main.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/main.go && export GOROOT=$(pwd)/external/golang-darwin-amd64/go/bin/.. && cd  bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && ../../../../../../external/golang-darwin-amd64/go/bin/go tool compile -o ../../../../../../bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a -pack -I . github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/doc.go github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/main.go github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/generator/generator.go github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1: bash failed: error executing command

  (cd /private/var/tmp/_bazel_David/965f3b0a69b74add319e09e7417880fb/bazel_go_rule_problem && \

  exec env - \

    GOARCH=amd64 \

    GOOS=darwin \

  /bin/bash -c 'rm -rf bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf && ln -s ../../../../../../../../../../../bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/proto.a bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/proto.a && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go && ln -s ../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/doc.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/doc.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/generator && ln -s ../../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/generator/generator.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/generator/generator.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/internal/grpc && ln -s ../../../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go && ln -s ../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/main.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/main.go && export GOROOT=$(pwd)/external/golang-darwin-amd64/go/bin/.. && cd  bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && ../../../../../../external/golang-darwin-amd64/go/bin/go tool compile -o ../../../../../../bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a -pack -I . github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/doc.go github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/main.go github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/generator/generator.go github.com/davidsantiago/bazel_go_rule_problem/external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.

Target @go-protobuf//:protoc-gen-go failed to build

INFO: Elapsed time: 0.405s, Critical Path: 0.22s

I tried to work around this, by duplicating the go rules related to go_prefix, and creating myself a local_go_prefix rule, which can then be passed explicitly to the go_prefix argument on the go_binary() rule for the external project. That succeeds in changing the go_prefix for the external repository, but it still can't build: 

$ bazel build @go-protobuf//:protoc-gen-go --verbose_failures

INFO: Found 1 target...

INFO: From GoCompile external/go-protobuf/protoc-gen-go.a:

github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/main.go:49: package main; expected documentation

github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/main.go:55: can't find import: "github.com/golang/protobuf/proto"

ERROR: /private/var/tmp/_bazel_David/965f3b0a69b74add319e09e7417880fb/external/go-protobuf/BUILD:25:1: error executing shell command: 'rm -rf bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/...' failed: bash failed: error executing command

  (cd /private/var/tmp/_bazel_David/965f3b0a69b74add319e09e7417880fb/bazel_go_rule_problem && \

  exec env - \

    GOARCH=amd64 \

    GOOS=darwin \

  /bin/bash -c 'rm -rf bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf && ln -s ../../../../../../../../../../../bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/proto.a bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/proto.a && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go && ln -s ../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/doc.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/doc.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/generator && ln -s ../../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/generator/generator.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/generator/generator.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/internal/grpc && ln -s ../../../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go && ln -s ../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/main.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/main.go && export GOROOT=$(pwd)/external/golang-darwin-amd64/go/bin/.. && cd  bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && ../../../../../../external/golang-darwin-amd64/go/bin/go tool compile -o ../../../../../../bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a -pack -I . github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/doc.go github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/main.go github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/generator/generator.go github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1: bash failed: error executing command

  (cd /private/var/tmp/_bazel_David/965f3b0a69b74add319e09e7417880fb/bazel_go_rule_problem && \

  exec env - \

    GOARCH=amd64 \

    GOOS=darwin \

  /bin/bash -c 'rm -rf bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf && ln -s ../../../../../../../../../../../bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/proto.a bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/proto.a && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go && ln -s ../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/doc.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/doc.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/generator && ln -s ../../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/generator/generator.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/generator/generator.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/internal/grpc && ln -s ../../../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go && mkdir -p bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go && ln -s ../../../../../../../../../../../../external/go-protobuf/protoc-gen-go/main.go bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir/github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/main.go && export GOROOT=$(pwd)/external/golang-darwin-amd64/go/bin/.. && cd  bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a.dir && ../../../../../../external/golang-darwin-amd64/go/bin/go tool compile -o ../../../../../../bazel-out/local_darwin-fastbuild/bin/external/go-protobuf/protoc-gen-go.a -pack -I . github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/doc.go github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/main.go github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/generator/generator.go github.com/golang/protobuf/external/go-protobuf/protoc-gen-go/internal/grpc/grpc.go'): com.google.devtools.build.lib.shell.BadExitStatusException: Process exited with status 1.

Target @go-protobuf//:protoc-gen-go failed to build

INFO: Elapsed time: 0.249s, Critical Path: 0.10s

This has changed the go prefix to the one that the external repository expects, but it still can't find the proper files, because the go rules will look for them under the local repository structure under the go prefix (the /external/go-protobuf part of the path after the prefix). Of course, it's possible I am using the rules wrong. Quite possible! And probably easiest for me to fix and get on with. But if so, I haven't been able to figure it out, so I've been looking at fixes. Also possible that this use case isn't intended to ever be supported. In my understanding, Google wouldn't ever run into this issue with how they structure their code. I hope that's not the case, because it would really limit the usefulness of bazel for go projects, and a solution doesn't seem very difficult.

It looks to me like there are two problems that both need to be fixed. The first is the way go_prefix is globally unique. That seems problematic with code that imports full paths, as is the case here, because it won't find its code in the directory structure it is expecting from regular go usage. The second problem is that just setting the go prefix isn't enough, because even if you can get the go prefix correct, it still gets prepended to the bazel directory structure for the external repository, which won't be correct from the code's perspective. 

My inclination to fix this would be to make some changes to how go_prefix works. First, I'd change its behavior so it is no longer globally unique. I think probably "the bazel way" to do this is to give the go_prefix() function a name= argument, that lets you name the go_prefix object and refer to it later by name. Not passing in the name= argument could maybe set the "default" go_prefix to use when there isn't one explicitly specified by a rule. Otherwise, the go_* rules can take a go_prefix argument (as they do now) that specifies a go prefix to use during the build of that target. 

Then I'd also change the go_prefix() function to take a second argument as well, which allows you to "delete" a part of the path. It could be something like overlay="deleted/path/". That would then use the -importmap argument to the go tool commands to "remove" part of the path from the point of view of the compiler when it is looking for imports. That would solve the "second problem" mentioned above. 

Any of this sound reasonable? Have I totally confused myself or missed something? I'm happy to work on it, but I'm not sure if any of this is either desired, or my solutions would be the desired fixes. 

[Han-Wen Nienhuys]

the Go rules were made to interoperate with the "go" tool, which
expects that code which is imported is placed under

vendor/github.com/golang/protobuf/

with BUILD files that have references that are fully qualified within
the Bazel repository.

In such a scenario, it is not Bazel who does the downloading, but
rather the project owner, and on downloading and committing the code
to the repository, they would have to update the BUILD files. Since
BUILD files (especially the ones for Go) can be created/updated
automatically, this is feasible. Unfortunately, I haven't gotten round
to exporting the tooling for that. I could try to prioritize it
though.

The upside is that it is completely compatible with the Go ecosystem,
and the downside is that it does mesh well with Bazel's download
functionality.

If you want to fix this up from the Bazel side, you would have to
revamp go_library() and friends, such that they become macros and a
label

//src/protoc:main

would be translated to

//external/golang-protoc/src/protoc:main

so Bazel accepts the BUILD files. Then, the rule implementation would
translate the //external/golang-protoc/src/protoc label to an
importmap entry "github.com/golang/protobuf/src/protoc".

The problem is where to store the 2 mappings, in such a way that they
can be accessed in the right places, ie.

1) in BUILD files under external/golang-protoc , //src/... labels in
macro arguments should be interpreted as
//external/golang-protoc/src/...

2) references to //external/golang-protoc/src should import as
"github.com/golang/protobuf/src"

for 1), you could move the original extension to somewhere else, have
the macros and translation mapping in /tools/build_rules/go/def.bzl

for 2), you could extend go_prefix to take a dict argument, which
would hold the mapping for the entire workspace.

The mappings in 1) and 2) have to match what WORKSPACE tries to
download, so this solution is pretty fragile, and is certainly not
straightforward for new users.

> --
> You received this message because you are subscribed to the Google Groups
> "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bazel-discus...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/bazel-discuss/78dd3d92-c4e0-4e34-b766-0b9dfc72ca22%40googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.



--
Han-Wen Nienhuys
Google Munich
han...@google.com

[Han-Wen Nienhuys]

On Wed, Dec 9, 2015 at 11:30 AM, Han-Wen Nienhuys <han...@google.com> wrote:
> The upside is that it is completely compatible with the Go ecosystem,
> and the downside is that it does

(insert "not" here)

> mesh well with Bazel's download
> functionality.

[David Santiago]

How does the /vendor directory come into play here? I'm not trying to depend on any of the code in golang/protobuf, I just need to build the protoc-gen-go binary so I can generate go protobuf definitions. Well, eventually I'll need some of the code in that repo, of course, but that's not related to just building this binary. And if I were to manually download this code into my repo, wouldn't I still need to rewrite the imports in it to work with my repo layout? 

In any case, I may have misunderstood bazel's go support. I was hoping for a system quite a bit different from the default go toolchain, and that does not seem to be the goal.  

[Han-Wen Nienhuys]

On Wed, Dec 9, 2015 at 11:52 AM, David Santiago
<david.s...@gmail.com> wrote:
> How does the /vendor directory come into play here? I'm not trying to depend
> on any of the code in golang/protobuf, I just need to build the
> protoc-gen-go binary so I can generate go protobuf definitions.

but the protoc-gen-go binary does depend on golang/protobuf, right?

> Well,
> eventually I'll need some of the code in that repo, of course, but that's
> not related to just building this binary. And if I were to manually download
> this code into my repo, wouldn't I still need to rewrite the imports in it
> to work with my repo layout?

the rules have support for the magical "vendor/" directory component,
so the source files don't need rewriting.

> In any case, I may have misunderstood bazel's go support. I was hoping for a
> system quite a bit different from the default go toolchain, and that does
> not seem to be the goal.
>

> --
> You received this message because you are subscribed to the Google Groups
> "bazel-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to bazel-discus...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/bazel-discuss/0affdbca-1a37-447f-9c8d-bdbea6ff3ca6%40googlegroups.com.

>
> For more options, visit https://groups.google.com/d/optout.

[Han-Wen Nienhuys]

I discussed this for a bit with Lukacs.

We came up with the following solution:

For 1), we don't really need to do something new, as labels for
external repos are already of the @protobuf:// rather than
//external/protobuf, so the mapping is implicitly given.

However, I don't think the rules currently do anything with the @repo
syntax, so some tweaking may be necessary.

For 2), according to Lukacs, the default (which is //:go_prefix) would
work within external repos, since the default inside @protobuf would
be interpreted as @protobuf://:go_prefix, which would point to
"github.com/protobuf".

However, currently the prefix is passed on between targets as a simple
string. For the rules to work with a mixture of repositories, it would
need to be a map, so it can setup the right filesystem tree entries
depending on the repository from which the dependency library came
from.

On Wed, Dec 9, 2015 at 11:30 AM, Han-Wen Nienhuys <han...@google.com> wrote:

[David Santiago]

Cool. I want to work on this, but I'm having a bit of trouble understanding what the solution you came up with is. 

Your last message appears paired with the numbers in your first reply. So as I understand what you're saying, "you don't need to do something new" to get //src/... labels interpreted as //external/protobuf/src/... labels in macros. But I don't understand how this inaction connects with the problem statement of getting those labels translated. Something about the @repo labels needing to be parsed and interpreted? 

For 2), that is about importing //external/protobuf/... labels as "github.com/golang/protobuf/...". In the most recent reply, you say "the default (which is //:go_prefix) would work within external repos...". I'm not sure what you mean by this. 

I do understand the parts about modifying the rules to parse @repo labels, and needing to modify the go_prefix rule to pass around a map of translations. I'm just not sure how you want those modifications to be made, from the statements I highlighted above.

Sorry if I'm seeming dense, but I don't want to waste your time on review. Seems better to sort it out now. 

[Han-Wen Nienhuys]

On Fri, Dec 11, 2015 at 3:50 AM, David Santiago
<david.s...@gmail.com> wrote:
> Cool. I want to work on this, but I'm having a bit of trouble understanding
> what the solution you came up with is.

I guess I misunderstood your problem. I thought you had a repository
which already has BUILD files embedded, but in your example, you
overlay one BUILD file over the repository instead. Let me see what
can be done.

> https://groups.google.com/d/msgid/bazel-discuss/6adedba5-6996-4656-9161-2a1b339cf8e6%40googlegroups.com.

[Han-Wen Nienhuys]

So, it appears that references within an external repo don't work as
we discussed.

https://github.com/bazelbuild/bazel/issues/700

Lukacs, I don't know how to go from here.

[David Santiago]

So now that issue 700 had a fix applied for it, I got back around to this today.  I'm still having a problem getting a build done. I'm not sure if this issue is related, but it seems like it might be, since it appears to involve an external repo. It's still just a stock usage of the go rules, just trying to use them in an external repo. I haven't gotten around to updating the go rules, because as far as I can tell, bazel doesn't even get as far as calling into them now.

I've updated the repo here: https://github.com/davidsantiago/bazel_go_rule_problem. When I try to build the protocol buffer file, it gets snagged:

$ bazel build proto/sadface-protos

ERROR: /Users/David/Documents/Development/bazel/base_workspace/tools/build_rules/go/toolchain/BUILD:15:1: no such package '@golang-linux-amd64//': BUILD file not found on package path and referenced by '//tools/build_rules/go/toolchain:toolchain'.

ERROR: Loading failed; build aborted.

INFO: Elapsed time: 0.075s

The problem is getting flagged here: https://github.com/bazelbuild/bazel/blob/master/tools/build_rules/go/toolchain/BUILD#L15

I'm not sure if I'm misunderstanding something, or maybe the error message is throwing me, but it seems like this should be a working statement. This is the file from the go rules in bazel, after all. And I never had a problem with the @golang-linux-amd64 repo being used by the go rules before. As I understand bazel's complaint, it's saying that there's no BUILD file in the @golang-linux-amd64 repo, even though that repo is made by the new_http_archive() in WORKSPACE, and that call provides a BUILD file as an argument. That definitely used to work, at least without using external repos. 

One last thing: sometimes the run complains about line 24 instead of line 15 of that file: 

$ bazel build proto/sadface-protos

ERROR: /Users/David/Documents/Development/bazel/base_workspace/tools/build_rules/go/toolchain/BUILD:15:1: no such package '@golang-linux-amd64//': BUILD file not found on package path and referenced by '//tools/build_rules/go/toolchain:toolchain'.

ERROR: Loading failed; build aborted.

INFO: Elapsed time: 0.072s

Thu Dec 17 02:20 AM David@David-Santiagos-Computer:~/Documents/Development/bazel_go_rule_problem

$ bazel build proto/sadface-protos

ERROR: /Users/David/Documents/Development/bazel/base_workspace/tools/build_rules/go/toolchain/BUILD:24:1: no such package '@golang-linux-amd64//': BUILD file not found on package path and referenced by '//tools/build_rules/go/toolchain:go_tool'.

ERROR: Loading failed; build aborted.

INFO: Elapsed time: 0.076s

Thu Dec 17 02:20 AM David@David-Santiagos-Computer:~/Documents/Development/bazel_go_rule_problem

$ bazel build proto/sadface-protos

ERROR: /Users/David/Documents/Development/bazel/base_workspace/tools/build_rules/go/toolchain/BUILD:24:1: no such package '@golang-linux-amd64//': BUILD file not found on package path and referenced by '//tools/build_rules/go/toolchain:go_tool'.

ERROR: Loading failed; build aborted.

INFO: Elapsed time: 0.073s

Thu Dec 17 02:20 AM David@David-Santiagos-Computer:~/Documents/Development/bazel_go_rule_problem

$ bazel build proto/sadface-protos

ERROR: /Users/David/Documents/Development/bazel/base_workspace/tools/build_rules/go/toolchain/BUILD:15:1: no such package '@golang-linux-amd64//': BUILD file not found on package path and referenced by '//tools/build_rules/go/toolchain:toolchain'.

ERROR: Loading failed; build aborted.

INFO: Elapsed time: 0.071s

Seems to be random, as far as I can tell. It seems like probably just bazel making different traversals through a graph, based on where things happened to hash on that run, or something like that. But I just wanted to mention it in case you get the other output, if you try it. 

[Han-Wen Nienhuys]

I cloned your repo, copied
https://bazel-review.googlesource.com/#/c/2561/1 into it, and then got

$ bazel build -s proto/...
INFO: Reading 'startup' options from
/usr/local/google/home/hanwen/.bazelrc: --watchfs
INFO: Found 2 targets...
>>>>> # //proto:sadface-protos_genproto [action 'Unknown proto/sad.pb.go']
(cd /usr/local/google/home/hanwen/.cache/bazel/_bazel_hanwen/eee7c9e33b1ae0cdb010c2bdce61620f/bazel_go_rule_problem
&& \
exec env - \
bazel-out/host/bin/external/protobuf/protoc
'--go_out=bazel-out/local_linux-fastbuild/genfiles/' -I.
proto/sad.proto)
INFO: From Unknown proto/sad.pb.go:
protoc-gen-go: program not found or is not executable
--go_out: protoc-gen-go: Plugin failed with status code 1.
ERROR: /usr/local/google/home/hanwen/vc/bazel_go_rule_problem/proto/BUILD:3:1:
null failed: protoc failed: error executing command
bazel-out/host/bin/external/protobuf/protoc
'--go_out=bazel-out/local_linux-fastbuild/genfiles/' -I.
proto/sad.proto.
INFO: Elapsed time: 0.161s, Critical Path: 0.07s


this looks like a problem with your proto rule; is protoc-gen-go
declared as input?

On Thu, Dec 17, 2015 at 11:25 AM, David Santiago

> https://groups.google.com/d/msgid/bazel-discuss/d374ccab-0487-4f2b-928c-e551a05ecf10%40googlegroups.com.

[David Santiago]

Right, the build definitely won't build successfully as the proto rule is written in that repo. What I was working on at the time was updating the Go rules, and had in fact already made the changes to def.bzl that you made in this CL. I tried to isolate what I was seeing down to a replicable case, because as far as I could tell, the Go rules weren't ever getting reached because it was complaining about the BUILD file and the external repo for Go. 

I've spent some time trying to replicate your output and have been unable to. I've applied your changes to def.bzl (the other two files are a README and shell tests, so don't appear related), and I've tried it with your def.bzl in the base_workspace under bazel, and also copied it into the tools/build_rules/go directory of the bazel_go_rule_problem repo. All I ever get is that same complaint (well, one of the two). When I type your command line in, I get:

$ bazel build -s proto/...

ERROR: /Users/David/Documents/Development/bazel/base_workspace/tools/build_rules/go/toolchain/BUILD:24:1: no such package '@golang-darwin-amd64//': BUILD file not found on package path and referenced by '//tools/build_rules/go/toolchain:go_tool'.

ERROR: Loading failed; build aborted.

INFO: Elapsed time: 0.138s

I've tried bazel clean, bazel shutdown, and redoing the repo. I must be doing something really dumb, because it works for you but for me it won't even load. 

[David Santiago]

OK, I managed to replicate your output. As I suggested in the last message, I was indeed doing something really dumb. Embarrassing as this is to explain, I was failing horribly at using cp -R to copy the tools/build_rules/go dir into the repo and it was silently going somewhere else, while files that I had put there by hand appeared to be what I had just "copied" because I wasn't checking the directory tree closely enough. The trailing "/" matters, dude. That accounts for the issue I was having, sorry about the false alarm.