rules_go: go_proto_repositories not loading dependencies?

[Tom Payne]

Hi,

I'm trying to build the Go gRPC Hello World example using Bazel and rules_go 0.5.0. The code is at:

  https://github.com/twpayne/go-greet

I've followed the instructions in go_proto_library.bzl to load and run go_proto_repositories() in WORKSPACE, and to build the proto library with go_proto_library.

However, when I build the project, Bazel is unable to find the package '@org_golang_google_grpc//':

$ bazel build //helloworld:helloworld_proto

ERROR: /Users/tom.payne/src/github.com/twpayne/go-greet/helloworld/BUILD.bazel:3:1: no such package '@org_golang_google_grpc//': Traceback (most recent call last):

File "/private/var/tmp/_bazel_tom.payne/ac39b00ec22625d0c7a5180966ef7ad0/external/io_bazel_rules_go/go/private/go_repository.bzl", line 45

ctx.execute([ctx.path(ctx.attr._fetch_repo),...", <9 more arguments>])

File "/private/var/tmp/_bazel_tom.payne/ac39b00ec22625d0c7a5180966ef7ad0/external/io_bazel_rules_go/go/private/go_repository.bzl", line 46, in ctx.execute

ctx.path(ctx.attr._fetch_repo)

Not a file: /private/var/tmp/_bazel_tom.payne/ac39b00ec22625d0c7a5180966ef7ad0/external/io_bazel_rules_go_repository_tools/bin/fetch_repo and referenced by '//helloworld:helloworld_proto'.

ERROR: Analysis of target '//helloworld:helloworld_proto' failed; build aborted.

INFO: Elapsed time: 0.126s

This is unexpected as it looks like go_proto_repositories() loads this package:

  https://github.com/bazelbuild/rules_go/blob/master/proto/go_proto_library.bzl#L346-L350

Have I missed a step somewhere?

Cheers,

Tom

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

This message is for the attention of the intended recipient(s) only. It may contain confidential, proprietary and/or legally privileged information. Use, disclosure and/or retransmission of information contained in this email may be prohibited. If you are not an intended recipient, you are kindly asked to notify the sender immediately (by reply e-mail) and to permanently delete this message. Thank you.

[Jay Conrod]

Hi Tom, I wasn't able to reproduce this. Your WORKSPACE and BUILD.bazel files look good to me.

From the error message, I think your project got into a bad state somehow when executing repository rules. fetch_repo is a tool from rules_go that we use to implement the "go_repository" repository rule. The fetch_repo binary is not built through the normal Bazel build process because we need it very early. So if there is a failure to build it, or if one of its sources or dependencies dependencies changes, it might not get rebuilt.

Couple things to try:

* bazel clean --expunge

* Try cloning the project into a new directory and rebuilding there. Might narrow down the problem.

If you have any earlier messages about fetch_repo not building, please post them.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAF08gXG2Xcsv%2B7ViWTt%2BU%3DBxbLZJJ2o-HmJAqJmKr1VVL5XXrg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[twp...@gmail.com]

Thanks for the fast response Jay :)

tl;dr I can reproduce you not reproducing my problem :P

I tried this on my home laptop and got the same error.

I ran `bazel clean --expunge`, re-ran `bazel build`, and got the same error.

I ran `rm -rf $GOPATH/pkg`, re-ran `bazel build`, and got the same error.

I saw that I had an existing `fetch_repo` binary in $GOBIN, so I updated it with `go get -u github.com/bazelbuild/rules_go/go/tools/fetch_repo`, then re-ran `bazel build`, and got the same error.

To isolate my system, I created a Docker image with Bazel and git installed using this Dockerfile:

----8< Dockerfile 8<----
FROM ubuntu:latest
RUN \
set -x && \
apt-get update && \
apt-get install -y curl && \
echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" >> /etc/apt/sources.list.d/bazel.list && \
curl https://bazel.build/bazel-release.pub.gpg | apt-key add - && \
apt-get update && \
apt-get install -y bazel git
----8< Dockerfile 8<----

And built this image with
$ docker build -t bazel-and-git .
(this step takes a while)

Within this image, I checked out and ran the code, and it worked:

$ docker run -it bazel-and-git /bin/bash
# mkdir -p ~/src/github.com/twpayne
# cd ~/src/github.com/twpayne
# git clone https://github.com/twpayne/go-greet.git
...
# cd go-greet
# bazel build //helloworld:helloworld_proto
Extracting Bazel installation...
.........
INFO: Found 1 target...
Target //helloworld:helloworld_proto up-to-date:
bazel-bin/helloworld/github.com/twpayne/go-greet/helloworld/helloworld_proto.a
INFO: Elapsed time: 94.890s, Critical Path: 28.61s

So, some state on my work and personal laptops is causing the problem. Note that both are macOS laptops, whereas the Docker container (and you, assuming you're running Goobuntu) are running Linux.

I have to run now, but hopefully this helps narrow down the problem.

Regards,
Tom

[Ian Cottrell]

fetch_repo is built in a very special way, it's not part of the build, it's part of the workspace loading, so it does not run in a sandbox or use dependencies in a normal way.

This means, unlike things built by the normal go rules, the build is not fully isolated from the machine it i is running on. We have tried to make it as close to that as possible, but probably there is a case we have missed where you have an unusual setup that we are picking up by accident.

I build on a mac laptop all the time, so it cant be (just) that.

The only bit of the output that actually matters is the line
Not a file: /private/var/tmp/_bazel_tom.payne/ac39b00ec22625d0c7a5180966ef7ad0/external/io_bazel_rules_go_repository_tools/bin/fetch_repo

The rest is just a consequence of that failure. If you could tell us what is in the io_bazel_rules_go_repository folder mentioned
  ls $(bazel info output_base)/external/io_bazel_rules_go_repository_tools
that would help, specifically if it has pkg and bin subdirectories and what is in those as well.

That folder should look and behave exactly like something that could be your GOPATH and build with go build, because that's essentially what we do to build fetch_repo right now (except we do it with the same toolchain we download to do the main build)

You could also compare that folder to the one in your succesful docker image to see what the differences are.

Both fetch_repo and gazelle are built the same way in that folder, a normal build with have them both sitting in that bin directory.

I just realized, I think this might be #536 (do you have GOBIN set?), in which case it's already fixed at tip, but you are using the last tagged build that does not have the fix in.

If it is that, could you either try the tip (8975f7a5222a8365d371f315d6f848dff076ecdd), unset GOBIN or wait a week for a new tag?

Ian

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/da8fa233-b768-4dc3-9999-0f0fe7e90cc9%40googlegroups.com.

[Tom Payne]

Thanks Ian, that's it.

I have $GOBIN set to its default value ($GOPATH/bin) on both my work and personal laptops (I have some automation to share my dotfiles across multiple machines).

From the current situation on my personal laptop:

$ bazel build //helloworld:helloworld_proto

... failure as above ...

$ unset GOBIN

$ bazel build //helloworld:helloworld_proto

... same reported failure ...

$ bazel clean --expunge

...

$ bazel build //helloworld:helloworld_proto

...

Target //helloworld:helloworld_proto up-to-date:

  bazel-bin/helloworld/github.com/twpayne/go-greet/helloworld/helloworld_proto.a

INFO: Elapsed time: 64.505s, Critical Path: 10.58s

i.e. SUCCESS :)

This revealed a couple of extra problems in my BUILD.bazel files that are fixed by https://github.com/twpayne/go-greet/commit/aa8ac9fa6c7ef9661773fe3593b3a1c5d5b73546.

For reference, the contents of $(bazel info output_base)/external/io_bazel_rules_go_repository_tools do look like something that could be in ones $GOPATH:

  $ ls -1 $(bazel info output_base)/external/io_bazel_rules_go_repository_tools

  BUILD

  WORKSPACE

  bin

  pkg

  src

  tools-3d92dd60033c312e3ae7cac319c792271cf67e37

Sorry for missing that this was the existing issue #536 - this was discoverable with sufficient searching.

Regards,

Tom

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

[Tom Payne]

Thanks to Jay and Ian for their fast resolution of this problem :)