Building Go projects with Bazel

[Kevin Burke]

Hi,

For fun, and since the Bazel homepage advertises you can build Go projects with Bazel, I thought I'd try to switch a Go project of mine to use Bazel. In particular, here are things I hoped to be able to do with Bazel (from this Makefile):

- Build my code. This is pretty well supported by Bazel and rules_go/gazelle.

- Run "go test ./...". This is fairly well supported by Bazel.

- Run "go test -race ./..." in a CI environment. This is not currently supported (https://github.com/bazelbuild/rules_go/issues/167) and I was wondering if/when I would be able to use this.

- Run "go vet ./..." or "go fmt ./...". Currently this requires the use of an external script; I have a small project and I'd rather not clutter up the output by adding a shell script and calling into it from Bazel. https://github.com/bazelbuild/rules_go/issues/511, https://github.com/kubernetes/test-infra/blob/master/verify/verify-govet.sh.

- Run third party checkers like the functions in go-tools (https://github.com/dominikh/go-tools), https://github.com/mvdan/unparam, or https://github.com/kisielk/errcheck. Most (all) of these require the GOPATH and GOROOT to be set, but when you run sh_test (which I think you need to do to run these), you can't set environment variables, so I need to run a script and set them in the script, I think? and then also make the right other libraries available in the build step. This is a little more cumbersome than I'd hope, or expect people to set up to get Bazel widely adopted.

- Run "go-bindata" to copy files from directories on disk (static, templates etc) into assets/assets.go, so I can compile them into the binary and distribute just that. Here's how I currently do it: https://github.com/kevinburke/go-html-boilerplate/blob/master/Makefile#L46-L50. I haven't tried this yet though since I got discouraged by the other failures.

It would be nice if there were more examples of open source Go projects using Bazel? Then I could copy what they do. Do any exist?

One nice thing about Make is that you can define shortcuts for common tasks, e.g. "make test", "make assets", etc. Is there a way to do this with Bazel? It can be annoying each time you want to run it to type out e.g. bazel test --test_verbose_timeout_warnings :all //assets:all.

Finally, if I manage to figure all of this out, I'd be happy to write about it!

Thanks!

Kevin

[Damien Martin-Guillerez]

+some rules_go maintainers

I think that is a very valuable feedback, feel free to send a PR on https://github.com/bazelbuild/bazel-blog to write a blog post about it :)

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/b4bef223-1ab6-4f3a-9ac7-d3cd08c7ada3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Tom Payne]

Here are a few snippets from our Bazel/make setup for Go, in case it is useful. Make is used as a shortcut for some common Bazel commands. Our third party dependencies are in vendor/.

Makefile:

build:

        bazel build -- //... -//vendor/...

test:

        bazel test -- //... -//vendor/...

gazelle:

        bazel run //:gazelle

Bazel rule for gazelle:

load("@io_bazel_rules_go//go:def.bzl", "gazelle", "go_prefix")

go_prefix("github.com/ourorg/ourrepo")

gazelle(

    name = "gazelle",

    external = "vendored",

)

BUILD.bazel genrule using go-bindata:

load("@io_bazel_rules_go//go:def.bzl", "go_library")

filegroup(

    name = "assets",

    srcs = [

        "foo.gif",

        "bar.tmpl",

    ],

)

go_library(

    name = "go_default_library",

    srcs = ["assets.go"],

    visibility = ["//visibility:public"],

)

genrule(

    name = "gen-assets.go",

    srcs = [":assets"],

    outs = ["assets.go"],

    cmd = "./$(location //vendor/github.com/jteeuwen/go-bindata/go-bindata) -nocompress -nometadata -o \"$@\" -pkg assets -prefix path/to/assets $(locations :assets)",

    tools = ["//vendor/github.com/jteeuwen/go-bindata/go-bindata"],

)

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/b4bef223-1ab6-4f3a-9ac7-d3cd08c7ada3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discuss+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN3hOS_kbSKROZ2v%3Df-NovY9oj_a0yxU0Fyinf_13CzYgEUUaQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

This message is for the attention of the intended recipient(s) only. It may contain confidential, proprietary and/or legally privileged information. Use, disclosure and/or retransmission of information contained in this email may be prohibited. If you are not an intended recipient, you are kindly asked to notify the sender immediately (by reply e-mail) and to permanently delete this message. Thank you.

[Jay Conrod]

Kevin, thanks for checking out the Go rules and for letting us know what you had trouble with. We often get isolated issue reports, but it's good to hear about all of the issues that are affecting a project.

> - Run "go test -race ./..." in a CI environment. This is not currently supported (https://github.com/bazelbuild/rules_go/issues/167) and I was wondering if/when I would be able to use this.

We've been waiting for platform / toolchain support in Bazel to land before starting on this because we wanted to express -race as a toolchain. This is necessary for supporting race detection when cross compiling. -race really important for some large projects, so I think we're going to support race detection sooner on the host platform only. It's easier to do host only because the SDK has precompiled binaries. This is a high priority for us, and I expect we'll have something in a few weeks.

> - Run "go vet ./..." or "go fmt ./...". Currently this requires the use of an external script; I have a small project and I'd rather not clutter up the output by adding a shell script and calling into it from Bazel. https://github.com/bazelbuild/rules_go/issues/511, https://github.com/kubernetes/test-infra/blob/master/verify/verify-govet.sh.

We're discussing how to do this. It shouldn't be too complicated, but it's not as high of a priority as cross compilation, race detection, or managing external dependencies. I'm not sure whether go fmt should actually format files or just warn you that files are unformatted when run from Bazel. Executables run by Bazel run from the execroot, not from the current directory, so it may have some unexpected behavior.

> - Run third party checkers like the functions in go-tools (https://github.com/dominikh/go-tools), https://github.com/mvdan/unparam, or https://github.com/kisielk/errcheck. Most (all) of these require the GOPATH and GOROOT to be set, but when you run sh_test (which I think you need to do to run these), you can't set environment variables, so I need to run a script and set them in the script, I think? and then also make the right other libraries available in the build step. This is a little more cumbersome than I'd hope, or expect people to set up to get Bazel widely adopted.

This will be simpler when "go vet" and "go fmt" are working. For that, we'll need to build something that returns a list of source files from a rule and its transitive dependencies. It should be straightforward to pipe the same sources to another go_binary. We'll need to evaluate whether that's still too complicated.

> - Run "go-bindata" to copy files from directories on disk (static, templates etc) into assets/assets.go, so I can compile them into the binary and distribute just that. Here's how I currently do it: https://github.com/kevinburke/go-html-boilerplate/blob/master/Makefile#L46-L50. I haven't tried this yet though since I got discouraged by the other failures.

This sounds similar to #304, which is a proposal for a go_embed_data rule. That should avoid the need to add another dependency and use a genrule (as Tom showed below).

> It would be nice if there were more examples of open source Go projects using Bazel? Then I could copy what they do. Do any exist?

We have a few things in the examples directory, but it's woefully inadequate. We've been deferring writing more examples and documentation because things have been changing quickly. I expect we'll be able to freeze our API by the end of the quarter, and we'll write more closer to that time.

> One nice thing about Make is that you can define shortcuts for common tasks, e.g. "make test", "make assets", etc. Is there a way to do this with Bazel? It can be annoying each time you want to run it to type out e.g. bazel test --test_verbose_timeout_warnings :all //assets:all.

Bazel has a test_suite rule that is useful for grouping tests. filegroup can be used to group other files and targets. I don't think there's any way to test Bazel flags for specific targets, but you can set them in general in .bazelrc files.

> Finally, if I manage to figure all of this out, I'd be happy to write about it!

Looking forward to it. Let me know if I can answer questions. Thanks for the feedback, and thanks for filing issues.

[Ian Cottrell]

In rules_go/pull/640 I added an example of using go-bindata

The rules I wrote are fairly limited, you could either start from them, or use them directly and we can improve them to meet your needs. 

On Tue, Jul 18, 2017 at 10:41 PM, 'Damien Martin-Guillerez' via bazel-discuss <bazel-...@googlegroups.com> wrote:

+some rules_go maintainers

I think that is a very valuable feedback, feel free to send a PR on https://github.com/bazelbuild/bazel-blog to write a blog post about it :)

On Tue, Jul 18, 2017, 10:12 PM Kevin Burke <kbur...@gmail.com> wrote:

Hi,

For fun, and since the Bazel homepage advertises you can build Go projects with Bazel, I thought I'd try to switch a Go project of mine to use Bazel. In particular, here are things I hoped to be able to do with Bazel (from this Makefile):

- Build my code. This is pretty well supported by Bazel and rules_go/gazelle.

- Run "go test ./...". This is fairly well supported by Bazel.

- Run "go test -race ./..." in a CI environment. This is not currently supported (https://github.com/bazelbuild/rules_go/issues/167) and I was wondering if/when I would be able to use this.

- Run "go vet ./..." or "go fmt ./...". Currently this requires the use of an external script; I have a small project and I'd rather not clutter up the output by adding a shell script and calling into it from Bazel. https://github.com/bazelbuild/rules_go/issues/511, https://github.com/kubernetes/test-infra/blob/master/verify/verify-govet.sh.

- Run third party checkers like the functions in go-tools (https://github.com/dominikh/go-tools), https://github.com/mvdan/unparam, or https://github.com/kisielk/errcheck. Most (all) of these require the GOPATH and GOROOT to be set, but when you run sh_test (which I think you need to do to run these), you can't set environment variables, so I need to run a script and set them in the script, I think? and then also make the right other libraries available in the build step. This is a little more cumbersome than I'd hope, or expect people to set up to get Bazel widely adopted.

- Run "go-bindata" to copy files from directories on disk (static, templates etc) into assets/assets.go, so I can compile them into the binary and distribute just that. Here's how I currently do it: https://github.com/kevinburke/go-html-boilerplate/blob/master/Makefile#L46-L50. I haven't tried this yet though since I got discouraged by the other failures.

It would be nice if there were more examples of open source Go projects using Bazel? Then I could copy what they do. Do any exist?

One nice thing about Make is that you can define shortcuts for common tasks, e.g. "make test", "make assets", etc. Is there a way to do this with Bazel? It can be annoying each time you want to run it to type out e.g. bazel test --test_verbose_timeout_warnings :all //assets:all.

Finally, if I manage to figure all of this out, I'd be happy to write about it!

Thanks!

Kevin

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/b4bef223-1ab6-4f3a-9ac7-d3cd08c7ada3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "bazel-discuss" group.

To unsubscribe from this group and stop receiving emails from it, send an email to bazel-discus...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/bazel-discuss/CAN3hOS_kbSKROZ2v%3Df-NovY9oj_a0yxU0Fyinf_13CzYgEUUaQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Centralway Numbrs AG | Tom Payne, Software Engineer, Backend

[Kevin Burke]

Thanks! These are all really helpful; I've got services running plain tests in Travis CI now and I'll work on the other stuff in short time.

In Travis CI, Bazel seems set on posting progress messages even though it can't rewrite the terminal output. Is there a way to configure this? I tried searching for "bazel tty" and "bazel disable progress messages" and didn't see anything promising. For example there's lots of this in the output. https://travis-ci.org/kevinburke/rest/builds/256303499

INFO: Cloning https://github.com/bazelbuild/rules_go.git: Resolving deltas (29\/ 2233)
INFO: Cloning https://github.com/bazelbuild/rules_go.git: Resolving deltas (25\7 / 2233)
INFO: Cloning https://github.com/bazelbuild/rules_go.git: Resolving deltas (49\6 / 2233)
INFO: Cloning https://github.com/bazelbuild/rules_go.git: Resolving deltas (799 / 2233)
INFO: Cloning https://github.com/bazelbuild/rules_go.git: Resolving deltas (10\08 / 2233)
INFO: Cloning https://github.com/bazelbuild/rules_go.git: Resolving deltas (12\32 / 2233)
INFO: Cloning https://github.com/bazelbuild/rules_go.git: Resolving deltas (15\08 / 2233)
INFO: Cloning https://github.com/bazelbuild/rules_go.git: Resolving deltas (19\

Also another thing I noticed on the Samtrans this morning is that Bazel caches network failures. If I run "bazel build //..." while unconnected to the network, Bazel continues to tell me I am not connected to the network when I rerun the command, even after I'm on wifi again. That seems like a bad idea? Here's the error message I got:

$ bazel test //...
ERROR: error loading package '': Encountered error while reading extension file 'go/def.bzl': no such package '@io_bazel_rules_go//go': Error cloning repository: g...@github.com:bazelbuild/rules_go.git: unknown host caused by g...@github.com:bazelbuild/rules_go.git: unknown host.
INFO: Elapsed time: 0.121s
ERROR: Couldn't start the build. Unable to run tests.

[Kevin Burke]

Ah - I figured out the "progress messages", I wanted --noshow_progress --noshow_loading_progress, although I still think it would be good to enable those by default if the output is not a TTY.

[Jay Conrod]

Hey Kevin, the issue you mentioned about Bazel caching network failures is troubling. I was able to reproduce this only when using --experimental_repository_cache. Were you using that flag too?

Hiding progress messages to some degree would be worthwhile, though some of those messages come from Skylark rules or external tools, and Bazel won't have the context to decide what's relevant.

[Kevin Burke]

>  I was able to reproduce this only when using --experimental_repository_cache. Were you using that flag too?

I was not using that flag. To reproduce:

# obv replace with wherever your bazel cache is located

$ rm -rf /var/tmp/_bazel_kevin/*

$ go get github.com/kevinburke/rest

$ cd $GOPATH/github.com/kevinburke/rest

# just in case you have these in that repo

$ rm -rf bazel-*

# ** Disable wifi and any Ethernet **

$ bazel test --test_output=errors //...

ERROR: error loading package '': Encountered error while reading extension file 'go/def.bzl': no such package '@io_bazel_rules_go//go': Error cloning repository: g...@github.com:bazelbuild/rules_go.git: unknown host caused by g...@github.com:bazelbuild/rules_go.git: unknown host.

ERROR: error loading package '': Encountered error while reading extension file 'go/def.bzl': no such package '@io_bazel_rules_go//go': Error cloning repository: g...@github.com:bazelbuild/rules_go.git: unknown host caused by g...@github.com:bazelbuild/rules_go.git: unknown host.

INFO: Elapsed time: 0.098s

ERROR: Couldn't start the build. Unable to run tests.

make: *** [Makefile:12: test] Error 1

# I ran the above command twice

# Now re-enable wifi:

# $ bazel test --test_output=errors //...

ERROR: error loading package '': Encountered error while reading extension file 'go/def.bzl': no such package '@io_bazel_rules_go//go': Error cloning repository: g...@github.com:bazelbuild/rules_go.git: unknown host caused by g...@github.com:bazelbuild/rules_go.git: unknown host.

ERROR: error loading package '': Encountered error while reading extension file 'go/def.bzl': no such package '@io_bazel_rules_go//go': Error cloning repository: g...@github.com:bazelbuild/rules_go.git: unknown host caused by g...@github.com:bazelbuild/rules_go.git: unknown host.

INFO: Elapsed time: 0.097s

ERROR: Couldn't start the build. Unable to run tests.

make: *** [Makefile:12: test] Error 1

It's important that you don't edit any files in between test runs, it works if you change a file. I'm happy to write this up as an issue report somewhere if you tell me where to send it.

--
Kevin Burke
925.271.7005 | kev.inburke.com

[Jay Conrod]

Thanks for writing this up. I was able to reproduce this following your instructions.

I think this is a bug with the git_repository rule or something in the Bazel core caching mechanism. Would you consider filing this as an issue on github.com/bazelbuild/bazel?