Is it safe to run go build in parallel?

[Yulrizka]

Hi,

Recently, I played around with our Jenkins pipeline. In the new set-up, I build multiple packages in parallel. This step executes "go build" for each of the services that we have at the same time.

One thing I also change is that for each service, it uses the same GOPATH.

Sometimes, I see failures like this

src/go/pkg/linux_amd64_netgo/project/vendor/github.com/prometheus/common/internal/bitbucket.org/ww/goautoneg.a" already exists and is not an object file

go build project/vendor/github.com/prometheus/common/model: build output "/var/lib/jenkins/workspace/project/src/go/pkg/linux_amd64_netgo/project/vendor/github.com/prometheus/common/model.a" already exists and is not an object file

go build project/vendor/github.com/struCoder/pidusage: build output "/var/lib/jenkins/workspace/project/src/go/pkg/linux_amd64_netgo/project/vendor/github.com/struCoder/pidusage.a" already exists and is not an object file

go build project/thrift/gen/denormalized: build output "/var/lib/jenkins/workspace/project/src/go/pkg/linux_amd64_netgo/project/thrift/gen/denormalized.a" already exists and is not an object file

My current guess is that this is because running multiple go build at the same time, but I'm not sure if this is actually the case.

If it is the case, what is the recommended way to build multiple packages when you have multiple Jenkins step/job that build the binary at the same time?

The reason that I'm using shared GOPATH for this Jenkins step/job is that to now download the packages in `pkg/mod`.

Cheers!

[ma...@influxdata.com]

https://github.com/golang/go/issues/26794 is "can't run go builds concurrently if they download modules".

In that issue, Russ says:

> There is a plan to make downloading of modules by parallel go commands safe but we haven't done that yet.

[Yulrizka]

Hi Mark, 

what I find interesting is that I haven't enable go module yet. 

As you can see in the errors that I'm still using a vendor directory

[Mark Rushakoff]

I saw that you wrote

> The reason that I'm using shared GOPATH for this Jenkins step/job is that to now download the packages in `pkg/mod`.

And I thought that meant you were using modules.

Discussion in https://github.com/golang/go/issues/26677 says that the build cache is explicitly safe, but if you have a reproducer that shows otherwise, that's probably worthy of its own issue.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/Pj7BVrrBSuw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Yulrizka]

It seems that after searching in google, I found couple of similar issue:

most of them are related to parallel build with "GNU parallel" which is in our case as well.

https://github.com/gravitational/teleport/issues/1708

https://github.com/grpc/grpc-go/issues/368

https://github.com/vmware/vic/issues/5230

This is even before GO 1.11 or go mod.

And one thing, I noticed that this is not about GOCACHE, looks like package installed binaries example:

/var/lib/jenkins/workspace/project/src/go/pkg/linux_amd64_netgo/project/thrift/gen/denormalized.a

while in our jenkins, `go env` says: 

...
GOCACHE="/var/lib/jenkins/.cache/go-build"

...

[Yulrizka]

Ok, to give some update apparently our Makefile uses `build -i` where it tries to install package dependency. 

This explains the conflicted `.a` files.

It was there before GOCACHE to speed up `go build`

Thanks for the help!