go install seems to do a lot of rebuilding in a container

[Shaun Crampton]

Hi,

I'm trying to optimize the build of my app.  Our build times have crept up to ~60s as we've added more dependencies and it's starting to become a problem.

The primary problem seems to be that we're building in a container, so we're losing the $GOPATH/pkg cache of build artefacts between builds.  However, even if I bind the same directory to that location for each run, it doesn't seem to help.  The artefacts (.a files) get built and added to that directory but the following build seems to ignore them and rebuild them anyway.

If I do two builds in the same container then it does pick up the artefacts.  Likewise, if I build outside the container it seems to work as expected.

I feel like there's something I'm missing.  Please can you help me understand how "go install" determines the freshness of artefacts and why they might not be being used? I tried running with -x but I can't see any info about freshness (but then I'm not sure what I'm looking for in that output).

Thanks,

-Shaun

[Dave Cheney]

Docker add, or whatever is pushing files into your container is probably not respecting timestamps so the .a files are older with respect to .go files (pkg sorts above src)

Why do you want to build inside a container, is that just adding an extra level of indirection to what is a simple process?

[andrey mirtchovski]

it's also possible that you need to do "go build -i" just in case your
dependencies are outdated. i had the same issue today with a windows
piece of code that rebuilt sqlite3 every time.

On Wed, Dec 7, 2016 at 2:57 PM, Dave Cheney <da...@cheney.net> wrote:
> Docker add, or whatever is pushing files into your container is probably not respecting timestamps so the .a files are older with respect to .go files (pkg sorts above src)
>
> Why do you want to build inside a container, is that just adding an extra level of indirection to what is a simple process?
>

> --
> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Dave Cheney]

s/go build -i/go install.

[sh...@tigera.io]

> Docker add, or whatever is pushing files into your container is probably not respecting timestamps so the .a files are older with respect to .go files (pkg sorts above src)

I'm bind-mounting the pkg dir in there so it shouldn't be a question of timestamps (not ruling out something going on there).  Those are the "real" files.

> Why do you want to build inside a container, is that just adding an extra level of indirection to what is a simple process?

Using containers gives us a lot of control over the build without mandating a lot of set-up on each developer's machine (they need make and docker installed).  For example, using a container locks the build to the correct libc and version of go and it ensures the that project documents (in code) all the required dependencies and doesn't depend on local dev environment.  It also lets developers use Linux Mac or Windows and still be able to build the product in the right environment.  Beyond that, it lets us build all kinds of artefacts that previously would have required several dedicated (crufty) build machines: we can use a container to build debs and RPMs, on a Mac, for example.

[sh...@tigera.io]

I've started with a clean pkg dir.

It's odd; the mtimes look correct (same inside/outside container).  the "buildid" to the compile commands s the same on each run too.

Does anyone know if there's a way to print out the "StaleReason" that the code collects, that might help me find out what gate I'm failing.

[Ian Lance Taylor]

On Thu, Dec 8, 2016 at 10:13 AM, <sh...@tigera.io> wrote:
> I've started with a clean pkg dir.
>
> It's odd; the mtimes look correct (same inside/outside container). the
> "buildid" to the compile commands s the same on each run too.
>
> Does anyone know if there's a way to print out the "StaleReason" that the
> code collects, that might help me find out what gate I'm failing.

go list -f '{{.StaleReason}}'

Ian

[sh...@tigera.io]

Thanks, that's given me a great thread to pull on.  All the rebuilds track back to os/user being stale due to build ID mismatch.  Probably an issue with the specific configuration we've set up for our build being different to the pre-build os/user (I think we disable CGO, for one).  Should be able to figure it out from here.

-Shaun

[sh...@tigera.io]

Yes, that was the problem.  We were building with CGO_ENABLED=0, which was meaning that built-in packages like "os/user" were always stale and causing rebuilds of almost every other package.

Putting this in my dockerfile (along with mounting the pkg directory as a volume shared between each build) fixed the problem:

# Disable cgo so that binaries we build will be fully static.

ENV CGO_ENABLED=0

# Recompile the standard library with cgo disabled.  This prevents the standard library from being

# marked stale, causing full rebuilds every time.

RUN go install -v std

-Shaun