Transferring host's go-mod-download cache into docker container

[greg...@unity3d.com]

Hi,

I'm converting over a project from using dep to using mod and I think I have a pretty good handle on it all... but:

This particular project has a dependency that is hosted on an internal git server that requires credentials.

In the past it wasn't an issue because we'd just copy the vendor directory into the docker container and the build wouldn't need to fetch anything.

But now, I don't know how I can essentially copy the mod cache (or whatever the right term is) like I'd have copied the vendor directory.

Any tips on this sort of thing? It seems like a docker container builds would be fairly common and that I'm not the the first to run into this.

[Tamás Gulácsi]

Use the vendor directory, or use a replace directive with a relative path.

[greg...@unity3d.com]

On Friday, January 11, 2019 at 10:23:01 PM UTC-8, Tamás Gulácsi wrote:

Use the vendor directory, or use a replace directive with a relative path.

Manually adding replace directives for all dependent packages (probably 30-40 now, who knows how many after a few years) sounds like a fairly horrible maintenance cost, but maybe I'm missing a way to use them easily? Is there a page somewhere with a good writeup of these directives?

Re: Using the vendor dir approach, I did re-read the `go help modules` text, the last section particularly:

-----

Modules and vendoring

When using modules, the go command completely ignores vendor directories.

By default, the go command satisfies dependencies by downloading modules

from their sources and using those downloaded copies (after verification,

as described in the previous section). To allow interoperation with older

versions of Go, or to ensure that all files used for a build are stored

together in a single file tree, 'go mod vendor' creates a directory named

vendor in the root directory of the main module and stores there all the

packages from dependency modules that are needed to support builds and

tests of packages in the main module.

To build using the main module's top-level vendor directory to satisfy

dependencies (disabling use of the usual network sources and local

caches), use 'go build -mod=vendor'. Note that only the main module's

top-level vendor directory is used; vendor directories in other locations

are still ignored.

-----

So I guess we can use `go mod vendor` and then `go build -mod=vendor` but is this a long term solution? It seems like the community desire is to move away from vendor dirs. If vendor dir support is eventually completely removed, will replace-directives be the only way to have network-less builds? I guess if this is the case we should probably just use vendoring until it's removed, then pay the price to switch.

[Sam Whited]

On Fri, Jan 11, 2019, at 23:03, greg...@unity3d.com wrote:
> But now, I don't know how I can essentially copy the mod cache (or
> whatever the right term is) like I'd have copied the vendor directory.

If you're suggesting that you already have the module cached on the machine that builds your containers, the cache lives at ~/go/pkg/mod by default so you could probably copy that tree into your build container.

—Sam

[greg...@unity3d.com]

Thanks for help, folks. As far as copying the `go env GOPATH`/pkg/mod from the host to the docker container: I don't think I can copy from outside the "context", which is the project repository. It's not horribly important; this was more of an experiment for me to gain a bit more go.mod understanding.  For now, I'm just going back to vendoring 'till some of this shakes out a bit more.

[Michael Poindexter]

Just as a data point, at my work we're using the `go mod vendor` / `go build -mod=vendor` route for exactly the reasons you mention, and it works quite well.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Philip Nelson]

You can't copy from outside the context but you can volume mount it. In my docker-compose.yml I have the following for a golang:1.11-alpine image derivative:

```

    volumes:
      - ${GOPATH}:/go
```

Phil

I `RUN go mod download` before copying my source

[greg...@unity3d.com]

On Monday, January 14, 2019 at 4:52:37 PM UTC-8, Philip Nelson wrote:

You can't copy from outside the context but you can volume mount it. In my docker-compose.yml I have the following for a golang:1.11-alpine image derivative:

```

    volumes:
      - ${GOPATH}:/go
```

Phil

Oh! That's interesting. Thanks!

 

[Andrew Walker]

There are a couple ways I know of that you can do this. You can vendor or copy as you mention.  Another approach is to create a "build" image that contains relevant credentials and a few `go env -w` directives to set up your build environment, which you build separately and derive other projects from.

```

FROM golang:1.16-alpine3.13

RUN apk --no-cache add ca-certificates git openssh

# cert not checked in to repo, (download with artifact script to rebuild base images)

ADD certs/mycert.crt /usr/local/share/ca-certificates/mycert.crt

RUN chmod 0600 /usr/local/share/ca-certificates/mycert.crt

RUN update-ca-certificates

# if you host a proxy, use these two lines or similar (see https://golang.org/ref/mod#private-modules)

# RUN go env -w GOPROXY=https://proxy.corp.example.com

# RUN go env -w GONOSUMDB=corp.example.com

# otherwise, make sure you don't reach out to public proxy/sum for internal packages

RUN go env -w GOPRIVATE=vcs.package.prefix/*

# gitconfig for vcs to use "ssh://git@" instead of "https://"

ADD gitconfig /root/.gitconfig

# internal VCS creds (probably want to use a "deployment key")

ADD ssh /root/.ssh

RUN chmod 0600 /root/.ssh/id_rsa /root/.ssh/config

RUN chmod 0644 /root/.ssh/id_rsa.pub

```

And then the gitconfig you copy in might be something like:

```

[url "ssh://g...@my.internal.bitbucket:7999"]

    insteadOf = "https://my.internal.bitbucket/scm"

```

This should create a build image that will know about your VCS and download modules from it directly.  In your project, you can either do the standard copy of your mod and sum files and go download, but you can also use the new buildkit features to mount a shared cache for both modules and artifacts:

```

# syntax = docker/dockerfile:1-experimental

FROM myregistry:base as base

# ...

RUN --mount=type=cache,id=myproject_mod,target=/go/pkg/mod \

--mount=type=cache,id=myproject_cache,target=/root/.cache/go-build \

go build

```