Transferring host's go-mod-download cache into docker container

748 views
Skip to first unread message

greg...@unity3d.com

unread,
Jan 11, 2019, 6:03:39 PM1/11/19
to golang-nuts
Hi,

I'm converting over a project from using dep to using mod and I think I have a pretty good handle on it all... but:

This particular project has a dependency that is hosted on an internal git server that requires credentials.

In the past it wasn't an issue because we'd just copy the vendor directory into the docker container and the build wouldn't need to fetch anything.

But now, I don't know how I can essentially copy the mod cache (or whatever the right term is) like I'd have copied the vendor directory.

Any tips on this sort of thing? It seems like a docker container builds would be fairly common and that I'm not the the first to run into this.

Tamás Gulácsi

unread,
Jan 12, 2019, 1:23:01 AM1/12/19
to golang-nuts
Use the vendor directory, or use a replace directive with a relative path.

greg...@unity3d.com

unread,
Jan 12, 2019, 8:39:33 PM1/12/19
to golang-nuts
On Friday, January 11, 2019 at 10:23:01 PM UTC-8, Tamás Gulácsi wrote:
Use the vendor directory, or use a replace directive with a relative path.

Manually adding replace directives for all dependent packages (probably 30-40 now, who knows how many after a few years) sounds like a fairly horrible maintenance cost, but maybe I'm missing a way to use them easily? Is there a page somewhere with a good writeup of these directives?

Re: Using the vendor dir approach, I did re-read the `go help modules` text, the last section particularly:

-----

Modules and vendoring

When using modules, the go command completely ignores vendor directories.

By default, the go command satisfies dependencies by downloading modules
from their sources and using those downloaded copies (after verification,
as described in the previous section). To allow interoperation with older
versions of Go, or to ensure that all files used for a build are stored
together in a single file tree, 'go mod vendor' creates a directory named
vendor in the root directory of the main module and stores there all the
packages from dependency modules that are needed to support builds and
tests of packages in the main module.

To build using the main module's top-level vendor directory to satisfy
dependencies (disabling use of the usual network sources and local
caches), use 'go build -mod=vendor'. Note that only the main module's
top-level vendor directory is used; vendor directories in other locations
are still ignored.

-----

So I guess we can use `go mod vendor` and then `go build -mod=vendor` but is this a long term solution? It seems like the community desire is to move away from vendor dirs. If vendor dir support is eventually completely removed, will replace-directives be the only way to have network-less builds? I guess if this is the case we should probably just use vendoring until it's removed, then pay the price to switch.

Sam Whited

unread,
Jan 12, 2019, 8:57:19 PM1/12/19
to 'Than McIntosh' via golang-nuts
On Fri, Jan 11, 2019, at 23:03, greg...@unity3d.com wrote:
> But now, I don't know how I can essentially copy the mod cache (or
> whatever the right term is) like I'd have copied the vendor directory.

If you're suggesting that you already have the module cached on the machine that builds your containers, the cache lives at ~/go/pkg/mod by default so you could probably copy that tree into your build container.

—Sam

greg...@unity3d.com

unread,
Jan 14, 2019, 11:49:05 AM1/14/19
to golang-nuts
Thanks for help, folks. As far as copying the `go env GOPATH`/pkg/mod from the host to the docker container: I don't think I can copy from outside the "context", which is the project repository. It's not horribly important; this was more of an experiment for me to gain a bit more go.mod understanding.  For now, I'm just going back to vendoring 'till some of this shakes out a bit more.

Michael Poindexter

unread,
Jan 14, 2019, 12:49:36 PM1/14/19
to golang-nuts
Just as a data point, at my work we're using the `go mod vendor` / `go build -mod=vendor` route for exactly the reasons you mention, and it works quite well.

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Philip Nelson

unread,
Jan 14, 2019, 7:52:37 PM1/14/19
to golang-nuts
You can't copy from outside the context but you can volume mount it. In my docker-compose.yml I have the following for a golang:1.11-alpine image derivative:

```
    volumes:
      - ${GOPATH}:/go
```

Phil

I `RUN go mod download` before copying my source

greg...@unity3d.com

unread,
Jan 15, 2019, 11:01:02 AM1/15/19
to golang-nuts
On Monday, January 14, 2019 at 4:52:37 PM UTC-8, Philip Nelson wrote:
You can't copy from outside the context but you can volume mount it. In my docker-compose.yml I have the following for a golang:1.11-alpine image derivative:

```
    volumes:
      - ${GOPATH}:/go
```

Phil


Oh! That's interesting. Thanks!
 

Andrew Walker

unread,
Mar 31, 2021, 8:55:19 PMMar 31
to golang-nuts
There are a couple ways I know of that you can do this. You can vendor or copy as you mention.  Another approach is to create a "build" image that contains relevant credentials and a few `go env -w` directives to set up your build environment, which you build separately and derive other projects from.

```
FROM golang:1.16-alpine3.13
RUN apk --no-cache add ca-certificates git openssh
# cert not checked in to repo, (download with artifact script to rebuild base images)
ADD certs/mycert.crt /usr/local/share/ca-certificates/mycert.crt
RUN chmod 0600 /usr/local/share/ca-certificates/mycert.crt
RUN update-ca-certificates

# if you host a proxy, use these two lines or similar (see https://golang.org/ref/mod#private-modules)
# RUN go env -w GOPROXY=https://proxy.corp.example.com
# RUN go env -w GONOSUMDB=corp.example.com

# otherwise, make sure you don't reach out to public proxy/sum for internal packages
RUN go env -w GOPRIVATE=vcs.package.prefix/*

# gitconfig for vcs to use "ssh://git@" instead of "https://"
ADD gitconfig /root/.gitconfig
# internal VCS creds (probably want to use a "deployment key")
ADD ssh /root/.ssh
RUN chmod 0600 /root/.ssh/id_rsa /root/.ssh/config
RUN chmod 0644 /root/.ssh/id_rsa.pub
```

And then the gitconfig you copy in might be something like:

```
[url "ssh://g...@my.internal.bitbucket:7999"]
```

This should create a build image that will know about your VCS and download modules from it directly.  In your project, you can either do the standard copy of your mod and sum files and go download, but you can also use the new buildkit features to mount a shared cache for both modules and artifacts:

```
# syntax = docker/dockerfile:1-experimental
FROM myregistry:base as base
# ...
RUN --mount=type=cache,id=myproject_mod,target=/go/pkg/mod \
--mount=type=cache,id=myproject_cache,target=/root/.cache/go-build \
go build
```
Reply all
Reply to author
Forward
0 new messages