Allow go mod download to store in subdirectory of current working directory

[jsal...@travix.com]

I'd like to be able to use go mod download to download packages to another directory than GOPATH/pkg/mod and automatically have all go commands be aware of this new location.

This is specifically for CI/CD to ensure I can download modules in one stage and use them in subsequent stages.

I'm the author of https://estafette.io/, and just like Bitbucket Pipelines and Drone.io it executes steps in (public) docker containers, but the only data passed from stage to stage are the files inside the working directory. So downloading modules to a directory outside of the working directory makes them get downloaded again in subsequent stages.

NPM tackles this by having the node_modules directory as a local subdirectory of your repository, but of course this loses you the performance advantage of having a shared cache for all your repositories.

Nuget uses a global cache as well, but they allow you to specify an alternative location that gets used by their other commands by running dotnet restore --packages .nuget/packages.

A similar approach as dotnet takes would be best for all go module related commands to be able to prevent CI/CD from downloading the same modules multiple times. This could look something like go mod download -dir module_cache.

[Marcin Romaszewicz]

Can you accomplish what you want with module vendoring? Before I set up my own Athens proxy, I was using that to avoid DDOSing github in my build automation, and it seemed to work fine. Your first pipeline step could vendor in all the modules, and subsequent pipeline steps would use those.

-- Marcin

--
You received this message because you are subscribed to the Google Groups "golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Manlio Perillo]

On Monday, March 4, 2019 at 5:49:06 PM UTC+1, Jorrit Salverda wrote:

I'd like to be able to use go mod download to download packages to another directory than GOPATH/pkg/mod and automatically have all go commands be aware of this new location.

This is specifically for CI/CD to ensure I can download modules in one stage and use them in subsequent stages.

I'm the author of https://estafette.io/, and just like Bitbucket Pipelines and Drone.io it executes steps in (public) docker containers, but the only data passed from stage to stage are the files inside the working directory. So downloading modules to a directory outside of the working directory makes them get downloaded again in subsequent stages.

NPM tackles this by having the node_modules directory as a local subdirectory of your repository, but of course this loses you the performance advantage of having a shared cache for all your repositories.

You can set the GOPATH to $PWD/.vendor.  It works with go1.12

I also tried with GOPATH=$PWD, but go does like it:

  $GOPATH/go.mod exists but should not

 

> [...]

Manlio

[thepud...@gmail.com]

Jorrit,

The simplest solution might be 'go mod vendor' to populate a 'vendor' directory, and then set '-mod=vendor' or 'GOFLAGS=-mod=vendor' when building.

If that is not workable for some reason, here is an alternative that is more similar to your request for a 'go mod download -dir module_cache':

The module download cache location is controlled by GOPATH. In particular, 'go mod download', 'go build', etc. populate the module cache in GOPATH/pkg/mod (or ~/go/pkg/mod if $GOPATH is not set).

In addition, when you want to use a particular module cache, you can tell the 'go' command to use a local module cache by setting GOPROXY=file:///file/path.

You can put those two things together, which I think gives you the control you were asking about:

 

    # Populate a module download cache in /tmp/gopath-for-cache

    $ GOPATH=/tmp/gopath-for-cache  go mod download

    # Build using the contents of the module download cache in /tmp/gopath-for-cache

    $ GOPROXY=file:///tmp/gopath-for-cache/pkg/mod/cache/download  go build

Note that even though you are setting the GOPROXY environment variable, there is no actual proxy process involved, and everything is just being read directly from the local filesystem.

 

Here is a more complete "Go Modules by Example" walk-through of that technique:

    https://github.com/go-modules-by-example/index/tree/master/012_modvendor

Regards,

thepudds

[jsal...@travix.com]

I indeed managed to use 'go mod vendor', which leads to much faster builds anyway. But it would be nice if the module tooling compared the vendor directory to match the checksums in go.sum, otherwise you can update on of your dependencies but forget about updating the vendored copy. Do you know whether it does this?

Thanks, Jorrit

[jsal...@travix.com]

I'll test the GOPROXY approach, although vendoring leads to faster builds (but requires more actions before committing changes).

Thanks, Jorrit