go modules, goproxy, building local caches of repositories

[Samantha Thompson]

Hi,

I'm trying to move to using the recently released module system with go.1.11. 

In particular, I'm trying to create a local registry of golang source repositories such that, once versioned and curated, local projects can be compiled without having to reach outside to github.com etc as this would be unacceptable for production builds within the environment in which I'm working.

So, I've been looking at goproxy which seems to actually describe such a local build cache or way to mirror repositories. With GO111MODULE=on and setting GOPROXY to my file:///~/go/pkg/mod/cache/download I can happily build and install using any packages/modules that are already cached there.

There are some packages however which I require which aren't yet updated to use a go.mod , maybe they never will be, others such as those in golang.org/ and gopkg.in/ I can't go get due to redirection, so have to manually git clone these into the correct path from the corresponding github.com/ location . That means checking them out and attempting to go mod init with a reasonable name. An example is golang.org/x/crypto which is really github.com/golang/crypto , perhaps I can git clone with the latter and then git mod init with the former such that the module can be found.

Well that's one problem. 

The other is how to locally curate such a module into my local proxy/cache.

I need to build the zip for that package, which I've attempted according to the guidance in go help goproxy. For example the above package would be zipped as golang.org/x/cry...@v0.0.1/    (just for simplicity I created the git tag v0.0.1 in this repo while I figure this out) and this would become $GOPROXY/golang.org/x/crypto/@v/v0.0.1.zip (there is a corresponding .mod .info and line in list for v0.0.1).

It appears to be of a similar format to automatically cached and zipped modules and the zip is located from a test project due to having a require golang.org/x/crypto v0.0.1 line in its go.mod , there is also an import ( _ "golang.org/x/crypto") line in a very simple main.go.

It seems to locate the v0.0.1.zip and attempt to unzip however complains with golang.org/ unexpected file. So I seem to be getting something subtly wrong with the zip structure/format there.

Are there any guides that describe using GOPROXY for this in more detail? Are there any examples/tutorials around how one might build such a local registry/repository ? I feel like I'm going round in circles with this the stuff in go help goproxy doesn't seem to be enough. Is there a way of using go mod to push a package to a local GOPROXY registry ?

[Ralph Corderoy]

Hi Samantha,

I'll leave others to help on your other points.

> It seems to locate the v0.0.1.zip and attempt to unzip however
> complains with golang.org/ unexpected file. So I seem to be getting
> something subtly wrong with the zip structure/format there.

Have you tried `unzip -l' on a ZIP file that Go's produced and then the
one you've built manually and looking for differences in directory
structure, e.g. root directory?

--
Cheers, Ralph.
https://plus.google.com/+RalphCorderoy

[komu wairagu]

Hi,

The shell/bash commands I have laid out in; https://gist.github.com/komuw/639f3811ad10a3453190fb9b4d971b37#gistcomment-2697042 
work for me for creating a file based proxy. 

[hoe...@indeed.com]

I've been hacking on a Go Module Proxy targeted for the internal hosting use case: https://github.com/modprox It's still being developed (read: pre-beta), but can be used as a GOPROXY for most open source projects as of today. The idea is to keep adding support for organizations that live "behind the firewall", and need features for things like authenticated repositories, domain redirecting, arbitrary path rewriting, version blocking, licence checking, etc. 

The redirection you describe is a feature of the go-get command: https://golang.org/cmd/go/#hdr-Remote_import_paths, and yes it is necessary to check for the redirect in those cases to determine where the repository actually resides. The name of the module in these cases must be derived from the first (not the final redirect) package name - assuming of course that's how other packages are referencing it. Take a look at GoGetTransform in this package we wrote, which among other types of URI manipulation handles the go-get style redirects.

For the zip file format, inside the zip, each file needs to live under a path that is the same as the name of the module. It looks like a redundant tree when you unzip it, because the zip itself is (probably) residing in a tree that is the same as the module name. Take a look at what modprox is doing to rewrite the zip files it gets from source repositories - there's also a more complex version of that code in the Go source tree.

The documentation for modprox is sparse to non-existent right now: https://modprox.org/ as the project itself is very much in the process of being developed. We do hope to make a general beta announcement in the next couple of weeks. Once the proxy is in a usable state for my own organization, I hope to write a series of blog posts describing in detail how the Go command makes the GOPROXY magic work, because it has turned out to be really interesting (and as you've discovered, sometimes frustrating - I've spent a lot of time digging through the go/cmd source code).

As for pushing a module into a registry, we plan on providing a little command to do that as well - but it won't be built into the go tool.

Regards,

Seth

[Samantha Thompson]

Hi,

I have checked zips in this way, they seem to be of the correct structure.

[Samantha Thompson]

Thanks for this, I have already managed to achieve this, the missing thing for me is being able to create that zip myself from scratch rather than getting go mod to create it when I run download or build.

[thepud...@gmail.com]

Hi Samantha, all,

Regarding manually building a zip file for a proxy cache, I had thought the go tooling would do this for you automatically when doing a `go get` or `go mod download` (including one of the listed use cases for `go mod download` seems to be to automatically create the on-disk structure that matches the requirements of a Go module proxy).

I do not have much direct experience with GOPROXY, so sorry if that is off base...

A quick test using golang.org/x/crypto (which was cited in the first post in this thread):

    $ go get golang.org/x/crypto

    go: finding golang.org/x/crypto latest

    go: downloading golang.org/x/crypto v0.0.0-20180830192347-182538f80094

    $ ls $GOPATH/pkg/mod/cache/download/golang.org/x/crypto/\@v/

    list

    v0.0.0-20180830192347-182538f80094.info

    v0.0.0-20180830192347-182538f80094.mod

    v0.0.0-20180830192347-182538f80094.zip

    v0.0.0-20180830192347-182538f80094.ziphash

If that is not what is desired, could you expand slightly on the problem statement?

Related snippet from the doc:

-------------------------------------------

https://tip.golang.org/cmd/go/#hdr-Module_proxy_protocol

-------------------------------------------

Even when downloading directly from version control systems, the go command synthesizes explicit info, mod, and zip files and stores them in its local cache, $GOPATH/pkg/mod/cache/download, the same as if it had downloaded them directly from a proxy. The cache layout is the same as the proxy URL space, so serving $GOPATH/pkg/mod/cache/download at (or copying it to) https://example.com/proxy would let other users access those cached module versions with GOPROXY=https://example.com/proxy.

-------------------------------------------

--thepudds

[komu wairagu]

On Tuesday, 4 September 2018 19:50:04 UTC+3, Samantha Thompson wrote:

Thanks for this, I have already managed to achieve this, the missing thing for me is being able to create that zip myself from scratch rather than getting go mod to create it when I run download or build.

This shell commands will create a zip file for you;

mkdir -p /tmp/zipper
  # download your module. in this case github.com/pkg/errors
wget -nc --directory-prefix=/tmp/zipper https://github.com/pkg/errors/archive/v0.8.0.zip
unzip /tmp/zipper/v0.8.0.zip -d /tmp/zipper
mkdir -p /tmp/zipper/github.com/pkg/errors@v0.8.0
cp -r /tmp/zipper/errors-0.8.0/* /tmp/zipper/github.com/pkg/err...@v0.8.0
  # create a zip file of the download module in the layout that Go expects
zip /tmp/v0.8.0.zip /tmp/zipper/github.com/pkg/err...@v0.8.0/*

[Baokun Lee]

You can use this open source project may be: https://github.com/goproxyio/goproxy

[Paul Jolly]

Hi,

> There are some packages however which I require which aren't yet updated to use a go.mod , maybe they never will be, others such as those in golang.org/ and gopkg.in/ I can't go get due to redirection, so have to manually git clone these into the correct path from the corresponding github.com/ location .

You don't need to do anything manual, not even for non-module
dependencies. If you do a go mod download in your module, that will
ensure the module cache (at $GOPATH/pkg/mod/cache/download) is primed
for use offline.

If you want to get the subset of $GOPATH/pkg/mod/cache/download
relevant to your module, you could do something similar to:

go mod download
td=$(mktemp -d)
GOPROXY=file://$GOPATH/pkg/mod/cache/download GOPATH=$td go mod download

Then $td/pkg/mod/cache/download will be primed with everything your
module requires for offline use.

I use exactly this approach, and push the results to
https://github.com/myitcv/cachex for use in my CI:

> The other is how to locally curate such a module into my local proxy/cache.
> I need to build the zip for that package, which I've attempted according to the guidance in go help goproxy. For example the above package would be zipped as golang.org/x/cry...@v0.0.1/ (just for simplicity I created the git tag v0.0.1 in this repo while I figure this out) and this would become $GOPROXY/golang.org/x/crypto/@v/v0.0.1.zip (there is a corresponding .mod .info and line in list for v0.0.1).

You shouldn't need to manually build any zips; the go tool will do
everything for you.

Hopefully the steps above help to clear things up a bit?

Thanks,


Paul

[stor...@foxmail.com]

Problems such as Internal Server Error or Timeout often occur when using GOMODULE. GOPROXY can solve the problem of timeout caused by pulling the package such as golang.org/x to some extent, and can also solve the 404 problem caused by project renaming, migration, deletion, etc. to some extent. However, it should be noted that using GOPROXY will result in you being unable to pull the private repository. Now, GOS solves this problem. https://github.com/storyicon/gos