performance of go mod verify

[Joseph Lorenzini]

All:

I'd like to run go mod verify as part of the CI/CD. If the verify fails, the build fails. However, the amount of time it takes to run for a small project is making me question whether that's viable. I don't actually know of any documentation that explains how slow or fast one should expect a go mod verify to take to run nor how much time it will increase as you add packages (e.g as you add packages is the increase linear or is it more complicated than that?).

I have a project with a go.mod with 113 require statements. Here's the file.

https://gist.github.com/jaloren/564d55145699c2a933798aec334d7ee9

I am go 1.11.1 on Mac OS. I've already run go build and primed the local cache. When I run go mod verify, it completes in 25 seconds. I have no way to know if that's slow or fast since I don't understand what the the internals are doing. All I found on modules FAQs is this:

"In addition, go mod verify checks that the on-disk cached copies of module downloads still match the entries in go.sum."

But I'd like to find out is if this is going to get pathologically worse as packages are added. A minute or two is an easy sell. Past 5 minutes becomes a much harder one.

Joe 

[thepud...@gmail.com]

Hi Joe,

As far as I know, I think it is computing the SHA256 of the dependencies:

https://github.com/golang/go/blob/release-branch.go1.11/src/cmd/go/internal/dirhash/hash.go#L25

Personally, I wouldn't expect that to get pathologically worse with more dependencies.

Also, just as you can often sequence 'go mod download' so that you can often take advantage of cached dependencies in CI, there is some chance you could do something similar with 'go mod verify' to avoid that cost every time?

In terms of how it all works, there is a bit more in the documentation here:

https://golang.org/cmd/go/#hdr-Modules_and_vendoring

"By default, the go command satisfies dependencies by downloading modules from their sources and using those downloaded copies (after verification, as described in the previous section). To allow interoperation with older versions of Go, or to ensure that all files used for a build are stored together in a single file tree, 'go mod vendor' creates a directory named vendor in the root directory of the main module and stores there all the packages from dependency modules that are needed to support builds and tests of packages in the main module."

Regards,

thepudds

[thepud...@gmail.com]

Hi Joe,

Sorry, I realized I pasted in the wrong quote from the documentation.

I meant to include this quote instead (from https://golang.org/cmd/go/#hdr-Module_downloading_and_verification):

"The go command maintains, in the main module's root directory alongside go.mod, a file named go.sum containing the expected cryptographic checksums of the content of specific module versions. Each time a dependency is used, its checksum is added to go.sum if missing or else required to match the existing entry in go.sum.

The go command maintains a cache of downloaded packages and computes and records the cryptographic checksum of each package at download time. In normal operation, the go command checks these pre-computed checksums against the main module's go.sum file, instead of recomputing them on each command invocation. The 'go mod verify' command checks that the cached copies of module downloads still match both their recorded checksums and the entries in go.sum."

Regards,

thepudds

[Joseph Lorenzini]

Hi pudds,

I’ve run go mod verify multiple times on my laptop for the same module. The elapsed time for the command to run is always 25 seconds or thereabouts. So I figured that whatever caching might be there, I would be getting it. 

I can try this with go mod vendor and see if that helps but I’d be surprised if it did. Why would the vendor directory instead of mod cache make a difference ? 

Joe 

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/EAiHWkxTc-o/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.