Go 1.22.0: Alpine: go test with coverage works but returns exit status 1

[Martin]

I am using the new `golang:1.22.0-alpine3.19` Alpine image to build and test a Golang 1.22.0 application. The test uses coverage like that `go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count`. The test command seems to work fine, but the exit status is 1. The created coverage.out seems also fine, as I could use go tool cover with it, see below.

Golang version: 1.22.0

Container Image: golang:1.22.0-alpine3.19

Reproducer:

main.go:

package main

import (
  "example.com/m/internal"
  "fmt"
)

func main() {
  fmt.Println(internal.Helloer())
}

internal/helloer.go:

package internal

func Helloer() string {
  return "Hello, world!"
}

internal/helloer_test.go:

package internal

import "testing"

func TestHelloer(t *testing.T) {
  want := "Hello, world!"
  got := Helloer()
  t.Errorf("Helloer() = %v, want %v", got, want)
}

go.mod:

module example.com/m

go 1.22

Shell commands for testing:

apk add --no-cache build-base
go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count || echo "flaky:$?"
go tool cover -html=coverage.out -o coverage.html
go tool cover -func=coverage.out

Is it a bug or exepected behaviour?

[Martin]

I need to add:

In the post the reproducer test is of course to short, so the test show that it fails all the time. But that triggers the problem anyway. Better would be this:

func TestHelloer(t *testing.T) {
  want := "Hello, world!"
  got := Helloer()

  if got != want {

    t.Errorf("Helloer() = %v, want %v", got, want)
  }
}

Also, the output of the go test call has a inidication why it has exit status 1:

$ go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count
example.com/m: open /tmp/go-build1679138031/b002/covmeta.f6e4431d5ec1fd71f02b3ce4e56eb691a86525173d917007425576a7d9db7c72: no such file or directory
=== RUN   TestHelloer
    helloer_test.go:8: Helloer() = Hello, world!, want Hello, world!
--- FAIL: TestHelloer (0.00s)
FAIL
coverage: 100.0% of statements in ./internal/...
FAIL    example.com/m/internal    0.003s

Also this happens with the Alpine based image, but also for the Debian based image as well.

[Brian Candler]

Is it a bug or exepected behaviour?

If a test fails, I would expect it to terminate with an error (exit code 1 in this case).

If I run your reproducer locally (not in Docker) with the modified TestHelloer, it works fine(*) and gives me an exit code of 0:

% go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count
?   example.com/m [no test files]
=== RUN   TestHelloer
--- PASS: TestHelloer (0.00s)
PASS

coverage: 100.0% of statements in ./internal/...

ok   example.com/m/internal 0.135s coverage: 100.0% of statements in ./internal/...
% echo $?
0

Therefore, if your problem only occurs when using Docker, then you should provide a docker-based reproducer (including the Dockerfile)

(*) However, I had to change the go.mod file to say version 1.21.  If it says 1.22, I get an error.

Under Linux (go1.21.7):

$ go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count

go: downloading go1.22 (linux/amd64)
go: download go1.22 for linux/amd64: toolchain not available

Under macOS (go1.21.6):

% go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count
go: downloading go1.22 (darwin/arm64)
go: download go1.22 for darwin/arm64: toolchain not available

I don't *think* this is the same problem as you're seeing, since you say that the coverage file is created, and presumably you would have noticed the "toolchain not available" error message. In any case, you're using a base image with go 1.22.0.

[Brian Candler]

I found the solution to the "toolchain not available" problem: put "go 1.22.0" instead of "go 1.22" in go.mod. Clues picked up from #62278.

It's confusing for people who've been using go for a while though, when go.mod used to contain "go X.Y" and it was invalid to put "go X.Y.Z". Now that appears to have swapped around.

[Martin Schallnahs]

Hi Brian,

thanks for checking out, yes that I wanted also to write you.

We need it currently in our CI as some dependency scanner tool does not work with the "go X.Y.Z." syntax, but I tried, and for my problem it did not was the cause.

> If a test fails, I would expect it to terminate with an error (exit code 1 in this case).

See my second mail, the test case should not fail, it was kinda a typo (tried to shorten the reproducer to much in my first mail).

> If I run your reproducer locally (not in Docker) with the modified TestHelloer, it works fine(*) and gives me an exit code of 0

Yes, when I run it with golang 1.21.7 it works fine as well, as my problem statement is about golang 1.22.0.

> Therefore, if your problem only occurs when using Docker, then you should provide a docker-based reproducer (including the Dockerfile)

Happens locally as well. And in my original setup it was using a fresh docker container from golang (in an CI/GitLab pipeline) and did this:

$ go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count || echo "flaky:$?"
example.com/m: open /tmp/go-build2233205084/b002/covmeta.f6e4431d5ec1fd71f02b3ce4e56eb691a86525173d917007425576a7d9db7c72: no such file or directory

=== RUN   TestHelloer
--- PASS: TestHelloer (0.00s)
PASS
coverage: 100.0% of statements in ./internal/...

ok   example.com/m/internal 0.004s coverage: 100.0% of statements in ./internal/...
flaky:1
$ go tool cover -html=coverage.out -o coverage.html
$ go tool cover -func=coverage.out
example.com/m/internal/helloer.go:3: Helloer 100.0%
total: (statements) 100.0%

But I just tried it locally (Windows) and there it happens as well:

C:\dev\git\golang-test-cover>go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count
example.com/m: open C:\Users\A1524415\AppData\Local\Temp\go-build2423189316\b002\covmeta.f6e4431d5ec1fd71f02b3ce4e56eb691a86525173d917007425576a7d9db7c72: The system cannot find the file specified.

=== RUN   TestHelloer
--- PASS: TestHelloer (0.00s)
PASS
coverage: 100.0% of statements in ./internal/...

ok      example.com/m/internal  15.260s coverage: 100.0% of statements in ./internal/...

So for now I checked it on:

- windows

- debian (via docker container)

- alpine (via docker container)

All with 1.22.0.

>  since you say that the coverage file is created, and presumably you would have noticed the "toolchain not available" error message. In any case, you're using a base image with go 1.22.0.

Exactly.

As seen in the output above, further commands (go tool cover) using the coverage.out work fine.

[Brian Candler]

$ go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count || echo "flaky:$?"
example.com/m: open /tmp/go-build2233205084/b002/covmeta.f6e4431d5ec1fd71f02b3ce4e56eb691a86525173d917007425576a7d9db7c72: no such file or directory

C:\dev\git\golang-test-cover>go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count
example.com/m: open C:\Users\A1524415\AppData\Local\Temp\go-build2423189316\b002\covmeta.f6e4431d5ec1fd71f02b3ce4e56eb691a86525173d917007425576a7d9db7c72: The system cannot find the file specified.

Those look like the underlying errors causing the exit code 1. But it works fine for me under both Linux and macOS, as long as I have "go 1.22.0" in go.mod. Maybe someone who knows more about Windows can help?

[Martin Schallnahs]

P.S.: According to the release notes of golang 1.22.0 the "go test -cover" was worked on, so thats why insist to find out if its a bug (and also because it prevents me from upgrading a project to golang 1.22.0 right now).

" go test -cover now prints coverage summaries for covered packages that do not have their own test files. Prior to Go 1.22 a go test -cover run for such a package would report"

[Martin Schallnahs]

Hey Brian,

dont get me wrong!

My usual setup is Continuous Integration Pipeline running in Linux VMs which, in case of my golang projects, start fresh golang docker container images from DockerHub.
In case of this problem I used golang:1.22.0-alpine3.19 and golang:1.22.0 (debian basd).
For both I have the problem I describe.
I just wanted to show, that its not a Docker problem and used my local Windows machine do prove that.

I grabbed my Macbook Pro (Intel), installed freshly Golang 1.22.0 and run the mentioned go test command on the reproducer, and for me here the problem also gets triggered.

As you explained I use the corrected reproducer, with the Test not failing, and have "go 1.22.0" in the go.mod.


$ go version
go version go1.22.0 darwin/amd64

$ go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count

example.com/m: open /var/folders/dm/k73ydgtx15l7dzwc3qk_wmhc0000gn/T/go-build3097649140/b002/covmeta.f6e4431d5ec1fd71f02b3ce4e56eb691a86525173d917007425576a7d9db7c72: no such file or directory

=== RUN TestHelloer
--- PASS: TestHelloer (0.00s)
PASS
coverage: 100.0% of statements in ./internal/...

ok example.com/m/internal 0.261s coverage: 100.0% of statements in ./internal/…

Bit confusing, that it works for you though.

On Thursday, February 8, 2024 at 6:11:15 PM UTC+1 Brian Candler wrote:

[Aldemar F]

I have the same issue here.

Running tests on a linux (ubuntu based).

It only happens when a use the flag coverpkg.

I have managed to "solve" it on an very bad way, creating an empty test file in packages that don't have tests

(removed the files paths)

```
open /tmp/go-build2070237917/b334/covmeta.ce414c14907a52c0962d732a8a53fd8b6639b1f9a004b9c6e110921165bab8c3: no such file or directory
open /tmp/go-build2070237917/b335/covmeta.41db8a5854133f388ac5e1dae6628818f4bfb391b0fed738b0e33c2fa26dd79b: no such file or directory
... ... ....
```

[Thomas McNulty]

While trying to upgrade a go package to 1.22.0 today, I ran into this same issue in my coverage step. From testing different combinations of options for `go test`, it seems that the 'no such file or directory' errors and resulting non-zero exit code occur when go test processes a folder/package that isn't included in the -coverpkg list- e.g. in your case, any package that isn't part of ./internal. When I set -coverpkg equal to the entire list of packages in the project, the error goes away.

Running with some packages (in this case, mocks) excluded:

% go version
go version go1.22.0 darwin/arm64

% CVPKG=$(go list ./... | grep -v mocks | tr '\n' ',')

% go test -coverpkg=${CVPKG} -coverprofile=coverage.out -covermode=count  ./...

  ... (tests passing)

github.com/<org>/<repo>/internal/mocks/a/b: open /var/folders/c4/kbr99g196216gsv36c94cp840000gn/T/go-build466145861/b849/covmeta.3a394ea9611306b457bfb2f5b2169adffc13123f3b07d667fd86b58eac717920: no such file or directory
github.com/<org>/<repo>/internal/mocks/c: open /var/folders/c4/kbr99g196216gsv36c94cp840000gn/T/go-build466145861/b851/covmeta.5fc2c6ff11ae04da5c2cdbbb4c9e9e64d515086b2a7a7f5660d5dc409cdf5724: no such file or directory
github.com/<org>/<repo>/internal/mocks/e/f: open /var/folders/c4/kbr99g196216gsv36c94cp840000gn/T/go-build466145861/b853/covmeta.2405ba52a1121a0f4a20da157347d9217eff95f8d18b9ae656cf0414aefe8221: no such file or directory

  ... (more tests passing)

% echo $?                                                                      
1

Vs. running with no packages excluded:

% go version
go version go1.22.0 darwin/arm64

% CVPKG=$(go list ./... | tr '\n' ',')
% go test -coverpkg=${CVPKG} -coverprofile=coverage.out -covermode=count ./...
.... (all tests passing)
% echo $?
0

[Martin]

Hey Aldemar and Thomas,

thanks for your hints.

>  While trying to upgrade a go package to 1.22.0 today, I ran into this same issue in my coverage step. From testing different combinations of options for `go test`, it seems that the 'no such file or directory' errors and resulting non-zero exit code occur when go test processes a folder/package that isn't included in the -coverpkg list- e.g. in your case, any package that isn't part of ./internal. When I set -coverpkg equal to the entire list of packages in the project, the error goes away.

You are right, if I adopt your command, the errors go away.

There seem several issues with the new code coverage design, see

https://github.com/golang/go/issues/65570

https://github.com/golang/go/issues/65636

In both issues its addressed, that with

$ GOEXPERIMENT=nocoverageredesign go test -v ./... -coverprofile=coverage.out -coverpkg=./internal/... -covermode count

The old behaviour is used and the error is gone as well.

I assume the root cause is related to the both issues linked above, but will create a new issue anyway.

I will include your findings as well.

[Martin]

Created a issue based on our discussion:

https://github.com/golang/go/issues/65653