Test coverage: joint unit test and integration test coverage

[Jan]

hi all, 

After reading the documentation, I managed to set up a process in which I compile and run my integration tests, and get coverage from them in the `$GOCOVERDIR` subdirectory.

Now I would like to have a combined unit tests + integration tests report, all in one go, and I'm not sure how to get that.

I was expecting that, if I go to my modules root directory, and I do:

```

go test --cover --coverpkg=./... --coverprofile=/tmp/cover_profile.out ./...

go tool cover -func /tmp/cover_profile.out > /tmp/cover_func.out

```

I  would get all the results, including integration tests (since they are called with GOCOVERDIR set). But instead I only see the coverage of the unit tests, and the information from the integration tests seems to be ignored, even though it is generated.

I'm sure it is generated because if I run the command above with `--work` (preserve the temporary files), and log the value of $GOCOVERDIR I can see where the test stores the coverage data files. And if I manually do:

```

go tool covdata func -i /tmp/go-build287472875/b001/gocoverdir

```

(where /tmp/go-build..../gocoverdir is the temporary directory reported for GOCOVERDIR)

I see the results I expected (some of my functions that I want to make sure are covered) are there. But they are not reported in `/tmp/cover_func.out` above.

Any ideas why ? 

Or any other suggestions on how to merge the report from unit tests (`/tmp/cover_profile.out` in my example) and integration tests ? 

thanks!

[TheDiveO]

Maybe similar to this? https://github.com/thediveo/lxkns/blob/cef5a31d7517cb126378f81628f51672cb793527/scripts/cov.sh#L28

[Jan]

Thanks, that's very helpful! 

I was doing something similar, but I was extracting the temporary coverage directory created by Go because I didn't know about the --test.gocoverdir flag. 

Still it feels wrong one has to do the trick of creating a temporary directory (or extract it from the go tool) , where for unit tests only it's not needed ... I wonder if this is done deliberately, or just a bug ?

cheers

[TheDiveO]

Hmm, my previous response got deleted for no reason. So here we go: the new mechanism is more general, and as you can see in my example I actually run the "same" unit tests twice. The reason is because some code paths might be only exercised when not being run as root, especially error handling. So there is some value in the new mechanism even "just" for unit tests. Admittedly, not all will need that in their unit tests.

[Jan]

Oh yes, it's true that the new mechanism is more general, and allows integration tests, it's a huge improvement for cases like this! Thanks for the design btw!

Now the ergonomics of the case of unit tests + integration tests could be made easier/more ergonomic -- I would assume it is the common case for projects with integration tests (because everyone has also unit tests). And a suggestion of maybe a mention / an example in the integration-test-coverage blog ?

[Jan]

I tried to use the `-test.gocoverdir` (*) and while the test(s) being executed use the given directory, unfortunately it still sets `GOCOVERDIR` to some newly created temporary directory. 

Since my integration tests are executed from a *_test.go test (it's a test after all), in the end the `-test.gocoverdir` flag contents are discarded (or not passed along in GOCOVERDIR). If the flag is set, shouldn't it be used for `GOCOVERDIR` as well ?

cheers

(*) Btw, I can't find documentation on -test.gocoverdir in Google, except the insides of code. Also `go help testflag | grep gocoverdir` doesn't return anything -- in go1.21.0.

[TheDiveO]

The only "documentation" is this issue comment https://github.com/golang/go/issues/51430#issuecomment-1344711300 and the reply to my question about the "official status" https://github.com/golang/go/issues/51430#issuecomment-1485320781