Changing PWD ruins test caching

[Kevin Burke]

We have some tests that are pretty slow. I would like to use Go's test caching to skip them in our CI environment, if nothing in the code or the environment has changed. Our CI environment has a set of _agents_, each of which can run multiple jobs simultaneously. As a result, they check out code into a different directory each time they run a job.

Go's test caching checks every env var loaded by the test program. If any of them change, then Go assumes that the cache is busted and the test must be run.

Any program that imports "os" hits this logic in the os package, which checks the value of the $PWD environment variable:

// We query the working directory at init, to use it later to search for the
// executable file
// errWd will be checked later, if we need to use initWd
var initWd, errWd = Getwd()

So checking code out to different directories means that any program that imports "os" cannot have test caching. This seems like a shame because the code is all laid out in the same place in the working directory.

Has anyone tried to fix this issue? Do you think this is worth trying to patch or modify the test caching behavior in Go itself? I could solve this by running a chroot or another layer of Docker, of course, but I'd prefer not to do these because of the difficulty of getting data in and out of each one, communicating with other Docker containers, etc.

[Kevin Chowski]

I haven't had this problem so this is a complete guess. But if PWD is the only env variable causing a problem, you could try hard coding it to a constant value (which would never match any real directory). That would allow the PWD to always be the same no matter which directory it was actually running, and briefly looking through the code it seems that the PWD variable is only used it if actually matches "."

Sorry for the noise if this isn't helpful - I'm on mobile so I can't test right now.

[Bryan C. Mills]

As far as I can tell the code in question is only built on AIX and OpenBSD — are you using one of those two platforms?

If not, perhaps the caching problem is coming from somewhere else.

That said, it does appear that calls to `os.Chdir` result in a spurious dependency on the PWD variable:

https://cs.opensource.google/go/go/+/master:src/os/file.go;l=340-345;drc=d961b12be9001cf8dbf8f52847607dbf84d94f8d

On Thursday, November 23, 2023 at 1:50:04 PM UTC-5 Kevin Burke wrote:

[Kevin Burke]

Sorry for slow reply. We are using Alpine, which I don't think is either of those flavors.

I discovered this by enabling GODEBUG=gocachehash=1, running the tests twice, and then checking the diff between the two test runs.

--
You received this message because you are subscribed to a topic in the Google Groups "golang-nuts" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/golang-nuts/74iG66JVA9s/unsubscribe.
To unsubscribe from this group and all its topics, send an email to golang-nuts...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/9507e736-9d8a-4929-9812-3a89358a1cf3n%40googlegroups.com.

[Sean Liao]

Using a basic example which just imports os
https://go.dev/play/p/BmX0ds901-4 (the rest copied from gobyexample)
copied to 2 different directories, I do see cache hits, so I don't
think it's from just the import?

23:12:37 ~/tmp/testrepo0530/a 0:00:00
main » go1.22.0 test .
testcache: example: test ID 7554777255756c714a6773663261576871416373
=> d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c
testcache: example: input list not found: cache entry not found: open
/home/user/.cache/go-build/d9/d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c-a:
no such file or directory
testcache: example: test ID 695f4e6944335a64497850503861366643364c48
=> e7d8d7272b091d2d5d155ff077e599c1e7c31eaad2d9bf3549e2d03149e94357
testcache: example: input list not found: cache entry not found: open
/home/user/.cache/go-build/e7/e7d8d7272b091d2d5d155ff077e599c1e7c31eaad2d9bf3549e2d03149e94357-a:
no such file or directory
testcache: example: save test ID
d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c =>
input ID 8671bffc7a3ec5201bb26a0eccecb5599259f0d0e63624375378b61cc00a0733
=> 516ef3f5e0d83ea893583f43f60e654775b3769b5f99f2de7b99487eae637298
testcache: example: save test ID
e7d8d7272b091d2d5d155ff077e599c1e7c31eaad2d9bf3549e2d03149e94357 =>
input ID 8671bffc7a3ec5201bb26a0eccecb5599259f0d0e63624375378b61cc00a0733
=> 42e8604e2c4807cd097efad167887d4d0ddfc76afda71e21f8145604d93ebce5
ok example 0.001s

23:12:42 ~/tmp/testrepo0530/a 0:00:00
main » cd ../b

23:12:46 ~/tmp/testrepo0530/b 0:00:00
main » go1.22.0 test .
testcache: example: test ID 7554777255756c714a6773663261576871416373
=> d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c
testcache: example: test ID
d9016fd4db0cc529026c3bdf959d21e8b37da70f61b3b9bd14cebcf4bcc41e0c =>
input ID 8671bffc7a3ec5201bb26a0eccecb5599259f0d0e63624375378b61cc00a0733
=> 516ef3f5e0d83ea893583f43f60e654775b3769b5f99f2de7b99487eae637298
ok example (cached)


- sean

> You received this message because you are subscribed to the Google Groups "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAKcy5egYecGK8oSaB1xA439BPQ-BnCKi9vHpafe_qzRtQ_D%2B2w%40mail.gmail.com.

[Kevin Burke]

You're right. I ended up inserting a shim to replace os/env.go in the standard library, that would dump the stack trace of anything that called os.Getenv("PWD"). It turns out that a popular testing library was doing this - https://github.com/onsi/ginkgo/issues/1355. 

To view this discussion on the web visit https://groups.google.com/d/msgid/golang-nuts/CAGabyPr2P_MaBdwuLRSoKzBxhrAg9Vun9TDRQMK7WWY_4eOcwA%40mail.gmail.com.