ANN: Basic test impact analysis (benchmark shows avg 29% reduction in test execution time atm)

[Markus Zimmermann]

We implemented a basic test impact analysis that identifies and then executes affected tests for a change, e.g. for a range of commits. Our benchmark for the repositories we looked at shows an average 29% reduction in test execution time. I am very much looking forward to hearing how your repositories perform!

Details of the benchmark and how the analysis/command works can be found here: https://symflower.com/en/company/blog/2024/test-impact-analysis/. The approach right now is to query a diff using Git and then check which Go packages are affected. In the next iterations, we will bring this analysis down to individual function and test-case level. The eventual goal is to use our symbolic execution engine to allow for even deeper granularity.

At one point i like to have it as a drop-in command for `go` but right now execution looks like this (for all changes of the last commit to now):

```
symflower test-runner --commit-from HEAD~ -- go test -v
```

Looking forward to your feedback!

Cheers,
Markus

[twp...@gmail.com]

How does this interact with Go's existing test result cache, and specifically the invalidation of that cache?

For example,  go test tracks which external files are accessed by tests and will invalidate the test results cache when those files are modified.

Will symflower test-runner also re-run tests when files accessed by tests are modified, even if the affected tests are not touched by the recent code changes?

Regards,

Tom

[twp...@gmail.com]

Constructively, here's a great starting point for understanding go test's cache:

  https://github.com/golang/go/blob/f117d1c9b5951ab2456c1e512ac0423fcf3d7ada/src/cmd/go/internal/test/test.go#L1750-L1773

Regards,

Tom