Proposal: Ability to print/persist stale test metadata (for dev tooling and CI)

[Noah Betzen]

Currently `mix test --stale` will do some checks for code that has been updated since the last test suite run. Relevant code: https://github.com/elixir-lang/elixir/blob/4fa224099730f59b19912e605a888bab68da6e5b/lib/mix/lib/mix/compilers/test.ex#L119-L176

For the purposes of larger Elixir projects and CI, it'd be nice to fetch the list of tests that need to be rerun but not rerun them in the same command.

This has partially been brought up before: https://elixirforum.com/t/is-there-a-way-to-predetermine-which-tests-and-why-would-be-run-on-mix-test-stale/48325

The most basic implementation idea I had for this was `mix test --print-stale` which prints all currently stale tests to stdout (or to a file?).

Another possibly nice feature would be the ability to provide a commit hash similar to Credo's git diff https://hexdocs.pm/credo/diff_command.html that shows which tests are "stale" between two points in time. The usefulness of this idea may be limited.

tldr I want to be able to configure my CI to not run ALL tests on every single PR. There are other ways to do this, but providing the data that `mix test --stale` uses to determine staleness seems like a great start!

[José Valim]

Can you please expand on how the files would be used compared to directly running the stale tests?

Note there is also https://github.com/elixir-lang/elixir/issues/14393, but it is currently unclear if it could be used for this purpose. I asked Andrea to add more context, I recommend following the discussion there too.

José Valim

https://dashbit.co/

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/elixir-lang-core/906a55e5-1b0d-4f4b-813e-43d5afd35b46n%40googlegroups.com.

[Noah Betzen]

> Can you please expand on how the files would be used compared to directly running the stale tests?

Initially I was thinking of being able to do something like:

1. At one point in time: `mix test --get-stale`
2. Later: `mix test --get-stale`
3. Diff the two outputs, and only run the tests that appear in both? (now that I type this out, I realize this probably doesn't help)

This might be a case of the "XY problem"; perhaps what I want is not "print the list of tests that `mix test --stale` would run", but something like "given the current state of the code, and these new code/module changes, give me the list of tests that should be considered 'stale' and should rerun".

Here's more of a typical business scenario I envision:

1. I have a project branch `main` in which CI runs all tests (say, 1000 tests) on every PR merge.
   1. Until then, CI should only run a subset of tests on each commit to a new PR.
2. On `main`, I run `mix test --stale` locally, and all 1000 tests pass.
   1. Ideally I could cache/snapshot this "state" or point in time, or CI could generate it directly on `main`.
3. I make a new branch `feature/a`, make code changes, add a couple new tests.
4. I run `mix test --stale` locally, lets say that only 100 tests need to run based on the code changed.
5. I submit a PR.
6. Current setup: CI tries to run `mix test --stale` but ends up running all 1000 tests again instead of just the 100 whose code/state was actually "stale".
7. Preferred setup: CI runs `mix test --stale` (or something else) and only reruns tests that would be considered stale after my commits were made.
8. PR merges into `main`, and just in case all 1000 tests are run again.

#14393 definitely looks interesting and I'll keep an eye on it!

[Noah Betzen]

My initial (probably incorrect) implementation: https://github.com/elixir-lang/elixir/pull/14499