Currently, ExUnit's `async: true` option would run test cases in this module synchronously,
but only run this module asynchronously along with other `async: true` modules.
This is to propose we add an option for ExUnit to run asynchronously by test cases.
# Background
1. Async by module was a surprise
My initial understanding of `async: true` is async by cases, instead of modules. It's a bit surprising the behavior is later.
2. Async by module would behave more like synchronous tests as a module gets more test cases
As we grow our libraries/apps, a test module will have more and more test cases.
It's tedious to break them into separate modules to speed up the test suite run.
And breaking them into modules has the cost of making related tests further from each other.
# Benefits
1. speed up test suite runs for libraries, apps almost effortlessly
2. more accurate `async: xxx seconds, sync: yyy seconds` metrics
# Caveats
1. Async by test cases may not run faster than async by modules:
- managing these test cases has a cost on its own
- communicating these test cases between ExUnit Server and Runner has costs as well
2. backward compatibility with current `async: true` behavior
some libs or apps may rely on the async by module behavior.
we should still allow user to use `async: ture` by default,
and make async by test cases an easily opt-in feature.
3. Async by test cases may complex the ExUnit implementation even further
# Potential solution
I looked into current ExUnit implementation a little bit
I think `async by test cases` is doable, but I don't have a concrete solution yet
A initial idea is to:
1. instead of saving modules in ExUnit Server, we save test cases (mfa) in ExUnit Server
2. when Runner asks for more async tests, ExUnit Server returns test cases (and also modules) for Runner
This seems to be a huge change,
so I'd like to know if this feature is desirable/feasible from the core team's PoV before I dig more into it.
Best,
Yiming