[Proposal] Use different exit codes for compilation errors versus test failures
90 views
Skip to first unread message
jonar...@gmail.com
Jul 13, 2021, 4:13:37 PM7/13/21
to elixir-lang-core
In our CI environment, we run our tests with this command:
mix test --warnings-as-errors <files>
This is intended to fail when someone has created a warning within a test file in addition to when the tests themselves fail. We have an earlier check (MIX_ENV=test mix compile --force --warnings-as-errors) that validates that our production code is warning-free, but this does not compile nor validate test files.
A few of our tests are somewhat finnicky, and will fail sporadically (e.g. due to PostgreSQL deadlocks). So we also have retry logic in our CI environment that will re-run tests when the above command exists unsuccessfully (exit code != 0). However, no amount of re-running will fix a compilation warning! Presently, both test failures and compilation failures return exit code 1, and so we are not able to differentiate between the two and only re-run in the former case.
Would it be appropriate to change the compilation failure case to return an exit code of, say, 2?
mix test --warnings-as-errors <files>
This is intended to fail when someone has created a warning within a test file in addition to when the tests themselves fail. We have an earlier check (MIX_ENV=test mix compile --force --warnings-as-errors) that validates that our production code is warning-free, but this does not compile nor validate test files.
A few of our tests are somewhat finnicky, and will fail sporadically (e.g. due to PostgreSQL deadlocks). So we also have retry logic in our CI environment that will re-run tests when the above command exists unsuccessfully (exit code != 0). However, no amount of re-running will fix a compilation warning! Presently, both test failures and compilation failures return exit code 1, and so we are not able to differentiate between the two and only re-run in the former case.
Would it be appropriate to change the compilation failure case to return an exit code of, say, 2?
Christopher Keele
Jul 15, 2021, 2:50:04 AM7/15/21
to elixir-lang-core
This is a compelling notion to me.
I'd prefer an implementation where the default was generic and the use-case was specific; ie have the complier return 1's for compilation errors/other, and have specific CLI use-cases extend the semantics, like mix test returning 2 for test failures.
José Valim
Jul 15, 2021, 3:06:10 AM7/15/21
to elixir-l...@googlegroups.com
Sounds good to me. PR is welcome!
--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/4d33bc53-017f-42f1-aa69-ceba1be8378cn%40googlegroups.com.
Reply all
Reply to author
Forward
0 new messages