[Proposal] Allow `mix test.coverage` to use custom coverage tools

[Ilya Borovitinov]

Hello!

Today I encountered a problem when trying to set up testing + coverage for the new Elixir project with Elixir 1.11.

We are using Gitlab as our source control and CI, and Gitlab can display covered lines if you were to provide coverage in the Cobertura XML format. That's all good, and Elixir/Erlang ecosystem even has `covertool` package for this purpose. 

I have an umbrella project with 3 apps currently, and so I read in the release notes that I could use `mix test` with `--export-coverage` flag to collect coverage from each app and then compile them with `mix test.coverage`. That's awesome.

Here's the problem though: `mix test` calls upon the specified tool to process coverage data, but `mix test.coverage` just generates HTML, so we cannot use any tools to properly compile and process coverage data across all apps (or partitions). Basically, I feel that there shouldn't be such a discrepancy in usage between these two commands. 

With great admiration for the language,

Ilya Borovitinov

[José Valim]

Hi Ilya, thanks for the report!

mix test.coverage is dependent on how the default coverage tool exports data. We cannot aggregate coverage reports from random tools because we don't know how these tools export their results. Therefore, it is up to these tools to do two changes:

1. Respect the :export option in their coverage tool

2. Provide their own mix task that aggregate the results

So please reach out to them and consider sending pull requests so they make this feature available,

Thank you!

--
You received this message because you are subscribed to the Google Groups "elixir-lang-core" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elixir-lang-co...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/848b4145-ef5e-48fa-863f-d0fb65f2a8cfn%40googlegroups.com.

[Ilya Borovitinov]

Hello again, thanks for your time and prompt reply!

I am not sure, I might not have been clear enough. For example, given an umbrella project, I can run `mix test --export-coverage` from the project root. Even if `:covertool` is specified as a tool in `test_coverage:`, the .coverdata files are exported correctly alongside XML within each folder.

The problem is: when I run `mix test.coverage`, it takes all .coverdata files and compiles them into an HTML report without any consideration for the specified coverage tool. Yes, this issue could be solved with a custom command within the package to operate upon coverdata files, but I still believe that this behavior is counterintuitive. Maybe another option could be specified explicitly for report generation based on multiple coverdata files?

One more example (although not my use case) could be the `--partitions` flag. It implicitly generates coverdata files, which are then expected to be compiled via `mix test.coverage`. So if a person has the tool specified in `test_coverage`, then running `mix test` will produce different reports from  `mix test --partitions` and `mix test.coverage` because one will use the coverage tool and the other will not.

Thank you again for your attention!

[José Valim]

The point is exactly that we cannot assume that the underlying coverage tool is using .coverdata or that it would accept it as input. They need to provide their own APIs that handle those. The only callback we have today for external coverage tool is handling out to them the whole coverage process, it is not meal peace based on .coverdata.

To view this discussion on the web visit https://groups.google.com/d/msgid/elixir-lang-core/9ceb51e7-82cf-4edb-8315-cd05df5a1bbfn%40googlegroups.com.