DotCover Results Disparity

[Paul Schoolden]

Hi, I'm moving this from the GitHub as I accidentally posted it in the Sonar Lint Issues as was hoping that someone may have some insight.

Description

We have been using dotCover which feeds into our Sonar Analysis for a little while without issue. To help resolve an issue we had with Sonar Lint we updated the the Sonar C# plugin last week to version 6.6. Immediatley after this update our code coverage metrics drastically changed. We did have coverage of 93% which then dropped to 73%. We utilize a dot cover configuration xml file to detail any module and class filter masks, as there is a reasonable amount of configuration classes or data access classes which are not covered by unit tests we need to filter these types out to not affect the actual coverage metric. On face value it appears that a lot of our filters are no longer always being taken into consideration in Sonar,where as in the past they always were and missing coverage for the filtered types was not reported.

As a first step I looked at the outputted dotCover HTML report (the file fed into Sonar) and it was detailing the same coverage as we were expecting (93% overall) and was not reporting classes that had been masked in the configuration file. I then tried a different version of dotCover, we were using 2017.1.3 on the build agent which was upgraded to 2017.2.2 which does generate a sligtly different report but the result in Sonar was the same, 73% opposed to the 93% shown in the dot cover report.

The image below shows the html report outputted by dotCover:

The image below shows the coverage stat in Sonar:

The image below shows our test coverage history in Sonar, as you can see, apart from a dip caused by a bad check in, after we configured our class masks our coverage stat was quite high and stable until the final analysis which is the date that we updated the plugin. This may be unrelated as there was around two weeks where that code base had not been worked on and analysed but it seems coincidental that it occurred the day that the plugin was updated along with fact that the coverage report from dot cover seems the same as before.

Repro steps

Run dotCover as part of the analysis and feed the output HTML report into Sonar.

Expected behavior

The test coverage metrics match the output of the dotCover report.

Actual behavior

The coverage metrics differ to the output of dotCover with missing coverage detailed for classes excluded by dot cover in the dot cover configuration file.

Known workarounds

None known.

Related information

VS 2017 - V15.2,
Sonar Lint V3.7.0.2645
Sonar Qube V6.4.0.25310.
Sonar C# Plugin V6.6

Tested with same results using dotCover:
dotCover V2017.1.3
dotCover V2017.2.2

Thanks

Paul

[G. Ann Campbell]

Hi Paul,

Believe it or not, this really is a feature, not a bug.

I'm assuming you upgraded SonarC# to 6.6, which introduces Executable Lines. The point of this new metric is to understand which lines are executable and therefore testable. For instance a Java Interface contributes to Lines of Code, but it's not itself executable. (Sorry I can't give you a C# example, but I'm sure the idea translates.)

Without executable lines, when we calculate coverage %, we can go only by what's reported from your coverage engine. For instance, assume I have 2 files, each with 100 executable lines. File A is 100% covered and file B has no tests at all. Since only A will be represented in the coverage report, then without executable lines, we can only calculate coverage at 100%. Really though it's 50%.

With the addition of Executable lines, then the denominator for the coverage % calculation becomes all the lines represented in your coverage report + all the Executable Lines from files omitted in your coverage report.

At this point, you probably understand what's going on. Your filters are still working to exclude the files you omit on purpose from testing, but Executable Lines is "fixing" that for you.

To get back to what you expect, set coverage exclusions on those files you really do want to ignore.

Ann 

[Paul Schoolden]

Hi Ann,

 

Many thanks for your reply, it is greatly appreciated. It’s not a surprise to me, I had wondered if Sonar was doing some kind of intersect on the coverage results from dotCover with what it had deemed as executable lines of code. Apologies, I should have read the documentation in depth to understand that this was actually the case. Whilst I think I can understand the value of this feature, sadly, this is not desired functionality for us.

 

What we are trying to achieve is to have friction-less analysis with local and server results to match as closely as possible. The reason we are using Sonar Lint is so that if an engineer writes some code which is deemed as a code smell they have immediate feedback which will match the build server’s analysis. One reason that we have not yet fully embraced Sonar is that we were waiting for an upgrade to Sonar Lint for it to sync with resolved issues so that the code smells reported by Sonar Qube and Sonar Lint matched one to one. We have been able to mostly achieve the concept with test coverage up to now. Before an engineer commits code we want them to have the option to run dotCover to see if they have impacted test coverage and we wanted that to match any server analysis. To achieve that with dotCover we used the xml config file which is used by the console runner locally and the console runner on the server.

 

We wanted to use dotCover in the IDE but sadly it did not use the xml file for coverage exclusions as it uses its own settings file instead which cannot be used by the console runner. We did not want to have to maintain multiple places for exclusions as this increases possible drift and friction for the developers which will ultimately lead to apathy to the tooling and process. We have used a Sonar analysis exclusion on a module that is both non-test and non-production code but this is definitely the exception and not the rule. I really want to avoid having to manage exclusions in dotCover and in Sonar, especially as this aspect was working perfectly for us until we upgraded the Sonar Plugin, in the past if dotCover reported 0% coverage for a type then the Sonar 65% coverage rule was reported to the team until that was either covered or the dotCover configuration file was amended. I’m not sure if I am missing something?

 

Hope that all makes sense.

 

Many Thanks

 

Paul

[devesh.a...@gmail.com]

Hello Ann,

We are facing the same issue now with SonarCloud and the problem on top of this is if we exclude using Analysis Scope we have same directory names in two projects.

and we just want to exclude one whole project from analysis.

Any Idea how to achieve it ?

@Sorry Paul, for posting this in between of your query as It seems we are in same boat :(

Regards,

Devesh

[Paul Schoolden]

HI,

Does anyone have any further insight on this? At the moment this feels like a breaking change, we were so close to fully embracing SonarQube & SonarLint across all of our teams. We thought the SonarLint update was the final hurdle but this issue has derailed our plans somewhat which is quite frustrating. Is there any chance of adding an option to toggle the new behaviour off?

Many Thanks

Paul

[Paul Schoolden]

Hi Ann,

Are you or you aware of anyone else who could respond to our query? I'm keen to understand if it will be possible for us to have the code coverage working again is was prior to the applied update. I understand the concept detailed in your previous post but we really do not want to managing multiple exclusion definitions, we were happy with how code coverage metrics were being reported locally and through SonarQube when using dotCover in the past. I do find it a little strange that a new feature like this was deployed without user control to disable it.

Many Thanks

Paul

On Thursday, November 30, 2017 at 2:01:32 PM UTC, G. Ann Campbell wrote:

[michal....@sonarsource.com]

Hi Paul,

An option more convenient than maintaining two lists is to mark the code you don't want to cover with ExcludeFromCodeCoverage attribute. That gets you consistent results across the board:
- Code on SonarQube won't be marked as uncovered.
- dotCover in the IDE understands this attribute with no extra configuration (as you pointed out, this does not seem configurable anyway)
- dotCover command line runner won't report it as uncovered either (although you need to manually configure exclusion filter for that attribute)

Thanks,
Michal

[Wilhelm S. Teinitz]

Hi,

first of all let me say that SonarQube does a good job in playing the gatekeeper and saves us a lot of time - that's why we try to establish it in our company.

It looks like people behind this decision to implement this feature don't share our views about what to write unit tests for and how .NET software architectures are set up.

In this thread Ann wrote that this "feature" was introduced because otherwise SonarQube "can only go by what's reported by the coverage engine". And that's the point - SonarQube should only go by what the coverage tools reports because the developers tell the coverage tool where business logic can be found and where not.

The point in using code coverage metrics is to measure how much of our (business) logic is covered by unit tests - best practice in .NET software dev suggests using a layered application architecture where each layer is a separate project/assembly.

A layer used for the domain model may only contain classes & simple getters/setters - do we want to test the c# = operator? Do we want to test abstractions layers who only do routing and enable mocking? Unit tests for a data access layer? Test Unit tests? Test variable declarations (which are also executable lines according to the docs)?  The answer is the same to all questions: no.

No one ever said test each line or test 80% of all executable lines - the point of unit testing is testing logic, not c#, not .NET or any other library.

That's why msbuild/vs (and all the other tools out there) allows to include/exclude whole projects/layers/assemblies with a runsettings-file. And that's why those tools don't report this files to you - and SonarQube C# understood, until 6.6.

According to SonarQube docs code coverage results get imported - that's the intention: import results, don't reinterpret them.

Currently there are about 40 software projects in our company being watched by SonarQube - before our fatal decision to update to > 6.6 all of them were shown with the correct coverage - afterwards everything is broken.

We get statistics both from Visual Studio Online - which of course if our first stop in checking the builds - and from SonarQube - VisualStudioOnline shows > 70% and SonarQube < 30 % - of course we expect both tools to show the same value (that's the point of importing after all).

Conclusio: we have to disable code coverage and associated quality gates in SonarQube - this update is a heavy set-back in our efforts to establish SonarQube as gatekeeper. Companies don't like such surprises.

This way of breaking common functionality, redefining "import" and redefining "code coverage" is a real show stopper.

Please fix it - I would suggest to simply import the code coverage results and add an checkbox to activate your fancy feature, but disable it by default. Then all current SonarQube installations would work correctly as before and whoever would want to experiment with this bug/feature could enable it at his/her own risk.

Meanwhile we'll try to limit the damage.

A Visual Studio Solution is a "project" in SonarQube, a project/assembly is a "component". Are you sure that the paths for file exclusions are relative to the Sonar QubeProject and not to the component? In the latter case file exclusions would be no option, in the firstcase the component name (the name of the folder containing the .net assembly) would be the name of the first folder, am I right? Because we don't get this to work.

Using ExcludeFromCodeCoverage seems like the better solution as it works with any tool, although rather clumsy considering that the development environment already knows which assemblies to ignore.

In .NET Core 2 there also seems to be an assemblyattribute for ExcludeFromCoverage ("google told me so", did not test it so far) - if I target .NET Standard 2.0 and use this option does SonarQube understand this? This would at least ease our life in newer projects,

Best regards & thanks in advance,

 Will

[Paul Schoolden]

Hi,

We can use ExcludeFromCodeCoverage however, we as a group of teams in our platform (five teams out a company wide total of around 30) decided against this approach many months ago, for two main reasons:

1. We agreed to avoid littering our code with attributes. This is not a big issues but we did all agree that this was something we would avoid, some people were more vocal than others but ultimately we agreed to use dotCover exclusions as it kept this concern out of our code and in the actual code coverage tool of our choice. This is a subjective opinion and choice, not every will agree but ultimately it was our decision and we do not really want to be forced to revert this decision simply because we upgraded a tool to a new version.

2. The attribute is not exactly sophisticated, we have much better control with exclusions in the dotCover XML file with namespace exclusions and wildcard uses. We found that this approach was working perfectly for us both locally and on Sonar.

Our main concern here is that from our perspective this seems like a breaking change, when I upgraded the plugin I was not expecting our code coverage stats to change at all, let alone the amount it did change. We appreciate new features being added, even if we are not getting the benefit from them but others are. However, we would not expect a new feature to be compulsory and break our use case, it seems strange that a toggle was not included with this change. As I have already stressed, the frustrating aspect is that we were waiting for a few months for a fix to Sonar Lint to allowing the syncing of resolutions before we started using Sonar Lint and Sonar Qube in anger and just as we thought we had that aspect solved we hit this issue after the mandatory update.It is key for us for developers to have the fastest analysis feedback possible and for there to be the same results locally as on a server.

Many Thanks

Paul

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/lz9lz3jVWyE/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/cb2f3cab-5130-4d0f-8a50-f7b2e4bbdeaf%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[michal....@sonarsource.com]

Hi,

We had a long internal discussion, weighted the solutions and decided not to change the current model.

The executable lines metric should be calculated for all plugins. We at SonarSource do not want to provide a way to opt-out of that.

We considered options such as parsing the exclusions file and applying the exclusions ourselves. However, current scanning pipeline makes that change quite hard, because our exclusions are file-based, but the dotCover exclusions format is code syntax-based (attributes/classes/namespaces) rather than file based, and as you know the two aren't always the same thing. Trying to translate would require quite a lot of work. We feel that time can be spent better on providing value in other areas.

To recap, there are two ways to exclude code from coverage:
- Use ExcludeFromCodeCoverage attribute.
- Specify exclusion list in SonarQube UI (file path-based exclusions).

Thank you for raising this feedback. Even though the solution may not be what you'd ideally want, we are listening.

The way the metrics are calculated probably wouldn't be much of an issue if it was done from the start and not introduced without warning. We apologize for not communicating it clearly. We will put more effort to avoid similar problems in the future.

Thanks,
Michal

To unsubscribe from this group and all its topics, send an email to sonarqube+...@googlegroups.com.

[Grégory Bittan]

I have a similar issue apart from i'm not using exclusion in my DotCover configuration but only inclusions so Sonarqube should understand which files to interpret or not.