disable "Some Quality Gate conditions on New Code were ignored because of the small number of New Lines"

[Roman Ivanov]

Hi ,

we got a info note " Some Quality Gate conditions on New Code were ignored because of the small number of New Lines"

and this allowed some tech dept leak to happen that suppose to be resolved on the moment of implementation and how it hard to resolve it.

We use Java project base on maven build.

Is there a way to never allow Sonar to ignore gate after version bump? no leak period is allowed ....

https://github.com/SonarSource/sonarqube/blob/master/sonar-core/src/main/resources/org/sonar/l10n/core.properties#L2286

https://github.com/SonarSource/sonarqube/blob/master/server/sonar-server/src/main/java/org/sonar/server/qualitygate/QualityGateEvaluatorImpl.java#L40

https://github.com/SonarSource/sonarqube/blob/master/server/sonar-server/src/main/java/org/sonar/server/qualitygate/QualityGateEvaluatorImpl.java#L94

out version:

SonarQube™ technology is powered by SonarSource SA

Version 6.7.1 (build 35068) - LGPL v3 - Community - Documentation - Get Support - Plugins - Web API - About

thanks,

Roman Ivanov

[Roman Ivanov]

Leak Period=previous_version

But we do not see how to set line numbers(MAXIMUM_NEW_LINES_FOR_SMALL_CHANGESETS) to 0.

[G. Ann Campbell]

Hi Roman,

This is not currently configurable. The reason it was introduced is because of experiences we had internally with failing quality gates with for example 1 changed, uncovered line. In this case, Coverage on New Code was 0% while covering that 1 (or 3 or 7) line(s) of code was a case of diminishing returns. 

You may be interested in SONAR-10231 which is designed to "improve" the way this feature behaves, although on review I see that from your perspective it's probably just the opposite of an improvement because it would allow larger change sets through the filter.

Can you share more about your use case and why you'd like to turn this feature off?

Ann

[Roman Ivanov]

Hi Ann,

https://docs.sonarqube.org/display/SONAR/Metric+Definitions

Lines to cover	lines_to_cover	Number of lines of code which could be covered by unit tests (for example, blank lines or full comments lines are not considered as lines to cover).
Lines to cover on new code	new_lines_to_cover	Identical to Lines to cover but restricted to new / updated source code.

looks like new_lines_to_cover will help.

Our workflow is simple - no leak periods, no violation, no uncovered code/conditions on each commit.

Very strict but such workflow works well, no merge to master till all work is done up to our quality standard.

But right now, leak happening as somebody do small changes and cover not all conditions in code, fix is merged, time is passed.

Some other engineer do bigger update and now validation did not pass, because previous engineer did a incomplete job, coverage of unrelated code cost a lot, and result in multiple distractions for other engineers.

We use simple principle: all changes(on each commit) should follow quality rules.

All other approaches just cause technical dept to grow and cause leaks that are not always possible to resolve due to business priorities.

Do you know a workaround ? 

is it possible to move such hardcoded constant to property to be defined from config file ?

we probably can provide PullRequest, as we have to hack SQ , to make our gates works.

thanks,

Roman Ivanov

[G. Ann Campbell]

Hi Roman,

After preliminary internal discussions I can tell you we're not likely to act on this directly. Instead, it will probably be addressed gradually as we flesh out Branch support. MMF-1118 adds Coverage and Duplication metrics to short-lived branches. We want to add "real" quality gates as well, but there's not even an MMF for that yet.

At the same time, I'd like to better understand your workflow. It seems that you ... do a full (sonar.branch=xxx ?) analysis on each ... commit(?) to ... a feature branch(?) with a very rigorous Quality Gate, which must be passed before merge is allowed?

Thx,

Ann

[Roman Ivanov]

Hi Ann,

thanks a lot for desire to understand a problem, I appreciate this.

On new project, without any legacy and existing tech dept.
We started new Sonar workflow: on each commit to master, no violations, no uncovered code at all for lines and conditions (complete 100% coverage).
It worked for us, there was some unexpected problems with unstable coverage detection that we did have chance to reproduce.
But now we caught a reason of problem.

Use case:
Project1 with some state on Sonar. Gate is passed, all code is covered.
User1 do small commit to master, sonar passed, all happy, User1 switched to another project1.
User2 do mid size update to project1, and it expects to resolve problems only from his part code, BUT Sonar now reveal problems in code from user1, user1 already on another task/project eh cannot come back to quickly fix.

So all that technical dept should be resolved by user2, that is a pain, that is unfair, it could take too much time.

Sonar should be configurable to define workflow. If some team are ok with tech dept  - it is ok, special config.

If some teams do not allow any tech dept (no leak period) - it should be possible.

Problem is in this hardcoded variable value. 

So we suggest as workaround (before you do all completely correct) to allow users to override this value from property in property file.

It could be even not well documented as you will not support it in long run and could remvoe at any time as you got better fix.

In same time it will allow teams to set up gates to Sonar to be strict in enforcing NO tech dept.

thanks,
Roman Ivanov

[Roman Ivanov]

Hi Ann,

> It seems that you ... do a full (sonar.branch=xxx ?) 

we use community Sonar for now, No branch support in it so we do some ugly hack (inconvenient) in sonar execution to validation from branch and affect main/master branch statistics.

Ones we will find that Sonar can enforce what we need, we might switch to Sonar version that support branching.

Gates for branches and master will be the same strict.

> analysis on each ... commit(?) 

yes

> to ... a feature branch(?) 

as implementation is done in branch, engineer do verification from branch against Sonar gate of master.

> with a very rigorous Quality Gate, which must be passed before merge is allowed?

yes.

thanks,

Roman Ivanov

[G. Ann Campbell]

Hi Roman,

Thanks for taking the time to explain. I wish I could reciprocate by telling you we'll give you a way to turn this behavior off, but we're not planning that. At least not in the short term. I can, however offer you some workarounds:

First, even though the QG condition is disabled when the changeset is small, the metric calculation is not, so you can check the metric values before allowing commit. Yes, I truly "get" how frustratingly redundant that seems (I feel a little silly even typing it) when the QG itself is supposed to be doing that. This double-check would be automatable using web services.

Another option is to use the QG on the main project as a backstop. The "ignore" behavior only kicks in when the changeset is small, and I'm assuming that your cumulative changeset between releases is large enough to tip over the threshold. So if you reanalyze the master branch after commit/merge, you could at least catch any issues before the developer moved on to the next project. 

Out of pure curiosity, is your industry safety- or health-related?

Ann

---

G. Ann Campbell | SonarSource

Product Manager

@GAnnCampbell

https://sonarsource.com

--
You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/qxQLr7OAioA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/3a01092b-07b0-4468-ac88-9e386cca84c4%40googlegroups.com.

For more options, visit https://groups.google.com/d/optout.

[Roman Ivanov]

Hi Ann,

thanks for workarounds...

Can you share here some issue that we can refer in future ?

We will wait when QulityGate actually become follow what user is set (no hidden and non modifiable allowances).

Looks like our workflow is  very unusual to you. 

Can you share what is Sonar team vision on enforcing no leak to technical dept ? (automatic build failure, no manual recheck of all metrics). 

Hot-fix release could still be one line change.

> This double-check would be automatable using web services.

This is good workaround. But it will demand to have security token to be placed in GIT or Jenkins or any other open for read to most resources.

As extra scripting is required, it is better to do this in other way .... see below. 

> Another option is to use the QG on the main project as a backstop.

We practice almost-continuous deployments, so version bump (in build number) is very frequent(each 1-3 days), 

NO leak on each commit is the only way out.

> Out of pure curiosity, is your industry safety- or health-related?

we are in financial sector.

===========================

As we can NOT rely on QualityGate in Sonar for coverage metrics .... the better approach for us is to move "build failure Gate" to maven build process level and enforce it by jacoco maven plugin which is actually create coverage statistic for Sonar.

Sonar will be just to show statistics in fancy view. 

thanks,

Roman Ivanov

[G. Ann Campbell]

Hi Roman,

Thanks for pushing on this topic. I'm going to come back to you soon with a ticket.

Ann

---

G. Ann Campbell | SonarSource

Product Manager

@GAnnCampbell

https://sonarsource.com

--

You received this message because you are subscribed to a topic in the Google Groups "SonarQube" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/sonarqube/qxQLr7OAioA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to sonarqube+unsubscribe@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/9245f20d-bab3-433b-aff2-68ba1a288adc%40googlegroups.com.

[G. Ann Campbell]

Hi Roman,

I've created https://jira.sonarsource.com/browse/SONAR-10485. It's unscheduled for now. We want to see what other feedback we get on the topic.

Ann