Python code coverage fails to handle multiline statements

ttei...@gmail.com

unread,

Nov 28, 2017, 3:48:48 AM11/28/17

to SonarQube

Hello,

when I analyze code coverage in a Python file with expressions that cover multiple lines (e.g. having a newline after the parenthesis of a function call and then arguments on the following lines) code coverage does not behave as expected:

1. Each line of the expression is counted as a separate line instead of one line for the whole expression (this may be a wrong expectation on my side).

2. Only the first line is marked as covered, the following lines of the expression are marked as not covered. The (first) line of the next statement is covered again.

The coverage report .xml file was created using py.test (for Python 2.7).

Is this the right place to report such a bug or is there another one? If you need additional information, please let me know.

Thank you,

Tim

Pierre-Yves Nicolas

unread,

Nov 28, 2017, 8:46:58 AM11/28/17

to ttei...@gmail.com, SonarQube

Hi Tim,

I set up a little test. Here's what I see:

Line 5 is not neither green nor red. That seems correct to me.

Do you have something similar?

Pierre-Yves

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.
To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/8b691c3b-3442-43e6-b0d6-abf8f28591de%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

step...@drive.ai

unread,

Dec 13, 2017, 5:29:47 PM12/13/17

to SonarQube

I've filed an other report for the same bug:

https://groups.google.com/forum/?utm_medium=email&utm_source=footer#!searchin/sonarqube/coverage%7Csort:date/sonarqube/NSRi-ej_uaw/A2TFCKbNAgAJ

Here is was SonarQube reports vs Jenkins parsing the same coverage report file where you can see how lines 104 and 105 are reported as uncovered instead of being neutral:

On Tuesday, November 28, 2017 at 5:46:58 AM UTC-8, Pierre-Yves Nicolas wrote:

Hi Tim,

I set up a little test. Here's what I see:

Line 5 is not neither green nor red. That seems correct to me.
Do you have something similar?

Pierre-Yves

On 28 November 2017 at 09:48, <ttei...@gmail.com> wrote:

Hello,

when I analyze code coverage in a Python file with expressions that cover multiple lines (e.g. having a newline after the parenthesis of a function call and then arguments on the following lines) code coverage does not behave as expected:

1. Each line of the expression is counted as a separate line instead of one line for the whole expression (this may be a wrong expectation on my side).
2. Only the first line is marked as covered, the following lines of the expression are marked as not covered. The (first) line of the next statement is covered again.

The coverage report .xml file was created using py.test (for Python 2.7).

Is this the right place to report such a bug or is there another one? If you need additional information, please let me know.

Thank you,
Tim

--
You received this message because you are subscribed to the Google Groups "SonarQube" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

step...@drive.ai

unread,

Dec 18, 2017, 1:13:18 PM12/18/17

to SonarQube

I'm guessing that the bug could be related to SonarPython still using the 5.6 API: https://jira.sonarsource.com/browse/SONARPY-235

My reasoning is that the LineOfCode definition seem different in SonarQube 6 vs 5, at least how I remember it.

jacki...@gmail.com

unread,

Dec 18, 2017, 1:26:16 PM12/18/17

to SonarQube

Hi Guys,

We are seeing a similar problem with coverage calculation for multi-line statements in Python 3. I.e. individual statements that span N lines would show a 1 green line and N-1 red lines.

We did not have this problem in Sonarqube 5, appears to be a regression post Sonarqube 6 upgrade.

This is starting to cause our developers to lose faith in coverage statistics - it would be really great if this can be looked into at some point.

Thanks,

Jackie

Pierre-Yves Nicolas

unread,

Dec 21, 2017, 11:05:15 AM12/21/17

to jacki...@gmail.com, SonarQube

Hi,

I was not able to reproduce the problem.

I set up a tiny project, generated a coverage.xml and ran an analysis with SonarQube scanner.

In SonarQube (6.7), I see:

Which matches the content of coverage.xml:

<lines>

</lines>

</class>

What do you see in your coverage.xml file?

Thanks,

Pierre-Yves

--

You received this message because you are subscribed to the Google Groups "SonarQube" group.

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/sonarqube/b2504ff6-ffc9-4e79-b6e3-fa606cf167ad%40googlegroups.com.

ad...@drive.ai

unread,

Jan 9, 2018, 8:45:02 PM1/9/18

to SonarQube

Hello Pierre-Yves:

I've tried to reproduce the issue here. Screenshot:

sonarqube version: 6.7.0.33306

coverage version:

pip freeze | grep coverage -> coverage==3.7.1

pytest output excerpt:

2018-01-09 17:09:46 ============================= test session starts ==============================

2018-01-09 17:09:46 platform linux -- Python 3.5.2, pytest-3.0.7, py-1.5.2, pluggy-0.4.0 -- /usr/bin/python3

2018-01-09 17:09:46 cachedir: .cache

2018-01-09 17:09:46 rootdir: /var/tmp/test_offboard_jenkins, inifile: pytest.ini

2018-01-09 17:09:46 plugins: pyfakefs-3.2, cov-2.5.1, timeout-1.2.0, factoryboy-1.3.1, xdist-1.16.0

2018-01-09 17:09:46 timeout: 300.0s method: signal

...

2018-01-09 17:10:36 platform_eng/coverage/tests/test_canary.py::TestCanary::test_covered_scalar PASSED

2018-01-09 17:10:36 platform_eng/coverage/tests/test_canary.py::TestCanary::test_covered_vector PASSED

...

2018-01-09 17:11:26 Name Stmts Miss Branch BrPart Cover Missing

2018-01-09 17:11:26 ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

...

2018-01-09 17:11:26 platform_eng/__init__.py 0 0 0 0 100%

2018-01-09 17:11:26 platform_eng/coverage/__init__.py 0 0 0 0 100%

2018-01-09 17:11:26 platform_eng/coverage/canary.py 8 2 0 0 75% 20-24

2018-01-09 17:11:26 platform_eng/coverage/tests/test_canary.py 13 1 2 1 87% 19, 18->19

coverage.xml excerpt:

<lines>

</lines>

</class>

Please let me know if I can get you any additional context or information.

Thanks!

Adam

To unsubscribe from this group and stop receiving emails from it, send an email to sonarqube+...@googlegroups.com.

alban....@sonarsource.com

unread,

Jan 10, 2018, 1:00:17 PM1/10/18

to SonarQube

Hi Adam,

I found a bug in SonarPython 1.8 that is related to your problem.

Can you confirm that you have the property "sonar.python.coverage.forceZeroCoverage" explicitly set to "true" in your context?

The bug that I found was introduced by the usage of SonarPython on SonarQube 6.2, that deprecate CoverageType (UNIT, IT, OVERALL) and require the plugin to merge all the coverage reports in one.

And the bug only occurs when "forceZeroCoverage" is true, that's why I'm asking.

I'm currently fixing this problem of compatibility with SonarQube >= 6.2 and Coverage.py <= 4.4.2 in SONARPY-227

It should be in the upcoming SonarPython 1.9

Alban

ad...@drive.ai

unread,

Jan 10, 2018, 1:42:56 PM1/10/18

to SonarQube

Hi Alban,

You're right, sonar.python.coverage.forceZeroCoverage is set to true, and SonarPython version is 1.8 (build 1496). Setting it to off gives us the expected multiline statement behavior:

But we generally prefer to have uncovered source count as uncovered. Looking forward to SonarPython 1.9. Hope that helps!

Thanks

Adam

G. Ann Campbell

unread,

Jan 10, 2018, 1:49:41 PM1/10/18

to SonarQube

Hi Adam,

When SONARPY-179 is addressed, you'll get the behavior you want. Feel free to vote! :-)

Ann

Reply all

Reply to author

Forward