[Python-Dev] C code coverage report with lcov

[Christian Heimes]

Hi,

I have done some experiments with GCC's gcov and lcov to get the C code
coverage of our unit test suite. You may find today's report at

http://tiran.bitbucket.org/python-lcov/

I'm working on a patch for our Makefile to include all steps in one
simple make tag. http://bugs.python.org/issue18481

Christian

_______________________________________________
Python-Dev mailing list
Pytho...@python.org
http://mail.python.org/mailman/listinfo/python-dev
Unsubscribe: http://mail.python.org/mailman/options/python-dev/dev-python%2Bgarchive-30976%40googlegroups.com

[Brett Cannon]

On Mon, Jul 29, 2013 at 1:15 PM, Christian Heimes <chri...@python.org> wrote:

Hi,

I have done some experiments with GCC's gcov and lcov to get the C code
coverage of our unit test suite. You may find today's report at

  http://tiran.bitbucket.org/python-lcov/

Thanks!

I took a quick poke around and it seems some things are legitimately not being executed, while others are error conditions that we wouldn't expect to occur (e.g. memory exhaustion). If we ever decide to get serious about code coverage (both C and Python code) we may need to have a discussion as a group about our feelings related to pragmas dictating when code should be left out of coverage reports.

[R. David Murray]

I suspect Victor will eventually have something for us for exercising the
memory exhaustion code :)

--David

[Christian Heimes]

Am 29.07.2013 19:58, schrieb Brett Cannon:
> I took a quick poke around and it seems some things are legitimately not
> being executed, while others are error conditions that we wouldn't
> expect to occur (e.g. memory exhaustion). If we ever decide to get
> serious about code coverage (both C and Python code) we may need to have
> a discussion as a group about our feelings related to pragmas dictating
> when code should be left out of coverage reports.

Yeah, object allocation and creation checks as well as verification of
operating system API call are reason for lots branch and line misses.
I'm not sure what we can do about that. I don't want to plaster the code
with LCOV_EXCL_LINE comments.

As first action we should look into function coverage. We may not be
able to execute every branch but at least we should be able to call and
verify each function.

[Brett Cannon]

On Mon, Jul 29, 2013 at 3:31 PM, Christian Heimes <chri...@python.org> wrote:

Am 29.07.2013 19:58, schrieb Brett Cannon:

> I took a quick poke around and it seems some things are legitimately not
> being executed, while others are error conditions that we wouldn't
> expect to occur (e.g. memory exhaustion). If we ever decide to get
> serious about code coverage (both C and Python code) we may need to have
> a discussion as a group about our feelings related to pragmas dictating
> when code should be left out of coverage reports.

Yeah, object allocation and creation checks as well as verification of
operating system API call are reason for lots branch and line misses.
I'm not sure what we can do about that. I don't want to plaster the code
with LCOV_EXCL_LINE comments.

As first action we should look into function coverage. We may not be
able to execute every branch but at least we should be able to call and
verify each function.

If there's a way to report just function coverage then I think that's a great place to start. 

[Antoine Pitrou]

On Mon, 29 Jul 2013 21:31:02 +0200
Christian Heimes <chri...@python.org> wrote:
> Am 29.07.2013 19:58, schrieb Brett Cannon:
> > I took a quick poke around and it seems some things are legitimately not
> > being executed, while others are error conditions that we wouldn't
> > expect to occur (e.g. memory exhaustion). If we ever decide to get
> > serious about code coverage (both C and Python code) we may need to have
> > a discussion as a group about our feelings related to pragmas dictating
> > when code should be left out of coverage reports.
>
> Yeah, object allocation and creation checks as well as verification of
> operating system API call are reason for lots branch and line misses.
> I'm not sure what we can do about that. I don't want to plaster the code
> with LCOV_EXCL_LINE comments.

Ideally, we should run coverage runs on different systems (Unices,
Windows...) and merge the results together, so that we know which paths
are really uncovered.

Regards

Antoine.

[Christian Heimes]

Am 29.07.2013 21:38, schrieb Brett Cannon:
> If there's a way to report just function coverage then I think
> that's a great place to start.

lcov's genhtml command doesn't support just function coverage. But I
have removed branch coverage. It makes the report a little bit more
readable.

[Christian Heimes]

Am 29.07.2013 19:15, schrieb Christian Heimes:
> Hi,
>
> I have done some experiments with GCC's gcov and lcov to get the C code
> coverage of our unit test suite. You may find today's report at
>
> http://tiran.bitbucket.org/python-lcov/
>
> I'm working on a patch for our Makefile to include all steps in one
> simple make tag. http://bugs.python.org/issue18481

Have fun with

make coverage-report

as documented in the devguide


http://docs.python.org/devguide/coverage.html#measuring-coverage-of-c-code-with-gcov-and-lcov

[Christian Heimes]

Am 29.07.2013 21:48, schrieb Antoine Pitrou:
> Ideally, we should run coverage runs on different systems (Unices,
> Windows...) and merge the results together, so that we know which paths
> are really uncovered.

I don't that is easily possible. The coverage report depends on GCC and
its gcov extension -- so much for Windows. I also don't know if gcov
supports cross-profiling on varying platforms and operating systems.

By the way gcov understands preprocessor output. It doesn't report lines
as uncovered when the lines or functions are #ifdef-ed out.

Christian

[Brett Cannon]

On Tue, Jul 30, 2013 at 8:28 PM, Christian Heimes <chri...@python.org> wrote:

Am 29.07.2013 21:48, schrieb Antoine Pitrou:

> Ideally, we should run coverage runs on different systems (Unices,
> Windows...) and merge the results together, so that we know which paths
> are really uncovered.

I don't that is easily possible. The coverage report depends on GCC and
its gcov extension -- so much for Windows. I also don't know if gcov
supports cross-profiling on varying platforms and operating systems.

By the way gcov understands preprocessor output. It doesn't report lines
as uncovered when the lines or functions are #ifdef-ed out.

In case are curious to try and make this work using clang, I found the flags to use at http://clang-developers.42468.n3.nabble.com/Code-coverage-on-clang-td4033066.html and how to make it work with lcov. I also discovered clang itself uses https://github.com/ddunbar/zcov instead of lcov. This is all untested, but I didn't want to forget the links and in case someone has clang installed with compiler-rt and wants to see how it works.