Testing coverage

[Thomas Robitaille]

I just thought I'd share a useful tip I found out this week - if you install the pytest-cov module (http://pypi.python.org/pypi/pytest-cov) and run the Astropy tests with:

    python setup.py test -a '--cov-report html --cov astropy'

Then this will generate a nice HTML report (which will be in build/lib*/) showing which lines of code were executed during the tests (a.k.a coverage). I've uploaded the coverage for the latest version of the Astropy repository to:

    http://www.mpia-hd.mpg.de/~robitaille/astropy/htmlcov/

and you can click on the columns to sort it in different ways. You can also use this on your own forks of Astropy, when developing new functionality.

While I'm by no means advocating we reach 100% coverage, this is very useful to see whether there are significant (in length or importance) situations that are not being tested, or whether there is code that can be deprecated (of course 100% coverage does not mean that *all* the situations are being tested).

Cheers,

Tom

[Erik Bray]

I'm glad you brought this up so that I didn't have to. It's something
I've been meaning to look into for a while. I wasn't sure how to add
coverage support to py.test, so I'm glad that it was so simple.

100% coverage is definitely not a worthwhile goal. For one, it's not
possible in most coverage reporting to turn off inclusion of specific
sections of code. For example, since we have code that should only
run in Python 3 (or code that should only run on Python 2) it's
impossible to ever achieve 100% coverage. I'd say ~85% is a more
realistic/useful goal. (Though I believe it is possible to exclude
certain packages from the report. For example, we might want to
exclude astropy.extern, and exclude the tests themselves).

As you said, test coverage does not by any means prove that you've
tested all possible use cases. But it is helpful in directing where
new tests need to be written, and in tracking down why specific bugs
were *not* caught by your tests.

The next question is, is there some way we can add coverage reports to
Jenkins for each build?

Erik

[Thomas Robitaille]

Hi Erik,

Good point about including the coverage reports in Jenkins. I came
across this thread which might be helpful:

http://lists.idyll.org/pipermail/testing-in-python/2011-February/003827.html

I will try this tomorrow on the Mac builds of Astropy.

Cheers,
Tom

[Michael Droettboom]

You can save build artifacts in Jenkins -- this is how the automatic
documentation build works.

Haven't actually tried this, but it should just be a matter of running
"coverage html" after running the tests, checking the "Archive the
Artifacts" checkbox in the Jenkins build config, and setting it to the
"htmlcov" directory.

Mike

--
Michael Droettboom
http://www.droettboom.com/

[Erik Bray]

On Mon, May 7, 2012 at 12:48 PM, Erik Bray <erik....@gmail.com> wrote:

> 100% coverage is definitely not a worthwhile goal.  For one, it's not
> possible in most coverage reporting to turn off inclusion of specific
> sections of code.  For example, since we have code that should only
> run in Python 3 (or code that should only run on Python 2) it's
> impossible to ever achieve 100% coverage.

I stand corrected. I did install pytest-cov, and it is in fact based
on coverage.py (as it should be). I had some vague recollection that
coverage.py did have some kind of hack for controlling how specific
branches are counted in the coverage report. And I was right:
http://nedbatchelder.com/code/coverage/excluding.html#excluding

I wouldn't go crazy with this, but this can at least be useful for
examples like different code paths for Python2/3.

> I'd say ~85% is a more
> realistic/useful goal.  (Though I believe it is possible to exclude
> certain packages from the report. For example, we might want to
> exclude astropy.extern, and exclude the tests themselves).

coverage.py certtainly allows this, but I can't find anything in
pytest-cov that exposes this functionality. Maybe it's just not in
the documentation.

Erik

[Michael Droettboom]

You can pass a coveragerc file to the --cov-config option. coveragerc
is described here, and I believe it includes the ability to omit
certain files from the report:

http://nedbatchelder.com/code/coverage/config.html

Mike

On Mon, May 7, 2012 at 12:56 PM, Erik Bray <erik....@gmail.com> wrote:
> On Mon, May 7, 2012 at 12:48 PM, Erik Bray <erik....@gmail.com> wrote:
>> 100% coverage is definitely not a worthwhile goal.  For one, it's not
>> possible in most coverage reporting to turn off inclusion of specific
>> sections of code.  For example, since we have code that should only

>> run in Python 3 (or code that should onlyIt run on Python 2) it's

>> impossible to ever achieve 100% coverage.
>
> I stand corrected.  I did install pytest-cov, and it is in fact based
> on coverage.py (as it should be).  I had some vague recollection that
> coverage.py did have some kind of hack for controlling how specific
> branches are counted in the coverage report.  And I was right:
> http://nedbatchelder.com/code/coverage/excluding.html#excluding
>
> I wouldn't go crazy with this, but this can at least be useful for
> examples like different code paths for Python2/3.
>
>> I'd say ~85% is a more
>> realistic/useful goal.  (Though I believe it is possible to exclude
>> certain packages from the report. For example, we might want to
>> exclude astropy.extern, and exclude the tests themselves).
>
> coverage.py certtainly allows this, but I can't find anything in
> pytest-cov that exposes this functionality.  Maybe it's just not in
> the documentation.
>
> Erik

[Erik Bray]

On Mon, May 7, 2012 at 1:22 PM, Michael Droettboom <mdb...@gmail.com> wrote:
> You can pass a coveragerc file to the --cov-config option.  coveragerc
> is described here, and I believe it includes the ability to omit
> certain files from the report:
>
> http://nedbatchelder.com/code/coverage/config.html
>
> Mike

Yup. I just figured that out. I dropped in a .coveragerc containing
the following:

[run]
source = astropy
omit =
astropy/extern/*
astropy/sphinx/*
astropy/*tests/*

That report gave us 64% coverage. There are probably some other
things we might want to omit. Maybe we should include a coveragerc
with the source?

Erik

[Thomas Robitaille]

> [run]
> source = astropy
> omit =
>    astropy/extern/*
>    astropy/sphinx/*
>    astropy/*tests/*
>
> That report gave us 64% coverage.  There are probably some other
> things we might want to omit.  Maybe we should include a coveragerc
> with the source?

+1 to including it. Also, good idea to exclude tests, I was noticing
that the files with the best coverage were the tests themselves, which
is cheating! ;-)

Tom

[Michael Droettboom]

+1 also on including a coveragerc.

Also nice would be another argument to "setup.py test" to perform the
coverage using the coveragerc. I don't think coverage should be the
default as it isn't exactly quick and has a couple external
dependencies. I can implement that.

Mike

[Michael Droettboom]

I have coverage tests up on ShiningPanda as part of the
`astropy-coverage` build. Click on that line, then click on
"coverage.py report" (it turns out ShiningPanda has a little plugin to
create that link -- how handy).

Mike

[Michael Droettboom]

Forgot to paste the direct link:

https://jenkins.shiningpanda.com/astropy/job/astropy-coverage/coveragepy/?

Mike

[Michael Droettboom]

A PR to make this easier/more obvious:

https://github.com/astropy/astropy/pull/227

Mike