testing coverage for Cython code

[Tal Einat]

The latest mention of coverage reporting for Cython code I've found was on this list back in August 2014. According to that thread, the discussed development of this feature never took off and the OP wrote a workaround.

Has there been any progress on this front since? I'd really like this for my project, fuzzysearch. I definitely want to know more than which functions were or weren't called.

- Tal Einat

[Stefan Behnel]

Tal Einat schrieb am 19.02.2015 um 16:22:
> The latest mention of coverage reporting for Cython code I've found was on
> this list back in August 2014

> Has there been any progress on this front since? I'd really like this for
> my project, fuzzysearch. I
> definitely want to know more than which functions were or weren't called.

I got line coverage working for Cython, yes. But it requires patching even
the latest coverage.py master (4.0 alpha), so there isn't a coverage.py
version that supports it.

I'm currently in discussion with Ned Batchelder, author of coverage.py, to
address this. We may be able to settle it soon, maybe within a couple of weeks.

Branch coverage may also eventually be supported, but don't hold your breath.

Stefan

[Tal Einat]

Stefan Behnel wrote:

Tal Einat schrieb am 19.02.2015 um 16:22:
> The latest mention of coverage reporting for Cython code I've found was on
> this list back in August 2014
> Has there been any progress on this front since? I'd really like this for
> my project, fuzzysearch. I
> definitely want to know more than which functions were or weren't called.

I got line coverage working for Cython, yes. But it requires patching even
the latest coverage.py master (4.0 alpha), so there isn't a coverage.py
version that supports it.

I'm currently in discussion with Ned Batchelder, author of coverage.py, to
address this. We may be able to settle it soon, maybe within a couple of weeks.

This sounds great! I'd love to hear whether this gets into covergage.py 4.0 (or later).
 

Branch coverage may also eventually be supported, but don't hold your breath.

 
Line coverage would be good enough for me.

- Tal Einat

[Stefan Behnel]

Tal Einat schrieb am 19.02.2015 um 17:13:
> Stefan Behnel wrote:
>>
>> Tal Einat schrieb am 19.02.2015 um 16:22:
>>> The latest mention of coverage reporting for Cython code I've found was
>>> on this list back in August 2014
>>> Has there been any progress on this front since? I'd really like this
>>> for my project, fuzzysearch. I
>>> definitely want to know more than which functions were or weren't
>>> called.
>>
>> I got line coverage working for Cython, yes. But it requires patching even
>> the latest coverage.py master (4.0 alpha), so there isn't a coverage.py
>> version that supports it.
>>
>> I'm currently in discussion with Ned Batchelder, author of coverage.py, to
>> address this. We may be able to settle it soon, maybe within a couple of
>> weeks.
>
> This sounds great! I'd love to hear whether this gets into covergage.py 4.0
> (or later).

Actually, I just tested it again and it seems that a pending bug in the
coverage tracer was fixed now, so latest Cython master + latest coverage.py
master work for me. Just give it a try.

http://blog.behnel.de/posts/coverage-analysis-for-cython-modules.html

Stefan

[Jérôme Kieffer]

On Thu, 19 Feb 2015 17:42:40 +0100
Stefan Behnel <stef...@behnel.de> wrote:

>
> Actually, I just tested it again and it seems that a pending bug in the
> coverage tracer was fixed now, so latest Cython master + latest coverage.py
> master work for me. Just give it a try.
>
> http://blog.behnel.de/posts/coverage-analysis-for-cython-modules.html

This looks great !!!

--
Jérôme Kieffer <goo...@terre-adelie.org>

[Matthew Brett]

Hi,

Thanks a lot for this - I think it will be very useful.

I just tried this on our repo, like this:

$ coverage run dipy/tracking/local/tests/test_local_tracking.py
nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
This tests that the Local Tracker behaves as expected for the ... ok
dipy.tracking.local.tests.test_local_tracking.test_trilinear_interpolate ... ok
This tests that the Probabalistic Direction Getter plays nice ... ok
This tests that the Maximum Deterministic Direction Getter plays nice ... ok

----------------------------------------------------------------------
Ran 4 tests in 0.451s

OK
$ coverage report
Plugin 'Cython.Coverage' did not provide a file reporter for
'/Users/mb312/dev_trees/dipy/dipy/tracking/local/tissue_classifier.pyx'

I have the latest Cython and coverage commits, I believe:

$ coverage --version
Coverage.py, version 4.0a6.
Documentation at https://coverage.readthedocs.org/en/4.0a6

I have linetrace and CYTHON_TRACE enabled.

Did I miss a step somewhere? Sorry if so.

Thanks again,

Matthew

[Stefan Behnel]

Matthew Brett schrieb am 20.02.2015 um 01:28:
> On Thu, Feb 19, 2015 at 8:42 AM, Stefan Behnel wrote:
>> http://blog.behnel.de/posts/coverage-analysis-for-cython-modules.html

>
> I just tried this on our repo, like this:
>
> $ coverage run dipy/tracking/local/tests/test_local_tracking.py
> nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
> This tests that the Local Tracker behaves as expected for the ... ok
> dipy.tracking.local.tests.test_local_tracking.test_trilinear_interpolate ... ok
> This tests that the Probabalistic Direction Getter plays nice ... ok
> This tests that the Maximum Deterministic Direction Getter plays nice ... ok
>
> ----------------------------------------------------------------------
> Ran 4 tests in 0.451s
>
> OK
> $ coverage report
> Plugin 'Cython.Coverage' did not provide a file reporter for
> '/Users/mb312/dev_trees/dipy/dipy/tracking/local/tissue_classifier.pyx'

Ah, yes, I hadn't taken the case of a separate reporting run into account,
only in-process reporting. I pushed a fix to the Cython master branch.

Stefan

[Matthew Brett]

Hi,

Thanks very much - that works very nicely,

Matthew

[Matthew Brett]

A small question. For one module, I get this kind of output:

dipy/reconst/peak_direction_getter.pyx 54 47
0 0 13%

I noticed the warning message during building that nogil code is not
covered. Does the 13% refer to all lines in the code, or just the
covered lines?

Thanks again,

Matthew

[Stefan Behnel]

Matthew Brett schrieb am 20.02.2015 um 17:42:
> On Fri, Feb 20, 2015 at 12:21 AM, Matthew Brett wrote:

I actually started working on that. Not sure if it will work out, but it
should be possible to make nogil code traceable by acquiring the GIL on
each code line (which will certainly result in a massive slowdown, but at
least for coverage analysis, speed is secondary at best). I'll try to give
it a separate C compiler option so that it can be enabled only at need.

> Does the 13% refer to all lines in the code, or just the covered lines?

The reference is currently the total number of source code lines for which
code was generated at all, i.e. basically all lines that have a '+' prefix
in the "cython -a" HTML page. That might even be a bit higher than the
number of lines for which code is ever *executed*, e.g. struct definitions
or ctypedefs have their own source line. It's all just emulated, and
there's certainly room for improvement.

I haven't tried the coverage report as HTML yet, would be cool to get that
in shape. The code is in Cython/Coverage.py, any help is appreciated.

Stefan

[Stefan Behnel]

Stefan Behnel schrieb am 20.02.2015 um 18:15:
> it should be possible to make nogil code traceable by acquiring the GIL on
> each code line (which will certainly result in a massive slowdown, but at
> least for coverage analysis, speed is secondary at best). I'll try to give
> it a separate C compiler option so that it can be enabled only at need.

I think I got it working. Please give it a try.

https://github.com/cython/cython/commit/3b6ec99c082d603161e9322846283d6138b951bc

To use it, you need to set the C compiler macros

-D CYTHON_TRACE=1
-D CYTHON_TRACE_NOGIL=1

(although I guess it should be enough if users only set the latter...)

Stefan

[Matthew Brett]

On Fri, Feb 20, 2015 at 11:04 AM, Stefan Behnel <stef...@behnel.de> wrote:
> Stefan Behnel schrieb am 20.02.2015 um 18:15:
>> it should be possible to make nogil code traceable by acquiring the GIL on
>> each code line (which will certainly result in a massive slowdown, but at
>> least for coverage analysis, speed is secondary at best). I'll try to give
>> it a separate C compiler option so that it can be enabled only at need.
>
> I think I got it working. Please give it a try.
>
> https://github.com/cython/cython/commit/3b6ec99c082d603161e9322846283d6138b951bc
>
> To use it, you need to set the C compiler macros
>
> -D CYTHON_TRACE=1
> -D CYTHON_TRACE_NOGIL=1

Thanks very much for this - that will make the coverage much more
useful for us (we spent a long time no-gilling our code, as you can
imagine).

Running on current master (1d926e8), with those flags applied, I get a
large number of errors like:

======================================================================
ERROR: dipy.denoise.tests.test_nlmeans.test_nlmeans_padding
----------------------------------------------------------------------
Traceback (most recent call last):
File "/Users/mb312/.virtualenvs/test/lib/python2.7/site-packages/nose/case.py",
line 197, in runTest
self.test(*self.arg)
File "/Users/mb312/dev_trees/dipy/dipy/denoise/tests/test_nlmeans.py",
line 14, in test_nlmeans_padding
S0n = add_padding_reflection(S0, 5)
File "dipy/denoise/denspeed.pyx", line 201, in
dipy.denoise.denspeed.add_padding_reflection
(dipy/denoise/denspeed.c:3781)
cnp.npy_intp [::1] indices_i = correspond_indices(arr.shape[0], padding)
File "dipy/denoise/denspeed.pyx", line 216, in
dipy.denoise.denspeed.correspond_indices
(dipy/denoise/denspeed.c:4294)
return np.ascontiguousarray(np.hstack((np.arange(1, padding + 1)[::-1],
AttributeError: 'bool' object has no attribute 'trace'

This is on our 'dipy' project. I know this is very much not a minimal
example, but just in case you want to replicate:

git clone git://github.com/matthew-brett/dipy.git
cd dipy
git checkout -t origin/cython-coverage
pip install -r requirements.txt
export DIPY_LINETRACE=1
python setup.py build_ext -i
nosetests --with-coverage --cover-package=dipy dipy

Thanks again,

Matthew

[Stefan Behnel]

Matthew Brett schrieb am 24.02.2015 um 01:53:

Did you also update coverage.py to latest? And rebuild its tracer? The
plugin API is still in flux, so there will likely be further changes on
both sides down the road. Cython is amongst the first providers of a
coverage plugin. The fact that the API was there when I started working on
the integration was pure coincidence.

Stefan

[Matthew Brett]

A happy coincidence for many Cythonizers ...

Just to check, I ran `hg purge` on the coveragepy source tree and
reinstalled coverage, but I'm still getting these errors. This is
the changeset 2020:7067350052e3 from Feb 22nd (for coveragepy).

There's only one unrelated test failure with the LINETRACE options disabled.

Is there another useful test I could do to diagnose?

Matthew

[Brandon Cazander]

Just adding in that I'm receiving the same exception when running coverage (@7067350).

Traceback (most recent call last):

  File "tests/tunnel_tests/test_runner.py", line 4, in <module>
    from tests.tunnel_tests.controlpacket_test import PingRequestPacketTestCase
  File "/usr/src/bi/bonding/tests/tunnel_tests/__init__.py", line 3, in <module>
    from controlpacket_test import *
  File "tests/tunnel_tests/controlpacket_test.pyx", line 121, in init tests.tunnel_tests.controlpacket_test (tests/tunnel_tests/controlpacket_test.c:8969)
    test_data = data_from_hex(
  File "tests/tunnel_tests/controlpacket_test.pyx", line 21, in tests.tunnel_tests.controlpacket_test.data_from_hex (tests/tunnel_tests/controlpacket_test.c:4578)
    return s.replace(' ', '').decode('hex')
  File "/usr/lib/python2.6/encodings/__init__.py", line 100, in search_function
    level=0)

AttributeError: 'bool' object has no attribute 'trace'

Let me know if I can provide anything to help troubleshoot this.

Cheers,

Brandon

[Stefan Behnel]

Stefan Behnel schrieb am 20.02.2015 um 18:15:

> I haven't tried the coverage report as HTML yet, would be cool to get that
> in shape.

That works now. I also integrated it into the annotated source HTML, so you
can now create an XML coverage report and pass it to "cython -a" with
"--annotate-coverage=coverage.xml" to get green and red markers on the
source code lines.

Stefan

[Stefan Behnel]

Matthew Brett schrieb am 24.02.2015 um 01:53:

> Running on current master (1d926e8), with those flags applied, I get a
> large number of errors like:
>
> ======================================================================
> ERROR: dipy.denoise.tests.test_nlmeans.test_nlmeans_padding
> ----------------------------------------------------------------------
> Traceback (most recent call last):
> File "/Users/mb312/.virtualenvs/test/lib/python2.7/site-packages/nose/case.py",
> line 197, in runTest
> self.test(*self.arg)
> File "/Users/mb312/dev_trees/dipy/dipy/denoise/tests/test_nlmeans.py",
> line 14, in test_nlmeans_padding
> S0n = add_padding_reflection(S0, 5)
> File "dipy/denoise/denspeed.pyx", line 201, in
> dipy.denoise.denspeed.add_padding_reflection
> (dipy/denoise/denspeed.c:3781)
> cnp.npy_intp [::1] indices_i = correspond_indices(arr.shape[0], padding)
> File "dipy/denoise/denspeed.pyx", line 216, in
> dipy.denoise.denspeed.correspond_indices
> (dipy/denoise/denspeed.c:4294)
> return np.ascontiguousarray(np.hstack((np.arange(1, padding + 1)[::-1],
> AttributeError: 'bool' object has no attribute 'trace'

Yes, that's a bug in the current tracer of coverage.py. I already reported
it to the author, but I guess it just got out of sight. I'll ping him again.

Stefan

[Matthew Brett]

Hi,

I just returned to this, but now, with latest tip of coveragepy, I am getting:

"Plugin module %r didn't define a coverage_init function" % module

coverage.misc.CoverageException: Plugin module 'Cython.Coverage'
didn't define a coverage_init function

I guess the API changed between version 4.0a6 and current hg tip of coveragepy?

Thanks again,

Matthew