Using coverage with Cython

[CFK]

Hi all, I'm slowly working on my code for my project, including unit tests.  I'm using nose's coverage plugin to make sure that I've covered all of my code, but it doesn't appear to work correctly with cython.  I found a blog post that gives a patch to coverage to make it work with cython, but that post is from 2010, so I'm not too sure if the patch is going to work with coverage at this point.  So, my question is, is there a known good method to make coverage and cython play nice with one another right now?  If not, are there any plans for this in the future?

Thanks,

Cem Karan

[Robert Bradshaw]

Yes, that should still work. If you compile with the CYTHON_TRACE
macro defined as true, it should even do line profiling these days.

> --
>
> ---
> You received this message because you are subscribed to the Google Groups
> "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to cython-users...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.

[Ian Bell]

I use line profiling in my project and it works just fine - a really really great addition to Cython! 

Now if we could easily do simultaneous python and C++ debugging without having to resort to cygdb.  I really dislike command line driven debuggers and in my opinion (let the flaming commence), Microsoft Visual Studio is and remains the gold standard in debugging.

Ian

[Cem Karan]

Neat! I'll enable it once I get back to work. Thanks!

Thanks,
Cem Karan

[Eleftherios Garyfallidis]

Hi guys,

Did someone here managed to get the nosetests coverage working with recent cython versions? 

If yes would it be too much to ask for a working example or link to some code which can do this successfully?

Best,

Eleftherios

[Stefan Behnel]

Eleftherios Garyfallidis schrieb am 01.08.2014 um 14:32:
> Did someone here managed to get the nosetests coverage working with recent
> cython versions?
> If yes would it be too much to ask for a working example or link to some
> code which can do this successfully?

Coverage (with coverage.py) doesn't work yet. This has been discussed
before, more work is needed, and patches are welcome.

Stefan

[Matthew Brett]

On Sun, Aug 03, 2014 at 10:25 AM, Stefan Behnel <stef...@behnel.de> wrote:

> from: Stefan Behnel <stef...@behnel.de>
> date: Sun, Aug 03 07:25 PM +02:00 2014
> to: cython...@googlegroups.com
> reply-to: cython...@googlegroups.com
> subject: Re: [cython-users] Using coverage with Cython

Can I check whether we have missed any pointers on the list?

* I think you (Stefan) opened a trac ticket about line profiling [1],
and referred to a preliminary pull request [2] on line number mapping
that didn't get merged yet. This came up (I believe) because of a
project proposal you made on the list [3] last year.
* This current thread mentions coverage and line profiling.

Is there other info we missed?

Can I ask - if you needed to estimate the code coverage of a large block
of Cython code right now, how would you go about it - in broad terms?

Sorry for the ignorant questions, and

Thanks a lot,

Matthew

[1] http://trac.cython.org/cython_trac/ticket/815
[2] https://github.com/cython/cython/pull/93
[3] https://mail.python.org/pipermail/cython-devel/2013-May/003652.html

--
Sent from Vmail

[Stefan Behnel]

Matthew Brett schrieb am 05.08.2014 um 20:02:

> On Sun, Aug 03, 2014 at 10:25 AM, Stefan Behnel wrote:
>> Coverage (with coverage.py) doesn't work yet. This has been discussed
>> before, more work is needed, and patches are welcome.
>
> Can I check whether we have missed any pointers on the list?
>
> * I think you (Stefan) opened a trac ticket about line profiling [1],

Yes. It lists the main steps for an implementation.

> and referred to a preliminary pull request [2] on line number mapping
> that didn't get merged yet.

I didn't merge it because it is useless by itself, without also making
Cython create only a single frame per function.

> This came up (I believe) because of a
> project proposal you made on the list [3] last year.
> * This current thread mentions coverage and line profiling.

Yes, those are the two main use cases.

> Is there other info we missed?

That's pretty much it. If someone wants to give it a try, I'll be available
for help and instructions.

> Can I ask - if you needed to estimate the code coverage of a large block
> of Cython code right now, how would you go about it - in broad terms?

The problem is that none of the common tools currently works, so getting
this implemented would be a great step forward.

> [1] http://trac.cython.org/cython_trac/ticket/815
> [2] https://github.com/cython/cython/pull/93
> [3] https://mail.python.org/pipermail/cython-devel/2013-May/003652.html

Stefan

[Matthew Brett]

On Tue, Aug 05, 2014 at 12:12 PM, Stefan Behnel <stef...@behnel.de> wrote:

> from: Stefan Behnel <stef...@behnel.de>

> date: Tue, Aug 05 09:12 PM +02:00 2014

> to: cython...@googlegroups.com
> reply-to: cython...@googlegroups.com
> subject: Re: [cython-users] Using coverage with Cython
>

> Matthew Brett schrieb am 05.08.2014 um 20:02:
>> On Sun, Aug 03, 2014 at 10:25 AM, Stefan Behnel wrote:
>>> Coverage (with coverage.py) doesn't work yet. This has been discussed
>>> before, more work is needed, and patches are welcome.
>>
>> Can I check whether we have missed any pointers on the list?
>>
>> * I think you (Stefan) opened a trac ticket about line profiling [1],
>
> Yes. It lists the main steps for an implementation.
>
>
>> and referred to a preliminary pull request [2] on line number mapping
>> that didn't get merged yet.
>
> I didn't merge it because it is useless by itself, without also making
> Cython create only a single frame per function.
>
>
>> This came up (I believe) because of a
>> project proposal you made on the list [3] last year.
>> * This current thread mentions coverage and line profiling.
>
> Yes, those are the two main use cases.
>
>
>> Is there other info we missed?
>
> That's pretty much it. If someone wants to give it a try, I'll be available
> for help and instructions.
>
>
>> Can I ask - if you needed to estimate the code coverage of a large block
>> of Cython code right now, how would you go about it - in broad terms?
>
> The problem is that none of the common tools currently works, so getting
> this implemented would be a great step forward.

We really need coverage metrics for Cython, and I'm guessing this would
be very useful for other projects like pandas, scikit-learn,
scikit-image, so maybe we could collectively find some development help
for this. How much work do you think this would be? If we (us heavy
Cython users) gathered some help for a sprint, maybe with a hangout with
y'all before, do you think a sprint would have a reasonable chance of
getting this done?

Cheers,

Matthew

[Robert Bradshaw]

I think this is totally reasonable to do in a sprint; that's a great
idea. I bet a couple of users hacking for a couple of hours should be
able to get it done.

- Robert

[Matthew Brett]

Hi,

OK - thanks - we will try and do that.

I know I don't know the internals of Cython or coverage at all well,
so at the moment I don't feel confident that I know what needs to be
done.

Can we ask for 15 minutes of hangout / skype / IRC from one of y'all
to get us oriented?

Cheers,

Matthew

[Robert Bradshaw]

What time zone?

> --
>
> ---
> You received this message because you are subscribed to the Google Groups "cython-users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to cython-users...@googlegroups.com.

> For more options, visit https://groups.google.com/d/optout.

[Matthew Brett]

On Thu, Aug 7, 2014 at 1:18 PM, Robert Bradshaw <robe...@gmail.com> wrote:
> What time zone?

At the moment it's me (UTC-7) and Eleftherios (UTC-4), but I will ask
around to see who else might be interested to help.

Cheers,

Matthew

[Matthew Brett]

Sorry, it's probably obvious, but we timed out on organizing this, and I ended up going for a quick and dirty solution analyzing the test file AST for calls to the Cython code:

https://gist.github.com/matthew-brett/ea2c713be6edc2ab29ca

Cheers,

Matthew 

[Francesc Elies]

Hi,

First of all thanks for this nice project.

As eleftherios wrote some time ago, I would be also interested in getting nosetests coverage working with cython.
So far I had no success, that's why I tried to replicate this particular test from the cython project https://github.com/cython/cython/blob/master/tests/run/coverage_api.srctree in a small project.

So far I tested this versions.
coverage=4.0b2 cython=0.23.1: coverage.misc.CoverageException: No data to report.
coverage=3.7.1 & cython=0.23.1: raised AssertionError: 'pkg/coverage_test_py.py' not found in coverage report:

In some posts I could read a patch being needed for coverage, in other places I could read it doesn't work yet.
I tried to follow Enabling line tracing & Enabling coverage analysis sections in http://docs.cython.org/src/tutorial/profiling_tutorial.html without luck, am I doing something wrong? Any hints are much appreciated.

Thanks in advance for your time and any updates you could give me.

Best regards,
Cesc

[Stefan Behnel]

Francesc Elies schrieb am 31.08.2015 um 10:34:
> As eleftherios wrote some time ago, I would be also interested in getting
> nosetests coverage working with cython.
> So far I had no success, that's why I tried to replicate this particular
> test from the cython project
> https://github.com/cython/cython/blob/master/tests/run/coverage_api.srctree
> in a small project.
>
> So far I tested this versions.

> *coverage=4.0b2 cython=0.23.1*: coverage.misc.CoverageException: No data to
> report.
> *coverage=3.7.1 & cython=0.23.1:* raised AssertionError:

> 'pkg/coverage_test_py.py' not found in coverage report:

Looks like coverage 4.0 keeps breaking its plugin API even during the beta
cycle. I'll have a look.

Stefan

[Stefan Behnel]

https://github.com/cython/cython/commit/067c6cca4be56b260fa3476d4b3a886a2e073fd5

Stefan

[Francesc Elies]

Lovely!

I'll give it a try, lot's of thanks

[Francesc Elies]

Hi,

just to report it back now the test pass, cool! and if I print the coverage lines from the test I get:

coverage =  4.0b2
cython = 0.23.1
['Name                      Stmts   Miss  Cover   Missing', '-------------------------------------------------------', 'pkg/coverage_test_py.py       7      3    57%   1-4, 10']
['Name                        Stmts   Miss  Cover   Missing', '---------------------------------------------------------', 'pkg/coverage_test_pyx.pyx       7      3    57%   1-4, 10']
['Name                            Stmts   Miss  Cover   Missing', '-------------------------------------------------------------', 'coverage_test_include_pyx.pyx       6      4    33%   1-6, 11', 'pkg/coverage_test_pyx.pxi           6      2    67%   4, 10', '-------------------------------------------------------------', 'TOTAL                              12      6    50%   ']

Does anybody managed to make it work with nosetests alltogether? I am still trying but no success so far

Once again thanks in advance for your help

Best regards,
Cesc