Require 100% Coverage on Patches

[Donald Stufft]

Hey,

There's a service called codecov, which is essentially just a little service
which can aggregate coverage across multiple test runs. I would like to turn
this on in the pip repository. In particular I would like to mandate that all
patches come with 100% coverage for lines of code that the patch itself
touches.

An obvious problem is that right now we don't really have CI except for Linux
(in particular, Ubuntu Linux) which means that there are some lines of code
which will never get touched in our current CI setup. These areas of code would
need to be marked with # pragma: no cover, to indicate to the coverage runner
that it's OK for them not to have coverage.

I think this would be a great way to get pip's coverage up, and hopefully at
some point we can be stricter about our coverage as a whole.

Thoughts?

---
Donald Stufft
PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

[Ionel Cristian Mărieș]

Is there anything preventing the test suite to be run on Windows?

AppVeyor got a huge upgrade recently: http://www.appveyor.com/blog/2015/06/23/new-oss-build-environment-and-xamarin-support

Seems to be as fast as Travis now (4x faster for the virtualenv rewrite tests, I suspect something similar for pip).

Thanks,
-- Ionel Cristian Mărieș, http://blog.ionelmc.ro

[Donald Stufft]

On June 29, 2015 at 12:52:49 PM, Ionel Cristian Mărieș (con...@ionelmc.ro) wrote:
> Is there anything preventing the test suite to be run on Windows?
>
> AppVeyor got a huge upgrade recently:
> http://www.appveyor.com/blog/2015/06/23/new-oss-build-environment-and-xamarin-support
>
> Seems to be as fast as Travis now (4x faster for the virtualenv rewrite
> tests, I suspect something similar for pip).
>

Someone to put in the effort to make it happen is probably the biggest thing,
though it's not strictly related because there are still going to be branches
that we simply aren't covering (for example, some branches of code only really
happen on AIX).

[Paul Moore]

On 29 June 2015 at 21:34, Donald Stufft <don...@stufft.io> wrote:
> On June 29, 2015 at 12:52:49 PM, Ionel Cristian Mărieș (con...@ionelmc.ro) wrote:
>> Is there anything preventing the test suite to be run on Windows?
>>
>> AppVeyor got a huge upgrade recently:
>> http://www.appveyor.com/blog/2015/06/23/new-oss-build-environment-and-xamarin-support
>>
>> Seems to be as fast as Travis now (4x faster for the virtualenv rewrite
>> tests, I suspect something similar for pip).
>>
>
> Someone to put in the effort to make it happen is probably the biggest thing,
> though it's not strictly related because there are still going to be branches
> that we simply aren't covering (for example, some branches of code only really
> happen on AIX).

I'll add this to my TODO list, although I'm just recovering from a PC
disk crash, so it'll probably be a while before I get to it...

The big problem with getting the tests running on Appveyor is that
they will almost certainly fail in a lot of ways initially (unless
someone put a lot of work into fixing the tests on Windows while I
wasn't looking :-)) So there'll be a chunk of work to do once the
tests run on Appveyor (and that's work I don't have the time to do,
unfortunately).

I suggest initially not running the integration tests on Windows, and
not making Windows test success be a prerequisite for merging.

Paul

[Matthias Bussonnier]

Hey, 

On Jun 29, 2015, at 07:46, Donald Stufft <don...@stufft.io> wrote:

Hey,

There's a service called codecov, which is essentially just a little service

which can aggregate coverage across multiple test runs. [...]

I tried codecov a few month back (December), and compared to https://coveralls.io/, 

and ended up using coveralls.io for whatever reason. ( my last commit being “if codecov

doesn’t work this time,I give up” might be a hint). 

If you like to investigate the alternative.

My 2 cents.

-- 

M