test runners

[memedough]

Hi,

So coverage now has a nose / pytest plugin. Is it intended to ship
coverage with these runners by default?

If so I'm interested in a couple of issues.

How do you select which plugin you want, as it may not be the coverage
supplied one? If nose/nose2 came out with something is it possible to
select between them?

There could be clashes, I notice both the coverage supplied one and
pytest-coverage use --cover as a prefix.

Invoking pytest / nose with -h may show options from an unused plugin
(or worse confuse some developers into trying to use the wrong one)
and currently quite a lot of them that get in the way if you want to
get help on some other aspect of pytest / nose.

Is the release cycle going to be affected by bundling? Will all of
coverage have to be released just to update the runner plugin due to
some improvement in it or improvement in pytest / nose?

I guess I'm biased since I have my own coverage plugin and will need
to continue using it. I will mainly be affected with many unused
options listed on help output (and -h is used and useful - though I'm
also guilty of clogging up the help output - something I'm currently
cutting down). A trivial downside would also be pushing around more
unused files to the many test areas, but it still all adds up.

All of these issues are easily avoided by releasing the plugins
separately. People can just pip install the one that it most suitable
for their needs. People know how to pip install since they did a pip
install for nose, pytest and coverage anyway. Both pytest and nose
websites list the plugins so they are easy to find.

I would prefer coverage to not follow nose with bundling. But I could
be a minority so I raise this as a question, do the advantages of
having a default installed plugin outweigh competition, clash
avoidance, help output noise and release cycles?

Either way I'm happy as I know I can make my situation work.

:)

[Zooko O'Whielacronx]

Heh, I'm thinking about a similar problem but from the opposite
direction. I'm maintaining a separately-packaged plugin to integrate
coverage.py with trial:

http://pypi.python.org/pypi/trialcoverage

I've been hoping to get it bundled into the coverage.py distribution
or else into the trial distribution. I guess my motivations for
wanting trialcoverage to be bundled into one of those other projects
(coverage.py or trial) are:

1. If coverage.py or trial changes, then trialcoverage might need to
adapt. If trialcoverage is bundled then the person who changes
coverage.py or trial is more likely to notice.

2. Easier for users if they don't have to install them separately.

3. Hopefully the maintainers of coverage.py or trial will start
maintaining trialcoverage so I don't have to. :-)

4. They might see what trialcoverage is doing and realize that they
could implement the same or better functionality directly into their
own project and do that, obsoleting trialcoverage.

Regards,

Zooko

[Ned Batchelder]

These are all interesting points. The original idea behind
incorporating the plugin into coverage.py was that coverage.py was
changing much faster than nose was. It also made it possible to
maintain a plugin for py.test in the same project.

But one of the reasons I added support for .coveragerc files was
specifically to allow people to use new features in coverage.py without
having to change their UI, for example, the nose plugin. This should
remove the need to constantly update the plugin as features are added to
coverage, since they'll be controllable without having to add tons of
new command-line switches to the plugin. In fact, I'm going to remove a
number of the current plugin options, leaving only the most basic
switches, and require the more advanced stuff be configured in the .rc file.

Also, I've been struggling with logistical issues around having the
plugin as part of the coverage.py kit anyway: in particular, testing the
plugins is a big departure from testing coverage.py itself.

So I could really see separating out the plugins into their own
project. At this point, there are a number of them: nose ships one,
coverage.py has a nose plugin and a py.test plugin, memedough has a
py.test plugin, and Zooko has trialcoverage.

Memedough, to address some of your questions: as of nose 0.11.3, a
plugin can be overridden with a same-named plugin from another package.
So if you install nose, then coverage.py, the cover plugin in
coverage.py will be used, and nose's built-in cover plugin will be
unavailable. When you use -h, you'll only see help for the
latest-installed plugin. I'm not sure how py.test deals with this.

--Ned.

[Ned Batchelder]

I've decided to leave out the test-runner plugins from the coverage.py
kit for now. I can't dedicate the time to give them their due, and
others seem interested in doing it, at least in some instances.

--Ned.

[memedough]

Hi,

I think there are some benefits to having the plugins separate anyway:
- separate release cycle is easier to track releases of pytest /
nose / coverage / ...
- easier for someone actively using a test runner to maintain the
plugin since they are using it daily
- plugins are a small way that others can help out the people behind
pytest / nose / coverage since those seem like big projects to me (hey
but i happy if those people want to hack on the plugin too)
- test runner can depend on external plugin to install automatically
for users so out of the box work

I read the emails last year about a common plugin for test runners.
Though I was dubious about how much common there would be.

There is some. I pulled the cov controller code into cov-core and
made pytest-cov more pytesty and made a nose-cov based on nose-cover
(which i had a go a hacking up a few months ago) and the nose plugin
in coverage.

All of this is based on the new source option in coverage so nothing
released - only on bitbucket. Also docs totally out of date or
missing (need overhaul which do when ready to release).

The nose-cov should work for sub processes too, but doesn't appear
work with multiprocess plugin (which is what i wanted). The pytest-
cov have tests to check pytest-cov / cov-core but no tests for nose-
cov :( If someone that use nose want to help that would be welcome
but otherwise I can maintain at least this plugin.

In case anyone interested to play:

cd /tmp
virtualenv --distribute --no-site-packages e1
source e1/bin/activate

cd /tmp
wget http://bitbucket.org/ned/coveragepy/get/bb16cf0b2ab7.tar.gz
tar xvzf bb16cf0b2ab7.tar.gz
cd coveragepy
python setup.py install

cd /tmp
wget http://bitbucket.org/memedough/cov-core/get/b50e6a7ed6f9.tar.gz
tar xvzf b50e6a7ed6f9.tar.gz
cd cov-core
python setup.py install

cd /tmp
wget http://bitbucket.org/memedough/pytest-cov/get/1f828fe6cabc.tar.gz
tar xvzf 1f828fe6cabc.tar.gz
cd pytest-cov
python setup.py install

cd /tmp
wget http://bitbucket.org/memedough/nose-cov/get/3c12ae94ccca.tar.gz
tar xvzf 3c12ae94ccca.tar.gz
cd nose-cov
python setup.py install

:)