Testing
26 views
Skip to first unread message
David Wilson
Mar 6, 2014, 9:21:27 AM3/6/14
to pypa...@googlegroups.com
Guys, we need to have a chat about testing. I've repeatedly deferred
writing this a few times over the past 6 months on the hopes of
writing it later when I'm a bit less emotionally involved.
The tl;dr is simply that setuptools (and preferably the rest of the
pypa stack) really needs some black box tests covering _minimum viable
functionality_ for recent/supported versions of Python, and those
version combinations need to be published somewhere. I'm 100% on board
for doing the trivial work required to make this happen.
As part of maintaining py-lmdb approximately once per month, I have a
need to observe the output of TravisCI, which executes setuptools on
my behalf against all widely deployed versions of Python starting from
2.5 (i.e. all except 3.0).
py-lmdb isn't interesting in itself, only that it attempts to remain
compatible with all versions of Python that might, somewhere behind
closed doors, be in use in some commercial environment. The package
does almost nothing fancy - it could be seen as approaching the
smallest possible test for a viable working packaging environment.
Despite sampling the present state of the packaging universe at most
once per month, almost reliably each month, I find 90% of my time is
consumed not in maintaining my package, but in shepherding the build
back into a working state, which more often than not has broken since
the last time I made any commits.
This has been going on for at least 4 months, and each time it
happens, I vow never to waste time on the problem again, and proceed
to incrementally remove yet another aspect of the
virtualenv/pip/setuptools/tox stack.
All the above tools have fundamental problems operating under at least
one recent version of Python - for example, setuptools was broken on
Python3.1 due to SSL assumptions, the Tox "vendored" version of
virtualenv recently started breaking against an older Python, and so
on.
As of today there is no single configuration of the above tools that
function reliably under all interesting versions of Python - and ISTM
often for unjustifiably trivial reasons (although I'm sure those with
experience of actually maintaining the stack may entirely disagree ;).
Today my build process no longer depends on any of virtualenv, pip, or
tox, yet still this month I've again found it broken, this time - most
damningly - due to a NameError in ez_setup.py master - a NameError
that makes it clear to me that code being committed to a critical
Python repository is being neither peer reviewed thoroughly nor tested
to any level of sensibility.
We should discuss how this stuff is tested - it's far too important to
the community to fall to the whim of random unreviewed commits. I
appreciate that various pypa tooling tests suck, and getting
comprehensive tests takes time, however writing an *ultra basic* smoke
test is very little work - in the ideal case it's a 30 line shell
script run for each "supported" configuration of the tooling.
I am more than willing to dedicate time to producing a CI
configuration and associated script that executes such a test, since
the effort in doing so is less than the time I appear to predictably
spend monthly trying to fix up the effects of their absence. But first
obviously there needs to be some consensus on this.
Glancing at setuptools .travis.yml hg log, I see that of the few tests
that exist for setuptools, none are run on Python2.5 any more. While
TravisCI is a cute and fashionable convenience, their decision to
remove Python2.5 from the base image *should not be dictating policy
for the wider Python community*, especially for such a widely deployed
Python release.
The proposed smoke test would simply install some popular Python
package that ideally includes at least one C extension. The test
succeeds if the package is installed and can be imported using the
version of Python under test. Preferably it would execute on each
commit, but even just running it before a release would suffice.
If you look at https://github.com/dw/py-lmdb/blob/master/.runtests-travisci.sh
you can see an example of how to get TravisCI base image into a state
where all recent versions of Python exist. This script is a direct
result of having spent a full weekend, abandoning all hope of getting
Tox/virtualenv/pip working in unison for all interesting Python
versions.
Initially, it would be great to get setuptools .travis.yml updated to
use something similar to the above script, and, say, installing the
lxml package (which has a nicely complex setup.py/compiler
dependency).
Thoughts?
David
writing this a few times over the past 6 months on the hopes of
writing it later when I'm a bit less emotionally involved.
The tl;dr is simply that setuptools (and preferably the rest of the
pypa stack) really needs some black box tests covering _minimum viable
functionality_ for recent/supported versions of Python, and those
version combinations need to be published somewhere. I'm 100% on board
for doing the trivial work required to make this happen.
As part of maintaining py-lmdb approximately once per month, I have a
need to observe the output of TravisCI, which executes setuptools on
my behalf against all widely deployed versions of Python starting from
2.5 (i.e. all except 3.0).
py-lmdb isn't interesting in itself, only that it attempts to remain
compatible with all versions of Python that might, somewhere behind
closed doors, be in use in some commercial environment. The package
does almost nothing fancy - it could be seen as approaching the
smallest possible test for a viable working packaging environment.
Despite sampling the present state of the packaging universe at most
once per month, almost reliably each month, I find 90% of my time is
consumed not in maintaining my package, but in shepherding the build
back into a working state, which more often than not has broken since
the last time I made any commits.
This has been going on for at least 4 months, and each time it
happens, I vow never to waste time on the problem again, and proceed
to incrementally remove yet another aspect of the
virtualenv/pip/setuptools/tox stack.
All the above tools have fundamental problems operating under at least
one recent version of Python - for example, setuptools was broken on
Python3.1 due to SSL assumptions, the Tox "vendored" version of
virtualenv recently started breaking against an older Python, and so
on.
As of today there is no single configuration of the above tools that
function reliably under all interesting versions of Python - and ISTM
often for unjustifiably trivial reasons (although I'm sure those with
experience of actually maintaining the stack may entirely disagree ;).
Today my build process no longer depends on any of virtualenv, pip, or
tox, yet still this month I've again found it broken, this time - most
damningly - due to a NameError in ez_setup.py master - a NameError
that makes it clear to me that code being committed to a critical
Python repository is being neither peer reviewed thoroughly nor tested
to any level of sensibility.
We should discuss how this stuff is tested - it's far too important to
the community to fall to the whim of random unreviewed commits. I
appreciate that various pypa tooling tests suck, and getting
comprehensive tests takes time, however writing an *ultra basic* smoke
test is very little work - in the ideal case it's a 30 line shell
script run for each "supported" configuration of the tooling.
I am more than willing to dedicate time to producing a CI
configuration and associated script that executes such a test, since
the effort in doing so is less than the time I appear to predictably
spend monthly trying to fix up the effects of their absence. But first
obviously there needs to be some consensus on this.
Glancing at setuptools .travis.yml hg log, I see that of the few tests
that exist for setuptools, none are run on Python2.5 any more. While
TravisCI is a cute and fashionable convenience, their decision to
remove Python2.5 from the base image *should not be dictating policy
for the wider Python community*, especially for such a widely deployed
Python release.
The proposed smoke test would simply install some popular Python
package that ideally includes at least one C extension. The test
succeeds if the package is installed and can be imported using the
version of Python under test. Preferably it would execute on each
commit, but even just running it before a release would suffice.
If you look at https://github.com/dw/py-lmdb/blob/master/.runtests-travisci.sh
you can see an example of how to get TravisCI base image into a state
where all recent versions of Python exist. This script is a direct
result of having spent a full weekend, abandoning all hope of getting
Tox/virtualenv/pip working in unison for all interesting Python
versions.
Initially, it would be great to get setuptools .travis.yml updated to
use something similar to the above script, and, say, installing the
lxml package (which has a nicely complex setup.py/compiler
dependency).
Thoughts?
David
Matthew Iversen
Mar 6, 2014, 4:26:31 PM3/6/14
to pypa...@googlegroups.com
I want to afore-mention that I think you raise some great concerns, this response only pertains to some parts of
what you talked about that I know.
On Friday, 7 March 2014 01:21:27 UTC+11, David Wilson wrote:
Today my build process no longer depends on any of virtualenv, pip, or
tox, yet still this month I've again found it broken, this time - most
damningly - due to a NameError in ez_setup.py master - a NameError
that makes it clear to me that code being committed to a critical
Python repository is being neither peer reviewed thoroughly nor tested
to any level of sensibility.
I'm not sure why you're pulling ez_setup.py from the master branch? Every link in existence
should advertise that ez_setup.py should be pulled from the bootstrap branch, i.e
If you know of any link pointing a user to use the master branch for this file, please do tell us so we can
fix it. I know of no guarantees that the 'master' branch will never break, nor that is advised to pull off of for general use.
Glancing at setuptools .travis.yml hg log, I see that of the few tests
that exist for setuptools, none are run on Python2.5 any more. While
TravisCI is a cute and fashionable convenience, their decision to
remove Python2.5 from the base image *should not be dictating policy
for the wider Python community*, especially for such a widely deployed
Python release.
You are aware that setuptools >= 2.x no longer supports Python 2.5 right?
2.5 was EOLed 2 years ago and is now more than six years old. It is *really* hard
to make packaging better, as you are asking for, and at the same time support this kind of software.
However, anyone of course is most welcome to as we all have different audiences we want to support;
if you wish to use a compatible setuptools, you should stay on the 1.x-maintenance branch. Accordingly, a
1.x bootstrap is available from the bootstrap-py24 branch, i.e
That is advertised to be used for packages / pythons of 2.4 and 2.5.
The proposed smoke test would simply install some popular Python
package that ideally includes at least one C extension. The test
succeeds if the package is installed and can be imported using the
version of Python under test. Preferably it would execute on each
commit, but even just running it before a release would suffice.
I'm absolutely sure Jason Coombs would be delighted to see pull requests for more
tests for setuptools, at the moment he is a one man team working on it.
Cheers,
Matt
Jason R. Coombs
Mar 6, 2014, 8:06:06 PM3/6/14
to David Wilson, pypa...@googlegroups.com
Hi David,
Your points here are largely valid and the whole team takes your concerns seriously. Thanks for posting them here.
I do apologize for the defect in ez_setup.py. However, you will observe that (a) the problem was fixed within two hours of being reported and (b) additional follow-up work was done to capture this type of failure early (prior to release). That is, I acknowledged prior to having read your e-mail that there was a fundamental problem here and sought to mechanize a solution for it.
If stability is your goal, I recommend you pin to a specific version of setuptools, and bump that periodically for freshness. That will allow you to address emergent issues on your schedule, but also likely avoid short-lived issues like the ez_setup issue you encountered this morning.
As for improving the test suite of setuptools, I'm eager to review and accept pull requests to improve the behavior. I accept pull requests on github (at jaraco/setuptools) as well as on the official repo. I'd rather if possible avoid spending too much time discussing the issues, but rather spend the time documenting them in tickets and hashing out the implementation. I think we all can agree that improving the process would be an improvement :)
Your suggestion about building a popular package does make some sense. It would be challenging to include something like that on Windows, so one will have to consider what it will take to skip that test on Windows. It also only covers a build step, but not other aspects. Neveretheless, it's probably worthwhile and certainly worthwhile if there's someone willing to put it together and shepherd any emergent issues that arise.
I appreciate your patience as we continue to push these projects forward with as much accuracy as we can afford.
Sincerely,
Jason
Your points here are largely valid and the whole team takes your concerns seriously. Thanks for posting them here.
I do apologize for the defect in ez_setup.py. However, you will observe that (a) the problem was fixed within two hours of being reported and (b) additional follow-up work was done to capture this type of failure early (prior to release). That is, I acknowledged prior to having read your e-mail that there was a fundamental problem here and sought to mechanize a solution for it.
If stability is your goal, I recommend you pin to a specific version of setuptools, and bump that periodically for freshness. That will allow you to address emergent issues on your schedule, but also likely avoid short-lived issues like the ez_setup issue you encountered this morning.
As for improving the test suite of setuptools, I'm eager to review and accept pull requests to improve the behavior. I accept pull requests on github (at jaraco/setuptools) as well as on the official repo. I'd rather if possible avoid spending too much time discussing the issues, but rather spend the time documenting them in tickets and hashing out the implementation. I think we all can agree that improving the process would be an improvement :)
Your suggestion about building a popular package does make some sense. It would be challenging to include something like that on Windows, so one will have to consider what it will take to skip that test on Windows. It also only covers a build step, but not other aspects. Neveretheless, it's probably worthwhile and certainly worthwhile if there's someone willing to put it together and shepherd any emergent issues that arise.
I appreciate your patience as we continue to push these projects forward with as much accuracy as we can afford.
Sincerely,
Jason
Reply all
Reply to author
Forward
0 new messages