Issues with running the unit tests

[Grig Gheorghiu]

Hi,

I am the coordinator of the Pybots project (http://www.pybots.org). One
of the Pybots contributors, Elliot Murphy, is running the CherryPy unit
tests on his buildbot buildslave. There are 2 issues with the tests:

1. At the very end of the test suite run, the user is prompted to press
enter; this obviously makes it very hard to run the tests in a totally
automated fashion (this was also described here:
<http://www.cherrypy.org/ticket/481>). As a result, the buildmaster
times out the test run after 1 hour.

2. One of the unit tests fails currently when run with the latest
Python binary built from the Python trunk. The failing test also waits
for user input, and the test run is timed out by the buildmaster after
1 hour.

Here is the test that fails:

test_02_wrapped_wsgi (test_wsgiapps.WSGIAppTest) ...
ERROR: 'Content-Type':'text/plain' not in headers
Show: [B]ody [H]eaders [S]tatus [U]RL; [I]gnore, [R]aise, or
sys.e[X]it >>

Is there any chance the tests can be modified so that they simply fail
without waiting for user input?

Grig

[fumanchu]

Sure thing. Fixed in http://www.cherrypy.org/changeset/1389 on ticket
http://www.cherrypy.org/ticket/481.

Robert Brewer
System Architect
Amor Ministries
fuma...@amor.org

[Grig Gheorghiu]

fumanchu wrote:
> Sure thing. Fixed in http://www.cherrypy.org/changeset/1389 on ticket
> http://www.cherrypy.org/ticket/481.
>

Great! Thanks for taking care of this.

Grig

[elliot]

Hi!

fumanchu wrote:
> >
> > Is there any chance the tests can be modified so that they simply fail
> > without waiting for user input?
>
> Sure thing. Fixed in http://www.cherrypy.org/changeset/1389 on ticket
> http://www.cherrypy.org/ticket/481.
>

Thanks for fixing this! I've updated my pybots build slave to now pass
the --dumb argument, and it is now working. We've now noticed a second
problem: cherrypy/tests/test.py does not return a non-zero exit code
when tests fail, so the builder is not marking the test stage as
failing. Would it be possible to sys.exit() a non-zero error code when
any tests have failed?

cheers,
-elliot

[Robert Brewer]

elliot wrote:
> We've now noticed a second problem: cherrypy/tests/test.py
> does not return a non-zero exit code when tests fail,
> so the builder is not marking the test stage as failing.
> Would it be possible to sys.exit() a non-zero error code
> when any tests have failed?

Woof. That'll be a bit more work, since all tests run in a separate thread, and we already use sys.exit to stop the CP server. I might not get this done until next week (as I'm still on vacation ;).

[Ian Bicking]

Robert Brewer wrote:
> elliot wrote:
>> We've now noticed a second problem: cherrypy/tests/test.py
>> does not return a non-zero exit code when tests fail,
>> so the builder is not marking the test stage as failing.
>> Would it be possible to sys.exit() a non-zero error code
>> when any tests have failed?
>
> Woof. That'll be a bit more work, since all tests run in a separate thread, and we already use sys.exit to stop the CP server. I might not get this done until next week (as I'm still on vacation ;).

Perhaps something like:

test_status = []

try:
run_tests(test_status)
except SystemExit:
if False in test_status: # or a more reliable test...
sys.exit(1)
else:
sys.exit(0)

--
Ian Bicking | ia...@colorstudy.com | http://blog.ianbicking.org